057 – Empowering Mathematics Educators with STACK

[00:00:00] Santiago: Hi, and welcome to the IDEMS podcast. I am Santiago Borio, an Impact Activation Fellow, and I’m here with David Stern, a founding director of IDEMS.

Hi David.

[00:00:18] David: Hi Santiago. We’re discussing STACK today, one of your favourite topics.

[00:00:22] Santiago: I do like STACK, yes. I have to say it is one of the defining tools in my career, I am very fond of it, I like the people behind it, I like the project itself and I love the impact that it can generate.

[00:00:42] David: Well, let’s start briefly with that impact from the perspective that we both value so much, which is the implementations we’re seeing in Kenya.

[00:00:51] Santiago: No, let’s start by giving a very brief outline of what STACK is for those listeners who haven’t listened to the other episodes where we explained it.

[00:01:02] David: Well, I come back to the fact that I don’t think we can do that without starting with the impact. So let me explain the impact from my perspective, and I hope that will make it clear.

[00:01:12] Santiago: Okay.

[00:01:12] David: The reason that this is such a powerful tool and it has achieved something which I struggled with for many, many years beforehand, is we have partners in Kenya who have classes of hundreds of students, sometimes even over a thousand.

[00:01:32] Santiago: At university.

[00:01:34] David: At university, where they have no teaching assistants, and as a single lecturer, they are expected to teach the class. The possibilities to give feedback are extremely limited, of course, in that context, and we had, in the past, tried all sorts of other things to help with the learning.

[00:01:56] Santiago: Can I interrupt?

[00:01:57] David: Of course.

[00:01:58] Santiago: It’s not just they have those classes, they also have all the admin related to the university to do. They have loads of other commitments within the university. It’s not just the teaching is too much.

[00:02:11] David: But let’s be clear, that’s not their only course. Other courses, you know, more advanced courses would tend to have much smaller numbers. But a course with hundreds or even over a thousand students could be the responsibility of a single lecturer. Responsible lecturers, and there are an incredible number of them working in these difficult environments, want to give a good education. And STACK has, in my mind, enabled this better than any of the other interventions I’ve tried, and I’ve tried a lot of them over the years. So broadly, that’s the impact. And let’s now get down to why it’s achieving that impact, and how, and what it is.

[00:02:56] Santiago: I think, a small bit of context, we started this in Kenya, with colleagues at Maseno University, and then it spread to other universities, but the standard was to give two written assessments for each course, and sometimes the strains were too high for the lecturers to be able, even though they wanted to, to be able to give any feedback whatsoever until the week before the exam and sometimes even after the exam. So in terms of student impact and helping the students understand where they’re standing, it was impossible.

[00:03:36] David: And let’s be clear, once STACK was integrated in a number of these courses. Students then had two assessments a week and one of them was a mastery assessment which they could attempt as many times as they could and so broadly you were multiplying by 10 the number of assessments with automated feedback that students were getting.

Now, I should be clear, we had tried other automated assessment systems beforehand, and they had had some success. In fact, one of them, used for statistics, is still one of the most impressive pieces of research a master’s student has ever done for me, in changing my thinking, where they used a system which did this and which actually got real student learning from this automated assessment.

So there are plenty of other automated assessment systems which in theory could have this role and there’s other ones which exist which have good question banks and which are open as well. It’s not the only open one.

[00:04:38] Santiago: And I think it’s worth mentioning the statistics one and the creator, it’s called CAST, and the creator is Doug Serling.

[00:04:46] David: Amazing work, a labour of love, over 10 years to produce what was a really incredible collection of statistics textbooks with these assessment components embedded in different ways. A real inspiration to me in many different ways, but not scalable in the way that was needed.

[00:05:03] Santiago: Yeah, and it has also influenced me in the way I create resources in STACK as well.

[00:05:11] David: And I think, you know, it’s still an incredible piece of work that he did. But it was not scalable. When we got it working, it was incredibly impactful, but it wasn’t scalable. And this was the thing. So STACK wasn’t the first impactful resource that we’d seen. And it isn’t the only thing out there. There’s many other things out there.

But the thing which was so powerful when STACK started getting used is the scalability and the flexibility and the general power to give meaningful feedback. You can create STACK questions very quickly, or you can spend a long time and create really good questions. And those layers of depth are really interesting.

[00:05:58] Santiago: Okay. I’m getting a bit uncomfortable not explaining what STACK is.

[00:06:03] David: It’s a question type in Moodle, and that doesn’t help. So I’ll let you explain better.

[00:06:07] Santiago: So it’s an open source tool that allows you to create mathematical questions or STEM related questions online. It’s open source, so it’s free to access, it’s not free to use necessarily because it takes quite a while to get it set up and get the right resources ready. But never mind.

[00:06:35] David: No that’s worth spending a little bit of time to actually dig into. And let’s be absolutely clear on here, although it can be used in other ways, it is really designed as something which plugs in to Moodle. Moodle is an open source learning management system and there are others out there and it can be compatible with others as well. But Moodle is one of the widely used open source learning management systems and it is a question type within that which enables you to give really deep mathematical feedback, and to be able to have questions which are randomized, which enables you to have mastery and other things. And I would argue that the last thing that I would give in the description is that because it has the computer algebra system behind it, it actually does mathematics. Which means that even if you’re not using it for mathematics, that is an incredibly powerful tool.

[00:07:35] Santiago: So I would argue there’s two aspects of STACK that differentiate it. One is as you said, the computer algebra system, which is called Maxima, which is also open source. It is deeply embedded into STACK, so STACK itself as a package can carry out very complex computations, and it can generate random variations of questions, so that students have different versions when they attempt the same question multiple times and different students have different versions of the same question. And it also allows students to input algebraic responses and mathematical objects as responses in a nice way in which other systems can but not to the same extent.

[00:08:25] David: Let me correct you on that. There are systems out there that can to the same sort of extent, not all of them are open, but the way that this comes together with the second point you’re going to do is something where again there are other systems which do elements of this but actually the whole package together, as a free open source system, there’s nothing at this point, it’s at the forefront of what’s out there.

[00:08:50] Santiago: And it is a labour of love from professor Chris Sangwin at the University of Edinburgh, over 20 years.

[00:08:57] David: And others. This is the other key thing because there’s other systems which are labours of love from other people doing wonderful work as well. But what Chris has done so successfully is built a real community about this. Not just a community of users, but also a community of developers. And therefore it’s going beyond an individual.

[00:09:17] Santiago: And a community of professionals that use it.

[00:09:21] David: Exactly. And for open source projects, community is everything. And so it does have that dynamism of the community as well as this. And you’ve not yet mentioned the feedback.

[00:09:32] Santiago: No, I was going to get back. And what Chris designed was remarkable. He had feedback at the heart and centre of the project. He understood the value of computer algebra and how it allowed for all sorts of advantages, but feedback was really at the heart and centre of it.

And he designed a system that allows you to not just assess a single answer and determine whether it’s right or wrong. It allows you to assess aspects of the answer, perform calculations with the answer, and check for specific areas. It allows you to do all sorts of things. You can ask as many questions as you’d like of an answer, or of results of calculations to that answer.

[00:10:27] David: Or multiple answers. This is the other important thing, of course.

[00:10:31] Santiago: Or multiple answers as well, you can combine multiple answers into a single evaluation process. You can, with that, create, with the advantage of the computer algebra system as well, combine both things to predict potential student mistakes, program them into the assessment process. And if those mistakes are made, give relevant, instantaneous, targeted feedback on a randomised question to the student right there, right then, in order to address misconceptions.

[00:11:12] David: And I want to be really clear here. It is not the only system in the world that does any individual component. And this is partly because Chris has been embedded in the world of maths education and his ideas have been shared with others and other people have shared their ideas with him.

What I think makes STACK stand out to me is that as a system designed by mathematicians, a lot of the details of these structures have been really carefully thought about, and I believe got right. They don’t necessarily make sense if you’re just wanting to do something simple. But when you want to do something which is really complex, when you’re giving really complex feedback, or wanting to dig in to identify something specific, that’s when STACK suddenly sings. It really enables you to delve into those layers of complexity.

[00:12:07] Santiago: Okay, now that we’ve described STACK, you mentioned that this is scalable.

[00:12:14] David: I want to say before that, I also mentioned that it’s game changing. I might not have used that terminology, but I want to. That in the context within which we work and we have been working, or I’ve been working for 15 years with a number of colleagues, this is, I believe, game changing. And we’re starting to see this with some of the results coming out with colleagues in Kenya and beyond, where the scalability comes partly of course because everything is open and therefore by developing open question banks and this is something which can then be shared and you can build communities. Again, good open source often requires community, good open educational resources requires community.

[00:12:59] Santiago: Okay, let me clarify, you mentioned open question banks. So creating a STACK question, simple STACK question with minimal feedback can take around two hours time. I have spent in a complex question about two weeks. So this idea of open question banks is crucial for scalability. It’s all the work that we’ve done to create questions is stored in an open repository that people can access.

[00:13:31] David: Let me be clear, we’re certainly not the first people to think about this. This amazing project, Abacus, I think it was an EU funded project, which actually tried to get these shareable question banks for and with STACK. And so lots of other people have thought about this.

I would argue that when Chris started out, there was one element where I think he would recognize this now, and this is Chris Sangwin who is the person really behind this. You know, when he started out, he was expecting lecturers to be question authors. He now recognizes the importance of what is, you mentioned, STACK professionals, people who actually, they’re specialised in writing and authoring STACK questions. And I think that, to me, is part of where the game changing element came in. And you came into IDEMS as a STACK author.

[00:14:28] Santiago: Yep, freelance initially.

[00:14:31] David: Yeah.

[00:14:32] Santiago: And then it became part of my work when I joined.

[00:14:37] David: Absolutely. And the point is that we still don’t play this role on quite the scale that we’d like to in the future, but we recognize that the game changing nature of STACK is around the reusability of really powerfully authored questions and the nature of the open processes to be able to do this and potentially, you know, actually deliver on what MOOCs promised many years ago. And that is to be able to use the massive nature of people interacting with questions to get the data to be able to improve them.

And that’s what I mean by scalability. That actually, we don’t have all the pieces of this yet. There aren’t even the structures yet to be able to combine the data in the ways that I think would be needed. And that’s a whole set of research which we’re still going to have to get involved in. But what really makes STACK so exciting to me is the fact that it’s put in place all the structures with really solid mathematical foundations to enable us to tackle what is, I believe, one of the hardest problems I know, which is how to give really good feedback to learners, which will help them to identify misconceptions, misunderstandings, and learn from them.

Ah, it’s amazing.

[00:16:05] Santiago: And I said earlier, it’s open source, but it’s not free in the sense that it takes a lot of resources to get it ready. But things like the Open Question Bank are what allows that scalability, because it removes some of the costs to actually implementing it.

[00:16:24] David: I think one of the things, what are those costs? Those costs are human, time, this is one of the big things, and there is a lot of problems around the fact that educators times, I believe, are often undervalued. The amount of time it takes to do, and to think, and to engage, as an educator, in educating, is often undervalued. And we recognize this in terms of what the administrative burden put on teachers, on lecturers in different contexts.

This is a worldwide problem. And so that educator time, saving that, ensuring that that can be used in the best possible ways is something that I believe very strongly. But more than that, the other big expense is the systems themselves and open source systems are free to deploy but you still need to have the hosting infrastructure, you need to have the expertise, the people to be able to manage them or run them or you need to pay someone to do that.

[00:17:24] Santiago: Sorry, there is a third one in this context which is the content creation.

[00:17:29] David: Originally, that was on the educators. And so this is what I was talking about when I was talking about educator time. That was originally the responsibility of the educators. And that’s what I think also needs to change.

So let me be clear about what I mean by that needs to change. I think that this expectation that educators can, in isolation, can have the time to really be the innovators on the content creation without inspiration from elsewhere. Open educational resources have the potential to both make it easier for these things to scale, but also be adapted to enable creativity of educators without requiring immense efforts in terms of, you know, time, commitment, and the rest of it.

Some educators go way over and above, as I’ve seen you do.

[00:18:22] Santiago: Not so much lately, unfortunately other commitments have taken me away from that. I would love to get back into it, and I’m sure it will happen at some point soon.

[00:18:33] David: Well, let’s be clear what you’re saying here, you love, and you left IDEMS for a year to go back to the classroom, because that’s where you are in your element. But when you came back to IDEMS, broadly it was because, basically, as an educator working in this, the time spent actually on that classroom interaction wasn’t as valued as the time you were spending on the administrative issues, which you were less interested in.

[00:19:04] Santiago: And there was one aspect that really frustrated me, the amount of marking that I was forced to do, because there were strict guidelines on the amount of marking that had to be done. And having this knowledge of an automatic system that could give better feedback to the individual than I could through marking. I could have spent the same amount of time creating wonderful questions that the students would get automatic feedback for, and I just wasn’t allowed because I had to actually show evidence of marking in the student’s books.

[00:19:44] David: And this is the thing, I’m not saying that actually, having elements of student marking and so on should be removed. But I am saying that done right, this is exactly where with automation, everybody wins. And there are many other areas where you as an educator going into the classroom, you’re not replaceable.

What you can do is not efficient, but marking in this way is a chore where actually you are less effective than a good automated system could be if you feed it into it well, and you don’t enjoy it. And the student doesn’t engage in it, so everybody’s losing. I want everybody to win. I want educators to be freed up to do the things which they’re really powerful at and really good at, engaging students.

[00:20:33] Santiago: Motivating students.

[00:20:34] David: How much time do teachers actually get to focus on those important aspects when actually a lot of their time is taken up with things which are less creative, and I would argue less impactful.

[00:20:48] Santiago: Exam practice is another one, but that’s a whole other podcast episode.

[00:20:54] David: Well, it is and it isn’t because the whole point is that exam practice is exactly the sort of thing where if you were thinking of that at the level of scale, then suddenly, instead of thinking about how you get the top teachers into the top students to make sure that those students are really drilled on exams, that’s actually creating inequality in our education system.

[00:21:16] Santiago: It is.

[00:21:17] David: And so, exam practice is something where, done really well, a good automated system should be able to outperform any human, however good they happen to be at drilling exam practice. Now motivating for exam practice, engaging people, that’s a whole different thing, there’s a human skill there which we’d want to encourage. And there are efforts to be able to build AI systems to do that and they may become more effective than they currently are and then I may get convinced that that is something which could be automated.

But right now, I would argue that the actual drilling, that piece, is a piece where we can and we should be doing. It’d be more equitable if we could make that available to everybody.

[00:22:04] Santiago: Yes. And I think that listeners who are enjoying this episode should look at the STACK internship episode as well, because that is a big part of how we are trying to make things scalable as well. The STACK interns, well, now STACK team in INNODEMS, our partners in Kenya, they are now taking responsibility of this work in Kenya, taking ownership of it.

[00:22:32] David: And I should be clear, we are not yet scalable in this. But what is happening is there is an element of viral scaling happening which I have been talking about for a lot of years and I’ve conceptualized quite a long time ago, almost, I think it’s 10 years now, ago.

But this is one of the first areas where I’ve really observed this happening with an educational intervention, where it is naturally spreading and it’s evolving and it’s emerging in different ways. And so the scaling is happening in a very natural way. Now the real scaling work that I think is needed, we’re not there yet.

We’re still small and we’re still only doing marginal efforts on this. But the potential is there, and somehow it’s coming in the right sort of ways. It’s emerging. I’m excited by how this is growing.

[00:23:22] Santiago: And we will surely update you in a couple of months with a new episode on this.

[00:23:29] David: I would argue before we finish this episode, I’m sure, as you say, there will be many more episodes on this topic, but before we finish this episode, the thing I would like to just share, in some sense, is that STACK itself isn’t the solution. And it may be in a few years time that there’s other technology which comes out and surpasses it in certain ways. However, what I believe right now is that STACK is an enabler and I have seen it as an enabler, enabling educators to be able to take on and do things they didn’t think possible.

And I would like to finish with one story on that because it’s still my favourite story and I won’t use any names here because I think it’s not fair. But there was a colleague who was trying to help the department and their colleagues take up STACK. And they basically said to their colleagues, okay, you don’t need to change the way you’re teaching, just teach as normal, but that continuous assessment, we’ll get the students to do that electronically for these courses. And one of the lecturers who was involved in one such course did exactly that. She taught as normal and she then had the students use the electronic assessment. And it was a tough course, it’s a course where normally most students fail, complex analysis course. Many students struggled with these concepts.

And so, you know, in the past, the pass rates had been relatively low. And she taught in the same way as normal, and she got to the exams, and the students were passing, they were succeeding on questions that they’d never done before. The pass rates were just beautifully normally distributed, exactly what you’d hope for a good, well taught course, well taught exam. And she hadn’t changed anything except the assessment system.

[00:25:34] Santiago: And there was one question in the exam that students always failed on. And suddenly, this time round, the majority got right.

[00:25:46] David: Exactly. And she’d never seen that before, there were always a few exceptional students who did well, but this was a differentiator question where almost everyone failed, which is why so many people were failing the final exam in the past.

[00:25:57] Santiago: And it was a sensible question.

[00:26:00] David: Absolutely. And so what she was finding was that there was real learning which had happened. And she then discussed it with the students and so on. And so she got a better appreciation. And she is now one of the great advocates of STACK in her university because of this experience.

And just that mind change, I believe this is scalable and the reason that I’m so excited about this as an initiative is it wasn’t more work for the lecturer in that first instance. They were able to do what they normally did with a little bit less work and that was already impactful and that was so exciting to them as a lecturer and as an educator that they then got engaged and they did put in the extra work but it wasn’t needed it was wanted.

And that’s what’s so powerful about this, that you can get people to sort of just be part of it. They can take it up and they can use it, and it is impactful even with them doing what they would normally have done. And that’s why this has been such an exciting innovation.

And I don’t believe it’s the only tool that could do this. So I’m not claiming that this is the only tool out there which could lead to these sorts of impacts. In fact, I can tell you, in certain contexts, there are thousands of software which claim to do this. What I can tell you is that in certain contexts, the open nature of this process is leading to evolution and spreading, which I am so excited about. Anyway, that’s my last thought.

[00:27:42] Santiago: Thank you very much, David.

[00:27:44] David: Thank you.