063 – Chris Sangwin: STACK

[00:00:00] David: Hi and welcome to the IDEMS podcast. I’m David Stern, one of the founding directors of IDEMS, and it’s my privilege today to be here with Professor Chris Sangwin, who is the founder of STACK.

Hi Chris.

[00:00:20] Chris S: Hi David, thanks very much for the invitation to join you.

[00:00:23] David: This is great. I’ve been looking forward to this for a long time because STACK is so important to the work that we’ve been doing. The opportunity to share this with others is great. So maybe we should just start by what is STACK and why did you start it? It’s quite a few years ago now.

[00:00:40] Chris S: Yeah, so thanks David. STACK’s an online assessment system which puts mathematics at the heart of assessment. So I’ve always thought that assessment drives learning, and assessment drives education.

Indeed, Hugh Burkhardt, a colleague at the Nottingham Shell Centre, said we should have tests worth teaching to. So putting assessment at the heart of what we do in education is important and digital tools need to support that. Mathematics always claims a distinctive nature because of the symbolism and the nature of assessments and the objective nature of the criteria by which we judge success in many situations.

And so we need special tools for assessment and STACK is a special tool which helps colleagues automate their assessments. And in particular, the student’s answer will typically be a mathematical expression rather than just a multiple choice question. And so that was the idea behind STACK.

[00:01:34] David: Absolutely. And it’s now, it’s over 15 years old!

[00:01:37] Chris S: Right. So there’s a continuous history of mathematicians building their own online assessment systems right back to the 1960s. And STACK is part of that long tradition, in particular using computer algebra systems as a library for getting the mathematics online. I saw that idea in 1999, and then we were using a system called AIM that originated in Belgium and had a second version from Sheffield, and then for various reasons, I decided that I wanted to write my own software. So I started doing that around 2003. And then the first version of STACK was around 2004, 2005.

[00:02:15] David: Wow. Almost 20 years.

[00:02:17] Chris S: Yes, so 20 years, yeah, but that’s part of a longer tradition as well. And in parallel there are lots of similar systems. I think it’s quite natural for mathematicians to want to ask questions, whether the answer is an expression, so a polynomial or an equation.

[00:02:30] David: Absolutely. And I think one of the things which drew me to STACK, I’d been aware of it for a long time, but it was when a colleague in Kenya was having to teach classes of four to five hundred students or a thousand students with no tutorial assistants. Well, the thing which really was central is your feedback trees.

[00:02:51] Chris S: Yeah.

[00:02:52] David: And the way you can give feedback. So maybe it’s worth just mentioning the importance of that feedback.

[00:02:57] Chris S: So there are three things actually there you’ve brought in. One is the feedback, two is scaling to large groups, and the third is writing your own materials. So we’ll just start with the feedback. It’s a good place to start.

Feedback should be based on what a student actually does. So, in classical control theory, you know, automatic control systems, there’s open loop feedback, which doesn’t depend on what the current state is, and then there’s closed loop feedback, which actually measures the current state and then feeds that back.

And the feedback really should be based on the student’s answer and, to be effective, it should help them improve their performance on the current task. And in order to do that, typically what you need to do is undertake some kind of calculation based on what the student has done, and then use the results of that calculation in the feedback.

So the calculation might be establishing some kind of equivalence with what you consider to be the correct answer, but it might be manipulating the student’s answer and then using the results of that and printing that in the feedback. Now that’s the tricky bit because you’re really dealing with what a student types, and so you have to be prepared to deal with anything that a user might throw at the system. But one of the distinctive features of STACK was being able to embed the results of calculations into the feedback for the students.

[00:04:22] David: And also to remove the validation step from the feedback and put that into the entry.

[00:04:27] Chris S: Right. So feedback comes in all sort of ways. One of the key design features in STACK was separating out what we call validation feedback from correctness feedback.

[00:04:37] David: Yeah.

[00:04:38] Chris S: So, the validation feedback should be largely independent of the question, and it should typically be always available even during an exam. So there are sorts of uncontroversial validation feedback, if you’ve missed one bracket, so you have an opening bracket, but you forget to close it, it’s impossible to make sense of the expression. And so that would certainly always be validation feedback and would typically always be available because it’s very rare that you want to penalize someone on a technicality.

Now it might be that you want to test whether they understand the notation. Okay, fair enough. But that’s really an edge case. Typically, you would always want to show feedback about syntax errors. But there are other sorts of things that you might put into validation. You might want to forbid floating point numbers because you expect an exact answer like root 2 or pi, or the student might have the wrong type of answer. So it might be that you’re expecting an equation and they’ve just typed in an expression and you might choose to signal that in the validation feedback.

Now that’s getting more towards correctness because you might judge the student on whether they know it should be an equation rather than expression. That might be part of the correctness, but it might be part of the validation. So although validation is mostly independent of the question, some things will depend on the question. 

With validate, rejecting something as invalid means I can’t even think about it in this context, right? So you don’t want to penalise a student on a technicality and there’s something seriously wrong with it. And then on the other hand, the teacher had to decide what makes an answer correct in this situation.

So it might be equivalent to the correct answer, it might have other mathematical properties, and that’s when you would then generate separate sorts of feedback. But there are all sorts of other sorts of feedback, including numerical marks, and so on, that you might also choose to display or withhold, and then there’s the timing of when that’s available.

So validation feedback will be typically always available, but in a practice quiz you might make the correctness feedback immediately available to encourage another try. In an exam situation, and online assessment is increasingly being used for high stakes exams, the validation feedback might be available, but then the correctness might not be available until it’s been checked and validated by an exam board. So that might have the traditional delay that you would expect in a normal assessment. It’s complicated, it’s a tool that’s used in many complicated setups.

[00:06:53] David: And I love the fact that you’re drawing this complication out, because this is one of the things that attracted me to this, compared to, as you say, there’s lots of other tools out there. But what makes, I think, STACK, you know, really special is the complexity which it holds, which you don’t have to engage with to do a question, but to author really good questions you have to do. So we might get on to the sort of authoring question of this.

[00:07:17] Chris S: Right, one of the goals of STACK was to make colleagues independent and to author their own questions. Yeah, so we could talk about that. I think I was, when I set this up, a little bit naive on how complex the whole thing was. And all the features in STACK are there because teachers have really needed them. There’s no marketing fluff. It’s an open source project. It’s evolved because of genuine need.

But that does mean it is sophisticated and requires sophistication from a user, and it’s perfectly possible to write nonsense questions or have nonsense outcomes, but it’s also very possible to give sophisticated feedback, which really helps the students, and that’s the goal in a formative setting. And also then in a summative setting, it’s possible to write questions, which actually test what’s relevant, and we can be confident that we judge the student’s ability based on that.

I mean, we have talked about what STACK is. STACK is a question type. And in this open source world, I’ve chosen to concentrate on the question type. And so currently the way that you would use STACK is through a quiz system. Version 1 of STACK, I wrote a whole quiz system, but that was just unsustainable and a bit silly. So at the moment, the STACK question type is embedded into a quiz system. And so we inherit a lot of the design decisions from the designers of the quiz.

So in particular how and when feedback is displayed, some of the style decisions, the way that the questions appear in the web browser, all the policies, but we get lots of advantages. So the quiz systems that STACK currently uses is Moodle and Ilias. I’m most familiar with the Moodle system, cause that’s what I use.

And we inherit a lot of very helpful features from that. So in particular, the ability to give individual users extra time, some of the accessibility settings, all of those things we inherit from the Moodle quiz system. I don’t need to worry about that at all. I have large classes and if we decide it’s right to give an individual student a second try for some technical reason, that’s all inherited from the quiz system. That’s nothing to do with me and my plugin.

We also inherit all the other question types. Now, I’m quite critical of multiple choice questions, but they certainly have their place, and there are a lot of other very interesting question types which help us in education alongside STACK. So STACK is absolutely about mathematics, but in teaching mathematics, we can easily use some of the other question types as well, and we certainly have multiple choice questions in our quizzes that go alongside that.

So as part of the whole process, we have the bigger quiz system and all the other tools, and I think that’s a really useful, a useful way to collaborate.

[00:09:48] David: Absolutely. I think one of the things which you’ve drawn out, which I think I’d like to just dig deeper in, is you mentioned Moodle and Ilias, but because of the API, which is now sort of coming out, and other ways of integrating between Moodle and other systems, this is actually accessible through a wide range of learning management systems or other forms, you know, it could even be embedded in a website in theory. And this is rather new as a concept.

[00:10:15] Chris S: So the STACK project is on four versions over the last 20 years. Version 1 was a completely standalone quiz system. It’s interesting to think back 20 years, it was difficult to get an equation to display on a web browser and so on and so on. And all these campus management systems have moved along just as the STACK plugin has. So version 1 was a standalone system. In version 2, we tried to extract just a question, and we tried to build that so that you could embed that into other quizzes. So version two is really designed around embedding questions much more flexibly. I wouldn’t use the word disaster, but it had some serious technical problems. And those were bound up with the difficulty of web services at that point. It just wasn’t ready.

[00:11:00] David: It was ahead of its time.

[00:11:02] Chris S: It was ahead of its time. And yeah, we did push forward web services through various projects, the development of web services. So it was ahead of its time, which means it didn’t really work very well in a live, large setting. And so we moved very quickly to version 3 where we committed to the Moodle plugin. It was also not clear to me at that time how difficult it is to actually separate out a question from a quiz.

So, for example, in a quiz system there will be certain settings, perhaps the most obvious one is the timing of releasing the worked solution. So that is really a function of the quiz setup. When is the worked solution going to be available, if at all? On the other hand, in order to generate the worked solution, especially when that worked solution depends on random parameters, that’s really a function of the question.

This is all terribly tedious to an end user who simply wants things to work, but your consistency is set at the quiz level, and then you want the question to do the right thing. So it’s not entirely clear how to extract a question from a quiz. And since each quiz system manages some of those things in very different ways, actually abstracting that out and still making it work in an individual quiz system was really quite difficult.

Now a lot of that has settled down, people have understood the kinds of things you might ask a question to do. But again, going back 20 years, it wasn’t uncommon for every question to be a hand crafted, individual thing. And so having hooks and names for things. Again, utterly, utterly tedious, but does a question plug in, return a score between 0 and 1, and then you scale it up in the quiz system, or does this quiz system tell you how many marks this question is out of, and then you somehow have to scale it on the plug in side? This is just, it’s all design decisions at every level.

[00:12:43] David: Yeah.

[00:12:44] Chris S: All of that has to be done. And then you get rounding errors and all sorts of very, very tedious things.

All of that’s come along and now we’re on version 4. We’ve gone back and we just released, um, a standalone version of the stat plugin, which makes it possible to embed those questions into individual web pages. The motivating example for that was high stakes exam systems. Universities have secure exam systems now. We need to use those tools, STACK, for a high stakes assessment. And so they needed to extract the STACK question type from the quiz management system. I mean, I’m confident that there’ll be much wider range of exams that are on screen and automatically marked. They’re separate issues, of course, but the automatic marking and the on screen is coming along. That’s a big change.

[00:13:36] David: And as you say, it’s something which in some sense is very recent because the pandemic pushed that along in ways that nobody really expected.

[00:13:45] Chris S: Absolutely accelerated the process. And it really comes back to the design of the materials and the authoring of the materials.

[00:13:55] David: So let’s come back to that authoring point, because this is actually something where, as you said, you started off with this perception that the authoring would be done by your colleagues, as you put it. You were building a system for fellow maths lecturers to be using and authoring their own questions.

[00:14:11] Chris S: Yes.

[00:14:11] David: I must admit, from when I really got engaged, that was never my vision. I think our visions are now closer than they were.

[00:14:20] Chris S: Yeah, so the goal was to make colleagues independent and allow colleagues to write their own questions. I come from a higher education background. Colleagues are largely independently minded and they want to do their own thing in their own courses, even when there are many courses with a large overlap. So especially first and second year calculus, algebra, linear algebra, vector calculus, there’s a huge overlap in what we teach and the way we teach it and, by and large, conservatism in the way that’s all done. However, colleagues do want to write their own materials, as do I.

[00:14:52] David: Let’s be clear here. You’re really talking from a UK, European perspective.

[00:14:58] Chris S: Yes.

[00:14:58] David: Because the American perspective is big textbooks, which is rather different. And they’ve gone almost down a slightly different route with that. Now I’m not saying that it’s totally different, and what you’re saying isn’t true in that context as well.

[00:15:11] Chris S: So I think that largely arises from the way that we teach. In the UK and Europe, we have large lecture courses with one member of staff and some teaching assistants. But I think in North America, there’s much more section teaching in groups of 30 or so. And that section teaching means people are culturally much more used to using textbooks rather than having lecture notes, which would originate from the lecturer.

And so for those reasons, I think they’re much more willing to teach in teams. That’s changing in Europe, I think.

[00:15:39] David: Yeah. And I want to come from the perspective I was coming from, because I spent six years at a Kenyan university, where we had large classes, and my normal lecture load was sort of three to four lecture courses a semester.

[00:15:51] Chris S: Yeah.

[00:15:52] David: Which meant that the time to spend on an individual course is totally different. And in that context, the idea of actually being able to share and collaborate with others, to get resources which are reusable, is really attractive. And the idea of using open resources alongside the open system is really what we found is so powerful in those contexts.

[00:16:16] Chris S: Yeah, and there’s a good history of maths exercises being reused, you know, word problems go back thousands of years. People copy exercises from one textbook to another. And so it’s sensible to reuse things and share them. And again, one of the political motivations for STACK was to produce an open source tool.

So the idea is to collaborate on the player and the tool and write an open source tool. And then what people do with that is entirely up to them, including closed commercial products. So STACK is being used in a closed and commercial way. It’s supporting a textbook. I’m very happy with that. On the other hand, there are large open question banks, which people are welcome to use and contribute to, and all of that can easily sit together.

But I do see a danger for the education world. I’m employed in a publicly funded institution. Whether the university is a commercial or not is not entirely clear. Some of their work is clearly commercial and some of it’s clearly publicly funded. But I think as a community, my view is that we don’t want to rely on commercial tools. I think we should be taking responsibility for our own core business and part of that is developing and maintaining the assessment platforms.

[00:17:25] David: And I think that’s also related to the fact that you’re embedded in Moodle because it was the Open University switching to Moodle which really was part of that becoming really at the forefront and accepted by many universities as a key player for that.

[00:17:44] Chris S: Right, and Moodle has been very successful, I think, and it sits alongside some of the commercial campus learning management systems. The Open University has been a key player in STACK. They explicitly funded some of the development in moving from version 2 to version 3, for which I’m very grateful.

And that’s really part of the business model, if people find these tools useful, we accept that quality needs to be paid for one way or the other. And in terms of the OU, that was assigning staff time to help with the development rather than actually paying cash to a commercial company for the finished product.

And so they’re just different ways of getting the quality that you want and the tool that you need. And that seems to be working very well so far.

[00:18:28] David: And I suppose that gets us almost to the knowledge transfer partnership, in which we’re collaborating.

[00:18:32] Chris S: Yeah.

[00:18:32] David: But maybe we should just come back to one point which has been slightly lost in the discussion, well I suppose it’s two points together, that if you’re looking in these real large scale environments, sort of my Kenyan colleagues teaching classes of a thousand students, the value for that scale and also the potential to be able to think about the lecturers taking responsibility for editing questions rather than necessarily authoring from scratch. I think it’s a really important point.

[00:19:02] Chris S: Yeah.

[00:19:02] David: And one of the things which I think is important there is it comes back to this element that you mentioned in passing that you can write terrible questions, or you can write very good questions. Actually, tweaking very good questions to make them suit your audience rather than starting from scratch and potentially writing bad questions is, I think, an approach which I definitely believe is a powerful way for us to go in terms of supporting lecturers with these large classes and across contexts.

[00:19:30] Chris S: Yeah. There’s quite a lot to discuss on that point. The first is selecting from existing question banks and making it easier for people to find and use the questions that they want in their course. That’s actually sort of a librarianship problem really, and it’s nothing to do with the STACK plugin because we’ll be using other question types.

It’s just a fundamentally difficult problem. It’s much easier to flick through a paper book and quickly scan them and reject them than it is to search through even the best designed web page. That’s a sort of practical problem. And then there’s the freedoms, freedoms to just slightly adapt. And the last thing I want to say is the time it takes to actually design, test and refine.

[00:20:17] David: Yeah.

[00:20:17] Chris S: And I think as a practical teacher, the first year I run a course, just getting the minimum thing up and working is quite a challenge. In the second year, improving the randomization, sometimes even completing the worked solutions. The honest truth is I don’t always have time to write the quality worked solutions that I might like in year one. So going back and doing that in year two, improving the randomization and improving the feedback. So I would say it’s really only in year three that you have something that would be something I’d want to share even, just something that I know works, works well.

And then there will be questions in a quiz that either have floor or ceiling effects that you just reject as being pointless. And there’s no point asking that question because everyone either gets it right or wrong and you don’t know that until you battle test it.

[00:21:02] David: Exactly.

[00:21:02] Chris S: If I can use that phrase. So this business of the freedom to be able to change things, which you can’t in a published commercial setting. And there’s really opportunities for everyone to contribute. Anyone who can teach the course could improve the worked solutions, or suggest feedback or look at what the students are doing and realize, ah, there’s a gap here, we’re not really testing this sort of understanding, or this particular topic is missing. So given the difficulty of curating materials, every teacher can contribute to the education endeavour by actually thoughtfully using the materials and suggesting improvements, even if they currently don’t have the technical ability.

So it’s just a cost benefit or a priorities issue. Anyone that can teach maths could sit down and learn how to write these sorts of questions. So it does come down to priorities and available resources in terms of their own time and how that balances against the other priorities that are demanding their time. It’s just as simple as that.

And so having the ability to gently improve things and then having an expert on hand to do the occasional highly technical task or provide an example that you can adapt is much more effective than having everybody write from scratch, which would not be easy.

[00:22:18] David: I want to contradict you slightly, that although I sort of agree that anyone could, it certainly isn’t the case that anyone would, it depends on personality types.

[00:22:27] Chris S: It does.

[00:22:27] David: Some people love just digging in and writing this and thinking about it from all angles, which is what you need to do to write a good question.

[00:22:35] Chris S: So part of my goal was to avoid the requirement that colleagues become programmers. So going back 20 years, I mentioned earlier, I think, that 20 years ago, every question was a handcrafted little mini web page. And so there is this design decision about where the library sits and how much resides within an individual question and where there are general libraries. So we clearly need a general library that will robustly establish algebraic equivalence between expressions. That clearly should be part of the system.

But then there are things that you only do in certain subjects, and there are assumptions made in certain subjects about the nature of symbols. So does n represent a natural number, for example? There are certain assumptions that are made regularly in topics. So is that in a library or is that in a question?

We have libraries for dealing with scientific units, right? So if we’re dealing with scientific units, then m is probably a meter rather than an arbitrary integer, n and m in a pure maths course would be different from m in a scientific course. And so the consistency and reliability versus individual freedom, it’s actually not clear where that boundary should lie. And those things evolve.

And so making it easy for colleagues to write those questions, yeah, that, I think I underestimated that. I think I also, from my own personal position, underestimated people’s aptitude for programming or their desire to program. So the goal was to get colleagues to think about the mathematical properties.

So, for example, I want algebraic equivalence rather than how do I actually establish it.

[00:24:15] David: Yes.

[00:24:16] Chris S: And so the way that I’ve designed the question type would encourage people to think in terms of properties, and then there should be libraries for establishing that. That might sound obvious, but the prior experience with other systems means that you are more of a programmer, or you’re required to become more of a programmer.

And so for a real expert, you can write, much more flexible questions, much more quickly if you’re a programmer and you have full control. On the other hand, you just can’t even get started if you’re temperamentally not interested in programming.

[00:24:44] David: And I think this is a key point, because this is why I’ve levitated towards STACK. I actually really strongly believe that good programming is highly mathematical.

[00:24:54] Chris S: Yeah.

[00:24:55] David: And in some sense, that’s really what you’ve got with STACK, is it’s not written by programmers for programmers. It’s written by a mathematician for mathematicians in some sense. And I guess what I’m interested in, and let’s get to the KTP, the Knowledge Transfer Partnership now, because we’re a not for profit social enterprise.

[00:25:13] Chris S: Yeah.

[00:25:13] David: We are interested in finding ethical, inclusive business solutions to be able to use this tool and get it out at scale.

[00:25:23] Chris S: Yeah.

[00:25:24] David: And one of the things which I think is really interesting in this is, as you say, you underestimated a little bit how complicated these things can be. And I would argue that from my perspective, what you’ve done is brilliant. You’ve built the foundation. But if I want to get this used at scale, most people can’t start from the foundation. They need lots of scaffolding. They need these open question banks with the libraries that help them get in. They need those services on top. And we need to have the sort of extra tools like the open question banks, but also experts at actually digging in and what makes good feedback, good feedback, which not every lecturer, not every educator wants to dig into in great depth.

And I want to just mention briefly, my passion for this came when I first heard about MOOCs. And you know, it was 2012, which was the year of the MOOC, which is actually a long time ago now. And when I first heard about MOOCs, it was all about the feedback that you could give if you had 100, 000 students doing a particular question. Then, if a very small percentage get it wrong, that’s still a lot of students where you get a lot of data to learn how to give better feedback.

And this is where, for me, the dream is to build on top of systems like STACK so that we actually get these common question banks, we get the data coming in from lots of different people using them, and actually that task of improving that feedback, you could eventually, potentially do better than any individual lecturer could do on these quite complex questions based on that data. This is what you could get by scale, by using scale.

[00:27:04] Chris S: Yeah, we do need large courses to get decent data. We reviewed an online test in Edinburgh using item response theory. We did that when I arrived in Edinburgh in 2015, and we had many years of data, and that was largely, but not exclusively multiple choice for those questions.

But in reviewing that test, we actually realised that we had far too many questions of particular types and that we’d missed out areas. So that statistical analysis allowed us to reduce the length of the test and increase the quality at the same time. But when we tried to apply item response theory to some of our mainstream courses, even where we had four or five hundred students on the course, we found we didn’t really have enough data to use those statistical techniques in a way that would have given us some meaningful recommendations. And mathematics is surprisingly international. There are important cultural differences, but certainly at university level, we do have that opportunity of scale.

[00:28:05] David: Yeah. And that’s one of the things which is really exciting. So we have this knowledge transfer partnership, which is funded by Innovate UK, and we should thank Innovate UK for this, where they are providing the funds for us to have an exceptional postdoc, he really is quite phenomenal, who is working on different aspects of developing STACK, but also doing so in a way which hopefully positions us, IDEMS, to be able to achieve some of these goals of actually getting it out, in particularly the low resource environments we’re interested in, but also in our vision to be able to offer these services about getting higher quality assessment into education everywhere.

[00:28:46] Chris S: Yeah.

[00:28:46] David: And he started off really well. And I guess the key question that I have for you on this is has this changed your thinking, or maybe this has come at a time where your thinking is changing anyway with respect to STACK, about how you see this going forward in the next few years?

[00:29:02] Chris S: Yeah, I mean, we’ve now got a very sophisticated tool that’s highly reliable, and I think we’re going to be moving into a law of diminishing returns in terms of adding features. It’s really now about community involvement, and that involves sharing materials and effectively sharing materials and having mechanisms for curating those materials. And that’s really the key part of the knowledge transfer partnership.

[00:29:26] David: Yeah.

[00:29:27] Chris S: If we stopped adding features to the plugin, we still would have a very useful tool. And so the biggest priority for the community is materials and material sharing.

[00:29:38] David: Yeah. And that’s hard.

[00:29:41] Chris S: Yeah, it is. And just as librarians are highly valued professional colleagues in a university environment with expertise, that business of curating and looking after the materials and making recommendations, and then providing professional advice on test development and materials development. I think there are real opportunities for that, which by and large, research mathematicians have not spent the time thinking about those things.

[00:30:06] David: Yeah.

[00:30:06] Chris S: All university mathematicians have succeeded in a traditional exam system. And so they have a lot of experience of tests. And so they can, we can all rely on that experience to write our own tests and exams.

[00:30:20] David: Yeah.

[00:30:21] Chris S: But I think that’s scope for expertise in test construction that would increase the effectiveness of education quite significantly.

[00:30:28] David: Well, and in particular, as we’re looking at AI coming in in different ways, and thinking about how technology is integrated in, in ways that many mathematicians didn’t experience. And so it’s bringing those sort of advances to play in a positive way to the education. And this is something we, you know, I was privileged to come up to Edinburgh and actually have this session, discussing how could AI be responsibly used in teaching and improving teaching rather than it being seen as a foe. That was a fantastic day, really enjoyed that session.

But that’s also part of it, that there is this sort of element of what’s new, the AI phase, which is coming in, and how that’s going to affect education, which is not in people’s experience.

[00:31:16] Chris S: So ultimately what matters is what the students can do and what they understand. And so the ability of an AI to flawlessly perform on an exam is not really of concern to me. It’s an interesting question, and it will certainly force us to question what we’re asking the students to do and why, just as we did when electronic calculators came along. But ultimately, there will be things we want the students to be able to do, and then the students have to be able to do that.

So I think the role of AI would be, in a positive way, will be to suggest automatic feedback, to suggest similar questions, but ultimately, I think a human teacher will remain responsible, and that we’ll need expertise in education and in the subject to guide the AI. So I’m not, don’t feel threatened by AI as a teacher, but I’m quite excited that there’ll be things that we just don’t need to do anymore that they’re usually the tedious things.

[00:32:09] David: If used right, AI should be a real boom.

[00:32:13] Chris S: Yeah.

[00:32:13] David: And it really shouldn’t be threatening.

[00:32:15] Chris S: So that’s fine.

[00:32:17] David: Great. I think this is probably a good point to end on unless there’s anything you’d like to bring out as a last reflection or a last thought.

[00:32:24] Chris S: No David, other than to thank you for your interest. I think the role of IDEMS and the way that we’ve set up this knowledge transfer partnership, this KTP, is a very exciting way of broadening the range of users outside the European university system. So thank you for your interest. And it’s a very exciting project.

[00:32:41] David: I should, I should be clear here. I mean, it’s already used in over a thousand institutions around the world, isn’t it?

[00:32:47] Chris S: Yeah.

[00:32:47] David: So this is already highly successful in terms of the users in different contexts. But I think you’re right, our interest is a sort of almost virgin market, where people are not currently using it. And so the hope is that together we can actually get this out to a lot more people and hopefully make it more powerful. It’s exciting times.

[00:33:06] Chris S: It is.

[00:33:07] David: Thank you very much.

[00:33:09] Chris S: Thank you.