121 – Reflections on Research Methods Support (RMS) Workshops

[00:00:00] Lucie: Hi and welcome to the IDEMS podcast. My name’s Lucie Hazelgrove Planel. I’m a Social Impact Scientist and I’m here today with David Stern, one of the founding directors of IDEMS. Hi David.

[00:00:18] David: Hi Lucie. We’re discussing our recent trip to Niger, Burkina Faso and Mali.

[00:00:25] Lucie: Yes, so we ran workshops for research method support as part of our research method support work for the Global Collaboration for Resilient Food Systems, exploring quite a few different aspects of research methods.

We’ve already had an episode to discuss the pre, our plans for it, and we were talking about the history of these workshops, which used to be more perhaps skills based trainings but this one was a new sort, I think, if my understanding is correct.

[00:00:57] David: I suppose it wasn’t that it was new in that we hadn’t done elements of this before, but I think in the format it was where we did the three country workshops and just two days in each country because that’s all we had time for and the focus of those two days was very conceptual. That’s something we’ve not done before and so it was in its nature something we haven’t done before.

[00:01:24] Lucie: We covered topics from qualitative research to big data and artificial intelligence, to having big goals, knowing how your research fits into the bigger goal, I guess is the way to put it and then how your different methods can fit into those.

[00:01:43] David: Absolutely, and all of this is in the context for a number of years where the program has been embracing agroecology. Agroecology has complexity inherently, it’s multi dimensional, it’s multi faceted. And traditional research is very discipline focused. And so it’s all in this view of actually pushing people out of their disciplinary focus into research which is taking into account complexity, which relates to agroecology in its multifaceted nature, and which therefore requires this diversity of methodology, which isn’t generally what’s trained, what scientists are trained in.

[00:02:28] Lucie: Yeah, I’m interested, you are very careful with your wording, I think, in how you’re explaining just there, and I think that’s really interesting, because it’s not a question of going deeper, for example, because, a lot of these researchers go very specific on soil, or on pests of a plant.

[00:02:43] David: In their discipline. They are disciplinary scientists. No, sorry, a lot of the scientists we work are no longer simply disciplinary scientists, but scientists are trained to be disciplinary.

[00:02:54] Lucie: Yeah, and we’re sort of encouraging them to think beyond that.

[00:02:59] David: Our role in the program is encouraging them to think about that, because it’s a program which sits at the boundary between research, development, impact. It’s very focused on the farmer centered approaches, which are inherently not disciplinary. Farmers are inherently having to deal with soils and crops and pests and everything is thrown at them. The climate change, all of these things affect the farmers.

And if you take that farmer centric approach, then it’s not that disciplinary science isn’t useful, it’s that it has its place within something bigger. And that’s very different from how we train scientists. No, sorry, let me rephrase that slightly. That’s very different from how scientists are trained and we are therefore trying to retrain scientists into this sort of other approaches. And it’s a challenge because this is in some sense putting us at odds with more traditional journals, pathways to success, institutions in general, where the disciplinary focus is desirable.

And we’re not saying disciplinary focus is bad for scientists. It isn’t. It is desirable in certain contexts. But if you’re wanting to be at that very impact focused, taking farmer centric in this context, and I should say that farmer centric is also criticised, because it’s not just about farmers, it’s also about consumers, it’s about society, community, but it isn’t just a discipline focused.

Thinking of farmer centric as opposed to discipline centric is a good analogy, even though it’s not just about the farmers, it’s about all actors within the…

[00:04:55] Lucie: Actor focused then.

[00:04:57] David: Actor focused, there you go. Never heard that one before.

[00:05:01] Lucie: No, yeah, it doesn’t quite have the same ring to it.

So I think in terms of the workshops that we did, it was the same structure of a workshop that was repeated in the three countries as the program works in the three countries. And I think there were differences in how, let’s say the workshop evolved in the three countries, even though there was loosely the same plan.

[00:05:22] David: And well, more than that, there were practical differences. You didn’t get a visa to Niger, which meant that you were not there, which actually was lucky because I was also not there for the first day because my flight was delayed and I arrived a day late. These are challenges that you have in this region, this part of the world, which was dealt with.

Yourself, Bettina, Batamaka, Aboubakari did really well on the first day actually managing, despite the fact that I was on a plane at the time. And interestingly, I don’t think it had a negative impact on the workshop. I think when I came in the feeling was that it had been a rich day of discussions, which was very fruitful and it led to a second rich day of discussions.

And the beauty of the nature of these workshops was it was very much less about us communicating specific things and more about creating this space for these researchers and students because there’s a whole range of people there’s also people from farmer federations, the whole range of people to have these deep discussions in the room. That was achieved in Niger and it was achieved differently in Burkina Faso and it was achieved differently in Mali.

[00:06:37] Lucie: Exactly what you’re saying there, in terms of the different actors in the room bringing different expertises. In the different sessions, it was interesting to see, for example, the NGOs being much more confident perhaps with the qualitative aspects, whereas a lot of the researchers were perhaps much more excited by the AI aspects. No, it’s not quite true to say that the researchers were more excited.

[00:07:02] David: What I think is true is that the qualitative methods for many of the biophysical researchers, they were outside of their comfort zone and they really weren’t clear how this could be rigorous. And this was a really important point, that it wasn’t that they didn’t see the value, and they very quickly saw the value, but that’s not science. Yes, it’s valuable, but it’s not science. Now this was very interesting, actually moving them beyond that. I don’t say that everyone moved beyond that, but to see how, well, yes, if you don’t put the work in behind it, it isn’t rigorous, but it is still useful.

But if you do put extra work to make it more rigorous in certain ways, then it is something which has a form of rigour which brings different information. And that I think some people really got. Whether they’ll have gained the skills to be able to apply it or not, who knows? Because you can’t teach that in two days.

Good qualitative methods will take years for people to really integrate. But what’s really interesting, and I found this in all three countries, was that once this was really presented with value, many scientists who were coming from much more quantitative approaches recognised that they were already drawing immense value from qualitative approaches without using formal methods.

And then I think recognised how adding formal methods to what they’re doing anyway could add further value. And I think that was something where this was recognised. You got some explicit requests for help. I’m quite excited at just what a big shift that is. And it’s really interesting that this is a shift which I wish this was happening more globally, this real respect for a diversity of methods.

[00:09:06] Lucie: I was just in a webinar yesterday about evaluation methods or monitoring methods for the agroecological transformations, and they were purely using quantitative data. And someone asked how about you use some qualitative research in there? And the answer was, ah it’s complicated and it’s time consuming and it’s difficult. Which, their answer is exactly what I think some of the researchers also that we work with, that was their a priori, that’s ah, it’s going to be hard to do.

We can do quantitative, and then we can get precise things, we can get the results we want, basically, and do it quickly. But it’s just because it’s what they’re comfortable with, as opposed to actually what will perhaps make sense for the research they’re doing with farmers and with the other actors.

[00:09:54] David: I want to challenge you there. I think they’re right. Doing quantitative stuff is much, much easier. Because you don’t have to worry about all the things you’re wrong.

[00:10:03] Lucie: You’re saying that as a sort of data scientist.

[00:10:05] David: No, I’m not. What I’m saying is, it is very easy to set up quantitative methods where you will get results. Whether those results are useful or not is another matter. And this is where I think the necessity of integrating qualitative approaches is there. And so I don’t think it is easy to introduce qualitative methods. I think it is necessary to do so, because otherwise you get misled. And so the fact that people say it’s easier, the answer should be clear, yes, but just because it’s easier doesn’t mean it’s right.

It is always time consuming to take in new ideas. Fundamentally, I love the distinction between quantitative and qualitative methods as being quantitative methods, you already know what the answers could be, and you’re quantifying the differences between the answers.

[00:11:01] Lucie: Yeah.

[00:11:01] David: Whereas with any qualitative method, you could get an unexpected answer.

So fundamentally, of course, if you get something unexpected, it’s going to take more time. But that’s where richness comes. So I’m not saying that they’re wrong to say it’s easier to do quantitative methods. No, of course it’s easier. It’s just not as good. Because you already know your answers, you’re just quantifying them.

In the qualitative methods, that’s where you get answers, which you may not have expected, which you may not know about in advance. That’s where richness comes. And you might not yet be able to quantify it, because it might just be one answer. But that one answer might give you insights which you wouldn’t have had otherwise, and therefore it does take more time.

If you’re not going to put time into qualitative methods, you’re not going to get value out.

[00:11:59] Lucie: I’m just wondering, there the time aspect is in the analysis as opposed the collection.

[00:12:05] David: Oh, yeah.

[00:12:05] Lucie: I know this is a side to all of our discussion here, but I guess it’s a side thing for me to reflect on how to speed up the analysis.

[00:12:15] David: My claim is you’re not wanting to speed up the analysis to make it quantitative. You could do that with AI, you just get AI to listen to it or pull out what it wants from it. But you’re still losing the richness. There are ways to speed it up. What I’m saying is the task at hand is not to speed it up.

What I find so powerful is that if we literally make that line of quantifying known versus discovering unknown, which is a simplification, a really big simplification, but an important distinction, an important line to consider. If we just consider that line, then, you know, well the side you can speed up is always going to be the quantification of the known.

The unknown, you don’t want to speed that up because that’s discovery. That’s where you actually could get things when you come out which open up new questions, which open up new avenues of research, which open up new things because there’s unknown in it.

So fundamentally, it’s not about minimising the time. It is about sometimes efficiently drawing out information and for that you can use tools which do make it more efficient, which draw your attention to certain things and so on. So yes, there are tools to speed up qualitative analyses now in ways which are fantastic and I believe AI is going to sort of improve that even further as people build better and better tools to draw people’s attention to things which need human attention.

At the moment, a lot of AI is focused on replacing humans, whereas my hope is as we move forward, it’s going to be about enhancing human capabilities. And if you think about that within the qualitative, if you had lots and lots of qualitative data, being able to have your attention drawn to interesting pieces would be a very useful and interesting AI tool. You don’t want it to do the analysis for you, but you might want it to guide you in some of the things you look at. Then there’s of course the question about…

[00:14:17] Lucie: You always worry that it’s going to be missing out things that are actually interesting, which it doesn’t consider interesting.

[00:14:23] David: There’s some wonderful examples, and I’m sure there was an episode in the past about AI missing out on the essence because it was too narrowly trained, and so on. So yes, you’re going to have those issues. But that doesn’t mean that it can’t be useful. This is the key. It’s a tool. It’s not intelligent. It’s a tool. That means that it can be useful. It shouldn’t be relied on. It’s all part of that discussion.

[00:14:48] Lucie: And AI was part of also the workshops that we were running. You led that discussion as a Q&A basically.

[00:14:55] David: It was totally different in the three countries. It was really interesting how just different people were thinking about different things, there were just different voices in the room. And the discussion was so different in the different countries. I found it very interesting. And in all three cases, I didn’t have an agenda beforehand as to this is what I want to communicate and get across. And so the conversation did naturally go in different directions just by the nature of the voices in the room.

But I think in all three cases there was, I would say from my perspective, surprisingly diverse sets of voices in the room. In all three countries, there were people who use AI, ChatGPT and other tools almost daily. And there were people who had never touched it and really didn’t understand anything about AI. That diversity in each room was very rich.

What’s interesting is, I think, a little bit about the difference between the different discussions was a little bit about what were the loud voices in each room? Were they people who were more familiar or less familiar?

[00:16:05] Lucie: Exactly. It’s group dynamics, basically that sort of perhaps stopped other people asking some of their questions in some cases.

[00:16:13] David: I don’t know, I tried my best not to have that, but I think, yeah, the conversation did tend to veer towards where there were people in the room who were willing to engage in discussion. Now, I think in all three countries, there were voices heard across the spectrum, but I think the discussion went in a different way in each country because there were people willing to engage in discussion who had different interests and different expertise.

[00:16:43] Lucie: Another aspect which was discussed was using big data in research which is another really interesting aspect, and we’ve discussed the concepts again, I think, previously within the community of practice, especially through climate data, I think.

[00:16:57] David: And other sources, many other sources of big data. And I think one of the things there, which I was really surprised that the discussions we’d had on qualitative and quantitative, once we got to big data, it was interesting how suddenly people were are we talking about qualitative or quantitative? Whereas to me, the big data it’s neither qualitative or quantitative data in the traditional senses, because I think of those three as being three separate things.

Qualitative data is a way of collecting data, which is qualitative in nature, and which could give you something unknown. Quantitative is a way of collecting data and designing ways to collect data to actually quantify things in rigorous ways. And big data, the nature of the data could be quantitative or qualitative, but you’re not designing it. The whole point is it already exists.

[00:17:53] Lucie: Exactly. So there’s so much of it though, also, that you can’t analyse it as a human oh, as in by hand.

[00:18:03] David: You shouldn’t analyse any data by hand nowadays. So that’s not really the issue. What I think is the case is that if the data is big enough, you can then use machine learning methods as opposed to just statistical or descriptive methods. And I think that is the thing where this idea of being able to use…

[00:18:25] Lucie: Sorry, that’s what I mean by hand, I mean doing things on the computer or something.

[00:18:32] David: It is a funny definition of by hand as not using artificial intelligence.

[00:18:37] Lucie: But it’s doing it yourself as opposed to…

[00:18:39] David: I understand what you mean. It’s an interesting statement of our society where that statement could even be made.

[00:18:45] Lucie: Yeah. I know you’re seeing the tool as an extension of yourself.

[00:18:50] David: Yeah. But AI is not yet a tool which is an extension of yourself. That’s different for you. But I think there is an interesting distinction, because machine learning, by definition, needs to have enough data that traditional statistics methods have capped out their need for data. Traditional statistical methods, they work in data scarcity, and so they are very good at making the best possible use of a small amount of data. But if you keep on adding more and more data, you very quickly get to the stage where the additional data isn’t adding much value, because the tools are designed to maximise the use of a small amount of data.

And so the additional data isn’t being used that effectively. This is how traditional statistical tools are designed. And machine learning, fundamentally, one of the ways of seeing it in its core is a way of making use of that excess data to have better learning than the traditional statistical methods.

This is not how many people would describe machine learning, but I find it a useful way to help people who understand statistical methods see why machine learning is different and adding potentially value. That it’s using that excess data that you don’t need to use your statistical methods because you’ve got enough data. And so that excess data can then be used to get additional learning using these machine learning techniques.

That’s one way to see this. And this is very different from how people think of artificial intelligence when they think about chat GPT, because that’s just mind blowingly different. The fact that that’s using excess data this is just different in the way it’s thought about. Generative AI is playing a different role in people’s minds, even though it’s fundamentally based on the same machine learning algorithms, but used on very large amounts of data in ways which are different.

[00:20:56] Lucie: Yeah, so there were lots of interesting conversations that were coming out of that in the three workshops. Do you have any learnings from running the workshops?

[00:21:05] David: It’s interesting that our discussion has really focused on the AI machine learning component, whereas I would argue one of my learnings was that it was actually the qualitative methods that was the most, important part of the workshop. I don’t think the AI machine learning part is going to be used. And we saw this when we got to sort of people presenting at the end. It was barely mentioned.

[00:21:31] Lucie: It’s new.

[00:21:32] David: It’s not just that it’s new, it’s that the qualitative resonated with things people are doing, but not formalising. Whereas the machine learning or the big data or the new data sources, the big data sources, it was just, this is beyond things that people can currently imagine how they could bring into their work.

[00:21:56] Lucie: Exactly. It’s a bit inaccessible still because you need, you know, even to find the right specialists, it’s harder.

[00:22:01] David: So one of the learnings which I took away is that if we want people to use AI, actually one of the key places I think could be through more use of qualitative. And this is where we come back to again the sort of point of actually formalising qualitative, integrating it into people’s studies more generally as a systematic approach at scale.

And then you get a lot of data which is time consuming to analyse, and actually having AI tools to help the analysis of that data is something which I would really imagine in five years time would become just natural, whether this is for the translation of that qualitative data into a language where you can do the analytics more, whether it’s on the identification on things where you draw people’s attention.

I’m not saying that this will replace, but in my mind, this is an area where in the work that we’re doing, if you want to hear the voices of many people, if you’re working with thousands of farmers and you don’t have AI tools, you’re not going to do the qualitative analysis.

And so I believe that AI tools will come in and complement some of these opportunities where we have partners doing field experiments with literally thousands getting on tens of thousands of farmers.

What if every single farmer as part of their field experiment recorded an audio in their native language, about their experience that season, or maybe multiple times through the season. What rich qualitative data that would be, what a goldmine of information, which would be totally inaccessible unless we have the right tools or an amount of manpower, which is unimaginable.

[00:23:58] Lucie: So, I find your vision a bit scary and I’m not sure about, I’m surprised by you saying it’s an amount of manpower that’s unimaginable, because what’s so interesting about some of the farmer federations and some of the other research groups in the region is that they have these structures of manpower which are incredible. The reason I obviously find it scary is because of things being missed and just mistranslated and taken out of context.

[00:24:22] David: So if you think about the manpower that they have, I’m not thinking of an AI which would replace that manpower. I’m thinking of an AI which would work alongside it, which would complement it. So that manpower is exactly the sort of one where you’re right, the hope would be that they would have that manpower, that although there’s tens of thousands of farmers for every 50 or so farmers, there is somebody who’s responsible for them, who would know all of those cases, who would have recorded those, and who would be able to sort of draw out and say wait a second, that’s not what was said. That’s not what he meant.

So there would be these potentially human verification loops in a way which could be extremely powerful and where you could be complementing the human and the artificial intelligence in very powerful ways. But I do claim that in terms of the research interest, we know that the manpower that’s in place does miss things which researchers would be interested in because it’s just, oh, that’s normal.

Yes, okay, they talked about the fact that this small change they made in their farming made a difference and they can now send their kids to school and they couldn’t before. Yeah, okay, that’s normal. But that’s not normal, this is a huge deal for a researcher to be able to draw out what a social impact could happen because of a change in farming methods or approaches.

So I think you’re underestimating the potential for symbiotic processes. I’m not saying symbiotic processes will happen, I’m saying they could.

[00:25:58] Lucie: Yes.

[00:25:58] David: And the whole point is to try and work towards understanding how could you create those symbiotic processes. That’s the sort of work which, of course, we can contribute to, but it’s much bigger than our partners. This is not in their hands. So these are the sort of things for us to think about and to be worried about, exactly as you’re saying. If we believe in this without being worried, we’d be irresponsible.

[00:26:23] Lucie: One of the things I found interesting from the workshop was a sort of distinction between, so lots of projects are doing social innovation in the sense that they’re designing groups for people to work together, but they’re not necessarily doing research about the effectiveness of that. So I guess, there’s an interesting distinction between doing sort of social research and doing social innovation. And so I think there’s something there that can be encouraged or discussed more within the community.

[00:26:52] David: No, I think this is very interesting because this is something which has emerged within this community over the years, where a lot of the projects have moved beyond their technical disciplines into areas of social innovation to be able to get to scalability of the outputs of their research.

I’ll give the concrete example of the entomologists who started by finding there was a natural wasp that could reduce the negative impact of a certain pest on millet. They then found that the rates of that wasp were too low to control the pest. But if you were able to increase the wasps in that environment to higher natural levels, which were still within the range of natural levels, then the damage on crops would be much lower. And this was actually well within what was natural for the region.

And so they’ve created mechanisms to be able to have these release bags of the wasp into the environment, which increased the natural levels. And it’s been very effective at controlling and reducing the damage of this particular pest across huge areas of society. And they’ve now gone into the social innovation aspect that you’re talking about of working with farmer federations to be able to have private units producing these release bags for their communities and working with local governments to sort of pay for this or to protect their villages.

So there’s all sorts of social innovation on that which the project has engaged with but has not really studied in the same way, because they are entomologists, so they’re mostly interested in studying the pests, the impact of the pests, the ecological system, if you want, between the pests and in this case the wasps and so on.

But they’ve engaged in these elements of social innovation without really studying them, where actually those social innovation components are maybe one of the most interesting pieces of the whole package that they’re working on. It’s that interesting balance, isn’t it?

And as you say, the opportunity to now formalise some of that learning to investigate that as a social innovation, it’s interesting that if this was in higher resource environments, you’d have had a whole different team do the social innovation. Whereas, often in the low resource environments, you don’t have that excess capacity. And so individuals get the opportunity to go outside their discipline and to work and to innovate in this way.

And I think there’s something exciting about that, that although it could be interesting, and it would be interesting to study it, I also think that there is something interesting about the fact that it has also emerged not from people who study how to do the social innovation. I’m not saying that people who study how to do social innovation wouldn’t add value. I’m saying I like the transdisciplinary nature of having people who are entomologists actually engaging in something outside their discipline in how to have social innovation. It’s not to say they do it better, but it is to say that I like that transdisciplinary aspect.

[00:30:39] Lucie: Do you have any final reflections about the workshops?

[00:30:42] David: We did stray from the workshops quite a lot, so maybe the last final reflection I’d like to have, which really is about the workshops, is that this is a region where there aren’t so many opportunities for interaction outside. Travel to and from the region is difficult and what I found very interesting was that these small two day workshops were enough.

I’ve always thought that the longer workshops are better. But actually in this particular case I felt the two day workshops they were enough to just stimulate discussion and stimulate reflection. And the fact that they didn’t have skills in them but were really about that reflection and discussion meant that it was very healthy length.

If it had been longer, actually people would have been out of the work that they need to be worrying about, without gaining concrete skills and so on, I think there would have been a frustration. But for two days to engage in these sorts of challenging discussions where they’re being exposed to thinking about research in different ways, and challenged to do that in their context, and then they go back to their normal work, I felt that was, surprisingly to me, enough and a good length.

And it’ll just be interesting to see how this emerges, or what emerges in terms of requests for support maybe by groups that don’t do qualitative work to engage in something more formal or what might come out. It’s an interesting process and it’s nice to me that, I think I mentioned this in a previous episode, I’ve been working in this region and this role for 10 years now and I’ve come out of these workshops feeling, oh, we’re starting something new in a way which I think is very interesting.

And it’s not that we haven’t had these sorts of deep reflection processes. It’s just the people in the room, the way they’re engaging, the way that they’re thinking, the way that it’s stimulating their thinking. They’re thinking about new things and therefore we’re engaged in new things. It’s really exciting. And it is an exciting place to be.

[00:33:00] Lucie: You’re right about this sort of short time period and, in a way, it is really a sort of privilege to be able to take just two days out of a project to really think about sort of the bigger picture again, basically to take yourself out of the project, take yourself out of the day to day grind of the skills and this applying these things and to think what else is out there? Yeah, so it’s going to be interesting to see what our next workshops will be like.

[00:33:28] David: Yes. See what the evolution, what the next process takes us to.

[00:33:31] Lucie: Great. Thank you very much, David.

[00:33:33] David: Thank you.