03 – Principled Evaluations

[00:00:00] Lucie: Hi, and welcome to the IDEMS Principle. I’m Lucie Hazelgrove Planel, a social impact scientist, and I’m here with David Stern, a founding director of IDEMS. Hi, David.

[00:00:18] David: Hi, Lucie. This is our first discussion together about principles, and we’re not digging into an individual principle today, are we? We’re discussing principles for monitoring and evaluation, I’d say, as part of our organisation. Is that how you see this?

[00:00:34] Lucie: Yeah, so IDEMS has a sort of tripartite view of monitoring and evaluation, and one of those legs, if we think of a three legged stool is principles. It’s not really clear to me, I’ll be honest if we’re evaluating what effect we have had, or if we’re evaluating our own principles.

I think we’re doing both.

[00:00:57] David: Yes. The principles are a way to help us evaluate what effect we’re having. But they’re also something where we should be evaluating those principles and how well we are abiding by them, using them, and how they’re serving us, what, what purpose they’re playing within us. I think it’s worth mentioning the other two legs, very briefly, that we have the idea of pathways of change, which comes from theory of change, but it’s a methodology which we’ve adapted a bit into something rather specific. And that’s really used for a specific intervention or activity where we want to actually say this is what we’re doing and this is why we’re doing it and we believe this will happen and if we don’t observe that then either we’ve done it badly or we were wrong.

So that’s sort of our pathways and we found this useful in a number of different contexts. And then there’s our value creation stories. They’ve actually got a new name for it. It’s come out of communities of practice, evaluating communities of practice.

[00:02:02] Lucie: From Wenger and Trayner.

[00:02:05] David: Yes, I think that’s correct.

And it’s something where they talk about five different types of value, and how actually by understanding and thinking about those different values and following them at different points in time, you can actually not only be aware of and monitoring if you want intended consequences, but also unintended consequences or, or simply interventions, which led to things which were then opportunistic, which is what often happens within a community of practice.

And so to, to understand the importance of a community of practice, you often need to, to be open to measuring in some form things that were not planned in the same way.

[00:02:51] Lucie: Absolutely, no, and it’s the side outcomes can be both positive and negative, so you should definitely be trying, trying to capture them and identify them in order to better learn also to see, were those things valuable or not, and how could they be avoided in future.

[00:03:07] David: So in some sense, our three legs are value creation stories serves exactly that role of capturing the different components and trying to bring in things which were either maybe intended or not, but cover that whole spectrum. The pathways is all about intentionality, and it’s about our understanding of how things work.

Does it happen in reality? Do we get what we expect when we do things in the way we believe they need to be done? And the principles – are they actually leading to the results that we hope they will be? Are they effective? Are we making the right choices? Are we causing ourselves harm by focusing on something which is actually misguiding us?

[00:03:50] Lucie: In terms of the principles — and principles is surely a clear word, but I think I just forget it — including principles in our evaluation approach is really about seeing if we’re right in our belief in what changes we want to create and how to create those changes?

[00:04:05] David: Well, that sounds much more like a pathway to me because there you’re saying this is the pathway to this change and this is what we need to do to get that change. These are..

[00:04:14] Lucie: Okay, idealistic changes then. Is there a difference between…

[00:04:17] David: but I don’t see principles being about changes. By definition, I see principles as being about choices. I hope as we go through each of the principles, we’ll be able to pretty much every principle say, well, actually, there’s a good alternative. If a principle is just, well, that’s what you should do, then it’s of no value.

A principle should help to make a choice in a particular case, in a particular context. And when I was with Santiago, I discussed very much this aspect of the guiding principles, the principles guiding and helping you make those choices.

But if you think about it from a more monitoring evaluation or research perspective, what’s the impact of making this choice? When should you make this choice versus another choice? Now, the whole point is that the principle shouldn’t be self-evident. It should involve some element of choice and therefore evaluating, it’s not always going to be the best choice. It’s going to have trade-offs. Understanding the trade-offs related to it, understanding when a principle is good and when it’s not so good. I mean it’d be fantastic if people actually sort of dug into us as a case study and actually try to untangle some of the, the mess of, of, I think our principles are rather interrelated, so to speak.

[00:05:37] Lucie: I think we’ll be seeing that in the next podcast, yeah.

[00:05:41] David: I don’t think I’ll be able to talk about one principle without actually getting distracted by another. They really do interrelate to one another. But I believe, and this is again where in terms of understanding, and the understanding we place on this, and people being able to study this, both internally and maybe externally.

If the principles are powerful and are valuable, then there should be complexity around them and around the choices that are made and about how we’re making choices which relate to the principles, or don’t, when we’re struggling with certain principles and when we’re not. I don’t know at this point, whether the principles themselves are going to be there to guide us or to show us that you’re not following your principles, you know, you said you’d do this, but you’re actually doing that. And that would then help us course correct. Or we change the principles, possibly. I don’t think so. I think, unlike the pathways of change, where you then change your theory because your theory is wrong, the principles are much more, they’re much more our values.

[00:06:47] Lucie: Exactly, that’s what I meant when I was trying to say that, as a word, it is fairly obvious principles, and yet, because these principles have a meaning in and of themselves, I’m sort of keep forgetting of the original word of principles, which is our values as an organisation and so, how our values affect our choices then.

[00:07:07] David: Exactly. Yeah,

[00:07:08] Lucie: I like that distinction you made between principles and the pathways of change. Of one being about choices and one being about actions and activities.

[00:07:17] David: Well, your pathways are built up from the activities. They’re the only thing you can control, whereas our principles, they’re about making the right choices in different ways. And really, how do we make our choices? And what do we think about when we’re making choices? So I guess the key question is then, well, what does it mean to have that as part of our monitoring and evaluation?

Part of what we need to do is we need to understand: are we making choices aligned with our principles? That’s a very simple thing to try and do. Can we, can we go through?

[00:07:51] Lucie: It’s a simple question, but I think it’s a difficult thing to actually evaluate in itself.

[00:07:55] David: Absolutely, because our principles are complex. And so, yeah, it is a very difficult thing to actually do.

[00:08:00] Lucie: And we make so many choices.

[00:08:03] David: Exactly.

[00:08:03] Lucie: At so many levels, every day.

[00:08:05] David: Yeah. And the big ones, of course, are easier in some sense than the many small ones. But it is about the small ones as much as it’s about the big ones. And that’s why the principles are so powerful, that they should be guiding us in our decision making at all levels all the time. And if not following that, then are we getting lost?

That’s where thinking of structures to be able to help us evaluate this, and have some elements of monitoring related to this is really important. And we should be clear and transparent, but at the moment we’re just too small. As an organisation, we don’t have the extra capacity to have a whole arm studying ourselves.

[00:08:48] David: But we already have the structures in place to enable that arm to grow as we grow as an organisation. We already want to have these opportunities. We actually do already have some interactions with academic partners who are possibly interested in studying us and studying our principles and maybe that’s the way this will start.

[00:09:08] Lucie: Yes, interesting possibilities there.

[00:09:10] David: Yeah. I think this will start in ways which are ad hoc, because it’ll have to be. We’ve had these principles, most of them since before IDEMS existed, in some form, they’ve changed a bit, but they’ve evolved – one of our principles, continually evolving – but things will evolve in the future as we dig into them, and as more people engage with them, and as we start to have them challenged more, and then hopefully as we start to have them studied.

My hope is that each of our principles become areas of study for us, where we can check, are we doing this? If we’re doing this, is this actually paying off? Is this of value to us as an organisation? And the communities we serve? If we had teams studying that for our principles, it would be amazing.

[00:09:56] Lucie: Well, so as a whole, the principles are in our monitoring and evaluation plan to really help us. I think you said this actually yourself just a few minutes ago – but to help us as an organisation, see if we are living up to what we want to do and to keep us aware of that and conscious of it.

[00:10:11] David: Yes. The difference between principles and values is very interesting. I like principles because in some sense it’s a way of manifesting our values. Do you understand that or should I try and dig into it?

[00:10:25] Lucie: I don’t understand the difference between principles and values.

[00:10:29] David: So I would argue that many organisations choose to have values. But values don’t necessarily help you make choices because they’re not worded as such. They’re not articulated in such a way. They say what you value, but a good principle is something where it actually helps to distinguish itself from something else which you could also value, but which you’re saying, our principle says we’re going to try and focus on this, we’re making this choice. It should help guide your choices.

[00:11:04] Lucie: And something else you mention sometimes with principles is that it’s not always clear that it’s going to be the right decision, or it’s not always clear that that principle will always be the case. Whereas I think values are much more you can’t really argue against that value.

[00:11:18] David: Exactly, exactly! And you shouldn’t argue against people’s values. People are allowed to hold to value what they want to value. You should be able to argue with people’s principles, and challenge them on them. There’s a very distinct difference in how they’re worded and what they represent. I will make an effort in the individual discussions.

Because this is our second talk about principles, and we haven’t actually gone through any of the principles at this point. But as we go through the principles, we’re going to try and make sure, wherever possible, we state what the principle is against. Which is also a sensible choice.

That it’s not that we’re saying we’re right and you’re wrong for not following this principle. In many of the cases I will be probably arguing, many people would be better off not following this principle but following that.

[00:12:09] Lucie: an unusual approach.

[00:12:10] David: Yes, that would be easier and it would be better in certain ways, but we’re doing this because, and we’ve chosen this principle because, and it then really helps people to understand where we’re coming from and why.

I think our principles are deeply entwined to our nature as a social enterprise. That’s something which will probably come out quite strongly. If we were just evaluated based on our profitability, then I don’t think our principles would be appropriate. I think as an organisation, we are a social enterprise, which means that we have multiple indicators of value. By definition, we are valued by our profitability because if we’re not profitable, we don’t exist. We are a business. And so we are valued on our profitability, but we’re also valued on our impact on how we serve our community. And so that nature of the business is, I believe, central to the principles.

These principles would not be the right choice if we only had a single measure of success, which was profitability. The fact that we actually have this more complex set of indicators for success, and that’s possibly what the principles are there to provide us. If we could be thinking maybe 10, 15 years down the line of actually evaluating ourselves against our principles as being a way of indicating our success.

We’re doing well on this principle and we’re doing well on that principle and ways of indicating our success that might be really useful. And if we did have that, then one of the principles would almost certainly have to become something different because we don’t have things like fundamentally profitable as one of our principles.

It is part of social enterprise and how we describe the social enterprise, but it’s not one of the principles. Maybe it should be, and there’s maybe a whole set of principles missing around our social enterprise nature. We do have another set of principles, of course, which are staffing principles.

[00:14:22] Lucie: Yeah, I like those.

[00:14:24] David: Yeah.

[00:14:24] Lucie: They’re quite unusual.

[00:14:26] David: Well, I’m glad you like them. They’re something which we thought long and deep about, and we don’t discuss them much.

[00:14:34] Lucie: We do in the mentoring scheme! We do in the company mentoring scheme.

[00:14:38] David: Oh, interesting. I’m delighted by that. I’m also slightly surprised I didn’t know that.

So I think one of the elements related to that, and this is where principles as an approach, do come in in different places within the organisation. And I think we haven’t yet figured that out as much as I would like to. I mean, the fact that we have staffing principles which are separate to our principles was necessary when we set out.

They both predate IDEMS. They both existed before we set up the organisation. Think about that for a second, because you know the staffing principles if you’ve gone through them in the mentoring scheme quite well; we had those principles when there were no staff. We didn’t have any staff in the beginning. It was myself and the co founder for almost two years!

So, well, I suppose the staffing principles were used earlier. But we didn’t have a full time member of staff until almost two years after we started, and yet those staffing principles were just sitting there dormant. And yet we do try and, and use them and think about them. They integrate into the way we work and the way we treat our staff.

[00:15:49] Lucie: Well, exactly. Some of the principles have similarities with the ‘the way we work’ principles because a lot of those principles are about internal things as well as external things.

[00:16:01] David: I’ve never called them ‘the way we work’ principles.

[00:16:04] Lucie: Sorry

[00:16:05] David: On our website, we have a section called The Way We Work. Under that Way We Work, we have a section called Principles. So it does make absolute sense for you to call them like that, but it’s just I’ve never heard them called in that way before. I don’t know whether I like it or not. I’m afraid I’m going to have to think hard about that one.

I’ve always thought of them as company principles, that, that’s how they were always intended. These are the principles of the organisation. Which is broader than just the way we work. And I, so I, I think, I think I don’t like ‘the way we work’ principles.

But it might be that there are other principles that would be good to have as way we work principles. And then maybe we can have a smaller set. ‘Cause 20 principles is quite a lot.

[00:16:47] Lucie: I think you just like principles.

[00:16:49] David: I do. I find principles incredibly powerful. I mean, we’ve got a whole nother podcast series on agroecology and that podcast series and the fact that we engage with agroecology as we do is 95 percent on my behalf because they have principles.

[00:17:07] Lucie: So what, what makes a good principle then? In terms of if we’re going to review a principle?

[00:17:12] David: Well, to me it’s this duality. Good principles are, as we described with Santiago, they’re about guiding decision making, but they’re also about being able to hold you accountable.

They’re about the monitoring and evaluation. You should be able to then, look at the principle, and judge your decisions against the principle and be able to evaluate and say, how are we doing with respect to this principle? Are we I don’t know, embracing diversity is one of our principles.

How well are we doing in terms of embracing diversity? Have we got areas where we need improvement? How well are we being inherently inclusive? That’s another one of our principles. If we’re not being inherently inclusive, then alarm bells should ring and we should change what we do. This should affect us.

Because the principles should guide the organisation in the decision making and they should also serve to help hold us to account and say, well, hold to account is maybe the wrong word. I don’t like that language. But they should help us to notice, to learn, to evolve.

I like continually evolving, that’s one of our principles. They should..

[00:18:21] Lucie: And the learning aspect too.

[00:18:22] David: Yes, yes, to help us in that learning, so that the evolution of the organised is directed. This is the point, they give directionality to how we as an organisation evolve. Without the principles I believe there’s a real risk that when we get down to the day to day details of what we do, how we work, we just get caught up in what we’re doing at that point in time and then we might end up somewhere we didn’t want to be.

The principles give us directionality. We talked about them as guiding principles and they help us to course correct if we monitor are we going in the right direction or not. So they should help with the course correction, they should help with the directionality of the organisation. That’s what they’re there for, they’re there as guiding principles, they’re there to help us with course correction through monitoring and evaluation.

[00:19:17] Lucie: And if if we look at what Michael Patton, who wrote the book on principle focused evaluation, and I think I’m correct in saying THE book on that. So I think he says then that we should not only use the principles to evaluate a company’s own progression, but also we should be evaluating the principles themselves. And if they’re useful, and I think you’ve mentioned this too.

[00:19:39] David: And this is part of this duality that it’s if we find that the principles aren’t guiding us in the right way, then of course, principles should be evolving. Our principles have already evolved. They’ve gone through a few evolution cycles.

And we’ve got a meeting coming up where we’re going to be seeing again whether they evolve. I think there’s, there’s a good chance that there’ll be at least a few wording changes, maybe not huge, but I would expect every time we revisit them as a group, we think hard about them, then there tend to be slight changes in the wording.

And that’s good, and this is improving and evolving to represent what we have as our culture, as our common cohesion. This brings the coherence within us as a group.

[00:20:25] Lucie: You’re telling us by changing the principles, you’re communicating the learning that you have developed, I guess.

[00:20:34] David: Yeah, we’re stating the difference between what we believed before and what we believe now. The principles represent what we believe we should be following.

[00:20:45] Lucie: And there’ll always be a reason for why we have changed.

[00:20:49] David: Well, and there’s a reason for why we have chosen this principle, and this is what’s been so difficult to communicate with the wider team. Broadly, I would argue that Danny and myself, the two founding directors, we very quickly agreed on every single one of the principles.

Well, actually there are only 16 to start with, there are now 20. So we agreed very quickly on the 16 principles. This was us putting down together, what we both collectively, or collaboratively, agreed. As our team has grown, people don’t understand them, and we don’t have the same beliefs, and we can’t communicate some of the differences between our beliefs.

But we have been able to put down the principles. What we’ve not been able to do is communicate these well enough to explain some of the depth, part of the reason for this podcast series.

[00:21:36] Lucie: Exactly, and so that’s something that’s interesting to go into at a later stage. How are we using the principles and how are we communicating them within the team? How are we encouraging people to reflect on them and use them in all of their decision makings?

[00:21:52] David: Well, and I think the important thing there is it’s not necessarily all of their decision making. It’s in their decision making. There’s a subtle difference there. These aren’t rules to be followed.

These are guiding principles. They’re principles to guide you. That doesn’t mean they have to always be followed. And, and, and therefore in the evaluation, recognizing there might be some principles which we really struggle to follow.

And that might be because the principle is one which, for whatever reason, we’re not able to live up to. Or we actually find that we believe something else in certain ways. Or it might be because we’re not doing things in the way we need to, and then it’s a way of actually identifying ways we could try and do things differently in the future. So using them to sense check the decisions that are happening across the institution is something which would be really powerful.

I do see the principles as a way of being able to distribute decision making. But I think almost certainly one of the things that will come up as we start doing that then is that the principles will be needed to be able to engage in discussing decision making with people as they make decisions which are not aligned with what we want in terms of the company.

This is something which is really useful. So I have struggled at times handing over decision making power in certain ways, because when people are suggesting certain things, they don’t align with the principles and pretty consistently, I’ve been able to use the principles to highlight why what somebody’s suggesting, doesn’t align with the principles.

Now, that’s powerful because I want to give you independence of decision making, thought and process. But if I disagree with you, and there isn’t a well defined reason why, then there’s real tension between me using my authority to undermine your role versus if I’m able to say I don’t think that’s right, because actually if you think about it with our principles, then this is the principle that it doesn’t align with.

Now, you have two choices. You can now say, ah, okay, I understand that, I like that principle, I want to include that principle, and then you think about how that comes into that decision making. Or you can say, wait a second, If that’s what that principle means, I don’t agree with it. I don’t like that principle. That principle doesn’t work for me. Both of those are valid reactions, and both of those are constructive. You, challenging our principles, saying we need to change these principles because, look, that principle went against me doing this, and I think that would have been good, it would have been good for me, it would have been good for the organisation.

The principle is holding us back. It’s wrong. Great… I look forward to those discussions. Similarly…

[00:24:46] Lucie: well, I think I have been in the case where I knew that one of the principles of the company was to be open and collaborative. And to me that doesn’t always come naturally.

[00:24:56] David: Yeah. That’s one of the ones which actually, I think people find really, really hard, especially people who have been in other organisations and having experience elsewhere, where the natural thing, the natural element of an organisation is once you’re in, you’re competitive against people who are out.

[00:25:13] Lucie: Yeah,

[00:25:13] David: You want to see your organisation succeed at the expense of others.

[00:25:19] Lucie: Unfortunately, exactly.

[00:25:21] David: And so this is collaborative by nature. That is a really, really tough principle. And I know that one’s come up with almost all staff members. This feels unnatural. Why would we be doing it like that? That doesn’t make sense. Collaborative by nature, I’m afraid.

Anyway, that’s a nice place to, I think, end because this is exactly where thinking about this as an evaluation tool where you come to me with an idea or with something you want to do and the way you want to do it. And if I can point to the principles, that helps you to then challenge your thinking with respect to the principles or with respect to what you’re doing. And it makes that discussion much more practical and I believe it enables this coherent decision making. It’s hard though. It is really hard because it’s counterintuitive. Some of the principles are really counterintuitive.

[00:26:15] Lucie: Yeah.

[00:26:16] David: They sound right, but when you try and apply them they can be counterintuitive.

[00:26:19] Lucie: Well, I look forward to the following discussions or to our next discussions where we dig into those and hear about their complexities.

[00:26:27] David: I’m afraid I think complexity might come out quite a lot. But that’s why we need tools to help us to evaluate it and to monitor things in that context. Anyway, thank you very much. And this has been an interesting discussion, and I look forward to this whole series. I mean, there’s 21 more episodes, 20 principles, and then I believe we’re going to have a conclusion.

So it’s a whole load more episodes about the principles.

[00:26:51] Lucie: We’ll see you soon then, David.

[00:26:53] David: See you soon. Cheers. Bye.