Brusset: You see light even in the grimmest situations

Continuing our series of interviews with evaluation team leaders, we spoke to Emery Brusset, director of Social Terrain, on his recent experiences, new challenges in evaluation and hopes for the future.

You recently led a corporate emergency evaluation of WFP work in Myanmar. What was characteristic about this evaluation, the situation and how it was conducted?

The one salient characteristic is that we’re dealing with a closed country affected by a wide number of conflicts. People tend to see it as a conflict between governments and rebels, but actually there are multiple armies, factions, regions. That’s not very different to many other situations but here there is an extremely high degree of control and confidentiality in today’s contexts. I think we’re going to see more cases of information being misused. And so evaluation, having a disclosure role, may inadvertently create tensions, even though WFP itself is doing all it can to remain neutral, impartial, independent, and humane.

The evaluation itself is not just presented at the (WFP) executive board, but also has meaning for country office relations with the government, sub office or field office relations, and relations with opposition groups. And all of them have a stake in a particular message going out. That really influenced the evaluation because we had to take great care to how evidence was presented in the report, to make sure that there were no issues with the sources or the type of information shared, without compromising our own rigour. The country context also had implications for the conduct of the evaluation because we had to wait for travel authorizations, we had to use national consultants to a much greater extent. We managed to finish it within the original timeframe, but that was a feat.

In what ways did you collect data and were there any challenges?

The WFP Office of Evaluation (OEV) were clever in building the evaluation in modules, with each module happening in a particular sequence so that there would be a discussion with the country office and regional bureau as we went through. The whole idea was that the evaluation would be highly responsive to evolutions on the ground and to the needs of the country office.

We used four different data collection strategies. We knew from the beginning that some would work better than others. The one that worked least well was the satellite imagery because the country office was not used to collecting GPS coordinates on asset creation activities.

A method that worked better was the mobile phone survey, which we applied to peri-urban populations around Yangon. Though difficult to reach, WFP has information about them including mobile phone numbers. But as the questions are designed before you run the survey, when you collect the information you’re not sure if you’re getting the real story, why people give certain answers and not others. So there’s always an element of possible bias.

The third data collection method we used, which worked quite well, was to rely on WFP’s own post distribution monitoring. The team in Myanmar was superb in allowing us to add several questions, on for example social cohesion, to their food security and nutrition surveys, which were being conducted at the same time as the evaluation.

Data collection method number four was focus group discussions. In particular, a workshop with the implementing partners, which gave us unique insights into how the conflict is lived by them. Their descriptions helped us chart events, trends and drivers. We then looked at how WFP dealt with these drivers, what it was able to reduce, adapt to or mitigate, and to what extent shocks and stressors were taken into account by WFP in programming.

The data collected was useful for case studies. Three national consultants covered five case studies, which, apart from the peri-urban population, led to direct contact with beneficiaries and affected populations. The case study approach is really useful for unpredictable situations, conflict affected countries, fragile countries, particularly closed countries, where you need to pursue opportunities. The big challenge for this evaluation was that the case study approach was not compatible with the evaluation matrix, which tends to create a very linear approach. It’s a model that obliged us to look at lines of inquiry, set them early on, and then pursue those same lines of inquiry throughout the evaluation.

What do you consider to be the best ways to involve and feed findings back to affected populations, in terms of “closing the learning loop”?

In terms of closing the learning loop, important is going back to the implementing partners and, using the illustrative data dashboard to explain what we have done, and how we have done it.

We tend to forget that implementing partners are the ones talking to the communities. They are often members of the community, and by not engaging with them in these kinds of focus groups, by not having feedback sessions with them about the evaluation, we forego an opportunity.

For WFP to get back to the communities, staff have to use information products generated with the communities. It doesn’t need to be gossip and rumour mongering. It doesn’t need to be flashy events. It shouldn’t be pictures of WFP delivering food or highly polished social media content. It has to be a dialogue. It has to be a back and forth. And it has to be done in a visual way — communities don’t cope with long text and methods.

In evaluation, I think we are only scratching the surface. The tools we use are too extractive of information without returning the information in a way that could be consumed locally.

What do you consider to be the most practical skills an evaluator should possess in carrying out their work?

I think the ability to combine quantitative and qualitative data collection is key. Not at the national level but within a case study framework. If I had complete freedom, my preferred option would be to use case studies and then within the case studies apply quantitative data collection and qualitative focus groups with semi-structured interviews using national consultants. I think that’s the best way of connecting data.

Then there is the team itself, having junior project managers who are willing to go over the report, go back to the data, go look at all the detail and ask questions for quality control. This ensures that it’s not all happening in the field work by heroic national consultants, but also the people sitting back in the headquarters of the consultancies who read the reports carefully and probe inconsistencies.

And maybe the last important skill is patience. In the case of Myanmar, we started the evaluation in August, but were only able to get to the field at the end of January, early February. It was very sudden — we had the visa and had to go very quickly.

Having been involved in numerous evaluations across the globe, what aspects of your work do you find most rewarding? What drives you on?

I think there are two, one is less personal one is more personal. The less personal one is reward from our ability to observe change. The challenge of evaluation is that people quickly close the door when the evaluation is done. They don’t want the evaluators to stay around and meddle. We’re led into the kitchen, but quickly and very politely led out again at the end.

OEV has done something really remarkable, which is that evaluation now has the same status in WFP as audit. They are regarded with a mix of fear and exasperation because they’re seen as important, not a tick-the-box exercise as in many other organizations. People pay attention to them in WFP. The country offices really pay attention. So that’s rewarding.

On a more personal level, it’s our ability to work with different people collaboratively. That’s what really makes it quite unique. I’m maybe a bit different from many other evaluation consultants, but I do enjoy the teamwork and the team and the experience.

All we have to do is listen and write down what they say. It’s fantastic in a selfish way. You could say we’re being paid to be educated. But beyond that, there’s this whole vision of the world and you can see light even in the grimmest situations, some beautiful stories and really, really fantastic people.