Dean Karlan: “We need to think bigger, more collectively to improve cost-effectiveness”
USAID’s chief economist shares his reflections with WFP’s Southern African spokesperson, Tomson Phiri, during the inaugural WFP Impact Evaluation Forum held in December 2023
THOMSON PHIRI: Today I’m in conversation with Dean Karlan. Dean Karlan is USAID’s Chief Economist. Dean, welcome. So good to have you here. How long have you been in the job now with USAID?
DEAN KARLAN: I have to stop saying I’m new. It’s just over a year now; a year and a couple of weeks.
One year into the job, if you had a crystal ball, how would you predict the development landscape to look like in 5 to 10 years?
Better, incrementally. Some of that is going to happen just because of growth. Overall long-term trends do tend to go up, but there’s a lot we need to do to make sure that the growth is inclusive. In terms of policy, we continue to see a positive arc of being more and more evidence-based over time. And I suspect that in five to 10 years, we’ll see cash transfers being more and more common as they deal with a lot of the issues that we face.
When we look at the world right now there are indeed multiple issues. What should be the key focus for development and humanitarian organizations? What are the main challenges, and how should we position ourselves to be able to respond better?
So two broad thoughts: One, we’re always going to have short-run and long-run problems. No one is ever advocating for an approach that says we should only deal with short-run, or we should only deal with long-run problems. Where exactly we draw that line in terms of resources, is a challenging decision to have on your shoulders.
Now, there is a sweet spot when you can address short-run problems and have long-run impact. This should be something that we continue to search for as a way of helping to bridge what seems like a tension.
From an evaluation perspective, one of the reasons why we have a hard time finding those sweet spots is because we sometimes get the timing off when we measure impacts. We don’t measure short enough and we don’t measure long enough. We are focussed on one-year results a lot of time, as the standard way of doing measurement. But we actually also need to know more about the really short-run impact and then the three-year, four-year impacts to be able to guide those decisions.
The other thing that we are going to see happening is that there are various ways of collecting data that are going to transform our ability to pre-empt issues and act more rapidly. As the world gets more and more data, more and more cheaply, we’ll see more abilities to predict catastrophes and pre-empt problems.
Everywhere you look you hear people saying that resources in the humanitarian and development sector are going down. How can we better pull resources and work together to overcome some of the challenges that you highlight?
There are some very clear public goods that we need to be better about sharing. One of those is knowledge. The knowledge about what’s cost-effective and what’s not then also helps us allocate scarce resources to do the most good we can.
There’s no point in keeping that knowledge to oneself. And likewise, the production of that knowledge, if done collectively, is actually cheaper.
We also need to understand more about the cost structure of implementing policies, where it’s not about the intervention per se, but about how we implement it. Doing things piecemeal is driving up costs without a benefit that accrues to people, so we need to think bigger, more collectively and that will help improve cost-effectiveness.
We are moving away from subjective feelings about programmes towards whether programmes achieve significant and sufficient impact now. What were the drivers for this?
We have an ethical duty with our resources to do the most good that we can not just for today but for the future as well.
It’s an ethical calling, so to speak, that we all share in the humanitarian and development space. And with that calling comes a demand for rigour to make sure that things are working, and that we achieve the things we seek to achieve.
Why is it that even with this deliberate shift towards focusing more on impact, many organizations still make relatively limited investments towards rigorous evidence generation?
Two thoughts: One is the world is a complicated place and there are lots of different opinions, and not one way of doing things. That’s just the nature of the world. The other recognition is that not everybody should be doing impact evaluations. That would be way too many impact evaluations. That would be too much money spent on data and impact evaluations, and not enough money spent doing things.
It is important that we as a community come together and identify what are the important knowledge gaps and what’s the most efficient way of addressing those knowledge gaps and building knowledge.
Not every single organization out there should be doing impact evaluations. There should be innovators that can test and pilot, and have an impact evaluation that’s rigorous to show their results. And then there will be other organizations that are simply using the best of the information to design their programmes, to adapt to what the evidence is saying works, and to run those programmes without doing impact evaluations.
What are some of the strategies that we can use to encourage the sharing of impact evaluations?
Gatherings like the WFP Impact Evaluation Forum are really important. But it’s also really important to do gatherings like this in countries where the work is with local government.
When we think about how we can make an impact in the evidence space, we’re small compared to the resources that local governments have. So why should we run a programme that is one-fifth the size of the government’s similar programme, independent of it? Instead, it could be collaborative, our resources can go towards helping to learn and being an on-ramp to improve government policy.
Obviously, different governments have different appetites and have different strengths of institutions so it’s not possible everywhere, but when it is possible, that is a clear path to scale in terms of improving policies.
What do you perceive to be the burning issues for evidence generation in the development and humanitarian context today, but also in the next five years?
There are two things we’re seeing rapid change in. One is what I just spoke about where we’re seeing more integration with local stakeholders, local government and an on-ramp to larger-scale policy.
Secondly, our data are getting continuously cheaper and cheaper, and multifaceted. For example, satellite imagery, cell phone data, and different remote sensing data are transforming both our predictive ability for targeting, but also potentially even for measuring impact, particularly when it comes to environmental and agricultural programmes.
We still have to apply our minds as to how we use such data. To measure impact, often it will be cheaper, but at the same time a bit more ‘blunt’ in terms of its ability to pick up changes.
Dean, it was an absolute pleasure and I’m wondering if you have any final thoughts?
It’s really exciting to be here with the World Food Programme and all the various parties and partner agencies. There’s a huge amount of energy. This area of the humanitarian development nexus is an absolute critical issue. It speaks to one of your earlier questions about the short-run and long-run, and that is the exact space working with populations that are vulnerable, but not in that emergency moment, just a little past that, so we can actually have the luxury of trying to address the short-run issue while also thinking about the long-run. And it’s an area that is really ripe for understanding how to generate stronger support and resilience for households and people facing those types of more recent catastrophes.
