Why would you ever want an impact evaluation?
Five years after the launch of WFP’s Impact Evaluation Strategy, demand for impact evaluation evidence is higher than ever. Using examples from WFP programmes, we uncover the reasons why…
Jonas L. Heirman, Simone Lombardini, with Jennifer Waidler
Impact evaluations (IEs) are great for revealing cause-and-effect relationships and finding out what works, and what does not. However, they are not easy.
For WFP country offices, an impact evaluation is demanding on programme implementation and requires close alignment with an agreed impact evaluation design, and significant investments in primary data collection. For the WFP Office of Evaluation (OEV), impact evaluations require specialised teams with the right combination of technical quantitative analysis skills, management skills, country partnership skills, and commitment to stay engaged with a single programme for years. To add to the difficulty: WFP operates in some of the most challenging contexts in the world, which requires constant change and adaptation.
We are lucky, though. Demand for impact evaluations in WFP is higher than ever. In September 2023, we launched a call for expressions of interest for WFP country offices to apply and join one of our impact evaluation windows. Many countries responded to the call, and many more continue to reach out for impact evaluation support.
The WFP evaluation policy states that impact evaluations are demand-led, meaning that impact evaluations are conducted only when a country office asks for them. The definition of impact evaluation in WFP also requires us to use the most rigorous impact evaluation designs, which usually entails a type of randomised controlled trial (RCT) or experimental lean impact evaluation (A/B testing) designs. Furthermore, all WFP impact evaluations must contribute to both (i) the global evidence (e.g. existing literature), and/or (ii) programme learning, which means that country demand usually needs to align with one of our thematic ‘windows’.
Given all these hurdles, the first question we ask a WFP country office is: Why does the WFP country office want to conduct an impact evaluation?
Like anyone familiar with Simon Sinek’s work, we know that when starting a new impact evaluation, it is crucial to have clear motivations and objectives. In this blog, we highlight examples of how impact evaluations meet WFP’s needs and why more country offices want to do them again and again.
1. Impact evaluations strengthen confidence in the effectiveness of WFP programmes
Impact evaluations are a particularly useful tool to strengthen the confidence of donors, governments, and other stakeholders in WFP’s work and its impact. Since 2019, many WFP country offices requested OEV support to include impact evaluations in their programme funding proposals, ensuring they generate rigorous evidence to demonstrate the impact of future interventions.
For example, Germany’s Federal Ministry for Cooperation and Development (BMZ) was the first major donor to directly fund rigorous impact evaluations (RIE) for WFP’s resilience programmes. Since 2019, RIEs have become increasingly common practice in the German development cooperation, supported by BMZ, DEval, GIZ, and the KfW Development Impact Lab. Donor governments also face trade-offs regarding where to allocate funding, therefore it is important for them to know if money is well-spent and whether programmes are achieving its objectives. Our team was recently in Frankfurt sharing findings from an impact evaluation of the first phase of a resilience programme in South Sudan with the German Development Bank (KfW), and we discussed certain adaptations to the design that could be implemented to maximize beneficiaries’ well-being. Following the successful completion of RIEs in Mali, Niger, and South Sudan, the WFP and BMZ country offices are using findings to increase the impact of future programming.
Another example is the KOICA-funded impact evaluation of WFP’s resilience programme in Rwanda. The Rwanda IE collected high-frequency data to understand the impact of the programme on household resilience across seasons. When the IE data identified early issues in programme take-up (participating in the programme after being offered to do so) the WFP country office was able to immediately address them, increasing the potential for impact.
The visibility gained from IE data increases the confidence of WFP and its partners in the effectiveness and impact of programmes.
2. Impact evaluations inform programme operations
Another common request from WFP country offices is for OEV to support them by generating rigorous impact evaluation evidence for fine-tuning new kinds of programming.
Impact evaluations, particularly in the form of lean impact evaluations, are cost-effective tools that inform programmes before scale-up. We recently wrote about two impact evaluation pilots from WFP’s school-based programme in Burundi and how the evidence helped inform programmes at key decision points.
For example, in Burundi, using an A/B testing type of approach, the lean IE compared the school meal delivery outcomes from 50 randomly selected schools enrolled in the new commodity voucher model (direct cash transfers to schools to procure food from local farmers) with 45 schools remaining in the previous centralised delivery model (where WFP is in charge of procuring and distributing food to schools).
Findings from the first year of the pilot indicate that schools enrolled in the new model have a 75% higher number of meal days compared to the previous model, informing the scale-up of the new model.
3. Impact evaluations provide evidence and technical assistance to governments
Governments are WFP’s principal partners. National governments face difficult funding decisions with real trade-offs for their constituents and the future development of their country. In this context, WFP country offices are regularly requesting OEV’s support to generate rigorous impact evaluation evidence that could inform how they advise national Government partners.
In Jordan, for example, the impact evaluation was conducted in partnership with the Ministry of Education (MoE) to evaluate the National School Feeding Programme as part of Jordan’s National School Feeding Strategy.
In Guatemala, WFP, in partnership with the Ministry of Education (MINEDUC) and Ministry of Agriculture (MAGA), developed a School Feeding Management Application app with the aim of facilitating purchases from schools to local farmers.
Impact evaluation brings additional knowledge, expertise and ideas into programmes.
4. Impact evaluations contribute to the global evidence base and shared learning agendas
Long before the 2019 Noble Memorial Prize in Economics was awarded to three pioneers in the field of development impact evaluation, many viewed the evidence generated from rigorous impact evaluations (e.g., RCTs) as the ‘gold standard’.
For WFP’s country offices, rigorous impact evaluations offer an opportunity to generate evidence that informs global evidence in the areas they operate. The global body of evidence also shapes knowledge, debates, government policies, and donor funding decisions.
For example, USAID’s Bureau for Humanitarian Assistance (BHA) awarded WFP with a grant to increase our knowledge and experience in designing and conducting impact evaluations in protracted crisis scenarios and where conflict is prevalent. We are using these funds to conduct impact evaluations on forecast-based financing, targeting, and climate adaptation and foster knowledge on the transition from humanitarian support towards a path of development.
Similarly, Norad is interested in better understanding the impact of home-grown school feeding for smallholder farmers. As this aligns well with the School-based Programmes impact evaluation window, Norad and WFP partnered to conduct an impact evaluation in Malawi to assess the impact of the Home-Grown School Feeding Programme on the local economy.
In conclusion, impact evaluations are difficult, but for many WFP Country Offices they are worth the effort, time, and investment.
Does all this sound exciting? Contact us and we will be happy to explore whether an impact evaluation or lean impact evaluation is it right for you.
