Experimental evidence in adaptive programming: rapid, cost-effective tools for scale-up
Two impact evaluation pilots from WFP’s school-based programmes in Guatemala and Burundi helped to inform programmes at key decision points. This is what we learnt…
Simone Lombardini with Astrid Zwager, Minh Phuong La, Roshni Khincha, Florence Kondylis, Jonas L. Heirman and Cox Bogaards, Hannah Irmela Uckat, Dahyeon Jeong, Thiago Scot, and colleagues working on the School-Based Programme Impact Evaluation Window*
Impact evaluations (IEs) provide quantifiable, credible evidence to inform decision-makers. However, conducting them within real-world programmes presents certain challenges.
First, especially in the early stages, programmes need to experiment and quickly adapt based on the evidence and learning to ensure they have a feasible intervention ready for an IE.
Second, once a suitable intervention is ready for roll-out, IEs typically require a large enough number of participants to ensure they are ‘powered’ to detect impacts.
Pilot impact evaluations help WFP overcome these two challenges.
When time and resources are limited and evidence is insufficient, pilot IEs support WFP to understand if something could work before programme scale-up. In addition, they help to better understand the context and assess the implementation capacity.
Pilots are particularly well suited to lean impact evaluation methods, which employ rigorous experimental methods to compare different intervention modalities (e.g. A/B testing).
Like other IEs, they require close alignment between programme implementation and evaluation design to enable rigorous comparisons. However, rather than focusing on outcome-level changes, pilot IEs focus on output-level changes, such as implementation quality and service delivery to targeted groups. The aim is to understand feasibility before scaling.
This blog highlights lessons learned from two pilot impact evaluations under the School-based Programmes Impact Evaluation Window and their contribution to informing programmes at key decision points.
Burundi — towards a decentralised procurement model
In Burundi, WFP’s school feeding programme used a centralised procurement model to deliver food to schools. In 2022, WFP piloted a new procurement model based on commodity vouchers with local cooperatives distributing directly to schools. The model aimed to increase the proportion of locally produced school meals.
In partnership with the Burundi country office, WFP’s Impact Evaluation Unit and the World Bank’s DIME department embedded a lean IE into the implementation of this new decentralised procurement model to assess whether it impacts the performance of meals distributed by schools (e.g., quantity, diversity, and quality of meals).
The IE compared the school delivery outcomes from 50 randomly selected schools enrolled in the new procurement model, with 45 schools remaining in the centralised model.
Findings from the first year of the pilot indicate that schools enrolled in the new community voucher model have a 75% higher number of meal-days compared to status quo schools, with the meal-day coverage in the new community voucher model reaching nearly 100% in February 2023, compared with 50% in schools in the old model.
While the scale of the community voucher model in its first year is not big enough to have power to detect statistically significant changes in farmers’ revenues and children’s outcomes, the results from this lean IE are informing the scale-up of the new procurement model and guiding the feasibility and design of a larger-scale impact evaluation.
The programme scale-up is currently under design, and the impact evaluation will assess the impact of Home-Grown School Feeding on farmers’ income and children’s learning.
Guatemala — technological solution aiming to connect farmers and schools
Guatemala has close to universal school feeding coverage. To encourage procurement from local farmers, WFP’s country office developed a smartphone app to improve schools’ procurement systems and better connect them with local farmers.
A pilot impact evaluation assessed the usage of the app and its ability to connect schools with farmers, before rolling the app out at a larger scale. The IE compared food procurement practices in 108 randomly selected schools where school officials and suppliers are trained in the use of the app, with 102 randomly selected schools where the app is not yet promoted.
Six months after the launch of the app, the first round of school survey data suggests that while 94% of the schools are aware of the app, only 5% completed a purchase using the platform. Internet connection is the most frequently mentioned challenge preventing app use.
The pilot enabled the evaluation team to document operational challenges and identify possible solutions in the rollout of this new technology. It also informed the evaluation team before investing in a larger-scale impact evaluation, which would have been affected by the low take-up of the app.
Lean impact evaluations provide evidence for scale-up
WFP’s experience in Burundi and Guatemala shows how lean impact evaluations during pilots can be rapid, cost-effective tools for generating valuable evidence before the scale-up of new intervention modalities. Pilot IEs also help ensure conditions are in place for successful implementation before conducting a full-scale impact evaluation.
*Colleagues working on the School-Based Programme Impact Evaluation Window include Anna Subirana Bofill, Assereou Atekou, Astrid Zwager, Benedetta Lerva, Cox Bogaards, Dahyeon Jeong, David John Horner Ibler, Florence Kondylis, Erin Kelley, Gregory Lane, Hannah Irmela Uckat, Jonas Heirman, Minh Phuong La, Paul Christian, Roshni Khincha, Simone Lombardini, Thiago Scot Vedarshi Shastry and all the colleagues from Country Offices, Regional Bureaus, and HQ’s School-Based Programmes.
