“Lean” impact evaluations: experimental evidence in adaptive humanitarian interventions
By Felipe Dunsch, Simone Lombardini and Jonas Heirman
Impact Evaluation is an important tool for assessing what works in development and humanitarian interventions, providing rigorous evidence to inform policy and contribute to the global evidence base. However, not every context or programme is suitable for traditional impact evaluation approaches. Many interventions supported by the World Food Programme (WFP) are delivered in rapidly evolving contexts and aim to meet immediate food needs. These rapidly evolving contexts require a lean and adaptive approach to testing and optimising life-changing interventions.
WFP is the world’s largest humanitarian organization, saving lives in emergencies and using food assistance to build a pathway to peace, stability and prosperity. WFP’s emergency operations use a wide range of interventions, including cash-based transfer, food assistance for assets, nutrition supplements, etc. depending on the country context and emergencies encountered. The objective in all contexts is saving lives while also enhancing the resilience of individuals, households, and communities, to ensure they maintain food security and access adequate nutrition during before, during and after shocks.
WFP’s novel humanitarian impact evaluation workstream is a cross-cutting initiative supported by USAID’s Bureau for Humanitarian Assistance (BHA), and delivered with partners including the World Bank’s Development Impact Evaluation (DIME) team. The workstream aims to bring the practice of impact evaluations into the humanitarian space by piloting and supporting approaches suited to emergencies, such as ‘lean’ impact evaluations. The new workstream cuts across WFP’s already existing impact evaluation windows, which generate demand for evidence in priority areas identified by thematic programme divisions.
Lean, adaptive impact evaluation approach
The term ‘Lean Impact’, as coined by Ann Mei Change, is a twist on The Lean Startup methodology, which promotes an iterative, data-driven approach to building profitable businesses. As former Chief Innovation Officer at USAID, and Senior Engineering Director at Google, Ann Mei recognises that addressing rapidly evolving humanitarian and development challenges requires problem-focused, iterative innovation and learning, based on the most reliable data and evidence.
At WFP, ‘lean’ impact evaluations adopt a similar adaptive, data-driven approach to support learning and optimising interventions. WFP’s lean impact evaluations will use variations of A/B testing, commonly used by private companies to optimise their services. However, instead of maximising profit, WFP will harness the power of A/B testing to maximise the impact of humanitarian and development interventions.
Lean impact evaluations compare two (or more) options for addressing operational challenges (e.g. targeting; uptake of services; adoption of technologies, etc.) identified by WFP and its partners. The proposed options (e.g. methods, modalities, timings, etc.) are randomly assigned to two or more comparison groups (groups “A”, “B”, “C”, n). Each group can be composed of individuals, communities, companies, or any other population levels or types targeted by the programmes. By comparing measured usage patterns, outputs, or outcomes between groups, the ‘A/B tests’ reveal which options are more effective at addressing problems identified. In this way, ‘lean’ impact evaluations quickly measure the effectiveness of different process or intervention choices during programme implementation. This information can then be used by programmes to learn and adapt based on rigorous evidence for what works best in their contexts.
Like traditional impact evaluations, ‘lean’ impact evaluations rely on experimental methods that randomly allocate intervention options, thereby minimising selection bias. In emergency contexts, the ‘lean’ approach may also help to address potential ethical concerns because it doesn’t require a pure control group (e.g. group that doesn’t receive any support). A WFP project or programme can iterate by implementing multiple experiments over time, depending on the questions and intervention options available. These tests can help WFP and its partners understand which variations produce the best results, and support an experimental approach to problem-based, adaptive innovation and learning.
4 features of lean impact evaluations at WFP
WFP’s ‘lean’ impact evaluations approach include the following key features:
• Adaptive learning and innovation — keeping the focus on problems that need to be addressed, lean impact evaluations support WFP to innovate and adapt, based on rigorous evidence.
• Rigorous experimental methods — harnessing experimental methods to enable rigorous comparisons during implementation. Similar to traditional impact evaluations, lean impact evaluations also require a close alignment between programme implementation and evaluation design.
• Supporting as many people as possible — all beneficiaries that can be supported are supported as soon as possible, and a pure control group isn’t necessary to identify which interventions are more effective.
• Minimising data requirements — harnessing all available data to make comparisons (for example SCOPE, the corporate results framework, or post-distribution monitoring data) to reduce reliance on separate impact evaluation surveys.
To ensure ‘lean’ impact evaluations focus on WFP’s evidence priorities, the humanitarian workstream is being implemented across the impact evaluation ‘windows’ developed in partnership with programme divisions. Windows are designed to increase the visibility and demand for priority evidence needs.
Each window focuses on a common set of priority questions, which can be answered across WFP’s country contexts. For example, the Cash-based Transfers (CBT) and Gender Window aims to test whether variants of CBT are more effective at supporting gender equality and women’s empowerment. Despite the expansive use of CBTs, there is still much more to be learned about which modalities of CBTs are most effective at achieving WFP’s target outcomes and to set people up on a successful path from humanitarian assistance to sustainable development.
Similarly, the Climate and Resilience Window aims to understanding how the timing and sequencing of interventions supports populations to absorb, respond and adapt in the phase of climate-related shocks and stressors.
Finally, the School-based Programmes Window aims at understanding how changes in implementation modalities, complementarity activities or procurement systems contribute to greater health, nutrition and learning outcomes as well as to local economies. In each window, WFP seeks opportunities to support learning by harnessing a ‘lean’ impact evaluation approach to test, adapt and scale the most effective interventions.
Sample questions for lean impact evaluations
Examples of questions that could become subject to a ‘lean’ impact evaluation include:
• Which transfer frequency is most effective in maintaining food security for different populations?
• Which transfer duration is optimal to achieve positive effects on food consumption?
• Which transfer modalities are more effective to improve food security (for example, steady transfer sums or seasonally adjusted transfers)?
Do you have experiences using lean impact evaluations or want to know more, explore and discuss? Please, reach out to Jonas Heirman, Felipe A. Dunsch or Simone Lombardini.
