Optimizing targeting of humanitarian food assistance in the DRC: Precision and coverage

Community-based targeting performs well against other (statistical) methods when identifying food insecure households in eastern DRC. It may serve as a tool for prioritization amidst the growing funding gaps in the humanitarian sector.

By Jonathan Garcia and Kristen McCollum with Felipe Alexander Dunsch, Andrea Guariso, Marcus Holmlund, Ghida Karbala, and Jonas L. Heirman

Although achieving universal coverage of assistance for vulnerable populations is a priority for global development, especially in emergency contexts, budget constraints imply that governments, development, and humanitarian actors need to identify and select those more in need, or who would benefit more from their assistance programmes.¹

This process of selection, known as targeting, is particularly challenging in humanitarian settings, given the growing funding gaps and the characteristics of the population affected: highly vulnerable and largely homogenous.²

Most existing evidence on targeting is focused on development programmes³, with a strong emphasis on measuring the targeting error associated with each method (targeting precision).⁴

Thus, the current body of literature and toolkit for targeting is predominantly relevant for “persistent” outcomes, such as poverty, and potentially less useful for humanitarian interventions that aim to address transient outcomes, such as food insecurity.⁵ Likewise, the focus on precision has resulted in limited evidence regarding the potential trade-offs faced when choosing between targeting methods, for example, differences in costs, speed, etc.

With the generous support from USAID’s Bureau for Humanitarian Assistance, the WFP Office of Evaluation and the World Bank Development Impact Evaluation department are supporting the WFP country office in the Democratic Republic of the Congo (DRC) to conduct a “lean” impact evaluation comparing two methods for targeting humanitarian assistance in the Tanganyika Province. To compare the two methods, the WFP country office randomly allocated 84 blocs (similar to village sections) to one of the two targeting methods. By tracking and comparing outcomes across the two groups, one can rigorously assess which approach leads to higher coverage, precision, community acceptance, social cohesion, and cost-effectiveness. This blog focuses on outcomes of coverage and precision following preliminary results from the ongoing evaluation.

Two targeting approaches

PMT Plus

Currently, the WFP country office uses a comprehensive household-level targeting approach called “PMT Plus”. This approach is primarily data-driven but combines limited community-informed elements. The main component is a common Proxy Means Testing (PMT) model that builds on training data from past outcome monitoring exercises and relies on machine learning methods to predict the Food Consumption Score (FCS) across households in all communities.

The results from the PMT model are then applied to a census database that allows classifying and ranking households based on their predicted vulnerability. In addition, three separate focus group discussions (FGDs) with men and women from the target region help identify a set of priority vulnerability criteria for all villages (the “Plus”). If any one of these “lock-in” criteria is met by a household, they are deemed eligible, regardless of their PMT vulnerability ranking.

This current standard procedure for targeting is costly, as it requires census-level data in addition to some qualitative data collection. Likewise, beneficiaries had voiced unease about the lack of transparency of the method, which resonates with findings in the literature about the difficulties in explaining PMT to communities.⁶ This motivated the DRC country office and the evaluation team to explore alternative targeting approaches that would still rely on data to ensure objectivity but also consider community preferences more directly.

Community-based targeting

A more community-based approach may be cheaper, faster, and met with greater acceptance from the community. We piloted a community-based targeting method, adapted from UNICEF, entailing the creation of committees within each bloc to define the criteria to identify households for inclusion in the programme. Each committee consisted of 12 members, with equal representation from both men and women, and representation from various vulnerable groups but excluding the village chief.

Instead of the traditional formula from the PMT’s statistical model and the lock-in criteria, the community-based approach determines eligibility solely based on the criteria set by the committee in each bloc. Any community member who possessed at least one of the criteria named by the committee was deemed eligible. Committees are expected to be more representative of the community than the traditional key informants used in focus group discussions, thus potentially enhancing local ownership and acceptance.

The approach also increases transparency and facilitates communication with eligible and non-eligible households about the selection process. Moreover, it could reduce the data burden associated with targeting, as it does not require detailed secondary data or a statistical specialist to develop the model. Thus, community-based targeting may better leverage the comparative advantages of field offices and further lower costs.

Do coverage rates differ between the approaches?

“Coverage rate” refers to the share of households within a community that are selected for assistance. Using PMT, the prediction model creates cut-offs to identify ‘true’ vulnerability (i.e. poor, borderline, acceptable), and as such, the number of beneficiaries is not capped a priori.

Common guidance for community-based programming recommends setting a quota for coverage as this type of approach lacks these thresholds to determine food-insecure populations.⁷ As a result, a theory is that community approaches can lead to larger coverage rates if the number of beneficiaries for each community is left uncapped. This evaluation tests this hypothesis. Our results show that contrary to the evaluation team’s expectations, the community approach led, on average, to 11 percent less coverage (see Figure 1).

Figure 1

Which method is more precise for selecting beneficiaries?

We measure the accuracy of the targeting approaches using three indicators: precision, and exclusion and inclusion error. Precision refers to the proportion of beneficiaries that are food insecure according to our “ground truth”, in this case, the observed FCS. Exclusion error refers to the proportion of true food insecure excluded from assistance and inclusion error refers to the proportion of beneficiaries who are not food insecure.⁸

Overall, both methods perform remarkably well. The targeting error rates are lower than those found in recent literature from similar experiments, where the median inclusion and exclusion errors are around 40 and 53 percent⁹, respectively.

Existing evidence suggests that PMT exhibits marginally better precision compared to, for instance, community-based approaches.¹⁰ When we perform a simple comparison across the two approaches, we find that PMT+ leads to lower exclusion error compared to community-based targeting; however, this difference is mechanically driven by the differences in coverage, as community-based targeting selects a smaller share of the community. When we adjust the comparison for the differences in coverage, the difference disappears, as shown in Figure 2.

Figure 2

Preliminary results from our midline follow-up indicate that, even though the community-based targeting approach results in significantly lower coverage than PMT+, community satisfaction and acceptance are similar. If this result holds, community-based targeting could be a cost-effective alternative for humanitarian targeting, without sacrificing targeting precision or community satisfaction. Stay tuned for more updates on results.

This evaluation is delivered in partnership with the World Bank’s DIME Department and is part of WFP OEV’s Optimizing Humanitarian Interventions workstream. This impact evaluation is registered here.

References

1. Coady, Grosh & Hoddinott (2004); Grosh et al. (2022)
2. Premand & Schnitzer (2021); Schnitzer (2019)
3. See Coady, Grosh & Hoddinott (2004) and Grosh et al. (2022) for a review of targeting for social protection.
4. See Alatas et al. (2012), Alatas et al. (2016), Basurto, Dupas & Robinson (2020), Beaman et al. (2021a), Beaman et al. (2021b), Karlan & Thuysbaert (2019), and Premand & Schnitzer (2021) for evidence on targeting experiments.
5. Premand & Schnitzer (2021)
6. Cameron & Shah (2014)
7. See Sabates-Wheeler et al. (2015) and Hillebrecht et al. (2020) for a review of community based targeting procedures.
8. Aiken et al. (2022)
9. Median inclusion and exclusion error calculated using reported errors from, Brown, Ravallion & van de Walle (2018) and Schnitzer & Stoeffler (2021), using data from 11 countries across sub-saharan Africa.
10. Alatas et al. (2012), Beaman et al. (2021b), Premand & Schnitzer (2021), Stoeffler, Mills & Ninno (2016), Hillebrecht et al. (2020)