Photo by FAO/Riccardo Gangale.

The following article is written by Iain Wright, deputy director general for research at ILRI.

On 6–8 Jul 2017, I attended a conference at the World Agroforestry Centre (ICRAF) on Impacts of International Agricultural Research: Rigorous Evidence for Policy organized jointly by the CGIAR Independent Science and Partnership Council (ISPC) Standing Panel on Impact Assessment (SPIA) and the CGIAR Research Program on Policies, Institutions, and Markets (PIM). I welcomed the delegates at this meeting in Nairobi, Kenya, on behalf of ICRAF and the International Livestock Research Institute (ILRI), the two CGIAR centres headquartered in Nairobi. A modified version of my address and personal reflections on impact assessment in CGIAR follows.

Iain Wright, ILRI.

We in CGIAR have committed ourselves to tackling some of the greatest challenges that the human population has ever faced.

How do we feed a growing population not only with calories but also with nutrients essential for good health, and do so in the face of climate change? We know that proteins, vitamins and minerals are essential not only for growth in children but also for children’s intellectual development, their cognitive and learning ability. We know that malnutrition can not only stunt children permanently but also damage a nation’s long-term economic development.

Agriculture forms, and will continue to form, the basis of economic development in many part of Africa. Agriculture is the route by which millions of people will escape poverty, not just through improvements to the livelihoods of individual farmers but also through commercialization of smallholder agriculture, which generates employment in farm input services and in the production of value-added products along agricultural value chains.

Although agriculture is often viewed in industrialized countries as harmful to the environment, farming holds the key to effective natural resource management and the provision of essential environmental services, such as reduced intensities of greenhouse gas emissions achieved via well-managed rangelands and trees that store significant amounts of carbon, or absolute reductions in greenhouse gas emission levels achieved through increased agricultural productivity and more efficient use of farm inputs.

While those of us who work in agricultural research recognize its importance, we must persuade the rest of the world of the case because agricultural research can deliver benefits such as these only if there is sufficient investment in this research. A few decades ago, agriculture received about 15% of official development assistance (ODA). Today, agriculture receives just 4% of total ODA (and of that 4%, agriculture’s livestock subsector receives just 4%, despite the fact that the livestock subsector contributes an average of 40% of the agricultural gross domestic product of developing countries).

Many studies show high rates of return to agricultural research but we need more specific evidence on what investment in agricultural research actually delivers. As donor organizations are under increasing political pressure to change their investment priorities, to better address, for example, domestic issues or the refugee crisis, it is timely to consider the role of impact assessment in CGIAR.

Agricultural research for development deals with complex agro-socio-ecological systems generating complex problems as well as benefits. We use complex research methodologies to solve the complex problems.

A major challenge in assessing the impacts of our research is having sufficiently robust methods to generate robust evidence. While we have methods to assess rates of uptake of a given technology or the welfare benefits of adoption of that technology, not all research is focused on a single technology. How do we assess the impact of research that delivers a mix of new technologies that are likely to be adopted and adapted in different ways by different farmers? In the livestock sector, for example, improved livestock genetics will have little impact if not accompanied by better livestock feeding strategies and health services, which themselves will require new institutional and marketing arrangements, which in turn will be effective only where there are policy environments conducive to such novel arrangements. In such cases, how do we discern what impacts our research is having?

Where CGIAR research focuses on influencing decision-making, the effects of such research on the complex political processes involved are often difficult to assess. Twelve years ago, I was at a workshop on the interface between research and policy organized by the chief scientist at the Scottish Government Rural Affairs Department, who at that time was Maggie Gill, now chair of the CGIAR ISPC. One participant presented a list of things a minister has to consider when devising a new policy. Technical or scientific evidence was only 1 of 23 things on that list. How do we know what impact our research is having on the other 22 factors being taken into account?

As we consider here impact assessment work in CGIAR, let us also continually ask ourselves how we can best deal with complex questions about impact. This will help us avoid focusing only on things easy to measure.

To meet the global challenges that CGIAR is researching, we will need not incremental but rather transformational change in smallholder agriculture.

If we focus on things that are easily measured, we will fail to provoke those transformational changes.

Do we have the tools and methods needed to measure the impacts of complex solutions to complex problems? I believe we need more methodological development of quantitative and qualitative impact assessments. I believe we have much to learn from other sectors, including public health and education.

So as we delve into impact assessment work this week, let us look not only at what we have achieved in the past but also at how we will demonstrate our achievements in the future.

---

Below are excerpts from selected conference presentations at the three-day Jul 2017 conference on Impacts of International Agricultural Research: Rigorous Evidence for Policy.

---

Doug Gollin, University of Oxford.

(1) The following comes from a presentation titled The rigour revolution in impact assessment written by Doug Gollin, professor of development economics at the University of Oxford and chair of the Standing Panel on Impact Assessment (SPIA) of the CGIAR Independent Science and Partnership Council (ISPC).

The CGIAR model of ex-post impact assessment arguably no longer provides the most useful information:

• Study of shift from ‘traditional’ to ‘improved’ varieties in aggregate no longer the most important question
• Model mute on distributional and other outcomes, e.g., poverty, inequality, environment, nutrition
• Difficulty of translating the model to NRM [natural resource management] / policy research areas
• Cherry-picking of winners limits potential for learning.

And the empirical practice lacks scientific credibility.

A rigor revolution is both a challenge to CGIAR and a huge opportunity.

CGIAR should aspire to help shape the direction of the methods/approaches for agriculture and natural resource management.

Need to develop core competence in data integration (Big Data platform a good start) to avail full range of possibilities.

Stronger partnerships with outside economics expertise remains necessary, as well as remote sensing, national statistics agencies, etc.

Role for SPIA in facilitating provision of system-wide public goods:

• High-quality, large-scale, open-access datasets on technology use for a few key geographies (provides basis for crowding impact evaluations to the same places)
• Development and validation of new tools and methods
• Continued need for capacity-building/networking role in CGIAR system.

Rigor is not limited to impact evaluation‚ and not limited to a single method, such as randomized controlled trials:

• SPIA has funded impact studies that rely on a range of methods, including observational data, qualitative methods, and large-scale models
• The challenge is to find the appropriate methods for the question that is posed.

The point is to bring a challenging and inquisitive mindset to evidence on impact:
Often at odds with pressures to produce evidence of success at the project level.

We need better evidence for the system to learn and to achieve greater impacts.

We should not confuse evidence with advocacy; learning requires that we not hide failure.

We know that not every research project will lead to success, and both theory and history tell us that a handful of successes will more than pay for all the failures.

Creating a culture of impact involves accepting these principles and pursuing good practices without fear, in the belief that the institution will be stronger with better evidence.

---

Phil Pardey, University of Minnesota.

(2) The following comes from a presentation titled Recalibrating and reassessing the global  returns to agricultural R&D evidence written by Phil Pardey, professor of science and technology policy in the department of applied economics at the University of Minnesota and director of the university’s International Science and Technology Practice and Policy (InSTePP) centre.

We conclude that the contemporary returns to agricultural R&D investments appear as high as ever.

---

Chris Barrett, Cornell University.

(3) The following comes from a presentation titled Comments on Papers on ‘From productivity increases to aggregate, long-run impacts’ written by Chris Barrett, the Stephen B and Janice G Ashley Professor of Applied Economics and Management and international professor of agriculture at Cornell University.

CGIAR mission: Science for a food-secure future
Primary impact pathway has always been farmer productivity increases through adoption of improved crop varieties. Critically important, as much now as ever, to rigorously estimate and document these impacts. With ever-improving [impact assessment] methods, more credible, and inclusive impact estimates increasingly feasible. This is exciting (and overdue)!

InSTePP data on agricultural R&D worldwide 1960–2011
Returns come 6–37 years after investments begin . . . Be patient!

Whether measured as commonplace (but inappropriate) IRR [internal rate of return] (median = 39%) or better MIRR [modified internal rate of return]/BCR [benefit-cost ratio] (median = 17%, BCR = 7.5), so returns are high . . . Patience pays!

But IRR-MIRR rankings correspondence decreases with discount rate . . . In present low [resource] environment, errors are easy!

Yet, MIRR mutes differences . . . So errors less costly.

Rates of return have not fallen over time . . . Keep investing!

Rates of return are no lower in LIC/MICs [low- and middle-income countries] than HICs [high-income countries] . . . Invest in developing world agriculture for high returns!

Key lessons from InSTePP studies
Choose Right Measure: MIRR/BCR over IRR to have credible estimates of returns to agricultural R&D (or any other sort!).

Shifting Patterns: Agricultural R&D increasingly led by MICs:

• Private sector plays a big role . . . And food not just agricultural firms
• Per capita agricultural R&D falling in LICs . . . Looming disaster.

Implications: Growing importance of private-sector partnerships for CGIAR (and [advanced agricultural research institutions]):

• Discoveries protectable by [intellectual property] increasingly firm-led, CGIAR space increasingly in NRM, policy and other traits with big externalities (e.g., pesticide resistance).
• The agricultural R&D investment gap growing in poorest countries.

Key lessons from Alwang et al. study
1. Importance of studying uptake reasonably promptly, especially if successful! 20 years is too long.

2. Which measure of variety to use? Is the true biophysical impact what we are after? What if management changes with variety? Won’t both measures underestimate gains (due to technical inefficiency)?

3. Still not capturing gains from consumer traits. The literature on gains from early childhood nutrition suggests that could mean BIG underestimates of aggregate, [long-run] impacts.

---

Cheryl Doss, University of Oxford.

(4) The following comes from a presentation titled Measuring gendered impacts written by Cheryl Doss, of the University of Oxford’s Department of International Development and the CGIAR Research Program on Policies, Institutions and Markets (PIM).

On Challenges & Pitfalls
Research questions and data analysis plan need to be developed before collecting the data.

Collect data to test your assumptions—on men’s and women’s roles, preferences, etc.

Many of the terms are used interchangeably:

• Bargaining power
• Empowerment
• Decision-making

Research needed to understand what ‘joint’ means in terms of asset ownership, decision-making, agricultural production, etc.

Have not discussed broader gender issues of how to analyze whether including women affects the impacts.

---

Alain de Janvry, University of California at Berkeley.

(5) The following comes from the conclusion to an invited paper titled The adoption puzzle—what can the CGIAR learn from field experiments of new agricultural technologies? written by Alain de Janvry, professor of agricultural and resource economics at the University of California at Berkeley.

Seven observations for discussion in addressing the adoption puzzle

Observation 1: Technology adoption in rainfed agriculture remains a first-order challenge
‘In spite of dispersed progress (LSMS-ISA), low technology adoption in SSA [sub-Saharan Africa] and rainfed SA [South Asia] (aggregate data) remains pervasive and important.

Reality is that supplying massively adoptable and profitable technologies to smallholder farmers under rainfed (risky and heterogeneous) conditions in SSA and Eastern SA is exceptionally difficult, yet essential for growth of agriculture-led countries/regions and to meet the [Sustainable Development Goals] 1 & 2.

Observation 2: Field experiments in the social sciences help better understand and support adoption
Field experiments allow greater precision in identification of:

• Causal determinants of adoption
• Impact of adoption
• Design of institutional innovations to help remove constraints

but progress still needed with methods to analyze the dynamics and scale of adoption:

• Design the complementarities of interventions
• Combine with natural experiments.

Observation 3: Rural poverty reduction needs more than a [Green Revolution]: also an Agricultural and a Rural Transformation
Technology adoption to achieve a [Green Revolution] is necessary but not sufficient to make a dent in rural poverty. Essential for this is to smooth labor calendars in agriculture through an [Agricultural Transformation] and to complement agricultural with agricultural-driven non-agricultural incomes in local [Rural Transformations].

Striving to achieve [Green Revolution] + [Agricultural Transformation] + [Rural Transformation] gives a useful conceptual framework in using technology adoption for development.

Observation 4: The presumed widespread existence of adoptable technology for smallholder farmers needs revisiting
In spite of some spectacular successes, the presumption of extensive existence of profitable technologies when adoption constraints have been lifted by institutional innovations needs to be revisited in view of the great degree of heterogeneity of circumstances: need to ascertain that technologies offered for adoption are indeed profitable in expected value and with low risk in local contexts.

It also suggests moving out of the difficult conditions of rainfed agriculture and investing more into water control.

Observation 5: There has been much progress with institutional innovations in removing adoption constraints on the demand and contextual sides
While research is incomplete due to heterogeneity of conditions and changing states of nature, much progress (by ATAI/SPIA/AMA-Basis and other research) has been made with removal of constraints on:

• The demand side: assets/property rights, behavior
• The contextual side: credit, insurance, market access, subsidies.

Observation 6: Improvement still needed on access to information for [smallholder farmers] and learning for adoption
To achieve adoption of available technologies, better access to information and learning options is still lagging, especially through demand-driven social learning, extension services, and motivated agents in value chains.

Extension services remain the poor child of development assistance.

Motivated agents in value chains as sources of information in interlinked transactions are also incipient (Neuchatel Initiative).

Observation 7: Also need increased local availability of technology for adoption under heterogeneous conditions
Secure the local availability of technology under adoptable conditions for smallholder farmers principally through commercial channels in value chains, especially accounting for heterogeneity of circumstances that can be characterized and managed (e.g., Mahajan et al.).

---

All presentations from the conference are listed and available here.