Ryan C Briggs

Human Rights Watch recently published a report on aid in Ethiopia entitled Development Without Freedom. They had researchers visit Ethiopia in 2009 and conduct about 200 interviews to examine how the ruling party was distributing foreign aid.

Their central finding is that “donor-funded services, resources, and training opportunities were being used as threats or rewards for citizens to join the ruling party and cease supporting the opposition, and that donor mechanisms for monitoring or controlling the misuse of aid programs were inadequate.” (p. 26)

The range of funds that were at affected ranges from fertilizer and other agricultural inputs (p. 36) to microcredit (p. 38) to food aid (p. 45).

One common theme running throughout the report is that aid was more politicized the more the service was decentralized. Various factors could influence this, but one is that monitoring is more difficult as responsibility for service delivery shifts further from the center.

Coming from someone who is currently writing a prospectus on the politicization of foreign aid before elections, my single largest problem with the report is that I have a hard time interpreting the report’s data. I find it impossible to believe that every person interviewed felt the way that they describe, and yet I can’t find quotes from interviewees who disagree with the report’s finding. Did they only interview people that they knew were upset? Did they cut out all the “boring stuff” after the fact? How did they create their sample? (At this point, everyone should go read what Texas in Africa has to say about the methodological differences between academics and advocacy organizations).

Their question—Is aid being captured by the ruling party?—is crucial and HRW seems to have found a suspicious pattern in the responses of their interviewees. Given the challenges inherent in conducting this kind of research, that finding alone is worthy of attention. Anyone with an interest in the politicization of aid should at least skim the report.

[update — Kate from wrongingrights also has a good post on the way that human rights lawyers and social scientists think about evidence]

Bill Easterly has a good post questioning the utility and ethics behind using randomized controlled testing to evaluate foreign aid interventions. His whole post is worth reading, but I want to elaborate on his second question, “Can you really generalize from one small experiment to conclude that something ‘works’?”

When it comes to aid the answer is pretty easy: no. There is a good reason for this, and it highlights one major difference between randomized controlled trials for aid and randomized controlled trials for medicine. First, the basics. A randomized trial is an experiment. In the experiment the researcher takes a sample from the population and divides the sample randomly into two groups, the control and the treatment. The experiment is set up so that the only systematic difference between the two groups is that the treatment groups gets the medicine and the control group doesn’t. The resulting difference between the two groups represents the effect of the medicine.

There is an obvious, but often overlooked, assumption at work here. The assumption is that the two groups that make up the sample are representative of the entire larger population. In the case of medicine, the control and the treatment groups are taken to be representative of the entire (much larger) target population for the medicine. The validity of RCTs for medicine hinges on this assumption being true. It usually is true because at a biological level most people are pretty much alike.

In political and social life this condition is rarely, if ever, met. A randomized trial in one country will tell you if the intervention worked in that specific place and time, but it will not tell you if the intervention works in general because different countries are far more different than different people. If tylenol works on someone in Botswana then it will probably work on someone in Vietnam. This is not true for aid interventions because the relevant social, political, and economic systems in any two countries are very different—far more so than the biological systems in different people. These systems also change rapidly when compared to human biology.

In short, an assumed equivalence between people underpins RCTs in medicine, but this equivalence is clearly lacking between countries. Because of this, RCTs can be used to see if an aid intervention worked, but it is impossible to use them to see if an intervention works. They are useful for small scale testing and can produce useful knowledge about what works in very specific contexts, but they cannot speak to anything larger. No one should expect them to do that.

—UPDATE: Jul 16, 2009—

Tim Ogden has a comment on Easterly’s blog that is worth highlighting. Here is an excerpt, “In fact, if you have a policy agenda, talking to Duflo, Kremer, Karlan et. al. can be maddening since it’s virtually impossible to get them to commit to a policy recommendation beyond the local context of an experiment they’ve run.” He is right, and I should have pointed out that the academics pushing RCTs for aid are well aware of the issues I raised in my post.