Ryan C Briggs

Bill Easterly has a good post questioning the utility and ethics behind using randomized controlled testing to evaluate foreign aid interventions. His whole post is worth reading, but I want to elaborate on his second question, “Can you really generalize from one small experiment to conclude that something ‘works’?”

When it comes to aid the answer is pretty easy: no. There is a good reason for this, and it highlights one major difference between randomized controlled trials for aid and randomized controlled trials for medicine. First, the basics. A randomized trial is an experiment. In the experiment the researcher takes a sample from the population and divides the sample randomly into two groups, the control and the treatment. The experiment is set up so that the only systematic difference between the two groups is that the treatment groups gets the medicine and the control group doesn’t. The resulting difference between the two groups represents the effect of the medicine.

There is an obvious, but often overlooked, assumption at work here. The assumption is that the two groups that make up the sample are representative of the entire larger population. In the case of medicine, the control and the treatment groups are taken to be representative of the entire (much larger) target population for the medicine. The validity of RCTs for medicine hinges on this assumption being true. It usually is true because at a biological level most people are pretty much alike.

In political and social life this condition is rarely, if ever, met. A randomized trial in one country will tell you if the intervention worked in that specific place and time, but it will not tell you if the intervention works in general because different countries are far more different than different people. If tylenol works on someone in Botswana then it will probably work on someone in Vietnam. This is not true for aid interventions because the relevant social, political, and economic systems in any two countries are very different—far more so than the biological systems in different people. These systems also change rapidly when compared to human biology.

In short, an assumed equivalence between people underpins RCTs in medicine, but this equivalence is clearly lacking between countries. Because of this, RCTs can be used to see if an aid intervention worked, but it is impossible to use them to see if an intervention works. They are useful for small scale testing and can produce useful knowledge about what works in very specific contexts, but they cannot speak to anything larger. No one should expect them to do that.

—UPDATE: Jul 16, 2009—

Tim Ogden has a comment on Easterly’s blog that is worth highlighting. Here is an excerpt, “In fact, if you have a policy agenda, talking to Duflo, Kremer, Karlan et. al. can be maddening since it’s virtually impossible to get them to commit to a policy recommendation beyond the local context of an experiment they’ve run.” He is right, and I should have pointed out that the academics pushing RCTs for aid are well aware of the issues I raised in my post.