Critics have called RCTs (Randomized Control Trials) unethical from the very beginning. Some have likened it to playing the role of God in choosing who receives treatment and who doesn’t. Defenders respond that in all development interventions, you are making a choice of who to treat (and therefore who not to treat) but RCTs at least make a conscious effort to measure the effects of the treated vs. the untreated.
As RCTs become the standard practice in most development interventions these days, there has been pushback. Francisco Toros at the Campaign for Boring Development recently wrote about a double-blind RCT done with high-yield seeds. In the paper that Toros discusses, the authors point out that in medical RCTs, in order for the results to be considered valid, they must be double-blind, ie neither party knows who is receiving the treatment and who is not. The authors realized that with high-yield seeds (which look the same as regular seeds) they could perform a double-blind RCT. And the results were rather disheartening. Basically, the only people that improved their harvest were the ones who were told they were receiving high-yield seeds. People who blindly were given high-yield seeds didn’t have higher yields. Toros saw this as a huge blow to RCTs in development (as evidenced by the title of his blog post ¨A TORPEDO AIMED STRAIGHT AT H.M.S. RANDOMISTA¨) Anyway, there were a lot of problems with this experiment. One problem that Toros identifies was that the experiment was on a secondary crop. This brings up certain questions as to whether the participants might have put less effort into the crop. But the other problem, and the thing I wanted to write about here, is that this information (whether the farmer is receiving a high-yield seed or not) does matter. People respond differently when they have certain information. A double-blind RCT in medicine makes sense. Researchers need to know whether the medicine is having the intended effect. The problem with this RCT is that we already know that high-yield seeds do produce more. The point of these interventions is to figure out how to get farmers to use them most effectively. The corollary to this experiment would be giving a farmer a regular seed and informing him that it was a high-yield seed. Of course they didn’t do this because nobody would really expect the results to be higher.
So the other question that this brings up is that of necessity. A recent paper entitled ¨The Unprincipled Randomization Principle¨ by Stephen T. Ziliak and Edward R. Teather-Posadas, argues that many RCTs in development economics violate one of the most basic tenets of running an experiment. Known as essentiality or equipoise, this ethical guideline requires that any intervention (and therefore lack of intervention in a control group) must be providing new information to be justified. The Ziliak paper uses a large study of Chinese students and eyeglasses as an example of a non-essential intervention. Of 19,000 students with bad eyesight, a certain number (the paper isn’t clear on the amount) are given glasses for a year and a control group receives nothing. The authors of the paper argue that it is obvious enough that bad eyesight will hinder your school performance that this experiment is entirely unnecessary and in fact very unethical for the control group. As a clinical trial, the authors are absolutely correct in stating that this brings no new evidence. But from a policy standpoint, some people argue that it is necessary in a benefit-cost analysis (I guess we are supposed to say it that way now).
Over at the World Bank’s Development Impact blog, Jed Friedman argues that these RCTs are not done to determine if eyeglasses improve schoolwork but rather to see if the impact is enough to justify supplying students with free glasses. The logic being that paying for glasses in a policy choice and we need to determine if it is the best use of resources. Friedman is correct in framing the argument in this way. Unfortunately, the argument is not framed that way by economists, leaving them open to the kind of criticism from the Ziliak paper.
Jeffrey Hammer, in a post entitled ¨The Chief Minister Posed Questions We Couldn’t Answer,¨ he tells the story of a conference of development economists who are asked by the Minister of Punjab province in Pakistan how he should allocate funding. Was it better to put it in infrastructure, social services, health etc? Nobody could answer the question.
Maybe the Chief Minister would be better off improving education or road networks or police or rural electricity. Some of these alternative policies could have more impact on food consumption than food distribution if we thought about how the world worked. Getting food to market securely (roads, better cold storage, trustworthy police and safe roads – this is Pakistan, which no one seemed to notice) may increase food availability much more than any tunnel-visioned food program Or not – maybe the food distribution system is better. We just don’t know. And none of us “experts” are trying to find out.
Hammer says that the lack of comparison (or maybe the lack of any connection to reality) makes all these RCTs useless for policymakers. Opportunity costs (hardly a novel idea) need to be taken into account.
When someone says “we should have more “X” because we have evidence that it works”, the response should be “compared to what?” What should we cut in order to promote your particular interest? My hobby horse these days is more sanitation in South Asia. I should have to defend it against (at least) a few alternatives.
Anyway, these are questions that have not been answered sufficiently. And until they are, these RCTs will continue to face criticism from both the ethical and the practical sides of the debate. After all, if dozens of highly trained economists can’t even answer the question of where to put the money, we are in trouble.