He briefly reviews the history of experimentation beginning with Galileo. Then, he provides examples from economics and criminology. He introduces the problem of causal density and writes that it, in effect, dooms many otherwise worthy randomized field trials.
He then moves into the business world citing the success of the credit card company Capitol One. That company answers its own questions via large scale, but relatively economical experimentation. He uses the example of deciding whether to mail customers solicitations in white or blue envelopes. Capitol One simply mailed 50,000 of each color to randomly selected customers and waited to see the results. Since then, other business have adopted similar methods.
He offers three conclusions:
- It is exceedingly difficult to demonstrate that any kind of social program works via traditional randomized and replicated trials. Social evolution with its attendant trial and error process may "trump" those methods.
- Far more social programs fail than succeed, especially programs designed to change the way people think or behave. Incentives work better.
- Most successful programs only lead to modest improvements, but that's ok.
