comparative history - new scientist article

Natural experiments: Working in the history lab

* 29 March 2010 by Jared Diamond and James A. Robinson

SOMETIME around AD 1600, Galileo Galilei is supposed to have climbed to the top of the leaning tower of Pisa and dropped two cannonballs of different sizes and weights over the edge. The point was to test Aristotle's hypothesis that the speed at which objects fall would be proportional to their weight. If Aristotle was correct, a cannonball 10 times as heavy as another would fall 10 times as fast.

Galileo, in fact, believed that in the absence of air resistance both balls would fall at the same speed. Of course there was air resistance in Pisa, but the densities and weights of the two balls made this effect so small that they hit the ground at almost the same time. Galileo was right and Aristotle was wrong.

Though this story may well be apocryphal, it is used as a defining example of an early scientific experiment. The experimental method is the basic technique by which scientists test hypotheses. To molecular biologists, chemists and physicists, experiments are synonymous with manipulative, controlled experiments like Galileo's. They involve examining in a controlled environment the consequences of manipulating the particular variable whose effect you want to investigate.

Now take a different case, the story of what happened in 1854 when a serious outbreak of cholera hit the Soho district of London. John Snow, a local doctor, did not accept the prevailing theory that the cause of cholera was "bad airs" or "miasmas". Instead, he believed the disease was caused by unknown bacteria transmitted in the drinking water. Snow drew a map plotting cholera deaths in Soho, and showed that they clustered around a particular water pump in Broad Street. But how to prove that it was polluted water from this pump that caused cholera? After all, everyone taking water from the pump also breathed the same air, so miasmas were a viable alternative hypothesis.

Believing pump water to be the cause, Snow would have considered it immoral to test his hypothesis by using manipulative experiments: in this case, dispensing Broad Street water to a group of Londoners remote from the pump, and then comparing their subsequent health to that of a control group who were not given Broad Street water. But Snow could use a "natural experiment": an experiment-like natural variation in the variable (drinking Broad Street pump water) whose effect he wanted to investigate.

He detected such an opportunity at the brewery on Broad Street. The men working there breathed the same air as everyone else in the area, but they drank only the beer they brewed or water from the brewery's well. This constituted natural variation in exposure to Broad Street water, holding constant other factors, such as the air. Crucially, working at the brewery was completely unrelated to people's inherent propensity to contract cholera. Lo and behold, Snow found that the brewery workers weren't getting sick with the cholera that was killing other citizens in the surrounding area. This was a crucial clue in a long and ultimately successful bid to show that miasmas were not to blame, and that specific bacteria were.

Snow's form of the experimental method is the rule in many areas of science now.

In Snow's case the difficulty was not that manipulative controlled experiments were impossible or illegal, but that they were immoral. In other sciences, manipulative controlled experiments are impossible, so, for example, astronomy, evolutionary biology, epidemiology and historical geology all use natural experiments. If you are studying planets, volcanoes or glaciers, you cannot manipulate them. The same goes for dinosaurs or other things that existed or happened in the past, so manipulative experiments are ruled out in such historical sciences as palaeontology, too. In social sciences such as economics, political science and sociology, manipulative, controlled experiments are ruled out on all three grounds - they are either impossible, or they are immoral or illegal. Investigators have no choice but to test hypotheses using naturally occurring experiment-like variations.
In many areas, controlled experiments are impossible, illegal or immoral

Yet there is one field where natural experiments could be used but seldom are: most historians resist them. That is odd because, after all, many of the sciences that use natural experiments are deeply historical.

Consider a question that has been front-page news this year: why is the average income of Haitians only one-sixth that of the citizens of the Dominican Republic, when the two countries are neighbours on the island of Hispaniola? One important practical result of this disparity has been that Haiti's government lacks the resources needed to repair the devastation caused by the recent earthquake.

The environmental differences between the eastern (Dominican) side of Hispaniola and the western (Haitian) side are only a minor part of the explanation. The main reasons for the different economic and social outcomes stem from their strikingly different colonial histories, and of being ruled during the 20th century by dictators who behaved in strikingly different ways. Those differences constitute a natural experiment resulting from human actions.

Through an accident of history, Spain established its colonial capital in the east of Hispaniola and French pirates their bases in the west. The island became permanently divided between a Spanish colony in the east and a French colony in the west, eventually (after independence) called the Dominican Republic and Haiti respectively. In the 18th century, France was richer than Spain, so it transported many Africans to work as slaves in its part of Hispaniola and developed a slave plantation economy. Spain did not, choosing instead to concentrate on its more profitable colonies of Mexico and Peru.

The plantation economy on the French half of Hispaniola led to massive deforestation and soil erosion, as ships bringing slaves departed for Europe with timber. It also created a distinct Creole language, isolating most Haitians from the rest of the world. After a ferocious revolt, Haiti achieved independence in 1804, the only nation born of a slave revolt and the first republic established by people of African ancestry.

Haiti became further isolated by its well-founded distrust of Europeans, who found obstacles placed in the way of their owning land and investing there. At the same time, the new nation suffered from European and American fears of seeing a country of former slaves succeed. The Dominican Republic, on the other hand, did receive immigration and investment that gradually enabled its economy to catch up with and overtake Haiti's.

Then, in the 20th century, Haiti's dictator François "Papa Doc" Duvalier did little to develop his country, while the economy of the Dominican Republic was being developed - in order to enrich its equally malign dictator, Rafael Trujillo. After Trujillo was assassinated in 1961, the Dominican Republic became increasingly democratic, presidents lost re-election bids and retired peacefully, and construction and exports boomed. None of this happened in Haiti, and the divergence resulted in today's glaring social and economic differences between the two countries.

As another illustration of the potential power of natural experiments, consider the long-standing debate among historians about whether the French revolution and Napoleon were good or bad for economic growth, technological change and industrialisation in 19th-century Europe. Although the period of the French revolution, between 1789 and 1799, and the Napoleonic wars of 1799 to 1815 caused great loss of life and chaos, they also prompted fundamental reforms favourable to economic growth, such as the abolition of feudalism and guilds, and the enshrining of equality (among men at any rate) before the law.

Historians selecting and describing different case studies have come to diametrically opposed views about the net effect of both the revolution and the wars. But an all-powerful extraterrestrial visitor equipped with a time machine could quickly settle matters with a manipulative experiment. The extraterrestrial would merely have to dial the year 1800 on the time machine, sprinkle Napoleonic armies at random over the map of Europe, and allow the Napoleonised patches to undergo Napoleonic-style chaos and institutional reform. He would return 50 years later to measure whether the Napoleonised or the non-Napoleonised patches were more developed economically.

Luckily for us, history carried out a similar natural experiment. True, the patches of Europe under Napoleonic influence were not selected at random, but controlled randomisation is only one way to guarantee that the designation of particular patches to receive the "experimental treatment" is not related to factors that influence the outcome of interest. Napoleon's conquests in Europe were instead based mostly on immediate political considerations and were unrelated to factors associated with later 19th-century economic performance.

In fact, in Germany the Napoleonised patches were initially less rather than more developed economically than the non-Napoleonised patches. But by around 1850, out of 29 areas of Germany, the Napoleonised areas had pulled ahead of the non-Napoleonised areas. So while Napoleon initially created a mess, his institutional reforms enabled the areas he conquered to industrialise more rapidly as the industrial revolution spread across Europe several decades later.

This conclusion could never have been reached by the individual case studies beloved of historians, because of the many confounding factors in each case. Only a large-scale simultaneous comparison of many cases permits us to extract Napoleon's signal from the background noise.

There are obvious complications of natural experiments compared to manipulative experiments. A chemist accustomed to taking two otherwise identical test tubes and adding a test compound to just one of them would be horrified at the inelegance of allowing Napoleon to select for conquest his own patches on the non-uniform map of Europe.

Moreover, both the Napoleon and the Hispaniola experiments are "blunt" in the sense that the experimental "manipulation" involved a whole vector of attributes. Haiti differed from the Dominican Republic in its European colonising power, language, proportion of slaves in the population, views of and by Europeans and Americans, and the behaviour of its 20th-century dictators.

The natural experiment of Hispaniola shows us that the combined effect of those multiple factors was enormous, but further information must be added in order to establish their relative importance. Natural experiments always require scrutiny to assess whether the "selection" of patches to be manipulated or not manipulated was truly arbitrary with respect to the perturbing manipulation, and whether the resulting "experimental variation" observed was really due to the perturbation itself or else to other associated or omitted factors.

Our new book Natural Experiments of History describes eight natural experiments which illustrate a range of approaches. They can be classified into two types: experiments in which the initial conditions are similar among the societies compared, and the key difference in outcome involves some societies experiencing a perturbation and others experiencing no perturbation (or different societies receiving different types of perturbation); and experiments in which the initial conditions differ among the societies compared, but the perturbations are similar among the societies compared.

The first type is exemplified by the two halves of Hispaniola, and by the 29 patches of Germany. Of our other two cases of this type, one natural experiment attempts to address the familiar debate about the effects of British colonial rule on India, by comparing 233 parts of India which the British formerly managed, using three different systems of tax collection.

The other case concentrates on the equally familiar debate over the effect of the former slave trade on modern African economies, by comparing the slave trade (or its absence) in 52 African countries. It turns out that African countries that were sites of high slave exports tend to have low incomes today - even after taking account of potentially confounding variables such as geographic location, climate, coastline, Islamic influence, natural resources, and the identity of the European ruler.

We also have four examples of the second type of experiment, where there are differing initial conditions. Two of them involve comparisons of different sets of Pacific islands, with different physical environments, all subjected to Polynesian colonisation, with varying outcomes of deforestation and sociopolitical complexity.

The third experiment compares economic cycles in seven 19th and early 20th-century non-European frontier societies around the world, including the American West, Canada, Argentina and Siberia, all characterised by explosive settlement by Europeans. The last experiment involves comparing three former colonies in the Americas - the US, Brazil and Mexico - which all needed banking systems after they became independent.

Natural experiments have become a familiar methodology in anthropology, archaeology, economic history, economics, political science and sociology. So why are historians still so sceptical or even hostile to them? We think there are a number of contributing factors. First, many historians see themselves as storytellers and non-scientists rather than as scientists. Students choosing to become historians rather than economists, political scientists or sociologists often do so to avoid having to learn mathematics, statistics and scientific methods. Multi-authored research collaborations, which allow scholars to pool their complementary strengths, are virtually unknown among historians.

Then there is the fact that most historians devote their careers to studying one geographic area and one slice of time. They might, say, study the American civil war, but not the American revolution or the Spanish civil war as well, and certainly not civil wars in general. Historians who pride themselves on having acquired an understanding of the American civil war after 40 years of study wouldn't dare discuss the Spanish civil war to which they had not devoted such study, and would bristle at a historian of the Spanish civil war having the temerity to discuss the American civil war. That's sad, because no historian can claim to understand the American civil war if she or he cannot explain why its course was so different from that of the Spanish, English, Russian, Mexican, Chinese and Japanese civil wars.

Yet another problem is that American historians in particular point to a school of quantitative history termed cliometrics, which was at its peak in the days of mainframe computers. Its methods were heavily criticised, and its "weaknesses" were then - inappropriately, we think - generalised to quantitative methods of all kinds. Today, we hear historians responding to suggestions that they use quantitative data and statistics by saying something like: "The cliometricians already tried that several decades ago, and we know it doesn't work."

Finally, historians feel that human history and motives are uniquely complicated and can't be expressed in numbers. But it makes no sense to maintain this position when psychologists and other scholars of human behaviour somehow manage to find ways to measure human motives in numbers.

Our concern, and that of many historians, is that the study of human history is suffering as a result of methodological rigidity and narrowness, and that economists, political scientists and sociologists are encroaching increasingly on its traditional territory. Historians tell us they receive few and small research grants, and they feel undervalued by their college deans and presidents, who, in turn, say the average salaries they pay to historians in the US are below those paid in most of the other social sciences.

We believe historians could become more effective and valued by adding natural experiments, quantitative methods and statistics to their traditional expertise. Scientific purists in labs might also benefit from accepting that natural experiments are not an inferior, second-class science but a necessity for the scientific understanding of important disciplines that do not lend themselves to lab methods.