The current semester is ending, and with it, a chapter that I started about six months ago. As someone interest in the political economy of industrialized countries, I thought I could do little of interest if I did not have a minimal understanding America, which is also the country I live in.
In the last year I have seen or done several replication exercises. When you are asked to do one of these for class, you usually choose one for which the data is available, and preferably the code too. In American politics, this is often the case: the data are pretty standardized and available. They are also pretty detailed. A good argument to study America is, in fact, that there are plenty of data available. And yet, systematically, results fail to replicate. Codes do not run. Arguments do not resist a second look.
This paints a grim picture of quantitative American politics. Kids are taken through all those math and stats classes to end up not discovering much. But it gets worse. If this happens for the case of American politics, in which the community that is monitoring your work is large and the data are large in size, standardized and of the best possible quality,… what can we possibly expect from less quantitatively disciplined branches? Think of the case in which people conduct their own survey or experiments in exotic countries. Think of historians and anthropologists that have to rely on observation and ‘judgment’. Or just think of the case of hypotheses that are only vaguely defined and difficult to identify.
I had this thought reading again through this psychological analysis of how the Trump victory implies the victory of white male sexism and racism. I can perfectly see, of course, that there is some visible aesthetical and substantive distance between Trump and Chimamanda Ngozi Adichie. But what people seem to imply is that this symbolic dimension, which is apparent, is likely to have a causal relationship with what is going on in people’s minds. This is something that as a social scientist, I feel, I should expect to be backed by something else than just impressions.
In my view, the main lesson about the replication crisis in quantitative science is not that quantitative work is hopeless, it is that conclusions, especially those in public debate and non-quantitative literature, are to be doubted proportionally on their strength.