Scientific Inference from Field and Laboratory Economic Experiments: Empirical Evidence
By Jonathan H.W. Tan, Zhao Zichen and Daniel John Zizzo
Scientific inference is an integral part of scientific methodology. A standard argument is that field experiments are superior to conventional laboratory experiments in the ability to derive inferences that are applicable to natural world settings. A key rationale for this argument is that field experiments are implied to be directly representative of natural world settings in ways that conventional laboratory experiments are not, with artefactual and framed field experiments being somewhere in the middle. In other words, they would be characterized by within‐domain inference in relevant domains rather than out‐of‐domain inference.
We test whether and to what extent this holds for conventional lab and artefactual, framed and natural field experiments in a dataset of 512 articles published in 21 top general and field journals in 2018‐19. We recognize that background assumptions are always made to construe interpretation from any specific experimental dataset, including that of field experiments, and that the scale of the inference problem depends on the claims being made. With these qualifications, we consider five domains for which we seek to verify whether there is a match between the specific experiment being conducted and the claims being made: profession, age and gender of experimental subjects, country of experiment, and experimental asset in relation to which a claim is made.
Conventional lab experiments are with standard university student samples, and, while they do not achieve a data‐claim match for profession, gender and country, their asset match rate of around 20% compares to that of artefactual and field experiments. Similarly, for those papers that provide gender breakdown, conventional lab experiments do roughly as well as any field experiment category (at 40‐60% each). Natural field experiments have the highest asset match rate, but this is still only around 60% and appears driven by policy testbed experiments. Profession match is highest with artefactual and framed field experiments, but still again only around 60%. Country match is always lower than 20%, and age match is always lower than 35%. Only 20% of field experiments achieve a match in at least 3 out of 5 categories. By and large, economists are satisfied with an out-of-domain inference approach (particularly outside policy testbed studies). Overall, natural field experiments do not have a systematically higher match than less pure categories of field experiments; and, given the claims they make, a large majority of field experiment articles are largely reliant on out‐of‐domain inference, limiting any potential superiority they may have on this regard relative to conventional laboratory experiments.