Cultures of Trial and Error: Replication in Social Psychology

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

For the last 13 years psychology has been in the grip of a replication crisis. The crisis was declared because time and again the results of earlier, often influential research could not be reproduced in replication studies. Social psychology in particular was affected, and much of the discussion has focussed on this field. Roughly speaking three different lessons have been drawn from these failed replications.

The first is that there must be structural problems in social psychology. Apparently social psychologists produce a lot of spurious results, which then get published and cited, and may even give rise to entire research programs, despite being false. A research culture has developed in which ‘getting it published’ is more important than ‘getting it right’, and it is easier to publish flashy studies than robust work (Nosek et al., 2012). This diagnosis was shared by a growing number of disaffected psychologists who banded together to reform the discipline. They proposed and implemented a number of solutions, many of them aligned with the principles of the broader Open Science movement.

A different kind of conclusion was already drawn in the 1970s, when failed replications had also attracted attention in social psychology. Ken Gergen, an American social psychologist, caused an uproar when he proposed that social psychology should abandon its search for general laws of behavior (K. Gergen, 1973). There are no such laws, he claimed, the phenomena that social psychologists study fluctuate over time and are culture-specific. Moreover, psychology itself causes its object of study to change: when people learn about the results of psychological research, they may resist and choose to behave differently. As evidence for the instability of behavioral phenomena Gergen pointed among other things to failed replications of famous studies in social psychology (K. J. Gergen, 1976). Clearly, prediction and control are not a viable aim for this field. Instead, Gergen argued, social psychology should be seen as a form of history, cataloguing how people behave, so that they may learn to behave differently if they wish. Social psychology should have an explicitly political, liberatory mission. Gergen’s ideas were indignantly rejected by the majority of his colleagues, but they did inspire a relatively marginal social constructionist school that survives to this day. The replication crisis is grist to their mill.

The third response to the failed replications in social psychology takes over Gergen’s ideas about the changeability of social behavior, but not his radical conclusions. These social psychologists argue that, indeed, social behavior is very sensitive to context, and behavioral phenomena vary over time and place. “In matters of social psychology, one can never step in the same river twice” (Crandall & Sherman, 2016). That is why replications so often fail to reproduce the original results. Because the phenomena it deals with are so fluid, ‘direct’ replications that follow the protocol of the original study are not useful. The theories of social psychology describe the stable mechanisms and processes underlying the variable phenomena. These theories should be developed with the use of ‘conceptual’ replications, different experiments that explore aspects of the same theory. According to these social psychologists, the failed replications that the reformers are up in arms about are not a sign of structural problems but of the special nature of social psychology’s object of study. It requires a slightly amended methodology, but there is no need to go crazy and abandon science the way Gergen had done.

On the basis of the failed replications alone it is not possible to choose between these three responses to the replication crisis. First of all, when a replication fails to reproduce the results of the original study, there are broadly speaking two possible conclusions: there is something wrong with the original study, or there is a problem with the replication. As sociologist of science Harry Collins pointed out long ago, it is impossible to decide on purely empirical grounds where the problem lies (Collins, 1985). As long as there is no agreement about the proper outcome it is always possible to put the results in doubt. Collins called this the experimenters’ regress: a debate about which of the two experiments (original or replication) was competently performed is at the same time a debate about what the right outcome of the experiment is, and the right outcome is that of a competently performed experiment.

Collins focused on competence, in particular on tacit skills. But discrepancies between original and replication may also lead to theoretical discussion. The original researchers may defend their theory by arguing that it is basically correct, but that the problem lies with its auxiliary hypotheses (Lakatos, 1970). Any anomalies can be explained away by modifying its protective belt of hypotheses about boundary conditions, proper instrumentation, and so on. Perhaps the replication failed because instrument XYZ is not such a good way to measure the dependent variable after all. Again, this kind of debate cannot be ended on purely empirical grounds, because any further experiment will be vulnerable to the same kind of argument.

Thus, no matter how many failed replications the reformers in psychology point to, each one can be contested, and indeed the crisis has seen plenty of controversies about replications, in which both arguments about competence and the “lakatosian defense” (Meehl, 1990) have figured prominently. Against the social constructionists, moreover, it can always be argued that further research may, one day, unearth the regularities that they believe do not exist. That argument was brought against Gergen in the 1970s (Manis, 1975; Schlenker, 1976) and it is put forward again today with the appeal to the ‘stable underlying mechanisms’. All in all, it seems that replication cannot be the lever for change that reformers and social constructionist hope that it is.

And yet, much has changed in psychology the past 13 years, due to a great extent to the steady stream of (failed) replications. There are of course researchers who continue their work much as they did before the crisis, but the discipline around them has certainly shifted. For one thing, there has been broad support for raising methodological standards, even though specific innovations such as preregistration remain contentious. Moreover, once-thriving research programs such as priming research have fallen in disarray because of the repeated failures to reproduce their results (Leys, 2024). A replication is never decisive, but multiple failures to replicate are an embarrassment and make it increasingly difficult to defend a claim or a theory. The problems in psychology have also given an opportunity to proponents of qualitative research methods to assert themselves and argue, for example, for the integration of qualitative methods into social psychology (Power et al., 2018) and interaction research (Albert & Ruiter, 2018; de Ruiter & Albert, 2017).

Finally, replication studies have encouraged a reflection on the way the field produces knowledge. As we observed in an ethnographic study of replication that I am involved in, trying to replicate someone else’s work confronts researchers with the minutiae of the research process and raises the question which of these details really matter, and which were merely convenient (Derksen et al., 2024). The often messy process behind the neat ‘methods section’ is foregrounded, and the role of tacit knowledge and skills becomes visible. In turn these replicators also consider their own research practice and how they describe it in their papers. These reflections may also include wider epistemological and social epistemological issues: what is a replication really, and who should be charged with conducting them? Thus, replication studies have led to a kind of autoethnography, in which this field of research observes and analyzes its own practices.

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393.

Bibliography

Albert, S., & Ruiter, J. P. de. (2018). Improving Human Interaction Research through Ecological Grounding. Collabra: Psychology, 4(1), Article 1. https://doi.org/10.1525/collabra.132

Collins, H. M. (1985). Changing order: Replication and induction in scientific practice. SAGE Publications.

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93–99. https://doi.org/10.1016/j.jesp.2015.10.002

de Ruiter, J. P., & Albert, S. (2017). An Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology. Research on Language and Social Interaction, 50(1), 90–107. https://doi.org/10.1080/08351813.2017.1262050

Derksen, M., Meirmans, S., Brenninkmeijer, J., Pols, J., De Boer, A., Van Eyghen, H., Gayet, S., Groenwold, R., Hernaus, D., Huijnen, P., Jonker, N., De Kleijn, R., Kroll, C. F., Krypotos, A.-M., Van Der Laan, N., Luijken, K., Meijer, E., Pear, R. S. A., Peels, R., … De Winter, J. (2024). Replication studies in the Netherlands: Lessons learned and recommendations for funders, publishers and editors, and universities. Accountability in Research, 1–19. https://doi.org/10.1080/08989621.2024.2383349

Gergen, K. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309–320. https://doi.org/10.1037/h0034436

Gergen, K. J. (1976). Social Psychology, Science and History. Personality and Social Psychology Bulletin, 2(4), 373–383. https://doi.org/10.1177/014616727600200409

Leys, R. (2024). Anatomy of a Train Wreck: The Rise and Fall of Priming Research. University of Chicago Press.

Manis, M. (1975). Comment on Gergen’s “Social Psychology as History.” Personality and Social Psychology Bulletin, 1(2), 450–455. https://doi.org/10.1177/014616727500100207

Meehl, P. E. (1990). Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631.

Power, S. A., Velez, G., Qadafi, A., & Tennant, J. (2018). The SAGE Model of Social Psychological Research. Perspectives on Psychological Science, 13(3), 359–372. https://doi.org/10.1177/1745691617734863

Schlenker, B. R. (1976). Social Psychology and Science: Another Look. Personality and Social Psychology Bulletin, 2(4), 384–390. https://doi.org/10.1177/014616727600200410