Cultures of Trial and Error: Field-specific Barriers to Performing Replications

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

“Reproducibility is a defining feature of science,” opens one article describing a large-scale replication project (Open Science Collaboration, 2012, p. 657). [1] Many advocates for replication begin similarly, framing replicability as a universal, incontrovertible quality of good science. Discussions about what hinders replications are similarly general, focusing on obstacles common to most fields: researchers are not incentivized to conduct replications and editors are not incentivized to publish them (Open Science Collaboration, 2014; Romero, 2018; Vachon et al., 2021), key parameters are not reported in methods sections (Clayson et al., 2019), infrastructures for sharing/reusing data are insufficient (Horbach et al., 2025; Vachon et al., 2021), and little funding is available specifically for replications (Lilienfeld, 2017).

In this piece, we call for more comparative research on field-specific replication dynamics. Replicability is not a goal that makes sense for all research fields, nor should it be taken uncritically as an indicator of research quality (Leonelli, 2018). Even in fields where replication is desirable, the obstacles to replication may be different in kind or scale. As work in science and technology studies (STS) has shown, scientific communities vary widely in their social structures, material practices, and the epistemic strategies they pursue (Knorr Cetina, 1999). Some disciplines invest heavily in increasing their control over the material world, bringing objects of interest into the laboratory where they can be purified, modified, and recombined (Latour, 1983). In other fields, in contrast, scientists constrain the messiness of the world through statistical manipulation rather than investing in 'bench-building' (Peterson, 2015, p. 1215). In this post, we explore how such differences impact the local texture of the challenges faced by those attempting replications, drawing on our interdisciplinary expertise in STS, history of science and technology, metascience, astronomy, and cell biology.

The Reproducibility Projects

The Center for Open Science’s large-scale replication projects demonstrate how access to specialized skills sets and resources constrain what is possible, and how the magnitude of these constraints differs among fields. The Reproducibility Project: Psychology (RP:P), launched in 2011, aimed to replicate studies from prominent psychology journals. Of the 158 studies selected, only 111 studies 'matched' with collaborators willing to replicate them, with the unmatched studies typically requiring specialized samples (such as people with autism or nonhuman primates) or specialized resources (such as functional magnetic resonance imaging or eye-tracking equipment) (Open Science Collaboration, 2015, aac4716-2). Of the studies selected, however, the group’s completion rate was high: 100 out of 111 attempted replications were completed on time and the meta-analysis of these results was published less than four years after the project began.

While the RP:P certainly faced problems with access to skills and materials, its sister project, the Reproducibility Project: Cancer Biology (RP:CB), faced even greater resource constraints. In contrast to the RP:P, which conducted their experiments with a grassroots network of volunteers, the RP:CB initially budgeted $25k USD for replicating 50 studies in commercial fee-for-service labs (funded by a private foundation grant and subsidized by donations from over a dozen biomedical reagent suppliers). Even these resources were not enough, however: project leaders reduced the number of studies to be replicated to 37 in 2015, then to 29 in 2017, and finally to just 18 in 2018 (Kaiser, 2018). Cost was a major barrier (studies cost $60k USD on average), but not the only one. In some cases reagents were no longer available from commercial providers; in other cases negotiating material transfer agreements took so long that the team opted to rebuild the needed materials from scratch (Errington et al., 2021a). In all, it took eight years from announcement of the RP:CB to publication, with each individual project taking an average of nearly four years from design to completion (Errington et al., 2021a; 2021b). Other projects have similarly been able to replicate a greater volume of social science studies. The Dutch NWO replication project, for example, completed 18 social science replications but only three medical replications (Derksen et al., 2024).

Comparing these two cases also reveals domain specific differences in seemingly similar problems. Both psychology and cancer biology face constraints on access to proprietary materials needed to conduct replications (Hays et al., 2017; Murray, 2010), but many of psychology’s core tools are protected by copyright while biology’s are protected by patents. This distinction is important because copyright accrues automatically to researchers, and US American universities have historically treated copyright as belonging to the author (AAUP, 1999). This creates the possibility for actions at the individual level which preserve access to scientific tools, such as using open source/open access licenses. In contrast, researchers must apply for patent protection, and the fees associated with this process limit who can engage with it and incentivize those who do to monetize their scientific tools. By law US American universities hold the rights to inventions made using federal funds (Patent and Trademark Law Amendments Act, 1980), and so would-be replicators must negotiate access to patented tools with institutions rather than individuals, as in the case of the RP:CB.

Replications in the nanosciences

The nanosciences span multiple disciplines, so attempts at replication in the nanosciences require a broad range of skills for working with different scientific objects and instruments. Thus, the first formalized replication project in the nanosciences - in which some of us are involved as replicator, advisor, or ethnographer - includes researchers from physics, biochemistry, and cell biology. The NanoBubbles replication project examines whether and to what degree (percentage-wise) nanoparticles escape intracellular vesicles called endosomes and access the cytosol where they can be used for sensing (and perhaps thus for diagnostics). Learning to produce (i.e., synthesize) nanoparticles and investigate their properties (i.e., characterize) requires skills that can take years to acquire. One advisory board member has “come up with 3 year[s] as estimate to really reproduce one paper with things new to the group” since “for all new materials one has to do all characterization from the beginning to really understand what happens” (Email to Research Participants 3, 2, and 1, 11.04.2024). This scientist thus plans for the replication of one paper to span an entire PhD project.

Acquiring an interdisciplinary skill set requires time but also money and social connections. Similar to RP:CB, costs represented obstacles for the NanoBubbles replication project and prompted the team to produce one organic molecule from scratch. As it was too expensive to buy, the biochemist-postdoc suggested synthesizing it. Since it was outside of the expertise of the team members, they collaborated with an organic chemist, who supervised the postdoc in synthesizing the organic molecule, which lasted four-months and was not even considered part of the replication as it was a preparatory step. For the replications, the biochemist synthesizes and characterizes the nanoparticles, while the biologist is responsible for studying them inside cells. In both parts, an in-depth understanding of different microscopy techniques is crucial. Lacking the time, money or personnel for establishing such internal division of labor and collaborations with external colleagues, which enabled the replication team to cover a broad, interdisciplinary set of materials, instruments and skills associated with them, can impede replications.

Working across different disciplines to conduct replications in the nanosciences also presents spatial challenges (and potential opportunities) (Choi & Shields, 2015; Yaneva, 2022). Rather than working in a “hybrid lab” (Yaneva, 2022, p. 738), the NanoBubbles replication project established a chain of labs, spatially illustrating how the nanosciences range from basic research with materials to applied science in biological environments. This chain of labs also challenges the homogeneity of the epistemic culture, as knowledge production is not based on norms, practices, and materials coming from a single field but of several that are moving across fields. A repercussion that stems from a heterogeneous epistemic culture as such is that scientists from different fields may have a different ways of reasoning, and different standards. For example what may serve as a necessary control for a biologist may seem needless for a chemist. The NanoBubbles team is based in an inorganic chemistry lab, while also working in the aforementioned organic chemistry lab. The replicators also use the chemistry repository Chemotion to re-characterize, store and share the data and samples of the organic molecule with other researchers. Additionally, the nanoparticles inside cells were planned to be investigated in a biology lab linked to a hospital. The replicators furthermore collaborated with two microscopy facilities, one in their own country and one in another country. Thus, the team created a network of labs, microscopy facilities, and repositories which enabled them to perform and verify different steps of their replication. Establishing such a network requires extensive preparation and effort, again hindering replications in this field.

Barriers between fields have been present since the beginnings of the nanosciences, and throughout that time disciplinary barriers have - occasionally - facilitated as well as hindered replications. One of the instruments credited with enabling nanoscience research was the scanning tunneling microscope (STM), invented in the early 1980s. However, the STM’s inventors were (mostly) unaware that they were replicating an earlier, similar instrument called the Topografiner. Other researchers, particularly surface scientists, believed that the Topografiner was a failed experiment and thus had not attempted to replicate it. In fact, surface scientists labored under an incorrect understanding of why the Topografiner failed (Mody, 2011); unencumbered by that disciplinary knowledge, the STM’s inventors (who were not surface scientists) merrily forged ahead. In turn, when the first STM results were announced, surface scientists flocked to replicate it but could not obtain the needed tacit knowledge from its inventors. Getting past that barrier required all the early replicators to meet together, thus seeding formation of an STM community that lowered barriers to future replications.

Indeed, they were perhaps lowered too far. In the early 1990s, multiple groups reported atomic resolution of organic molecules deposited on graphite in air - a result surface scientists tried, unsuccessfully, to convince other fields was improbable. Certain features of graphite and air STM were, however, later found to make ‘replication’ of spurious artifacts laughably easy. Air STM of biomolecules was thus largely abandoned, though no results were retracted - and barriers between fields (especially between experimental physicists on the one hand and materials scientists, chemists, and biologists on the other) have hindered flows of knowledge about common microscopy artifacts and thus fostered similar controversies over improbable ‘replications’ (Cesbron et al., 2012).

Observational astronomy

Astronomy instructively illustrates what replications entail and how equipment shapes the ability to undertake replications. Large telescope projects - both ground-based ones like the Square Kilometre Array (SKA) and space-based ones like the James Webb Space Telescope (JWST) - take several decades to plan, design and build, involving many thousands of individuals with a range of skills. The imperative to coordinate around large publicly-funded infrastructure significantly molds research cultures of observational astronomy. By the time they come on-line, explicit trust in large telescopes is high (the aberrations in the Hubble Space Telescope’s primary mirror being a rare example of astronomers’ doubts being made public). Once operational, the time available on these devices is limited and strictly rationed between elite professional astronomers (McCray, 2000). The volume of data collected on even a single night’s observation is vastly greater than what could be collected in a lifetime a century ago, presenting problems for data retention; for some datasets “it is not possible to keep all the raw data after analysis is complete” (Pepe et al., 2014, p. 1). Astronomy also shows how the boundary between replicability and reproducibility can be blurry. As an observational science, data collected is understood to be in principle unrepeatable, although given the timescale over which most astronomical systems evolve this is only a theoretical concern. Nonetheless, replication of measurements is quite rare. On the other hand, with strong confidence in underlying theoretical mechanisms, observations of other similar target systems are considered as probative as direct replications and are treated equivalently. As a result, for astronomers, replication primarily entails discovering, locating, retrieving and storing pre-existing data (Sands et al., 2013). Most working astronomers no longer control primary data collection and disputes are addressed through the assemblage of resources and producing interpretations. These re-use practices are no more likely to refer to primary data; as one participant in the study by Sands et al. (2013) noted “there are probably ten people on the face of earth that ever re-reduced Sloan images.” Data collected in vast amounts that are almost unmanageable, through a small number of observational facilities, create research cultures of data reuse that shape the possibilities of replication studies.

Towards field comparative research

These cases from psychology and cancer biology, the nanosciences, and astronomy illustrate that limited access to specialized and proprietary research objects and equipment, the skills associated with them, their timescales, and the infrastructures in which they are embedded can constrain replications in those fields. The understanding of what a replication entails can also differ greatly across research cultures, illustrated for example by how astronomers consider observations of other similar target systems as evidential as direct replications. While some barriers might be similar across fields, their magnitude can differ among disciplines, as illustrated by the comparison between the Reproducibility Project: Psychology and the Reproducibility Project: Cancer Biology. This can slow replication projects in certain disciplines and reduce their sample of replicated articles. In nanotechnology, replications are made more difficult by the variety of scientific objects, instruments, and standards coming together in this highly interdisciplinary field. Building a team with skills in materials science, cellular biology, and different microscopy techniques and establishing a network of labs ranging from basic to applied science can be time-consuming and resource-intensive. The history of the STM community, which formed partly because of a meeting aimed at understanding failed replications, furthermore illustrates how barriers between fields have hindered replications of earlier scientific instruments and have sparked controversies over easily replicable microscopy artifacts. The material infrastructure in astronomy complicates further the universal calls for replications since scientific equipment may need decades to be built, access is limited, and the equipment generates a large amount of raw data, which is collected by only a few observational facilities and is hard to preserve. We encourage more elaborate comparative research on field-specific barriers to replications that may help predict how likely some disciplines are to engage in replications and that can also provide insights needed to facilitate replications in these fields.

Acknowledgments

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393. CSFB, MMS and CCMM have individually received funding from the ERC-NanoBubbles project.

Footnotes

[1]: Early works use reproducibility and replication interchangeably; in this post we use the more currently accepted definition of replication as collecting new data by closely following the approach of a previously conducted study.

References

American Association of University Professors. (1999). Statement on Copyright. https://www.aaup.org/reports-publications/aaup-policies-reports/topical-reports/statement-copyright

Cesbron, Y., Shaw, C. P., Birchall, J. P., Free, P., & Lévy, R. (2012). Stripy nanoparticles revisited. Small, 8(24), 3714-3719. https://doi.org/10.1002/smll.201001465

Choi, H., & Shields, B. (2015). A place for materials science: Laboratory buildings and interdisciplinary research at the University of Pennsylvania. Minerva, 53(1), 21-42. https://doi.org/10.1007/s11024-015-9265-6

Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019). Methodological reporting behavior, sample sizes, and statistical power in studies of event‐related potentials: Barriers to reproducibility and replicability. Psychophysiology, 56(11), e13437. https://doi.org/10.1111/psyp.13437

Derksen, M., Meirmans, S., Brenninkmeijer, J., Pols, J., de Boer, A., van Eyghen, H., Gayet, S., Groenwold, R., Hernaus, D., Huijnen, P., Jonker, N., de Kleijn, R., Kroll, C. F., Krypotos, A.-M., van der Laan, N., Luijken, K., Meijer, E., Pear, R. S. A., Peels, R., … de Winter, J. (2024). Replication studies in the Netherlands: Lessons learned and recommendations for funders, publishers and editors, and universities. Accountability in Research-Policies and Quality Assurance, 32(7), 1285-1303. https://doi.org/10.1080/08989621.2024.2383349

Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021a). Challenges for assessing replicability in preclinical cancer biology. elife, 10, e67995. https://doi.org/10.7554/eLife.67995

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021b). Investigating the replicability of preclinical cancer biology. elife, 10, e71601. https://doi.org/10.7554/eLife.71601

Hays, R. D., Weech-Maldonado, R., Teresi, J. A., Wallace, S. P., & Stewart, A. L. (2018). Commentary: Copyright restrictions versus open access to survey instruments. Medical Care, 56(2), 107-110. https://doi.org/10.1097/MLR.0000000000000857

Horbach, S. P., Cole, N. L., Kopeinik, S., Leitner, B., Ross-Hellauer, T., & Tijdink, J. (2025). How to get there from here? Barriers and enablers on the road towards reproducibility in research. https://osf.io/gx9jq/download

Kaiser, J. (2018). Plan to replicate 50 high-impact cancer papers shrinks to just 18. Science, 243(10.1126). https://doi.org/10.1126/science.aau9619

Knorr Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University press.

Latour, B. (1983). Give me a laboratory and I will raise the world. In K. Knorr Cetina & M. Mulkay (Eds.), Science observed: Perspectives on the social study of science (pp. 141-170). Sage.

Leonelli, S. (2018). Rethinking reproducibility as a criterion for research quality. In L. Fiorito, S. Scheall, & C. E. Suprinyak (Eds.), Including a symposium on Mary Morgan: curiosity, imagination, and surprise (pp. 129-146). Emerald Publishing Limited. https://doi.org/10.1108/S0743-41542018000036B009

Lilienfeld, S. O. (2017). Psychology’s replication crisis and the grant culture: Righting the ship. Perspectives on psychological science, 12(4), 660-664. https://doi.org/10.1177/1745691616687745

McCray, W. P. (2000). Large telescopes and the moral economy of recent astronomy. Social Studies of Science, 30(5), 685–711. https://doi.org/10.1177/030631200030005002

Mody, C. C. M. (2011). Instrumental Community: Probe Microscopy and the Path to Nanotechnology. MIT Press.

Murray, F. (2010). The oncomouse that roared: Hybrid exchange strategies as a source of distinction at the boundary of overlapping institutions. American Journal of sociology, 116(2), 341-388. https://doi.org/10.1086/653599

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660. https://doi.org/10.1177/1745691612462588

Open Science Collaboration (2014). The reproducibility project: A model of large-scale collaboration for empirical research on reproducibility. In: V. Stodden, F. Leisch, & R. D. Peng (Eds.) Implementing Reproducible Research (pp. 299-324). CRC Press, Taylor & Francis Group.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Patent and Trademark Law Amendments Act, 35 U.S.C. §§ 200–212 (2018).

Pepe, A., Goodman, A., Muench, A., Crosas, M., & Erdmann, C. (2014). How do astronomers share data? Reliability and persistence of datasets linked in AAS publications and a qualitative study of data practices among US Astronomers. PLoS ONE 9(8), e104798. https://doi.org/10.1371/journal.pone.0104798

Peterson, D. (2015). All that is solid: Bench-building at the frontiers of two experimental sciences. American Sociological Review, 80(6), 1201-1225. https://doi.org/10.1177/0003122415607230

Romero, F. (2018). Who should do replication labor?. Advances in Methods and Practices in Psychological Science, 1(4), 516-537. https://doi.org/10.1177/2515245918803619

Sands, A., Borgman, C.L., Wynholds, L. & Traweek, S. (2013), Follow the data: How astronomers use and reuse data. Proceedings of the Association for Information Science and Technology, 49(1), 1-3. https://doi.org/10.1002/meet.14504901341

Vachon, B., Curran, J. A., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., ... & Grimshaw, J. M. (2021). Changing research culture toward more use of replication research: a narrative review of barriers and strategies. Journal of clinical epidemiology, 129, 21-30. https://doi.org/10.1016/j.jclinepi.2020.09.027

Yaneva, A. (2022). The architecture of the hybrid lab: Spacing graphene research. Social Studies of Science, 52(5), 733–757. https://doi.org/10.1177/03063127221114721