Cultures of Trial and Error: The Promise of Open Notebook Science

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

A lesser known yet powerful aspect of the Open Science movement, open notebook science challenges the traditional ways we produce and share knowledge, creating a more transparent and collaborative approach to research that could foster science correction.

Imagine a world where scientific research isn’t hidden behind paywalls or only shared after months - or even years - of polishing and peer review. Instead, every step of the journey, from brainstorming to experimentation (even the ones that fail), is open to everyone, everywhere, in real time. That’s the essence of open notebook science, a groundbreaking approach to how science could be done and shared, that was first implemented in chemistry education programs for training future professionals, and in research areas with limited commercial viability, such as antimalarial compounds.

In this post, we introduce this innovation through findings from comprehensive documentary research focused mainly on Jean-Claude Bradley's records - who we identify as a recruiter of followers for a new epistemic culture and its knowledge machinery (Knorr-Cetina, 1999). Vividly articulated by Bradley, who is a chemistry professor and researcher at Drexel University (USA), open notebook science challenges the traditional practice of keeping scientific findings confidential until they are formally published in a scientific journal, inviting anyone interested to dive into the details - from raw data to unresolved challenges - as they unfold in open notebooks. Ultimately, it aims to promote “faster science and better science” (Bradley, 2006a) through transparency, data provenance and open collaboration enabled by digital platforms and open licenses.

According to Bradley, open notebook science means “making all your research freely available to the public and in real time” (Bradley, 2010b). He defined open notebook science in the following way:

“[…] I will use the term Open Notebook Science, which has not yet suffered meme mutation. By this I mean that there is a URL to a laboratory notebook (like this) that is freely available and indexed on common search engines. It does not necessarily have to look like a paper notebook but it is essential that all of the information available to the researchers to make their conclusions is equally available to the rest of the world. Basically, no insider information.” (Bradley, 2006a)

Open notebook science, as a concept, was described in numerous blog posts by Jean-Claude Bradley and stemmed from both his advocacy for open knowledge and his personal ambition to expand the impact of science - from educational and research practices to innovative approaches in drug discovery. However, the practical implementation of this innovation was far from a solo endeavor. It relied on a broad and collaborative network of educators, students, researchers, synthetic chemists, computational chemists, biochemists, chemical companies, programmers, mathematicians, journalists, librarians, open knowledge advocates, and a variety of digital tools and platforms. [1] Together, they supported the development of open notebook prototypes and their surrounding research ecosystem. [2]

A new form of scientific communication: from lab secrets to open laboratories notebooks

Science, as often idealized, holds a unique position in shaping our understanding of the world. For centuries, its foundation has relied on the narrative of a specific methodology - the production of facts through careful experimentation, carried out by a select few within the controlled environment of laboratories. These individuals, often perceived as geniuses and extraordinary builders of facts, have been entrusted with the discovery of hidden truths about nature. Their findings, meticulously crafted into matters of fact, became the cornerstone of what we now consider Modern Science (Shapin & Shaffer, 1985).

The scientific article, as we know it today, is the primary instrument (or literary technology) for sharing these discoveries (Shapin & Shaffer, 1985). Published in peer-reviewed journals, it presents research as ready-made science (Latour, 1987) - polished and persuasive narratives that establish the validity of a new scientific fact. This perspective, known as the diffusion model (Latour, 1987), focuses on delivering coherent stories designed to convince the audience of the discovery's truthfulness.

According to Shapin and Shaffer (1985), the origins of this literary technology can be traced back to the 17th century, with figures like Robert Boyle and other natural philosophers. In their experimental essays, they aimed to communicate phenomena observed in private laboratory spaces to those who could not witness the experiments firsthand. Their work pioneered the concept of virtual witnesses (Shapin, 1984) - readers who could imagine the experiment's execution through detailed descriptions and visual representations, bridging the gap between observation and understanding.

To achieve this goal, early scientists developed a distinct style of reporting, one that brought experiments to life for readers. These reports were complemented by visual aids - drawings and diagrams - to fill gaps in imagination and ensure clarity. Over time, this literary method became intertwined with material technologies, like scientific instruments, and social conventions, such as peer validation, to legitimize findings and navigate controversies. Until today, scientific articles tend to limit scientific communication by showcasing only 'successful' experiments, which are filtered by editors from an avalanche of submitted manuscripts.

On the other hand, open notebook science advocates for a translation model (Latour, 1987) that illuminates in real time the complex network of contributors and interactions involved in research activity. This innovation aims to reveal the full research process in detail, including hypotheses, experimental procedures, setbacks, incompleteness, and so-called 'failure' experiments.

The origins of open notebook science

As a science worker, both researcher and university professor, Bradley identified that the peer review process (closed and double-blind) has several structural constraints and inherent biases that harm teaching and research activities, limit innovation, and restrict access to valuable information. His primary critique highlighted that scientific journals, including the most renowned ones, had limited capacity of evaluation since peer review is restricted to anonymous evaluators who deal with important gaps in information - this critique is particularly relevant if we consider that, at the time (early 2000s), scientific journals did not ask authors to submit raw data as supplemental material. Besides that, Bradley considered the voluntary work of reviewers precarious given the shortage of time to issue an opinion and the impossibility of replicating experiments.

In that sense, the author believed that a major and structural flaw of the evaluation process - established to ensure that a scientific paper meets established standards of quality, validity, and relevance - is its heavy reliance on trust. The way he saw it: readers trust editors to choose qualified anonymous reviewers. In turn, reviewers trust authors to summarize their findings accurately. And main authors trust their collaborators, students, and postdocs to produce accurate data (Bradley, 2007b). [3]

Bradley believed that this perspective dialogued with a core principle of the Scientific Revolution - the shift from blindly trusting authorities to systematically questioning everything. He considered it deeply troubling that even in the 21st century, researchers could still make claims based primarily on their position of authority rather than open, transparent, and reproducible evidence. As he puts it: "In chemistry, facts need references back to experimental data. Only opinions can stand without proof - and then the author's reputation becomes the only reference" (Bradley, 2005).

Secondly, the author criticized that scientific articles format often present experimental procedures in a highly condensed and imprecise manner - frequently lacking the level of detail necessary for even a competent peer to replicate the work (Bradley, 2012a). In some cases, the omissions were so significant that he, a senior professional, could not determine whether the error lay in his own replication attempt or in the original publication. For this reason, he envisioned unrestricted access to laboratory notebooks and raw data as the most effective way to share scientific knowledge. Such openness would enable others to trace every step of an experiment without having to rely on assumptions or incomplete information.

Finally, another criticism focused on what he viewed as a needless and systematic waste of valuable research being lost due to restrictive publication practices of only publishing what is considered a successful result. Bradley estimated that 87% of his own scientific production would never go beyond the walls of his lab because it consisted of experiments that had failed to achieve the expected results, being therefore considered as failures by gatekeepers who define what is to be published in scientific journals (Bradley, 2013). To him, science is made up not only of success histories, but also of failures.

For him, this distortion in scientific communication creates a pervasive reluctance among researchers to share ongoing work, a procrastination driven by two interconnected disfunctions: a culture of secrecy rooted in fears of being scooped, and an obsessive need to present only 'completed' scientific facts. The consequence is a systemic bottleneck in knowledge dissemination, where findings remain concealed until they can be polished into seemingly flawless, incontrovertible claims.

In contrast to the prevailing peer review process, structurally built on an ‘excess of trust’, Bradley advocated for open notebooks to minimize assumptions and enhance transparency in scientific communication. His primary concern was not malicious fraud but rather what he termed the ‘trusted source cascade’: errors introduced into literature under the pressure to publish that are subsequently propagated through citations in future articles. In his words, “once these mistakes are published, they are difficult to identify and even harder to correct - if they are discovered at all. Their spread is further amplified by the ease with which content is copied and disseminated via online databases” (Bradley, 2007b). [4]

In this context, making research data and detailed documentation openly available through open laboratory notebooks would greatly enhance the potential for scrutiny, correction, refutation, enrichment, collaboration, validation, and learning across a broader scientific community.

Training skeptical scientists: from information consumers to curators

As a professor of ‘Chemical Information Retrieval’, Bradley encouraged his students to question trusted sources and emphasized that peer review should not be seen as an unquestionable validation. To challenge the students' mindset, he asked them to gather information about a common experiment (synthesizing dibenzalacetone) from five different sources and compare them. The objective was to analyze protocols and specific requirements described in each article to make students identify and understand inconsistencies in literature. To his surprise, several learners used ethyl acetate in the recrystallization step, which didn’t make sense to him since ethyl acetate does not mix with water. Intrigued by how this mistake became widespread in the classroom, Bradley investigated major chemistry references and found that the mistake has been perpetuated since a paper was published back in 1903 (Bradley, 2012b; 2013).

Through this kind of exercise, he aimed to change the students' belief system and attitude toward scientific literature, transitioning their role from passive information consumers into active information curators. Or even act as peers that correct (and open up) science. He believed that future professionals should be capable of collecting, selecting, and evaluating relevant content in Organic Chemistry. Above all, they should know how to verify the reliability, integrity, and reproducibility of experimental data. In his words: “This exercise should teach them not to trust a source implicitly. [...] There are no trusted sources” (Bradley, 2010a).

Asking someone to be skeptical of others’ work may sound disagreeable, as sincere mistakes can happen to anyone. It’s important to make clear that this proposal did not aim to create a surveillance-like environment of constant oversight over others’ work. On the contrary, the genuine purpose of open notebook science was to highlight and value something fundamental to good science, often limited to a small audience: high quality research documentation regardless of the experiment's outcome. In his own words:

“Any chemistry grad student can tell you that there is tremendous value in discussing failed experiments with others who are equally or more knowledgeable. However, this discussion is usually limited to lab co-workers. By recording ongoing experiments in blogs, I can help you just by knowing what you are trying to do, even if you have not yet succeeded.” (Bradley, 2006c)

Open notebook science is part of a strategy to encourage more consistent scientific discussions and promote open collaboration. In that sense, data provenance must be preserved since it is essential to access the history of the shared dataset, trace its origin, understand the transformations it underwent, and review the analyses and interpretations made by scientists. Due to the importance placed on documentation, we refer to its epistemic culture as a matter of proof (Clinio, 2016; 2017), contrasting with the matter of fact of Modern Science.

What does open notebook science look like?

When Jean-Claude Bradley began exploring open notebook science in 2006, he used the Blogger platform to document and share research activities. Blogger’s simple features, like posting updates, enabling comments, and providing RSS feeds, were enough to get started. However, as Bradley's vision of open collaboration grew, he realized that true collaboration required more than commenting or critiquing. It demanded tools for editing, rewriting, and reshaping content.

To meet these needs, Bradley's UsefulChem Project adopted a bliki format - a blend of a blog and a wiki. [5, 6] The blog component was used to update a general audience on the project’s progress, while the wiki recorded detailed experimental work. This structure allowed researchers to collaborate actively, making changes and contributions directly without needing administrator permissions. For the wiki, Bradley introduced a standardized format for experiment pages to ensure consistency and encourage reuse of data. The experiment page consisted of nine sections: 1) the number of the experiment; 2) the graphic representation of the experiment, 3) the name of the researcher; 4) the objective; 5) procedures; 6) results; 7) discussion; 8) conclusion and 9) log.

Figure 1. Experiment n. 269, done by the student Matthew McBrid

Available at: http://open notebook sciencechallenge.wikispaces.com/EXP269
Accessed on: 10 Aug. 2015

Bradley (2006c) emphasized that the log section is the most critical part of an experiment record. Ideally, it captures every step of the experiment as it unfolds, ensuring accuracy and a rich level of detail. The log is written in a clear and objective style, avoiding unnecessary embellishments. It answers the essential questions: who did what, where, how, when, and what partial results were achieved.

This approach removes the need for a polished narrative, which is often required in traditional scientific articles. Instead, it creates what Bradley (2007a) calls a ‘storyless experiment’ - a straightforward, result-focused record that places equal importance on all outcomes, whether or not they align with the experiment's original goals.

Figure 2 : Experiment n. 269 page detail – the log section

Available at: http://open notebook sciencechallenge.wikispaces.com/EXP269
Accessed on: 10 Aug. 2015

Building an ecosystem for open collaboration

Bradley conceived open notebooks as hubs for collaboration. Its ‘architecture of attention’ aims to connect the right people to the right problems at the right time. In that sense, its ecosystem would function as a social networking platform that connects people that share interests and challenges. On this attempt of ‘reinventing discovery’ (Nielsen, 2011), openly sharing ‘negative results’, incomplete, or ambiguous experimental data should be interpreted as an invitation to collaborate with others. The idea is to take advantage of latent micro-expertise to provide just-in-time solutions for specific research bottlenecks, amplifying collective intelligence (Nielsen, 2011). Nonprofit initiatives such as the OpenSource Malaria project, with which Bradley collaborated, and more recently, the Montreal Neurological Institute-Hospital (The Neuro) are initiatives that have embraced open notebooks as a tool to boost research for neglected diseases.

Open notebook science has great potential to redefine how researchers solve problems. However, in order to achieve its objective to promote ‘faster science, better science’ (Bradley, 2006b), its open collaboration ecosystem requires robust infrastructures and a new culture between science workers. In brief, these shifts could promote that scientific knowledge would be communicated to others without prior control by an editorial committee (de-editorialized), through an open peer review process that is inclusive (accessible to everyone), transparent (visible to all), continuous (with no expiration date), interactive (allowing for responses and counter-responses), and dynamic (welcoming new comments and corrections over time).

For Bradley, ’faster and better science’ could be achieved through open notebook science, which fosters rigorous debate among all scientific stakeholders, such as professional researchers, students, amateurs, and even bots. It could also reduce redundant or misguided research and minimize wasted time and resources. By advocating those well-documented experiments be recognized as valid scientific contributions - regardless of their outcomes - open notebook science promises to foster meticulous record-keeping and a culture of continuous improvement. It could transparently establish priority among scientists through three key mechanisms: time-stamped contributions (automatically recorded), version-control systems (tracking all edits and additions), and third-party administration (ensuring impartial record-keeping).

Collectively these features create an auditable trail of intellectual contributions that could help resolve authorship disputes while maintaining collaborative integrity. Moreover, open notebook science could significantly enhance replicability and encourage open collaborations, potentially proving more effective than ready-made (and closed) science frameworks.

This text is based on the thesis “New laboratory notebooks and new epistemic cultures: between the politics of experiment and the experiment of politics” (2016), written by Anne Clinio. For further information about open notebook science in English, please consult the scientific article “Open notebook science as an emerging epistemic culture within the Open Science movement” published on the Revue Française des Sciences de l'information et de la communication.

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393.

Footnotes

[1]: The idea that open notebooks can accelerate progress and enhance the quality of science was a common argument among advocates like Matthew Todd, Anthony Salvagno, and Steve Koch, as well as open science evangelists such as Cameron Neylon and Peter Murray-Rust. Matthew Todd, for instance, a professor at the University of Sydney who collaborated closely with Bradley on an open drug discovery project, highlights a key advantage: “Selfishly, from a scientist’s point of view, it helps to be open because other people can correct your mistakes before you make them.” This openness, he explains, prevents wasted time on unproductive research paths (Todd, 2014).

[2]: During the prototyping phase, early enthusiasts strategically customized well established information tools to minimize learning curves and adoption barriers, incorporating corporate solutions when practical. They established an integrated digital ecosystem combining wiki-based experimental pages, Blogger for research communication, and Google Spreadsheets for collaborative data management. This documentation framework was enhanced by multimedia platforms - Flickr for experimental imagery and YouTube for procedural demonstrations. The ecosystem also implemented advanced chemical informatics through semantic web standards, InChI/InChIKey identifiers, and dual indexing via both Google and ChemSpider for optimal data discovery. Early enthusiasts developed specialized search through Google Custom Search and leveraged the Collaborative Drug Discovery (CDD) platform for targeted data access. The innovation adopted Creative Commons licenses, which was supplemented by a visual four-icon system denoting documentation granularity (full/partial) and dissemination timing (real-time/delayed). Their sharing practices also included material technologies, such as reagents and instruments, when a new lab partner accepted to adopt open research practices. By that time, task automation had already emerged as a key discussion point.

[3]: Professionals who often lack access to raw data, as well as the time and resources necessary to replicate experiments.

[4]: Preceding quotes in this paragraph come from the same source: Bradley, 2007b.

[5]: Originally named for its research directive ("chemists focusing their attention on chemical solutions to real immediate problems" (Bradley, 2006a)), UsefulChem became the prototype open notebook ecosystem, evolving from simple blogging to integrated, structured documentation platforms.

[6]: See: http://usefulchem.blogspot.com, and http://usefulchem.wikispaces.com

Bibliography

Bradley, J.-C. (2005, July 24). Peer review and supported documents [Blog post]. Drexel Coas e-Learning. http://drexel-coas-elearning.blogspot.com.es/2005/07/peer-review-and-supported-documents.html

Bradley, J.-C. (2006a, May 7). Bird flu chemistry progress [Blog post]. UsefulChem. http://usefulchem.blogspot.com.br/2006/05/bird-flu-chemistry-progress.html

Bradley, J.-C. (2006b, September 26). Open Notebook Science [Blog post]. Drexel COAS E-learning. http://drexel-coas-elearning.blogspot.com.br/2006/09/open-notebook-science.html

Bradley, J.-C. (2006c, February 13). Peer review in Google age [Blog post]. Drexel Coas e-Learning. http://drexel-coas-elearning.blogspot.com.es/2006/02/peerreview-in-google-age.html

Bradley, J.-C. (2007a, August 13). C&E News article on writing [Blog post]. UsefulChem. http://usefulchem.blogspot.com.br/2007/08/c-news-article-on-writing.html

Bradley, J.-C. (2007b, October 4). Science is about mistrust [Blog post]. UsefulChem. http://usefulchem.blogspot.com.es/2007/10/science-is-about-mistrust.html

Bradley, J.-C. (2010a, January 2). Dangerous data: Lessons from my Cheminfo Retrieval class [Blog post]. UsefulChem. http://usefulchem.blogspot.com.es/2010/01/dangerous-data-lessons-from-mycheminfo.html

Bradley, J.-C (2010b, September). The impact of Open Notebook Science: Interview with Richard Poynder. Information Today. https://www.infotoday.com/it/sep10/Poynder.shtml

Bradley, J.-C. (2012a, August 20). Shining a light on chemical properties with Open Notebook Science and open strategies [Video]. YouTube. https://www.youtube.com/watch?v=7mK6SY-jXqk

Bradley, J.-C. (2012b, September 27). A recrystallization app based on open services [Video]. Drexel Chemistry Mini-Symposium. https://www.youtube.com/watch?v=vMndyT-DSXM

Bradley, J.-C. (2013, February 12). Open education [Video]. Tuesday Tech Talks. https://www.youtube.com/watch?v=BN8UjULNG9A

Clinio, A. (2016). New laboratory notebooks and new epistemic cultures: Between the politics of experiment and the experiment of politics [Doctoral thesis, IBICT/UFRJ-ECO]. RIDI IBICT Repository. http://ridi.ibict.br/handle/123456789/943

Clinio, A., & Albagli, S. (2017). Open notebook science as an emerging epistemic culture within the Open Science movement. Revue française des sciences de l’information et de la communication, 11. https://doi.org/10.4000/rfsic.3186

Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University Press.

Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Harvard University Press.

Nielsen, M. (2011). Reinventing discovery: The new era of networked science. Princeton University Press.

Shapin, S. (1984). Pump and circumstance: Robert Boyle's literary technology. Social Studies of Science, 14(4), 481–520.

Shapin, S., & Schaffer, S. (1985). Leviathan and the air-pump: Hobbes, Boyle, and the experimental life. Princeton University Press. https://doi.org/10.1515/9781400838493