Cultures of Trial and Error: Hoaxes, Thought Experiments & Science Sleuths: Sniffing Out Ways to Correct the Scientific Record

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

In August 2024, a peculiar letter to the editor titled “Practice of Neurosurgery on Saturn” was published in the International Journal of Surgery Case Reports. The title suggests that the article is either a spoof or a hoax. However, continued reading reveals a scientific and formal tone adopted by the authors as they describe three clinical cases of chirurgical interventions on SATURN, in a country called ILLUSIONLAND, where 60 million Homo sapiens sapiens have lived since migrating from Earth (Mostofi & Peyravi, 2024). In fact, after reading the entire article, it becomes clear that the authors are conducting a thought experiment, using absurdity and humor as rhetorical devices to highlight issues that happens in real settings.

This piece might have gone unnoticed by the broader neurosurgical community, given that the Elsevier journal in which it was published has an impact factor of 0.6. This indicates that, on average, articles published in the journal were cited 0.6 times in recent years. Consequently, at least 40% of the journal’s publications receive no citations, while the remainder are cited only rarely, highlighting the journal’s relatively marginal status within the field of surgery. Nevertheless, this particular piece managed to catch the attention of “science sleuths”.

“Sleuths” is the most common name used by journalists to refer to a group of individuals “who are willing to read texts, evaluate images, design screening tools, run through statistical analyses of a publication’s data, and share their findings and views on websites, blogs, wikis and social media” (Biagioli & Lippman, 2020, p.17). The findings that science sleuths share or flag often involve inconsistencies, errors, or even fraud cases within the scientific and academic literature. They believe that the academic corpus should be clean, correct and pure. [1] From their perspective, an article like “Practice of Neurosurgery on Saturn” may represent a blemish – a spot of contamination that undermines the credibility of the scientific record. This raises important questions: Is there truly no place for humor in scientific literature? What can be learned from such unorthodox contributions? And how do science sleuths, peer reviewers, and the broader academic community respond to thought experiments, hoaxes, or spoofs?

There are, in fact, different ways in which hoaxers and sleuths aim to address and correct the scientific record. Hoaxers typically expose weaknesses in the editorial or peer review process by submitting intentionally flawed or absurd work, while sleuths focus on identifying and flagging specific cases of problematic research. They often use PubPeer, a post-publication peer review platform, to flag issues, leave comments, and publicly express their concerns and dissatisfaction about scientific articles. On this independent platform, anyone can comment on any published paper. This system has sparked criticism, particularly around the question of who is qualified to assess the quality of scientific work (Horbach & Halffman, 2018). In addition, some argue that PubPeer has become more of a space for denunciation, or an incubator for sleuths, rather than a forum for constructive scientific dialogue (Blatt, 2015). Nevertheless, several significant fraud cases – and the subsequent corrections and retractions – were addressed following comments on PubPeer (Dubois & Guaspare, 2019). PubPeer is likely to play an increasingly important role in reshaping how scientific articles are collectively evaluated. Articles once certified as valid through peer review can become contested objects through discussions on PubPeer, prompting greater awareness and initiating a process of reassessment by some editors.

Unsurprisingly, Practice of Neurosurgery on Saturn sparked some reactions on PubPeer, where commentators demanded answers: “Could the authors and the journal explain the rationale behind the publication of this ‘case study’?” (Anonymous, 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #1) In other words, how did a piece like this make it into the scientific literature? Some comments were more provocative: “At least informed consent was obtained from all participants. No violations of research ethics on Saturn.” (Anonymous, 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #3) One commenter addressed the journal editor directly: “My guess is that this spoof was primarily designed to expose poor peer review and editorial standards. It took the journal eight days to accept! Publishing charge $1,945!” (Thornton, Jim G 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #5)

The authors quickly disclosed on Pubpeer that the article was a parody. Their intention was to highlight the challenges neurosurgeons may face in certain professional contexts – challenges that could not be illustrated using real clinical cases due to ethical constraints. As a result, the scenarios had to be presented hypothetically, which is why the authors chose to set the case on another planet.

The use of hoaxes in the scientific literature

Some hoaxes have gained popularity and become highly cited, but what has the academic community learned from them? Are they an effective way of correcting the scientific record? I am particularly interested in examining the reaction from the academic community, including the sleuths, towards articles later recognized as hoaxes, as well as the shared expectations that were disrupted or disappointed upon their exposure.

One of the most prominent academic hoaxes is Alan Sokal’s article, “Transgressing the Boundaries: toward a Transformative Hermeneutics of Quantum Gravity”, published in the journal Social Text. (Sokal, 1996b) The article was filled with nonsensical and illogical claims in physics, but it was so densely packed with jargon, buzzwords and references that it was unlikely to be recognized as nonsense, especially because those claims were beclouded by the main message, which was ideologically and politically aligned with the field’s purpose (Cultural Studies) and the editors’ views. Immediately after publication, Sokal revealed the hoax in another journal, Lingua Franca. The story was quickly picked up by major newspapers and received extensive media coverage (Lynch, 1997).

In the Sokal hoax, the primary aim was not to challenge the peer review process, but rather to critique the field of Cultural Studies. The hoax was used to question its epistemic standards and its complicated language. In contrast, other hoaxes have primarily targeted editorial practices, particularly in response to the rise of fee-charging Open Access journals with poor editorial standards, commonly known as predatory journals.

The Bohannon’s “experiment” serves as a clear example of this type of hoax. The science journalist John Bohannon conducted an “experiment” in which he submitted a fabricated and deliberately flawed pharmaceutical study to 304 journals with a biological, chemical, or medical title (Lagerspetz, 2020). Of the 255 journals that rendered a final decision within the project’s timeframe, 157 accepted the paper, while only 98 rejected it. In each case, Bohannon withdrew the submission prior to publication and disclosed the hoax in an article published in Science (Bohannon, 2013).

The article Practice of Neurosurgery on Saturn isn’t exactly a hoax; it’s rather a thought experiment, an imaginary scenario designed to make a point or test a theory, much like Schrödinger’s cat. [2] While hoaxes typically aim to deceive or mislead, the authors of Practice of Neurosurgery on Saturn employ humor to provoke reflection, without ever claiming to present genuine surgical cases. Nevertheless, some sleuths have misread and responded to the piece as though it were a deliberate hoax.

The blurred boundary between fraud and hoax

What is the difference between fraud and hoax, particularly in the cases cited above? According to The Proceedings of the National Academy of Science, one of the first U.S. federal frameworks and educational guides developed around the term: “Fraud encompasses a wide spectrum of behaviors. It can range from selecting only those data that support a hypothesis and concealing the rest (“cooking” data) to changing the readings to meet expectations (“trimming” data) to outright fabrication of results.” In other words, “fraud arises when scientists intentionally publish inaccurate results.” (National Academy of Science, 1989) [3]

In Sokal and Bohannon cases, the authors violated editorial standards by submitting papers containing results they knew to be inaccurate. However, an important question arises: do hoaxes constitute ethical transgressions, or are they a legitimate means of exposing deficiencies in the editorial and publishing process? Can it be argued that the end justifies the means? To address this, we must first clarify the desirable end of both fraud and hoax.

Fraudulent publications are typically motivated by personal gain. The incentives for committing fraud may include obtaining or maintaining an academic position, gaining prestige or influence, attracting research funding, or confirming a belief in a particular hypothesis by “cutting corners” to reach a desired conclusion. In all cases, the fraudulent author aims to benefit by adding a new publication to the scientific record.

In contrast, hoaxers often have different motivations. Their primary aim is usually to expose institutional practices or deficiencies in the editorial and peer review processes of scientific journals. By successfully publishing demonstrably nonsensical or deliberately flawed work, they seek to reveal that editorial standards are insufficient and that manuscripts are sometimes accepted without appropriate scrutiny by technically competent reviewers. Thus, while a hoax is technically similar to fraud in definition, its underlying motivations differ significantly.

Reactions to hoaxes

In order to achieve their purpose – namely, to test how institutional practices function – hoaxes need to dismiss any accusation of ethical violation. As Lagerspetz observes, the term hoax is often used in a more conciliatory manner. “The perpetrators themselves tend to favor the term “experiment”, implying both knowledge creation and a context that can be argued to justify manipulative practices.” (Lagerspetz, 2020, p. 406)

The Sokal hoax received extensive academic and media attention, winning a place in the public spotlight, due in part, though not exclusively, to Sokal’s dismissal of any accusation of fraud, as noted by some authors (Hilgartner, 1997 & Lynch, 1997). Sokal claimed that all the works cited were real, and none were invented. According to him, the only deliberate inaccuracy concerned the physics content, which, as he stated, “any competent physicist or mathematician would realize… is a spoof” (Sokal, 1996a). This claim creates a paradox for those who accuse Sokal of scientific fraud. Such an accusation would hold weight only if his article had been submitted to and accepted by a physics journal – which it was not. Social Text is a journal in humanities; it does not publish empirical studies or experiments subject to verification for scientific accuracy (Lynch, 1997).

On the other hand, the Bohannon hoax reaped both praise and criticism, particularly regarding publishing ethics and research design (Buckland et al., 2013). Surprisingly, some sleuths expressed strong disapproval, raising several concerns on PubPeer (PubPeer, 2013). These reactions trigger an important question: why do hoaxes tend to frustrate and disappoint certain sleuths, whose primary goal is to correct and clean the scientific literature? Are hoaxes, in fact, an ineffective or inappropriate strategy to correct the scientific record? In conversations with some of these sleuths, a common sentiment emerged – readers do not want to feel deceived by scientific publications. From their perspective, authors are expected to maintain a tone of seriousness and transparency, leaving no room for irony or satire. In their view, authors need to stand firmly behind their work and ideas.

As one sleuth clearly stated on Pubpeer in reference to the case of Practice of Neurosurgery on Saturn: “Satire is rare in scientific publishing, and readers are generally not expecting to have to spot such pieces; also, sadly, ostensibly serious articles that express ideas almost as bonkers that of neurosurgery taking place on another planet do appear in the literature with depressing regularity” (Brown, Nicholas 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #7)

Conclusion: What we could learn from hoaxes?

In terms of concrete outcomes for the academic publishing system, Bohannon’s hoax prompted the Directory of Open Access Journals (DOAJ) to implement stricter quality control measures, resulting in the delisting of 114 journal titles. (Teixeira da Silva & Al-Khatib, 2016) The case of Practice of Neurosurgery on Saturn, triggered a public letter from the editor of the International Journal of Surgery Case Reports, in which he endorsed the explanation provided by the authors on PubPeer. In his words: “Letters are not reports that churn out rare or strange cases, and I believe we can, at times, publish letters that are of interest. It is of no surprise to me that people have posted this letter on social media, as they feel it is different, light-hearted but with a serious message.” (Rosin, 2025)

From a sociological perspective, hoaxes can offer valuable insights. They can bring a productive friction that stimulates critical reflection within scholarly communication. Placing hoaxes at the center of the analysis means we can learn from the responses they provoke to the “appearance-reality puzzle” they embody as well as the countermeasures they elicit and anticipate (Coopmans, 2022). One particularly important line of critical thinking raised by hoaxes concerns the limitations of peer review and editorial practices. Similarly, the thought experiment, Practice of Neurosurgery on Saturn, has raised questions about what constitutes an acceptable style in scholarly publication.

In the context of science sleuths, hoaxes can serve as instructive provocations. By mimicking the structure of fraud, while remaining distinct, they compel sleuths to clarify, refine, and sometimes revise the criteria they use to flag, critique, and protect the scientific record. In this way, hoaxes illuminate the values that sleuths hold, particularly their commitment to ‘clean’ the scientific literature. What is this notion of purity or cleanness in the literature they covet so much? Is there truly no room for mockery, irony, or sarcasm in the academic record, even though many comments on PubPeer are laced with precisely those tones?

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393.

Footnotes:

[1]: I can affirm this based on my fieldwork research on and with science sleuths, through which I’ve developed a keen interest in the underlying ideal of 'purity' that drives many of their interventions.

[2]: Schrödinger's cat is a thought experiment in the field of quantum mechanics. A hypothetical cat in a sealed box is considered both alive and dead due to its fate depending on a random subatomic event.

[3]: Later frameworks employed alternative terms to distance themselves from the implication of intentional wrongdoing associated with the term fraud. For example, the 1992 report Responsible Science introduced a framework centered on the terms misconduct in science, questionable research practices, and other misconduct. (National Academies of Sciences, Engineering, and Medicine, 2017)

Bibliography

Biagioli, M., & Lippman, A. (Eds.). (2020). Introduction: Metrics and the New Ecologies of Academic Misconduct. In GAMING METRICS: Misconduct and Manipulation in Academic Research (pp. 2–24). The MIT Press.

Blatt, M. R. (2015). Vigilante Science. Plant Physiology, 169(2), 907–909. https://doi.org/10.1104/pp.15.01443

Bohannon, J. (2013, October 4). Who’s Afraid of Peer Review? Science, 342(6154), 60–65. https://doi.org/10.1126/science.2013.342.6154.342_60

Buckland, A., Eve, M. P., Steel, G., Gardy, J., & Salo, D. (2013). On the Mark? Responses to a Sting. Journal of Librarianship and Scholarly Communication, 2(1). https://doi.org/10.7710/2162-3309.1116

Coopmans, C. (2022). Learning from Fakes: a Relational Approach. In S. Woolgar, E. Vogel, D. Moats, & C.-F. Helgesson (Eds.), The imposter as social theory: thinking with gatecrashers, cheats and charlatans. Bristol University Press.

Dubois, M., & Guaspare, C. (2019). “Is someone out to get me?” : la biologie moléculaire à l’épreuve du Post-Publication Peer Review. Zilsel, N° 6(2), 164–192. https://doi.org/10.3917/zil.006.0164

Hilgartner, S. (1997). The Sokal Affair in Context. Science, Technology & Human Values, 22(4), 506–522. https://www.jstor.org/stable/689833?seq=1

Horbach, S. P. J. M., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3(1). https://doi.org/10.1186/s41073-018-0051-5

Lagerspetz, M. (2020). “The Grievance Studies Affair” Project: Reconstructing and Assessing the Experimental Design. Science, Technology, & Human Values, 46(2), 402–424. https://doi.org/10.1177/0162243920923087

Lynch, M. (1997). A so-called “fraud”: moral modulations in a literary scandal. History of the Human Sciences, 10(3), 9–21. https://doi.org/10.1177/095269519701000302

National Academies of Sciences, Engineering, and Medicine. (2017). Context and definitions. In Fostering Integrity in Research. National Academies Press. https://doi.org/10.17226/21896

Mostofi, K., & Peyravi, M. (2024). Practice of neurosurgery on saturn. International Journal of Surgery Case Reports, 122, 110139. https://doi.org/10.1016/j.ijscr.2024.110665

National Academy of Science. (1989). On being a scientist (pp. 9068–9069). National Academy Press. https://www.pnas.org/content/86/23/9053.full.pdf

PubPeer. (2013, October 4). Who’s afraid of peer review? https://pubpeer.com/publications/E902004AA49A4636D480087B2A6BE8#0

PubPeer. (2024, September). Practice of neurosurgery on Saturn. https://pubpeer.com/publications/306A99F8F7084E6F58CBBFEB84FA99

Rosin, D. (2025). Regarding the letter to the editor: “Practice of neurosurgery on saturn.” International Journal of Surgery Case Reports, 126, 110665. https://doi.org/10.1016/j.ijscr.2024.110665

Sokal, A. (1996a). A Physicist Experiments With Cultural Studies. Lingua Franca, 62–64. https://physics.nyu.edu/sokal/lingua_franca_v4/lingua_franca_v4.html

Sokal, A. (1996b). Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity. Social Text, 46/47. https://physics.nyu.edu/sokal/transgress_v2/transgress_v2_singlefile.html

Teixeira da Silva, J. A., & Al-Khatib, A. (2016). Questioning the ethics of John Bohannon’s hoaxes and stings in the context of science publishing. KOME, 4(1). https://doi.org/10.17646/kome.2016.16