Founding JOTE: A Conversation with Stefan Gaillard: “I Believe That Positive Publication Bias is Actively Harming Science”

When establishing JOTE, its founders already had different views on the values and direction of the Journal. In this third and last part of the series on JOTE’s founders, we discuss the past and future direction of JOTE with its current chair Stefan Gaillard, who has a distinctively different view on failure and academic publishing than his former colleagues.

He was editor-in-chief until 2022, and took up the chair position from Martijn van der Meer in 2024. As a PhD candidate he studies overpromising in nanobiology at the Radboud University.

How did you get involved with JOTE?

"Together with the other two co-founders as well as Alex Visser, we attended a lecture on Open Access. One of the big focusses of the lecture was how Open Access would enable a general public to get access to the results of science. However, we disagreed with this take, because there is a lot that does not get published, so even if everything that is published is published Open Access, the general public (and us scientists) won’t have access to all the results of science.

"At the time that was a big part of our motivation in founding the journal. Over time I have become more sceptical whether the general public is really interested in having direct access to the results of science, be they positive or negative. I also believe that my viewpoint at the time was altruistic and that scientists should primarily focus on what benefits themselves, which is high-quality knowledge production (and knowledge of what doesn’t work or what is not the case, is definitely part of that).

"In addition, especially for Max and me, there was personal frustration with how scientists dealt with failed research. During my psychology premaster I conducted an experiment on subliminal messaging. My group failed to replicate the findings of our supervisors – the euro coins that we presented subliminally had no effect on the subsequent task that participants had to perform. Rather than being considered an interesting finding in itself, our supervisors almost immediately assumed that we had done something wrong or there was something faulty with the technology. Of course it was reasonable of them to assume so – I think it’s likely that it was either of those - but the fact that only those two options were considered, was what bothered me. There was also never any follow-up with regards to what exactly must have gone wrong with regards to either how we conducted the research or the used technology.

"So taken together, these were two important reasons to found the journal. After the Open Access lecture, the five of us went to a local café and started hammering out some preliminary ideas. After a while and a few more meetings we figured out that Martijn, Max and I would do most of the grunt work in establishing the journal and laying out the vision, and that Alex would stay peripherally attached and help us out where they could."

What was your role in establishing it?
"That’s a surprisingly difficult question. There was so much going on in the beginning, you could almost do historical research on it. A while ago I was going through some old emails and WhatsApp conversations and there really is a lot to reconstruct. I was involved in trying to set up the website and the submission portal with the help of a far more knowledgeable friend of mine. Still, it was only when Thomas Jorna joined our team that this really got up to speed.

"In addition, we had to work out the vision of the journal and our target audience. There were many, many hours long discussions on what exactly failure and trial-and-error mean, if we should use the word failure, what failure and trial-and error look like in the Humanities and much much more. We then also had to put that on paper, for example in our Manifesto.

"At the same time we were promoting the idea of the journal at various conferences which we divided between us. You might think that that would occur after we had fully worked out our vision and audience, but we took a more iterative approach. During these conferences we gathered feedback that we then took with us when working on the vision and strategy. Besides these main tasks we also divided “smaller” tasks between us such as getting an ISSN number and signing up for the archival service CLOCKSS.

Can you elaborate on your personal motivations, ideas and ideals when establishing it?

"Personally I was mostly motivated by my own dismay with how science operates. Science is our best way of producing knowledge and its contribution to human flourishing is without equal. Precisely because science is such a great enterprise I am very motivated to keep improving it and to protect it from harm. I believe that positive publication bias is actively harming science and at the very least holding knowledge production back.

"Although I think traditional for-profit publishers provide a valuable service, I do believe that from their viewpoint, universities and libraries would be better off spending their money on financing Diamond Open Access journals directly rather than paying for traditional or Green/Gold Open Access journals, because it is cheaper and the quality is, in my opinion, the same. However, it is up to us Diamond Open Access journals to convince universities and libraries of that in the marketplace of ideas. Even though there is a lot of talk from universities and libraries about Diamond Open Access being the supposed gold standard, there is currently almost no funding for it. Clearly, universities and libraries are less enthusiastic about Diamond Open Access than they say they are. Being more mindful of how we spend money as academics was thus also a big motivation, but never because of any anti-profit motives.

"During the founding, I was also convinced that we had a responsibility towards the general public to actively change their picture of the scientific enterprise, but I no longer believe this is something that we should be overly concerned about as scientists, let alone that it is our responsibility or even duty."

How did your ideas differ from the other founders?

"My ideas differed fundamentally in almost every way from the other founders. Sometimes I think that really the only thing we had in common was wanting to tackle positive publication bias. Max and Martijn are both very critical of for-profit publishing in general. I personally think that being a non-profit Diamond Open Access journal is the way to go for JOTE: it lowers the barrier for people to publish their failed research, which is precisely what we want. However, I feel that for other journals it makes sense to be for-profit. They provide valuable services that I myself also use in my capacity as a researcher. As Adam Smith wrote, ‘It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest.’ Despite all the complaining by scientists, evidently they also think the services of these publishers are worth it because they keep using them.

"Furthermore, I take a fundamentally different view towards how we approach failure. Unlike Martijn and Max, I do not believe it is our mission to problematize failure. Humans are goal-oriented creatures. Success means that we reach our goals, which contributes to our flourishing. In that sense success will always be superior to failure. However, at the same time we are not omniscient creatures. That means that inevitably we will fail – failure is part of life. To deny or ignore that would be foolish. Rather, we can learn from our failures to prevent making similar mistakes in the future, and ideally, we can also learn from the mistakes of others so we can prevent making our own. To me, that’s what the journal is about. But failure is always instrumental in that sense. It helps us succeed."

Does JOTE differ from your first conception of the project, and if so, how?

"Partially yes. When we first conceived of the project, we also had this idea of providing a platform to whistleblowers. In many cases, early career researchers do not own the data of their research, so they are dependent on their PIs to be able to publish their failed research. Those PIs often do not give permission. At the same time, for example in medicine, it is often mandated that these types of results are published, although this is almost never checked. In those cases I think a whistleblower platform for failed results would be a great idea and it’s something I’d still like to implement one day. In other cases intellectual property rights should be respected even though it is a shame that this means that failed research does not get published. Although I am sure my co-founders would disagree with my view on intellectual property."

"Another way that JOTE differs from my first conception is that we now also publish meta-research. I am very happy that we do, because I personally find it a very fascinating field (although not as new as it pretends to be), and also because I think it fits the scope of JOTE really well. Science itself is a process of trial-and-error, so reflecting on that process fits us very well in that sense."

What was the most productive or most notable failure in the establishment of JOTE (and what did you learn from it?)

"One of the most notable failures was the day that an article got published about JOTE in the Volkskrant, a national Dutch newspaper. A day where we expected a great increase in traffic to the website. However, just that very day an error occurred which led to the website being flagged as unsafe to visit. This led to a lot of lost traffic. I was very annoyed by this, which led me to become a bit terse with the people who pointed out that the website was failing. One visitor rightly called me out on this. Thus, that day was a good lesson for myself to be mindful that errors can occur and are there to be overcome, rather than let my annoyance get the best of me."

What do you hope for the future of JOTE?

"I hope that we continue to inspire other journals and platforms to address positive publication bias and perhaps the file drawer problem more broadly. Daniel Lakens and Eline Ensinck recently published an op-ed in Nature with regards to the broader file drawer problem, where not only negative and null results end up unpublished, but also sometimes positive results – for example because the lead researcher gains a new position at another university and starts with a new project there. I would be interested in helping address that issue. In addition, I still want to provide the aforementioned whistleblower platform.

"Eventually it would of course be great if we could solve positive publication bias and the file drawer issue. One could argue that at that point in time the journal becomes superfluous. Just like we don’t have the Journal for Positive Results, we would no longer have journals solely tailored to negative and null results. Every journal would just publish a mix of both, and without skewing towards positive results. However, I think even then, JOTE would have a raison d’être as (one of) the first multidisciplinary metascience journal(s).

"We have a Trial and Error Award which I would like to see expand in scope in the coming years. We are also starting a collaboration with the Museum of Failure, which I hope to see flourish in the future. To be honest there are too many things to count that I hope for the future of JOTE. I have more project ideas than I have time, money or manpower to conduct."

Do you have other thoughts or comments that you would like to add?

"I would sincerely like to thank the Descartes Centre, and in particular David Baneke, Hieke Huistra and Bert Theunissen for their support during the founding and their continued support over the years."