Talk on Descartes Colloquium: Initiating a Journal for Trial and Error in Science

On the 16th of June, project leader Martijn van der Meer gave a lecture at the Descartes Colloquium for the History and Philosophy of Sciences and Humanities at Utrecht University. The audience contained philosophers, historians and sociologists of science.

Ladies and Gentlemen –

My name is Martijn van der Meer, and on behalf of Max, Stefan, Nayra, Jobke and Davide, I am proud to present you our idea and progress on the Journal for Trial and Error.

I am absolutely honoured that we – as a team of mostly first year HPS students – are invited at this stage to present you hands-on output of our Masters programme. We are both excited and scared. Excited, because this moment is the kick-off of a small promotional tour in Europe at several conferences, ranging from a biomolecular conference in Cambridge, to a posthumanist conference at Lille, and we consider this audience as our (future) peers. This is a safe space, so to say.

On the other hand, we are also a bit scared, because we consider this audience as our (future) peers. As I will show in a minute, we try to contribute to the scientific community in a constructive – not necessarily a constructivist – way. In that respect, we present ourselves as Master's students who aim to apply the academic knowledge we are intrinsically interested in. And as I have learned from many local histories of science, those who apply knowledge do more than often meet skepticism from peers. Nevertheless, we are standing here, presenting an idea which is the product of months of weekly discussions. And we are asking for feedback, criticism and help.

The next two months, Utrecht will be the global center of philosophy and history of science. Or – to put it even more idealistically – Utrecht will in the next couple of weeks, be the center of reflection on science. After all, besides arguing that history and philosophy of science is – as I personally think – one of the most beautiful academic endeavors one can think of, we are young and hopeful and – at this point at least – don’t have jobs as researchers. That is why we are still asking questions such as ‘How can we contribute to society?’, ‘what does this reflection we apparently do precisely mean?’, ‘How are history and philosophy connected?’ and – my favorite – ‘Do we as students of history and philosophy of science, actually like science?’

I think all five of us do. We believe in the value of producing scientific knowledge within the context of institutions such as universities and research centers. History and (some) philosophy of science has taught us that throughout history, scientific knowledge has been constructed in the context of social, political, cultural and institutional influences. Despite that some inspiring intellectuals have – based on the observation that scientific knowledge has been socially constructed – challenged and in some cases radically relativized the universal aspects of all scientific knowledge, we think that there is something unique about science. We optimistically understand science as bringing society forward and that reflection should help scientists to do so.

This is relevant in the context of our project, because we think that on this quasi-philosophical basis, some cliché conclusions in the world of history (and also philosophy) of science, can be relevant – even useful – for science today. Let’s call it a ‘constructive’ rather than ‘constructivist’ position. The project we are about to present, aims to transcend an attitude of throwing intellectual criticism over the fence of hard-working scientists.

In the context of our struggle with the question about how we can contribute to the scientific community we actually like and which fascinates us in a constructive way, the Journal of Trial and Error was born. At the Christmas colloquium last December – we presume you don’t remember: we heard that some of you actually fell asleep during our ‘pitch’ – we presented the idea as a thought-experiment. Well, let’s be clear, there is no finished issue yet. But we have started to believe more and more in our idea, our plans are developed, there is even a call for papers sent into the world: the Journal for Trial and Error is not an idea anymore, it is here to stay.

First, let’s turn to the three clichés in history and philosophy of science that we want to turn into constructive applications.

1. Picturing the development of scientific practice as a culmination of grand discoveries, done by Western-White-Male Scientists is an oversimplification – and most of the time full of wrong interpretations of what ‘actually happened’. Both scientists themselves, as well as philosophers, sociologists and historians of science have increasingly been highlighting the importance of science in the making. A more faithful picture of Science, the one of practices and fine-tuning methodologies, seems to be at odds with the unrealistic public image of big-discovery Science. After all, science is made by humans.
2. There is a gap between what is published and what is researched. We know positive publication bias pressures scientists to conceal methodological mistakes, while desdaining research containing negative findings. This often threatens proper interpretation. In the face of failed research —which we define as outcomes of Science that do not meet the initial aim of the individual researchers— scientists have two options at hand: not publishing or framing the results as productive by adding ad-hoc hypotheses in an inadequate way. We understand this as a consequence of the expectations of big-discovery Science and the publish-or-perish politics of this Science.
3. There is a replication crisis. Since scientists validate their results in terms of replicability, the present-day situation of unreplicable experiments is a serious problem. Debate on this replication crisis has focused on the misuse of statistics by scientists, on methodological carelessness, or theoretical inappropriateness.

Based on those three clichés, we want to address, market, think about and facilitate Trial and Error in Science, understanding it as the elementary process by which scientific knowledge is acquired.

Let’s be more specific with what I precisely mean with ‘trial and error’. On the one hand, trial and error refers to Methodological Errors in a practical sense. We believe that if those errors were more widely known, they would foster improvement in the understanding and application of techniques. Methodological Errors are here understood in a broad sense, those that go beyond the learning of the individual researcher and have an impact at the scale of the scientific community.

A nice example is the case of multiple comparison problem in fMRI scans. As you see can see in this picture, by means of fMRI, brain activity in a salmon is measured ––– the only problem is that this fish is actually dead.

As you might know, in MRI, the brain is divided up into units called ‘voxels’ –– of which there are over 40,000 in a typical scan. Most fMRi analyses treat every voxel independently, and test to see if each voxel is “activated” by a certain stimulus or task. So that’s at least 40.000 separate comparisons going on. As a consequence, there is a high probability that fMRI analysis provides, by chance, some weird results –– that explains this picture. Publishing this kind of research would be very welcome in our journal.

On the other hand, Trial and Error refers to Conceptual Flaws, arising from hypotheses being confronted with conflicting observations. When the initial hypotheses are inappropriate in the face of empirical evidence, scientists improve or reject theoretical frameworks by developing alternative theses aimed at increasing empirical adequacy. Not only hits (positive results), but also misses (negative results) are key to scientific progress.

Within chemistry for example, one important subfield is the optimization of the synthesis of chemical compounds - either by speeding up the process or making it cheaper. Unsuccessful efforts to do so often remains unpublished. As a consequence, many different unconnected chemists try the same optimization process to no avail, due to the fact that there is no available record of previous unsuccessful attempts.

In spreading the three lessons we’ve learned as good history and philosophy of science students, we think that we need to operate in the context of Open Science. But above all, we want to emphasize the necessity of dialogue. As proud children of the Descartes Centre, it may be of no surprise that we identify a highly specialized academic community, aiming to tackle and reflect on social and intellectual challenges in a frequently unproductive way.

Because of the scattered organisation of university departments and faculties, a constructive dialogue between different tribes of cutting-edge Science is missing. In the context of the earlier mentioned problems of a harmful public image of Science –––– the positive publication bias –––– the replication crisis –––– and inaccessible Science, the lack of communication has to be addressed even more urgently. In the face of these multifaceted problems, we need useful solutions for the future of Science.

So. How do we think to solve these problems by squeezing Trial and Error into scientific practice in the context of a scattered institutional context? What do we have to contribute?

We are going to launch a ‘scientific’ journal, so that we can publish Trial and Error in the form of methodological errors which have productive conclusions for the scientific community at large, and conceptual errors in the form of negative results.

That does not mean that we want to publish sloppy science. We think it is necessary that every published science article should be double-blind peer-reviewed by more senior scientists. We believe that in talking about errors, and taking this very seriously, scientists can learn about the do’s and don’ts of their methods and concepts. As well, because negative results are highly informative, this would help alleviate the issue of publication bias, and reframe the replication crisis.

We ask scientists to publish their experiences of trial and error in the form of a short article. This article should include the IMRAD structure (Introduction, Method, Results and Discussion), but also a ‘purpose’-section and a take-home message. An IPMRAD+T-structure, so to say. We want to experiment with this format, hoping that asking scientists about their initial purpose and asking to formulate a take home message, a constructive and intriguing format will come into existence.

However, as icing on the cake: on every published ‘science’-article, a subject specialist, or a philosopher, historian, anthropologist or sociologist of science will be invited to reflect, thereby answering and problematizing the question “what went wrong?”. This combination aims to ensure novelty and quality in our journal.

So, we aim to create an open access journal that contains pairs of science- and reflection articles, in which trial and error in science is published, and on which scientists and – especially – humanities scholars will reflect. That we understand as progress.

Now, I hear you wonder: Is this new? On the one hand, it is not. There do exist journals which aim to publish negative results, and of course, there are many journals in which reflectors of science can publish (for their own community). But we focus especially on young researchers, since only they are the hope for a change in Science, we believe. Therefore, we do take their work and ideas seriously. We aim to publish high quality work of early-career scientists, peer-reviewed and edited by more senior scholars. In concrete terms, that means that we at this point want to focus on articles based on Master's theses. But of course, we hope that this will be a slippery slope, leading to a new way of publishing in the future.

This focus on young scientists that believe change is necessary, has a pragmatic side as well. Of course, we don’t expect that there is any serious senior scientist risking his or her career by publishing work that did not meet his or her initial purpose (‘failure’?) in a journal without any academic reputation. We are not naive. However, we do hope that ambitious Master's students that cannot publish their research because they had to work with some faulty database or theorized an impossible experimental setting, will publish in our journal.

Of course, it is necessary that it looks like that we are serious. That’s why it is necessary that we are institutionalized. What are our traces?

1. We have written a Manifesto for Trial and Error in Science - on which this talk is based
2. We have a website: jtrialerror.com
3. We have a blog about our own activities on our website. After all, we are also part of science in the making and want to be transparent about it
4. We are institutionalized as ‘Open Science Initiative’
5. We have already some money from the Utrecht University Fund, being very supportive and enthusiastic
6. We have promotional material
7. We’ve sent a call for papers to several Master's programmes
8. We are even part of the UtrechtInc Student Entrepreneurship Incubator
9. We are going to give talks. We started in Strasbourg last week, got the stage at this very moment, will give a short talk at Woudschoten, are present at a post-humanist conference in Lille and are invited speakers at the GSA symposium of Molecular Biology at the University of Cambridge
10. There will be a newsletter, by which we can keep those being interested updated. Please sign up at this moment

We have sent a pre-call for papers, calling for abstracts from the 1st of June. Our planning is to start the call for papers on August 1st until the 1st of December. The peer-review and editing cycle will hopefully be finished at around the summer of 2020. We hope to choose about five suited articles. Because three of our project-members hold a Master's and have a network in neuroscience and psychology, our first issue will thematically (and pragmatically) be scoped as about ‘sciences of the mind’.

As I’ve said at the beginning of my talk: we are here to receive critical questions and suggestions. Of course, we hope you all like our idea. However, the worst thing that could happen at this moment, is that you all give us a tap on the back, say ‘good job’, and meet us with great silence.

Thank you.