JOTE in Conversation: Michael Burke. "Why are publishing companies getting paid for our peer review?"

After talking with Daniel Lakens about metascience, we now present the third instalment of our series ‘JOTE in conversation with researchers.’ In this series, we ask researchers how they experience 'failure' and 'success' within their research practices - whereby 'practices' can range from the research proposal and funding to experimental constraints and publication. How does failure arise in these different phases? And how do researchers in different fields and career stages deal with failure?

In this conversation, JOTE co-founder Stefan Gaillard talks about failure, peer review and interdisciplinarity with Michael Burke, professor of rhetoric at Utrecht University.

Has the academic view on failure changed throughout your career?

Absolutely. It was different when I started, and I am 55 now. Although I don’t think we’ve really talked that much about failure at the time, but it appears to be better hidden these days. Before, if a research project wouldn't work out and produce nothing to show, you'd still try to share that knowledge and even try to submit it to a journal. But these days you just don't bother and say “let's just forget it and move on”. Failure is more hidden because the system doesn't allow for it. Everybody has to be a winner – which is in itself preposterous.

Why does everybody have to be a winner? Is this tied to funding, you believe?

I think so. Departments I previously worked in were kind of well-balanced football teams. There were people who could score, good midfield players, defenders, keepers – everyone did their bit. Similarly, my departments had good teachers, good administrators, good researchers and good supervisors. But these days, everybody has to be a striker: everybody has to score goals. Imagine a football team with eleven players solely focused on scoring goals. It is nonsensical, but apparently this is supposed to work for academia.

Is there a particularly interesting instance of failure from your own career that you’d like to share?

I remember we did a large study on rhetorical strategies in the discourse of students in 2013. We took an old framework from Cicero, and subjected around 600 Dutch students to both qualitative and quantitative analysis. It took ages to squeeze something out, to get some result out of this huge project, and although it was very interesting, we ended up with nothing. So after a year we just abandoned it and never spoke about it again.

I think that had we told people about what we had done and why we had done it, the study might have been valued better. We spent a lot of time designing the study, had several outside experts contributing to the development of the design, and collected a lot of data – all of which ended up not being used. But we actually learned a lot. Methodologically but also about ourselves and how to handle disappointment. That we didn't do something with it is quite regrettable. Although the system asks: “Where is your publication? Why can't we find it in the best journal in your field?”, but the things we learned would not have been valued in this system.

I think this is part of a broader pattern in which failure has lost its importance. Years ago, I feel, researchers still saw falsification as a productive result of research. You try to disprove something, and in failing you actually succeeded by increasing the robustness of the theory. Nowadays there is no room for any of that. You have to produce something that is unique and fantastic all the time, and if you don't you might lose your job. But in reality, many scientific innovations and discoveries are preceded by a period with lots of failures and few publications.

Do you have any views on how that shift came about in the management?

I am not sure, shifts like these happen really quickly. When I was younger and still working towards becoming an associate professor, full professors were appointed based on their research output, but they were also expected to supervise PhD students and give keynote speeches at international conferences in their field. When I became a full professor in 2011, I was ready to accept this, but the reality didn't look like that at all! I had to bring in large grants for the university, try to get my name in the newspaper – or better, my face on television. So, before I knew it, the job of full professor changed to being some kind of academic celebrity and a money generator for the university.

This new role of the professor as money-generator can be related to changes in the academic publication system. It was probably Elsevier who was leading it. They are reputed to have bought this publishing model, or scheme, to fleece universities. I read a well-researched article in The Guardian about it. This model now generates 600 million a year for shareholders using public money. If I try to explain the model to anybody outside academia, they usually don't get it. And I'm not surprised, because I don't get it either!

So let me try to explain it: I'm employed by the university, the university gives me time to do research and write up my research, but it doesn't belong to me nor the university. Because I have to publish it in a journal, I don't get paid for it; I have to pay to publish it. And then the publishers refuse the university to access my research, so no colleagues or students can read it, and if the university wants it back, they have to pay a huge amount of money that goes up tremendously every year. The money then ends up being given to their shareholders. It is almost as if the government is giving millions of euros directly to private individuals. You might as well just cut out the academics and give it straight to the shareholders!

In doing so, it commercializes peer review. Peer review is the key to quality research and entails that your peers have critiqued your work and have demanded alterations and approved it. This happens for free, and that is fine; it is part of my job as an academic. But while both writers and reviewers are not getting paid, scientific publishing companies are currently getting paid for it, and that's very strange. What if we just publish our articles on the internet or some university's server, have the names of the people who review it next to it, and make it accessible to everybody around the world for free? Then the university won't have to give millions of euros to private individuals every year, and could invest more in teaching and research. Why are we not doing this currently? Surely this form of open science is a better way forward.

So you are suggesting that very substantial peer review contributions should actually count as co-authorship?

Yeah, why not? You are working together towards a final paper – although I suppose that no paper is really final, certainly not in the humanities – so why not recognize these contributions? If people are pointing out errors, failures in research designs and help find solutions, then I would consider them part of the team. If they are prepared to put their names next to your paper, it means they believe in it. I like that system.

Do you think that failure plays a different role in scholarship that is more focused on qualitative research?

I think that the concept of failure is different in qualitative methods because the outcomes in a qualitative study are less precise. Of course, you can go in different directions with your data in quantitative studies, but the expectations to come up with a precise result are higher. Quantitative research – and this especially happens a lot in the social sciences – needs to find some kind of effect in order to publish. When you find nothing in SPSS and you've tried all the drop-down menus, you've turned the whole research upside down, you are even inclined to change your hypotheses to fit something, to squeeze something out that you can actually publish.

A lot of academics don't like doing that, but when there is a system that says: “you will have to have an effect, and if not you will be a nobody, perhaps even a nobody without a job”, then you will do that, I think. So, I feel that not getting a result in quantitative research is punished more readily than in qualitative work.

I recently talked with some humanities scholars, and they told me that in the humanities, failure happens not during the execution of the study itself but in the application process. Proposals for humanities research are more often turned down than those for more social scientific work, they say.

Oh yes, sure! A lot of proposals are judged based on how they fit societal goals, like the United Nations' Sustainable Development Goals. If you are a humanities scholar who just works in the humanities and doesn't look over the walls of their own discipline, it might be hard to find a SDG that appeals to you. Maybe you can find one that appeals to you, but then you might get rejected anyhow because all humanities scholars are proposing studies aimed at the same goal.

I think that research should therefore be moving – and it is already doing so – towards an interdisciplinary and multidisciplinary approach. This system where you have humanities scholars applying for a humanities project in the humanities might be coming to its end. Since many societal problems have an interdisciplinary character, you also need people with specializations from across different disciplines to come together and find a solution.

Humanities scholars are absolutely essential within these teams. Without them there is no ethics, no effective communication and no borders or questions. There are many absurd stories about countries bombing other countries thinking they can solve local political problems, without anybody speaking the languages, and while totally disregarding the local culture and history.

Last question. How do you try to encourage more openness about failure in students or PhD candidates you supervise?

That's a good question. I do tell PhD students about the failure of other PhDs and my own failures as well. Writing a PhD is the most difficult thing you'll ever do in your life, so I use anecdotes and I tell them that it is important to fail. It's critical that you fail. The idea that everything you write will be an instantaneous success is preposterous.

If someone is working on a project at a technical university, they'll go through perhaps fifty prototypes before they actually get to the finished project, and every one of those prototypes can be seen as a step of failure – but these steps are crucial. You can't just go from model one to model fifty; you need those forty eight prototypes in between. Although this is clear in technical fields, for many other disciplines like the humanities it is expected to go from step one to step fifty in one go. And perhaps even skip step one as well. Instead of this, I would like to encourage a culture in which failure is shared and embraced. After all, failure is a stage in the road to success, a necessary stage that has to be valued as such.