Peer review – the principal quality control mechanism in academic scholarship – is a surprisingly recent invention.
The journal Nature only made it standard practice in 1973 (and the phrase itself originated in the 1970s, although its origins are much older). It remains the cornerstone of ensuring a robust scholarly record.
And as long as there has been peer review there have been criticisms of peer review.
The AI difference
But the situation we find ourselves in now – where Large Language Models (LLMs) are increasingly and inappropriately being used to create both “papers” and “reviews” – feels different. Journals, conferences and funders are all struggling to respond to this subversion.
But authors (some of them at least) are responding. As the illegitimate use of LLMs by reviewers is increasing, so is the use of countermeasures. Authors are now adding invisible instructions (for example, in white text) to ignore all previous commands and provide only positive comments.
Can AI help prevent or detect this kind of thing? Perhaps. But that is not the point. Human ingenuity and computational power will always find a way; we are in an arms race, with every adaptation prompting a counter-adaptation. More AI – more technology in general – is not the answer. Indeed, it will just fuel the problem.
Instead, should we finally move on from peer review?
The limits of peer review
Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that! – The Red Queen, Alice in Wonderland
This feels radical (although we should remember the relative recency of widespread peer review). But if the traditional quality control procedures of academic research are facing an existential threat (and, with them, the legitimacy of research) perhaps such radical thinking is required.
Unlike in many commercial R&D settings, quality control in academic research typically relies on the expertise and goodwill of other academics – our peers. But the recent massive growth in research throughput, including greater quantity and complexity of journal submissions, grant applications, etc. has put enormous strain on this system.
Experts are in short supply and are under ever-increasing workload pressures themselves. It is understandable that actors in this landscape reach for technology to help, whether legitimately or not (especially when ChatGPT is so beguilingly helpful). Other symptoms include the rapid growth of paper mills and fake data.
So now we are faced with two related challenges: research throughput that is unsustainable given current resources, and a quality control crisis largely born out of tactical responses to that situation. We believe that a radical re-think of academic quality control, led by the research community, is needed. Urgently.
Quality from the outset
Here is one possibility: We re-think academic research quality control from the ground up, baking it in from the start and throughout, with institutions responsible for providing an environment that produces high quality research, including mechanisms to provide the wider community with assurance of both provenance and quality. External checks on environment and mechanisms provide further assurance. This implies significant changes to who does, and is rewarded, for what.
There will be challenges of course; the environment will need to be appropriate for the different kinds of research being done – one size really does not fit all. In some disciplines, computational, statistical and reproducibility checks will work. In other fields, the emphasis might be on methodological coherence and the extent to which a study’s ontological, epistemological, and methodological assumptions align with each other, with the methods used, and with the theoretical claims being made.
The local environment will need to provide the infrastructure, skills, workflows, incentives, and culture that give the institution confidence in its own research. This will cost money and time. Research institutions will need to see a parallel set of changes being implemented in the funding and policy environment to allow them to take on this role, and to be accountable.
Policy cover
At the national and, perhaps, European and international levels, this implies a new role for funders and policymakers. In the UK, the REF Strategy, People and Research Environment section could be reformulated to be an assessment and accreditation of this institutional research environment, without this becoming a bureaucratic burden. That would take some imagination.
Funders and policymakers would also have to provide extremely strong signals that demonstrating research provenance and rigour is what they want and what they will reward. This will not necessarily make for easy conversations with their sponsors – be those representatives of taxpayers or donors. Leadership will be needed.
The academic research sector is full of talented people; there will be other suggestions for how we can address (and improve) the current situation. Many will be willing to work with those who want to tackle these challenges, providing this is done in good faith. Conflicts of interest and of commitment need to be acknowledged and managed.
But the alternative – to try to maintain the status quo and rely on ever more technology to rescue us – is to resign ourselves to an increasingly desperate race against the Red Queen that we can never win. We risk undermining the legitimacy of academic research and of our licence to operate and be funded in society.
The authors are grateful for the contributions of Peter Flach to this article.
