Peer review is broken, and pedagogical research has a fix

An email pings into my inbox: peer reviewer comments on your submission #1234. I take a breath and click.

Three reviewers have left feedback on my beloved paper. The first reviewer is gentle, constructive, and points out areas where the work could be tightened up. One reviewer simply provides a list of typos and points out where the grammar is not technically correct. The third reviewer is vicious. I stop reading.

Later that afternoon, I sit in the annual student assessment board for my department. Over a painstaking two hours, we discuss, interrogate, and wrestle with how we, as educators, can improve our feedback practices when we mark student work. We examine the distribution of students marks closely, looking out for outliers, errors, or evidence of an ill-pitched assessment. We reflect upon how we can make our written feedback more useful. We suggest thoughtful and innovative ways to make our practice more consistent and clearer.

It then strikes me how these conversations happen in parallel – peer review sits in one corner of academia, and educational assessment and feedback sits in another. What would happen, I wonder, if we started approaching peer review as a pedagogical problem?

Peer review as pedagogy

Peer review is a high stakes context. We know that we need proper, expert scrutiny of the methodological, theoretical, and analytical claims of research to ensure the quality, credibility, and advancement of what we do and how we do it. However, we also know that there are problems with the current peer review system. As my experience attests to, issues including reviewer biases and conflicts, lack of transparency in editorial decision-making, inconsistencies in the length and depth of reviewer feedback all plague our experiences. Peer reviewers can be sharp, hostile, and unconstructive. They can focus on the wrong things, be unhelpful in their vagueness, or miss the point entirely. These problems threaten the foundations of research.

The good news is that we do not have to reinvent the wheel. For decades, people in educational research, or the scholarship of teaching and learning (SoTL), have been grappling both theoretically and empirically with the issue of giving and receiving feedback. Educational research has considered best practices in feedback presentation and content, learner and marker feedback literacies, management of socioemotional responses to feedback, and transparency of feedback expectations. The educational feedback literature is vast and innovative.

However – curiously – efforts to improve the integrity of peer review don’t typically frame this as a pedagogical problem, that can borrow insights from the educational literature. This is, I think, a woefully missed opportunity. There are at least four clear initiatives from the educational scholarship that could be a useful starting point in tightening up the rigour of peer review.

What is feedback for?

We would rarely mark student work without a clear assessment rubric and standardised assessment criteria. In other words, as educators we wouldn’t sit down to assess students work without at least first considering what we have asked them to do. What are the goalposts? What are the outcomes? What are we giving feedback for?

Rubrics and assessment criteria provide transparent guidelines on what is expected of learners, in an effort to demystify the hidden curriculum of assessment and reduce subjectivity in assessment practice. In contrast, peer reviewers are typically provided with scant information about what to assess manuscripts for, which can lead to inconsistencies between journal aims and scope, reviewer comments, and author expectations.

Imagine if we had structured journal-specific rubrics, based on specific, predefined criteria that aligned tightly with the journal’s mission and requirements. Imagine if these rubrics guided decision-making and clarified the function of feedback, rather than letting reviewers go rogue with their own understanding of what the feedback is for.

Transparent rubrics and criteria could also bolster the feedback literacy of reviewers and authors. Feedback literacy is an established educational concept, which refers to a student’s capacity to appreciate, make sense of, and act upon their written feedback. Imagine if we approached peer review as an opportunity to develop feedback literacy, and we borrowed from this literature.

Do we all agree?

Educational research clearly highlights the importance of moderation and calibration for educators to ensure consistent assessment practices. We would never allow grades to be returned to students without some kind of external scrutiny first.

Consensus calibration refers to the practice of multiple evaluators working together to ensure consistency in their feedback and to agree upon a shared understanding of relevant standards. There is a clear and robust steer from educational theory that this is a useful exercise to minimise bias and ensure consistency in feedback. This practice is not typically used in peer review.

Calibration exercises, where reviewers assess the same manuscript and have opportunity to openly discuss their evaluations, might be a valuable and evidence-based addition to the peer review process. This could be achieved in practice by more open peer review processes, where reviewers can see the comments of others and calibrate accordingly, or through a tighter steer from editors when recruiting new reviewers.

That is not to say, of course, that reviewers should all agree on the quality of a manuscript. But any effort to consolidate, triangulate, and calibrate feedback can only be useful to authors as they attempt to make sense of it.

Is this feedback timely?

Best practice in educational contexts also supports the adoption of opportunities to provide formative feedback. Formative feedback is feedback that helps learners improve as they are learning, as opposed to summative feedback whereby the merit of a final piece of work is evaluated. In educational contexts, this might look like anything from feedback on drafts through to informal check-in conversations with markers.

Applying the formative/summative distinction to peer review may be useful in helping authors improve their work in dialogue with reviewers and editors, rather than purely summative, which would merely judge whether the manuscript is fit for publication. In practice, adoption of this can be achieved through the formative feedback offered by registered reports, whereby authors receive peer review and editorial direction before data is collected or accessed, at a time where they can actually make use ot it.

Formative feedback through the adoption of registered reports can provide opportunity for specific and timely suggestions for improving the methodology or research design. By fostering a more developmental and formative approach to peer review, the process can become a tool for advancing knowledge, rather than simply a gatekeeping mechanism.

Is this feedback useful?

Finally, the educational concept of feedforward, which focuses on providing guidance for future actions rather than only critiquing past performance, needs to be applied to peer review too. By applying feedforward principles, reviewers can shift their feedback to be more forward-looking, offering tangible, discrete, and actionable suggestions that help the author improve their work in subsequent revisions.

In peer review, approaching comments with a feedforward framing may transform feedback into a constructive dialogue that motivates people to make their work better by taking actionable steps, rather than a hostile exchange built upon unclear standards and (often) mismatched expectations.

So the answers to improving some parts of the peer review process are there. We can, if we’re clever, really improve the fairness, consistency, and developmental value of reviewer comments. Structured assessment criteria, calibration, formative feedback mechanisms, and feedforward approaches are just a few strategies that can enhance the integrity of peer review. The answers are intuitive – but they are not yet standard practice in peer review because we typically don’t approach peer review as pedagogy.

There are some problems that this won’t fix. Peer review relies on the unpaid labour of time-poor academics in an increasingly precarious academia, which adds challenge to efforts to improve the integrity of the process.

However, there are steps we can take – we need to now think about how these can be achieved in practice. By clarifying the peer review practice, tightening up the rigour of feedback quality, and applying educational interventions to improve the process, this takes an important step in fixing peer review for the future of research.