The REF is dead, long live the REF

For many, today marks the end of the REF period. The results are out! Press releases abound. Post-match analysis is everywhere. For research managers, however, the REF never ends.

When many of our academic colleagues went on much-deserved annual leave after the May 2021 submissions, we were just starting an eight-month REF audit period dealing with hundreds of queries as the panels probed and prodded our submissions to check if they held true. And when that was finally over, the long process of preparing the institution for the results began.

Even now the results are out, further nail-biting begins as we try to predict what this might actually mean for our finances over the next seven years. Because let’s not forget, nobody knows this yet. It’s all carefully calibrated post-hoc by the four funding bodies depending on the results and what’s in the pot.

Thank you, next

But the bigger concern for many of us is not the last REF but the next one. Because given the last exercise examined research between 2014-2020, we are now already 18 months into the next assessment period. And without a single clue as to what that might look like.

The REF Future Research Assessment Programme (FRAP) charged with designing the next assessment has started to think about it of course. However, early consensus is that there is no consensus. On the principle that if you put five academics in a room, you’ll get ten different points of view, this is hardly surprising. And of course we have a significant range of universities in the UK – some fairly nascent, some centuries old – all of which will largely favour an assessment that will largely favour them.

The problem with leaving it this late to make these decisions is that in the absence of any other intelligence, most institutions will have started gearing their thoughts towards a process six years from now which is broadly like the one we’ve just delivered.

I’m certain that 157 REF “Lessons Learned” processes have already been run by the 157 institutions submitting to REF2021. No doubt the focus of many of these reviews considered how they might run the same process differently if they did it all again, not bearing to think about the fact that when they do it all again, the process itself could be significantly different.

Impact staff are already busy preparing the ground for the next set of case studies. Ongoing output reviews are being put in place to save the headache of a last-minute peer-review fest. Decisions are being made now in the light of how they might read in their next REF Environment Statement.

Is change needed?

I’d love to be proved wrong but I fear that because of the delays, it’s unlikely that higher education institutions (HEIs) will be in the market for serious changes to REF. And without a mandate from the UK HEI community, the funding bodies are unlikely to be able to sell anything more radical to government.

As Research England’s Catriona Firth says, the most common feedback she has about the REF is, “it’s terrible, don’t change it”.

But we desperately need a revolution and not an evolution when it comes to university assessment. I don’t have space to rehearse all the argumentsbut here’s a few:

1. The REF is ridiculously burdensome. And no, that isn’t because universities make a meal of it. Read the Guidance. All three sets. And the Letters to HEIs. And the FAQs (180 of them). And the Submission System help. And the individual query responses…
2. REF largely embeds the status quo: the bigger you are, the better you do. The REF 2014 Power Rating can be replicated to a correlation of 0.996 on the basis of FTE submitted.
3. REF is summative rather than formative. Despite being a qualitative peer-review process, each institution ends up with a spreadsheet full of numbers and only a few lines of confidential written feedback. No guidance is given as to how HEIs can improve.

One of the key problems with the REF for me is that it makes a competition out of something that is inherently collaborative. Whilst we are all individually celebrating our results today, the truth is that 73 per cent of outputs (as indexed by SciVal) produced by UK institutions over the REF period (2014-2020) were co-authored by other UK (16.2 per cent) or international (56.5 per cent) institutions.

With the exception of single-author disciplines, research is performed collectively but rewarded as if it were not. The Wellcome Trust taught us that 78 per cent of researchers think high levels of competition have created unkind and aggressive conditions in the sector and we’re all desperate to improve our research culture – including the REF. I’m just not sure how that can happen whilst the REF outcomes are so, well, rankable.

I don’t want to be the spectre at the feast. Of course we have things to celebrate today. UK research is world-changing. And the unhypothecated Quality-Related (QR) funding that results from REF has significant benefits to institutions.

But please can we keep an open mind about how institutions come to merit that QR funding and the impact of university assessments on our wider research culture? Frankly, as a research manager, I’d rather spend my time supporting colleagues to do better research than to do better at REF.