Beware the lure of simplicity in reforming the Research Excellence Framework

The complexity and cost of the Research Excellence Framework (REF) have long made it a juicy target for self-appointed saviours of the UK’s research funding system.

Last week’s proposal by new-kids-on-the-science-policy-block UK Day One to replace the REF with a “low-bureaucracy system” is the latest in a long line of arguments – often coming from the right of British politics – to scrap or radically simplify the exercise. The UK Day One authors position theirs as a pragmatic contribution from the centre-left, designed to reduce bureaucracy in response to the “22 billion fiscal black hole” that the Chancellor has identified in Whitehall budgets.

And to their credit, UK Day One do attempt to advance a workable alternative model, which would see quality-related funding, currently distributed to universities on the basis of the REF, allocated instead in line with the volume of external research income won through grants. They also seek to address one obvious drawback – that this would disadvantage the social sciences, arts and humanities, which have access to fewer sources of grant income – by proposing a 25 per cent uplift in the budgets of the Economic and Social Research Council (ESRC) and Arts and Humanities Research Council (AHRC).

As a package, these proposals have a superficial appeal – and some in the sector whom I respect, such as former DSIT adviser Ben Johnson, have welcomed them. I am less persuaded for three reasons – their maths doesn’t add up, they misrepresent the international picture, and they fail to grasp the progressive potential of responsible research assessment.

Dodgy maths

UK Day One misrepresents the costs of the REF and exaggerate the savings that would flow from its abolition. Their headline claim – that this would “save universities £430m” – is the research policy equivalent of “350m a week for the NHS” plastered on a Brexit bus. It’s eye-catching, simplistic and essentially false.

And it is perhaps unsurprising to see one of those responsible for that bus, who has been consistently uninterested in the actual working of systems he seeks to destroy, warmly endorsing these proposals. UK Day One cites the Technopolis review of the costs of REF 2021 – which states clearly that the costs of REF 2021 amounted to between three per cent and four per cent of the QR funding distributed as a result – and then inflate this to five per cent, with no explanation.

More importantly, the authors ignore what the Technopolis study goes on to say:

While this is a significant cost…it continues to look reasonable when compared with the cost to the community of…a peer review system for project-based funding [which is] 2-3 times the cost of the REF.

Some estimates place the cost of grant funding even higher – at up to 14 per cent of the funds allocated.

Any proposal to shift the basis of QR funding to one that aligns with volumes of grant income needs to take seriously the higher costs and inefficiencies of the grant funding system itself – not least from the 75-80 per cent of proposals to most UKRI grant funding calls that are routinely unsuccessful. We also need to consider the behavioural consequences of such a shift, which would almost certainly see a dramatic spike in volumes of grant applications across the system (as this becomes the only route to both grant and QR funding) – thus pushing success rates down and costs up further.

The myth of internal cost reduction

UK Day One argues that its reforms would also lead to a reduction in internal university costs being allocated to “impact case officers, research assistants and external consultants” for the REF. As anyone who has worked in university management would recognise, this is naïve. Many internal functions that now operate under the umbrella of REF preparation would still be required if the REF didn’t exist, as these are the means through which universities manage their research portfolio and performance. If the dual support model of funding was replaced with one linked purely to grant income, then universities would redeploy staff or hire new ones to support grant applications – as there would be far more at stake in their success (a point echoed by colleagues working in research management, such as Adam Forristal Golberg).

UK Day One also says nothing about the additional costs that would be generated by such a radical overhaul part-way through a REF cycle, when universities and researchers are already preparing in line with the model that was announced in the summer of 2023. This is non-trivial – if you talk to university managers, they usually highlight changes to the rules as the biggest source of additional burden, as these changes need to be internalised, modelled and explained to researchers before being rolled out.

Having cherry-picked data to support their case, it is ironic to see UK Day One position its proposals as part of moves towards greater use of metascience. As a longstanding advocate of metascience myself, and co-chair of UKRI’s recent £5m funding call in this area, UK Day One’s rather slapdash approach to the complexity, costs and consequences of reforming the funding system does no-one any favours – least of all those of us working to strengthen the uptake of metascience. And chucking a few extra million at DSIT’s Metascience Unit (of which I am a big fan) won’t remotely compensate for taking a sledgehammer to dual support, and the fragile ecosystem of research and impact across UK universities which it helps to sustain.

The international picture

UK Day One further justifies its proposals on the basis that New Zealand and Australia have cancelled their research assessment exercises. In neither case is this an accurate account of what is happening. In 2022, Australia paused its “Excellence in Research in Australia (ERA)” exercise and is now redesigning it, with a new framework set to be announced in 2025. Meanwhile, a series of revisions to the Australian Research Council Act have been passed by the Australian parliament which formally enshrine in law the ARC’s responsibility to “evaluate the excellence, impact and depth of Australian research”.

So assessment has been made more, rather than less, central to the overall management of Australia’s research system. Similarly, in New Zealand, a comprehensive review of the entire higher education and research system is underway, led by Sir Peter Gluckman. Its REF equivalent, the Performance Based Research Fund (PBRF), is one of many elements of the system being examined.

But having hosted a workshop recently for the NZ review team, I can report that there is a clear expectation that some kind of assessment framework will be maintained, albeit in a simplified form. And anyone who has read Sir Peter Gluckman’s International Advisory Group report on the future of the REF would be unsurprised to learn that he remains supportive of the potential for research assessment to be used as a positive tool of change and reform.

Elsewhere, it is wrong to pretend that the tide has turned against national assessment frameworks. Several countries are in the process of introducing, expanding or recalibrating their research assessment systems. As just one example, in July 2024, the Royal Society hosted a workshop for senior policymakers from Ukraine, where the government is looking to introduce an assessment framework based in large part on the REF. From the UK’s side, science minister Patrick Vallance spoke about the contribution of the REF over four decades to the effective management of our research system – and from Ukraine, Deputy Minister Denys Kurbatov, outlined how they saw assessment as a positive tool in strengthening their research system – even in the midst of war, with all the enormous pressures that this brings.

Progressive assessment

This links to a final, important point. UK Day One pretends that the sole purpose of the REF is the allocation of QR funding, when in reality the exercise has evolved to perform several other purposes that are equally important. The inclusion of impact within REF since 2014 has given impact-related activities a welcome status and visibility in the university system, and created an economic incentive for supporting and delivering impacts of various kinds.

Moves towards open access have been accelerated by the REF. And the proposals now on the table for an expanded focus on “people, culture and environment” are in direct response to serious concerns that the health, diversity and sustainability of research is being undermined by problems in the culture of UK research.

These changes to the REF reflect a broader shift now underway in assessment worldwide – from summative, audit-driven approaches, towards formative approaches, that incentivise and reward change in the orientation and operation of research systems. One can characterise this in simpler terms as a move from sticks to carrots in the way we use assessment to shape, strengthen and improve research. The UK Day One proposals have little to offer to this agenda. Axing dual support in favour of a system that incentivises and rewards one basic mode of activity – winning grant income – would exacerbate hyper-competitiveness and career precarity in UK research. And it would do nothing to encourage societal impacts, or broader institutional efforts to improve and enhance research cultures.

Progressive potential

For a Labour government, the progressive potential of assessment should be a serious focus for debate, alongside any focus on burden reduction. Of course, we can all agree that the REF should be as cost-effective as possible – as long as it can also deliver on its agreed purposes and goals. I welcome attempts – including these by UK Day One – to open up a debate about how we approach this task. But let’s not pretend it’s simple – or that cost savings can be generated at a stroke.

We are now midway through testing and applying a model for the next REF that was announced in 2023, following two years of extensive consultation. As Patrick Vallance has said, a process is underway, and we need to see this through. The time to look afresh at the system is after the current cycle has completed.

We have an obvious moment of opportunity coming up in 2026 – forty years on from the original introduction of assessment in the UK – for DSIT, UKRI and others to initiate an evidence-informed debate about where assessment goes after 2029, here in the UK and around the world, drawing on leading examples of assessment design and practice, and from initiatives like CoARA (the Coalition for Advancing Research Assessment). Let’s seize that opportunity rather than make any hasty moves to implement ideas which may seem attractive on “day one” but not withstand the test of time.