The problem with REF is that it isn’t just one thing. It is at least three things hiding within a single research excellence exercise.
The official objectives of REF are to:
- provide accountability for public investment in research and produce evidence of the benefits of this investment
- provide benchmarking information and establish reputational yardsticks, for use in the higher education sector and for public information
- inform the selective allocation of funding for research
It is meant to peer assess the quality of research, allocate funding on this basis, and then provide public information on where research funding is being spent. The challenge with REF, and at the heart of the debate about its future, is about the primacy and importance of each function.
Sanjush Dalmia and Richard Carr have made the argument for UK Day One that scrapping the REF would save £450m that could be more usefully spent elsewhere. They believe that the REF is expensive and ineffective at allocating research funding. Their argument is that REF does not drive up research quality and could instead be replaced by a combination of a system where funding is allocated in proportion to external research income won, where there is a reallocation of funding specifically to arts, and where a floor of funding is introduced in proportion to the number of FTE research staff. They further argue that investment in the mechanics of research, improving peer-review, reducing bureaucracy, and looking more deeply at meta-science could improve the system overall.
A welcome debate
While the impact of some of their suggestions are debatable, there is value in a wider debate on why REF not just which REF. If you believe that the REF is ultimately about the allocation of public money to improve UK research then it should be assessed on this basis not just whether it effectively assesses the research that is currently produced.
Dalmia and Carr are reasonable to suggest that the REF isn’t as effective as it could be in supporting blue-sky research, that it misaligns with wider government policy in places (in old language it does nothing for levelling up), and that REF is a bureaucratic exercise. It is also reasonable to suggest that the failure to achieve these goals isn’t necessarily a problem situated within REF but a problem of the wider research system.
For example the Higher Education Innovation Fund (HEIF) is calculated using knowledge exchange indicators related to business and commercialisation activity. This funding in turn can only be used for business and commercialisation activity like developing IP, business engagement, and local economic development. Last year there was £260m made available for this kind of funding while economic analysis suggests it returns £8 of value for every £1 invested. If the aim is to reward or encourage research commercialisation then one lever would be to rebalance funding to HEIF as a counterweight to REF.
Incentives
The authors recognise that:
While universities may respond differently to the incentives under a new system, and any reforms to the system will create winners and losers, the available evidence suggests that a system focused on external research income will result in a QR funding allocation similar to existing plans for REF 2029, but without the associated bureaucracy.
Universities, naturally, would then gravitate to systems which encourage external research income. The authors further argue that reducing the bureaucracy within the system would allow for more time for blue-sky research. This argument feels less convincing. The kinds of activities which are likely to attract private external funding are not necessarily the blue-sky, experimental, and as yet unfounded. A major benefit of public sector funding is that it doesn’t have the same kind of profit motive of the private sector.
The second is that universities would simply invent a new range of bureaucratic activities in order to meet new requirements. It is a different argument if this version of bureaucracy would be more socially useful. Again, it’s reasonable to speculate that attracting funding through business and philanthropy is a more economically or socially useful activity but it would change the kinds of research activity providers do and the things they do to attract research funding.
The charge that the REF doesn’t drive up research quality, or indeed makes it work, is more debatable but again the REF measures the research that is put in front of it. If the argument is that the REF’s incentives make research actively worse then the argument could equally be the REF has the wrong incentives not that scrapping would improve research quality. For all sorts of reasons measuring research quality only by the quality of its outputs is a bad idea for the environments, cultures, and sustainability of research, but that would more readily align research and current quality outputs.
Good and bad bureaucracies
Is REF too bureaucratic? Possibly, but this is as much a political decision as policy decision. For example, correlating QR with FTE of researchers would result in similar funding allocations as the current system and save a lot of money. It would also incentivise universities to recruit more FTE researchers, again which might be a good thing, but it would also mean research being evaluated without someone looking at it, which would be a massive cultural change in higher education. It also assumes the link between FTE researcher numbers and quality of research output would remain consistent even without the scrutiny of regular research assessment which, while possible, seems quite optimistic.
The challenge that UK Day One puts forward is whether the purported purpose of REF is being met by its current structure. If the purpose is to benchmark research across the entire country within a multitude of disciples then it does that and it is seen as a robust way of doing so. If the purpose is to align external funding with internal resources, crowd in business funding, level up the country, encourage blue-sky thinking, or promote technological diffusion then no, it does not do these things in and of itself.
The solution might not be in scrapping the REF but looking at the wider ecosystem of research funding. Making the case for more investment in the variety of the research environment more likely looks like growing HEIF, ensuring the success of ARIA, reprising Regional Innovation Fund, co-funding of large grant funding between government and business like the metascience programme, while reducing bureaucracy.
Dalmia and Carr’s proposals on reducing bureaucracy in learning from both Adam Tickell’s independent review of research bureaucracy and UKRI’s review of peer review would involve more randomisation, less form filling, and streamlining of services. There are elements of this work that should be adopted in any system. However, the question is not necessarily whether to have REF or not but what are the objectives for the UK’s research system and what levers does the government and its funders have to meet those objectives. In defence of REF, or a version of REF (and potentially a less bureaucratic REF), having part of the research mix as the peer review of excellence seems, on balance, to be an important means of accountability for public spending and the benchmarking of performance.
Good article. I would prefer a REF that was slightly stripped back and focused on the basics of assessing research quality as a mechanism to distribute QR funding. It has become too big and complex partly because it is rightly recognised as a way to create incentives to encourage particular policy goals, e.g. Open Access, or more inclusive research cultures. If we want to encourage OA we would do better to have a separate funding stream and approach to encourage rather than adding that onto REF. It’s being made to do too many different things.