Who is in and who is out of REF

In REF 2029 a provider’s return will be based on an average number of staff with significant responsibility for research rather than a headcount of staff.

The current proposal is that institutions will be required to “submit 2.5 outputs per FTE of volume-contributing staff in each disciplinary area where they have activity.” In REF 2021 institutions had to submit 2.5 outputs per 1 full-time equivalent staff member. However, each individual had to submit at least one and no more than five outputs.

In contrast to REF 2021, the risk is that REF 2029 encourages institutions to return fewer staff with significant research responsibility who they believe will perform well rather than a broader number of staff with more representative research output. Although the funding councils say institutions will have to justify their sample, the mechanism, penalties and incentives around this, are not yet clear.

For example, if the total volume score based on FTE of staff with significant research responsibility was five a unit could, in theory, enter five outputs from one staff member.

This could mean that every staff member with significant research responsibility is eligible for REF but not every staff member with significant research responsibility will be entered into REF.

The consequence of this could be pressure on fewer researchers to produce more “REFable” outputs. It could also profoundly impact university employment, reward and recognition processes because of the prestige attached to being entered into REF.

The long run impact on whose research REF assesses is as yet unclear. The total impact on institutions and the research system as a whole will not be known till well after the exercise is complete.

REF 2028

The Stern Review of REF 2014 found that Black, Asian and non-EU staff, staff with disabilities, and women were underrepresented in institutional submissions. The 2021 exercise included all staff with significant research responsibility, but even then, research to date demonstrates there are still inequalities in number of outputs and output scores.

The UK funding councils have taken a different approach for REF 2029 by using an average number of staff with significant research responsibility. The idea is that this depersonalises REF by emphasising whole institutional performance as opposed to a cross-section of the total research activity in an institution; a subtle but important difference. The downside of this approach is that it incentivises universities to optimise, whatever that means in their context, their pool of submissions.

The volume measure also risks pushing against the People, Culture and Environment element of REF which is designed to place greater emphasis on the conditions in which research is produced. For example, one way to change the pool that REF is concerned with is to move more staff toward teaching-only contracts. Another is to, despite the efforts of Stern, focus high levels of support on a handful of prolific researchers.

And deciding who is or is not a researcher in the first place is not straightforward.

In their response to the initial REF decisions, the UK funding councils have confirmed that staff on contracts with “no research-related expectations” will not be eligible to participate in REF 2029. However, this does not include roles like technicians, where they are contributing to a unit’s output.

It is a laudable aim to formally recognise technicians’ contribution to research, and numerous pieces of research have demonstrated they are routinely undervalued in their work. For the exercise, questions remain on whether the inclusion of technicians should be used as an indicator of a broader, perhaps more collegiate, research environment. For institutions, there are implications of how the work of technicians and associated roles should be consistently recognised and remunerated if they are playing a greater role in the production of REF submissions.

REF 2029

The UK funding council’s response to their initial decisions has further narrowed the pool of potential submissions. PhD students, except as co-authors, will also not be eligible for REF submission. It is an example of where the theory of an exercise which can capture a wider range of research outputs runs into the complexity of trying to develop a comparable set of measures for fundamentally different activities. PhD students are researchers, but they are not the same kind of researchers before they complete their studies as they are after.

Putting pressure on PhD students, and their supervisors, to produce and co-produce internationally leading research could also be seen as antithetical to the learning process of an already pressured PhD programme. The logistics of deciding at what point a PhD student would become eligible for REF, and how long their work would remain attached to their institution if they have left, are also complex.

For their part, the funding councils are entirely aware of some of the risks around submissions. They are considering sampling methods, codes of conduct, statements and reintroducing maximum limits per staff member. All of these come with trade-offs on administrative complexity and the potential for gaming, but this is the fundamental problem of taking an institution approach rather than an individual one to REF.

The incentive for institutions is to return the highest-scoring research possible. The aim of REF is to represent the broadest range of research activity. Without maximum and minimum numbers of individual submissions, these two aims could be incompatible.

It is also essential to view these changes in the context of how outputs become associated with institutions. The funding bodies are undertaking more work to support institutions in showing a “demonstrable and substantive” link between staff and submitting institutions. This is partially to prevent providers from recruiting staff solely for the purpose of capturing their outputs for REF. The language within the updated guidance suggests that more than one provider could be recognised for a single submission.

The REF after this one

The 2021 approach of including all staff with research responsibilities is not perfect or gaming-proof, but it is relatively clear.

For 2029 it may well be that changing volume measures is the only means to break the link between staff with significant research responsibilities and institutional submissions, but doing so adds some significant complexity that needs testing. The entirely worthwhile principles will burn up on impact if it needs to be clarified, or made consistent between institutions, on who is or is not submitted.

This is not to say some audit regime would not work, but the question is whether it is administratively worthwhile beyond measures like having a floor and ceiling for individual submissions.

In the debate of how research performance is measured it is important not to lose sight of whose research is being measured.