The updated guidance on Contributions to Knowledge and Understanding (CKU: formerly known as outputs) will be seen as the moment it became clear what REF is.
REF is not about solely, or even mostly, measuring researcher performance. Its primary purpose is to assess how organisations measure research excellence.
It is the release which signals that research may be produced by individuals but it is assessed at an institutional level and the only measure that matters is whether the institution was responsible for supporting the research that led to the output.
2014 Redux
It is worth rehashing how we got here.
REF is the tool Research England and its devolved equivalents use to decide how much QR funding universities will receive. One thing it measures is the research output of universities. The research output of universities are the outputs of the researchers that work there (or a sample of the outputs.)
The question that REF has always grappled with is whether to measure the quality of research or the quality of researchers. The latter would be quite a straightforward exercise and one that has been done in different formats over the years. Get a cross-sample of researchers to submit their best research at a given point in time and then ask a panel to rate its quality.
Depending on the intended policy output the exercise might make every researcher submit some research to ensure a sample is truly representative. It might limit how much any one researcher can submit to ensure a sample is balanced. It might tweak measurements in any number of ways to change what a researcher can submit and when depending on the objectives of the exercise.
The downside of this approach is that it is not an entirely helpful way to understand the quality of university research across an entire institution. It tells you how good researchers are within a specific field, like a Unit of Assessment, but it does not tell us how good the provider is at creating the conditions in which that research takes place. Unless you believe, and it is not an unreasonable belief, that there is no difference between the aggregate of individual research outputs and the overall quality of institutional research.
Individuals and teams
To look at it another way. Jude Bellingham looks very different playing for England than he does Real Madrid. He is still the same footballer with the same skills and same flaws. It is that for Real Madrid he is playing for a team with an ethos of excellence and a history of winning. And for England he is playing for a team that consistently fails to achieve anything of note.
The only fair way to measure England is not to use Jude Bellingham as a proxy of their performance but to measure the performance of the England team over a defined period of time. In other words, to decouple Bellingham’s performance from England’s overall output.
As put in a rather punchy blog by Head of REF Policy Jonathan Piotrowski,
REF 2029 shifts our focus away from the individual and towards the environment where that output was created and how it was supported. This change in perspective is essential for two key reasons: first, to gather the right evidence to inform funding decisions that enable institutions to support more excellent research and second, to fundamentally recognise the huge variety of roles and outputs that contribute to the research ecosystem, including those whose names may not appear as authors and outputs that extend beyond traditional journal publications.
Who does research?
The philosophical questions are whether research is created by researchers, institutions, or both and to what degree. And in a complex system involving teams of researchers, businesses, and institutions, whether it is any easier or accurate to ascribe outputs to researchers than it is to institutions. The policy implication is that providers should be less concerned about who is doing research but the conditions in which research occurs. The upshot is that the research labour market will become less dynamic, there is less incentive to appoint people as they are “REFable”, which will have both winners and losers.
The mechanism for decoupling in REF 2029 is to remove the link between staff and their outputs. The new guidance sets out precisely how this decoupling process will work.
There will be no staff details submitted and outputs will not be submitted linked to a specific author. Instead, outputs are submitted to a Unit of Assessment. This is not a new idea. The 2016 review of the REF (known as the Stern Review) recommended that
The non-portability of outputs when an academic moves institution should be helpful to all institutions including smaller institutions with strong teams in particular areas which have previously been potential targets for ‘poaching’.
However, it is worth emphasising that this is an enormous change from previous practice. In REF 2014 the whole output was captured by whichever institution a researcher was at, at the REF census date. In REF 2021 if a researcher moved between institutions the output was captured by both. In REF 2029 the output will be captured by the institution where there is a “substantive link.”
Substantive links
A substantive link will usually be demonstrated by employment of a period of 12 months at least 0.2 FTE equivalent. The staff member does not have to be at the provider at the point the output is submitted. Other indicators may include
evidence of internal research support (for example, funding for research materials, technical or research support, conference attendance) evidence of work in progress presentations (internally and externally) evidence of an external grant to support a relevant program of research.
In effect, this means that the link between researchers and REF is that their research took place in a specific institution, but it is ultimately the institution that is being assessed. The thing that is being assessed is the relationship between the research environment and the creation of the output. Not the relationship between the output and the researcher.
As the focus of assessment shifts so do the rules on what can or cannot be submitted. As we know from previous guidance there is no maximum or minimum submissions from staff members. There may be some researchers at, or who were at, a provider who find their work appears in an institution’s submissions a number of times, and maybe even across disciplines (there will no be now no inter-disciplinary flags but an output may be submitted to more than one UOA and receive different scores.)
The obvious challenge here is that while providers should submit representative outputs the overriding temptation will be to submit what they believe to be their “best” and then work backwards to justify why it is representative. The REF team have anticipated this problem and the representativeness of a submission will be assessed through the disciplinary led evidence statements. The full guidance on what these contain is yet to be released but we know that
The important issues of research diversity, demographics and career stages will be assessed as part of the wider disciplinary level evidence statements
Research England’s position is that aligning outputs to where they are created, not who creates them is a better way to measure institutional research performance. This should also end the incentive for universities to recruit researchers and in doing so capture their REF output. The thinking is that this favours the larger universities that can afford to poach research staff.
Debates had and debates to come
In a previous piece for Wonkhe Maria Delgado, Nandini Das, and Miles Padgett made the case that portability is key to fairness in REF. The opposite argument that is being put forward by Research England. Maria, Nandini, and Miles made the case that whether we like it or not one of the ways in which academics secure better career prospects is by improving the REF performance of a provider’s UOA. Research England makes the case that
The core motivation is to minimise the REFs ability to exert undue influence on people’s careers. To achieve this, institutional funding (remember, QR funding does not track to individuals or departments) should follow the institutions that have genuinely provided and invested in the environment in which research is successful. Environments that recognise the collaborative nature of research and the diverse roles involved, rather than simply rewarding institutions positioned to recruit researchers to get reward for their past output.
It is possible that both arguments may be right. If outputs are tied to institutions the incentive for institutions who want to do well in REF is to capture a greater number of high quality outputs to include in their submission. The way to do this is to have more researchers supported to do high quality work. On the other hand, at an individual level and in a time of financial crisis for the sector, there are likely some researchers who benefit from being able to take their research output with them when they move institutions.
In the comments of our initial portability piece it was flagged that researchers’ work could form part of an assessment where they had no relationship with the provider. This feels particularly egregious if they have been made redundant as part of wider cost saving. The message being that the research output is high quality but nonetheless it is necessary to remove your post. The REF team have considered this and
Outputs where the substantive link occurred before the submitted output was made publicly available, will not be eligible for submission where the author was subject to compulsory redundancy.
The guidance explains that there may be times where there is a substantive relationship but the research has not yet been published. On the face of it this seems a sensible compromise but if the logic is that a provider is the place where research outputs are created it seems contradictory (albeit kinder) to then limit the conditions through which that work can be assessed. It is possible there will be some outputs which were in the process of being published but not yet assessed which would fall into this clause.
The guidance confirms a direction of travel that was established as far back as REF 2021 and made clear in the guidance so far for REF 2029. While the debate on who should be assessed in which circumstances continues the wider concern for many will be that there is still significant guidance outstanding, particularly on People Culture and Environment, and the submission window for REF closes in 30 months from now.
A direction has been set. The sector needs to know the precise rules they are playing by if they are going to go along with it. There is undoubtedly a lot of good will around measuring research environments, culture, and the ways in which outputs are created more comprehensively. That good will, will evaporate if guidance is not timely, clear, or complete.
REF should never have become a way of hopping from institution to institution. That is an individualist mindset that undermines good research. REF never should have been anything to do with career advancement. It should always have been about improving research. Instead, it created a transfer market that only benefitted a small number of ‘rockstar’ academics. The sooner portability is consigned to the bin, the better. The majority of staff in universities do not and have not got any benefit from portability. It’s just a con.
Simply false. Portability has been especially important for those who did not yet have a permanent position and could take their outputs produced during temporary positions into a permanent position.
Agree 100%. I was in just this position and the portability of some top outputs was one of the few cards I and others had in a very tough job market, which is now even worse. Stopping portability thus undermines the noises made about supporting ECRs etc in the PCE part of REF.
The “transfer market” was always small, and while “poaching” did happen (especially before 2014), in the current financial situation the short-term hiring of “superstars” is not realistic anyway. However, many academics change institutions for perfectly acceptable reasons, such as family, institutional closure, research funding. The restrictions on portability will make life harder not only for early-career researchers and everybody temporary contracts, but also for those who had to leave academia but would like to get back into research. However, the ruling on institutions not benefitting from forced redundancies is welcome.
Institutions do not produce research – just as they don’t teach students. Individual academics, or groups of academics, do. Institutions provide an infrastructure within which research and teaching take place. That infrastructure certainly includes all the things that institutions have to describe, tediously and somewhat tendentiously, in REF entries. But their most important contribution, by far, the key bit of infrastructure if you like, is that they pay the academics. That is where the Bellingham analogy comes unstuck – Real Madrid pays his wages; England doesn’t (or not significantly). And, like goals, outputs in the end are all that matter not the build-up or the manager’s post-match press conference. The further the REF moves away from assessing outputs, the less credible and the more bureaucratic it becomes.
Agree with this. It has become more complicated as people have made it a driver for other things they deem desirable, eg impact, OA, research culture. A narrower, simpler REF would be less bureaucratic and these other things could be promoted in other ways.
Sure, but REF exists to allocate QR research funding to institutions, not to individual researchers. The funding flows to universities, who then use it to support research infrastructure, pay salaries, and create research environments.
REF isn’t asking “who produces research?” but rather “which institutions should receive public funding to support research?” BC the money goes to institutions, it surely makes sense to assess institutional research environments and outputs.
If paying researchers is the key institutional contribution, then assessing institutions (who receive the funding to pay those salaries) is logical.
Re the Bellingham analogy. The point isn’t about who pays wages, but about how the same individual can perform very differently in different institutional contexts. That supports the idea that institutional environment matters for research quality.
Also for me the idea that institutions “just provide infrastructure” understates what universities do – from providing expensive equipment and facilities, to managing grants, facilitating collaborations, providing administrative support, ensuring ethical compliance, and creating interdisciplinary research communities. Research, especially in many fields, is increasingly a collective enterprise that depends heavily on institutional support.
I get the feeling that a lot of people have internalised REF as a measure of individual worth rather than what it actually is – a bureaucratic tool for dividing up institutional funding. My sense is that universities would be healthier if they stopped treating REF as a source of personal validation and recognised it for what it is – a funding mechanism, not a judgment on individual scholarly merit.
Put another way, if your primary source of professional validation comes from REF scores rather than recognition, impact, or actual research outcomes, then you need better institutional management, mentorship, and a healthier academic culture that values the full spectrum of contributions.
For me the shift to institutional assessment might also finally recognise the contributions of PGRs and other research staff who often do the heavy lifting on “star” outputs but rarely see their names in REF submissions or benefit from the current transfer market system.
I think this is wishful thinking as the reality is that whatever form REF takes there will be game-playing etc by institutions due to reputational prestige of the university rather than just individual scholars.
I would repeat the point that a slimmed down, narrower REF focusing on research quality based on outputs would be a more effective mechanism for distributing money. By trying to make it do more things it has become increasingly complicated and amorphous, which I don’t think has been beneficial.
I would like to believe you are right. Alas my fear is the new rules mean there is now no limit on the number of outputs that can come from a single author or a single group. An unintended consequence is that we might find that the new REF ceases to assess the diverse output of a unit and instead focusses on the outputs of perceived “research stars”. I would prefer a REF in which the diversity of authors, the diversity of ideas, the diversity of subfields is represented and hence valued.
#confused ! If a “rockstar researcher” has a 12 month 0.2FTE contract with an institution (with a demonstrable substantive link, obvs) is it just the rockstar’s publications in those 12 months that count or all publications during the REF period. If the latter, there will be huge gaming of the system and rockstars will be more in demand than ever, surely ? (Sorry if you don’t like being called that)
I’m not a fan of the REF – the RAE/REF has been a great engine of intellectual conformity over the past thirty five years or so. Research outputs nonetheless remain the only worthwhile metric of research quality. Taking into account research culture sounds nice but assessing it is hard – it depends on such things as institutional memory. While I would be very much in favour of punishing university managers who have gone out of their way to destroy such memory and betray their founding principles (Manchester, Sheffield and Cardiff spring to mind) judgement here is inevitably subjective and arbitrary.
For short form: during employment definitely counts; after employment, withing a defined period, only if additional indicator(s) of substantive link, both of which which we don’t have defined yet. They have said additional indicators of substantive link will be things like acknowledging grant funding held by the submitting HEI.
Long form or long process: counts during, and before or after, within defined periods. We don’t know the periods or the definition of long form or long process.
Asked to chose between people & institutions, the answer is people create research. But the REF is not primarily about research.
Just a guess, without portability what you have is the collapse of research. Look around, the direction of higher education is reduced research support, increased teaching demands. If not internally motivated, why would researchers sacrifice sleep, family and personal time when there is no tie between their research performance and their benefits. Without portability nobody moves. With portability, if universities don’t provide research support, productive researchers would leave… That at least would be an incentive for universities to support research. Today that doesn’t exist. Universities are heading to be skill-training centers, forget about research and science.
Changing the focus of REF away from actual research quality to second-guessing what enables good research will make the REF more bureaucratic, more easily gamified, and less credible. The silver lining is that the changes will hopefully hasten the replacement of the whole rotten edifice with something cheaper and less time-consuming.
REF might be “about” institutions rather than individuals, but should it operate at the cost of and to the detriment of individuals who are the creators of research and knowledge? No. Not to mention that the coupling of outputs and institutions reinforces the idea that research is an institutionalised product and it marginalises the instituition-less and those experiencing career breaks but produce research/knowledge nonetheless. If REF claims to profess in creating a positive research culture in the UK, then rethink portability.
Research is driven by people, individuals who dedicate their evenings, weekends, and personal lives to advancing knowledge, often far beyond what is required in their job descriptions. Yet rarely, if ever, are researchers asked if they felt supported, how their work was enabled, or whether their research was conducted during paid hours. Nor are they asked whether the QR funding their outputs generated was reinvested to support their future research. The answer is always NO. These questions are avoided because they would expose the truth for many institutions: institutions benefit from research outputs without truly recognising or rewarding the individuals behind them.
Without portability, early-career researchers are penalised for changing jobs, and career breaks become career-ending. Those who power the research economy, often out of passion, commitment, and a belief in the social and academic value of knowledge are told that their work belongs to the institution, not them. So there is no incentive to promote or support them, and if you care about your field, you become the ideal candidate to be exploited by poorly managed institutions.
This situation will not change while universities are treated primarily as businesses rather than communities of scholarship. The removal of portability simply reinforces this model. This is not only unjust to researchers, it is fundamentally damaging to the future of research itself in the UK, so maybe it is time to give up, pack up and head over to Europe, as many of our colleagues already did in the past years.
As a member of professional staff I look on and find myself faintly bemused. What is it about the academic field that should make past achievements in themselves, rather than the expectation of future achievements, the most important thing in the job market?
When applying for a job, I present evidence of my past achievements as part of an argument that I can be expected to achieve similar things in the future for a new employer. If, for example, I successfully manage a project for Barchester University, I may then get a job at Wessex University on the strength of that, but that’s because they believe I will do well managing new projects for Wessex – not because they believe the benefits of the original project at Barchester will somehow now start to accrue to Wessex. The idea is somewhat bizarre when taken out of the research context.
Absent portability, will universities not simply start to make judgments on academic candidates in a similar way – i.e., which candidate’s record implies that they have the best potential to do good academic work in the future… and to help others around them do the same?
Allow me to take your question “what is is about the academic field” literally, rather than in a polemical sense. One of the problems is the speed at which you can demonstrate your capabilities to your new employer. In research-focused jobs, this is mainly through publications and grant capture. Both take considerable time to set up and bring through fruition, which means that if somebody tries to change jobs two years before REF, they might not have enough time to have another 4* article published or to secure a major grant. And in the current climate, many institutions won’t want to wait.
Thanks, that’s a helpful answer and does illuminate it for me. If I’m understanding it right, the risk of removing portability is therefore that movement between institutions would become harder in the later stages of a REF cycle. I can see that that is an undesirable consequence for most if not all parties.