On the face of it, I can understand why the REF team have pressed pause on their guidance development for 2029.
The sector is in serious financial difficulties, and while most are keen to see a greater focus on People, Culture and Environment (PCE), the challenges experienced by pilot institutions with the proposed assessment mechanism were real.
We cannot get away from this.
But of course, where there’s a vacuum, people will rush to fill it with their own pet peeves and theories, up to and including a full reversion to the rules of REF 2021.
PCE and EDI
One of the biggest fallacies being promoted is this view that PCE is what Iain Mansfield, Director of Research at the Policy Exchange Thinktank, and former Special Adviser, called “a euphemism for Equity Diversity and Inclusion (EDI)”. This conflation of REF PCE with EDI is entirely false. In fact, the PCE pilot included five different enablers of research culture, only one of which related to inclusivity. Of the others (strategy, responsibility, connectivity, and development) two were already themes in REF 2021 Environment Statements (strategy and collaboration) so not exactly a dramatic shift in a whole new direction.
Indeed, the Code of Practice and Environment elements of REF 2021 already placed a significant focus on EDI. Equality Impact Assessments had to be performed at every stage of the submission, EDI training for REF decision-makers was an essential requirement for even submitting to the REF, and both institution- and unit-level environment statements demanded narratives as to how equality and diversity in research careers were promoted across the institution. So anyone seeking a reversion to REF 2021 rules in order to eliminate a focus on EDI is going to be deeply disappointed.
Perhaps the biggest disappointment about this attempt to row back on any deeper focus on research culture in the next REF is that having a thriving research culture is an integral part of any definition of research excellence, whilst being perhaps the second biggest challenge facing the sustainability of the research sector after funding. The Wellcome Trust report, and the Nuffield report that preceded it, taught us that poor incentives, highly competitive & toxic environments, precarious research careers, and unmanageable workloads, are leading to questionable research practices, increased retractions, a loss of talent and reduced trust in science. And all this at a time when we really need more talent and greater trust in science. It wasn’t that long ago that this all led to a Government R&D Culture Strategy making a clear case for better investment in research culture for the benefit of society, but still, in the recent DSIT survey of the UK Research & Development workforce, only 52 per cent of higher education respondents said the culture of their organisation enabled them to perform their best work, compared to 85 per cent in the private sector.
The point of adding greater weight, and a clearer assessment mechanism, to a broader range of culture elements in the next REF was thus to address exactly these issues. As a reminder, the international advisory group for the next REF recommended a split of 33:33:33 for PCE, outputs and impact. Reducing the weight allocated to PCE would not only reduce the attention given to promoting positive research cultures, but actually increase the weighting allocated to the element of REF that is most responsible for driving poor research cultures: publications. We know that the publish-or-perish culture is causing significant problems across the sector. Re-calibrating the assessment to put greater weight on publications would run counter to the Coalition for Advancing Research Assessment’s first commitment: to recognise the diversity of contributions to research.
Outputs
I do think the pause in REF is an opportunity to think about how we recognise, incentivise and reward the better research cultures we clearly need. I’ve written before about how many elements of our research culture are essentially hygiene factors and as such should not attract gold stars, but be established as a basic condition of funding. There is also an opportunity to supply culture-related data (e.g., research misconduct reporting, and research staff pay gaps) alongside the other environment data already supplied to support REF-decision making. This could be formative in and of itself, as could the use of case studies (a tested REF assessment technology) by which HEIs report on their research culture interventions.
Whatever is decided, no-one working in a research-intensive institution can deny the power of the additional weight allocated to PCE in REF 2029. The knowledge that 25 per cent of the next exercise will be allocated to not just E, but P and C, has naturally been a lever staff have pulled to get culture issues up the agenda. And we’ve seen significant improvements: policy changes, new initiatives, and culture indicators moving in a good direction. So whilst it might feel like an easier move to simply revert back to the rules of REF 2021, there is an opportunity cost to this. A lot has already been invested in preparing institutions for a greater focus on research culture, and more will need to be invested in reverting back to the rules for REF 2021.
Because of the REF’s direct link both to (unhypothecated) gold and (international) glory, nothing really motivates universities more. To row back on efforts to recognise, incentivise, and reward the thriving research cultures that are at the very heart of any ‘excellent’ research institution therefore makes little sense. And it makes even less sense when financial constraints are putting those environments under even more pressure, making it more important than ever that we put people first. Can we do it in a more sensitive and manageable way? Yes, of course. Should we ditch it and run for the cover of REF 2021 rules? Absolutely not.
The article talks about the opportunity cost of reverting to the REF 2021 weightings, but this is a two-way street. One of the main reasons for pushback on the proposed weightings for REF 2029 is that research outputs would be worth 45% — less than half the weight of an exercise that is supposed to measure research quality. For comparison, the figure was 60% for REF 2021 and REF 2014 and 70% for RAE 2008.
I agree with much of what Elizabeth says about research culture, but I wonder if REF is the best way to achieve this. REF has become increasingly complicated and unwieldy over time, absorbing more money and time in terms of preparation partly because it has been used as a way to ‘drive’ behaviour and promote various things that have been deemed desirable (e.g. impact, OA, and now research culture). I agree that these things are desirable, but I would personally prefer a leaner, stripped back REF that focused more on measuring outputs, and promote the other agendas via a different… Read more »
The REF is an exercise to understand the quality of research undertaken by publicly funded institutions. If there were evidence that a better research culture led to better outputs, then it would be reflected in those outputs, and we wouldn’t need to measure culture at all. Good cultures would be recognised for producing good research. The REF would be an incentive for experimentation and improvement within autonomous institutions. On the other hand, if there is weak or no evidence of a link between research culture and quality – and so far I have seen none – then why is this… Read more »
But a focus purely on outcomes is precisely what produces a negative culture. If you have a target to e.g. reduce waiting lists it leads to a culture of corner cutting to drive numbers down, which has all manner of negative consequences that don’t show up in the measurements. If you have a target to only produce more high quality research outputs you end up with a culture geared towards gaming the system to make those numbers go up, with all manner of negative consequences for the broader culture. When its tied to funding the only things that matter are… Read more »
I am disappointed by the assumption in some of the comments here that research “quality” can only be assessed by the extent to which other academics are impressed by academic publications. REF is a system for assessing taxpayer value (which has to work on a timescale longer than the current flavour of government) – it is a mechanism for allocating funding, and providing accountability for public funding. Impact – meaning societal and economic benefit from research – is an absolutely legitimate aspect of research value, perhaps the most legitimate aspect. The introduction of PCE assessment emphasises that the training and… Read more »
“Reducing the weight allocated to PCE would not only reduce the attention given to promoting positive research cultures, but actually increase the weighting allocated to the element of REF that is most responsible for driving poor research cultures: publications.”
Really? Is the expectation to publish 2.5 articles in 7 years the primary driver of poor research culture?