Since the first Research Assessment Exercise (RAE) was introduced in 1986, such endeavours have always attracted their fair share of critics.
The Association of University Teachers (a precursor to UCU) argued in 2008 that “the RAE has had a disastrous impact on the UK higher education system, leading to the closure of departments with strong research profiles and healthy student recruitment.” Other commentators claimed the exercise placed undue pressure on academics to publish or perish, created unhealthy competition, nurtured gaming by universities and encouraged individuals to go to press with half-baked work – or in some cases, work that was based on falsified data.
And yet, over the same period, the reputation of UK universities for producing brilliant research soared. Britain may no longer be the workshop of the world; its standing as a global political superpower is a distant memory and its economy may have been overtaken by those of other nations; but overall, the UK performs extremely well in global university league tables, only surpassed by the endowment-rich cream of the US private sector.
By other indicators, too – publications, citations, attractiveness for international research talent, patents and reputation surveys – the UK’s most research-intensive universities are recognised as amongst the very best. The foundations of their success are surely based on the excellence of their research outputs and their quality is underpinned by the robust system of peer review which has run continuously since 1986.
Input/output
The next Research Excellence Framework will cover the period 2021–2027 but the rules of the game are only now emerging. A consultation document issued jointly by Research England, the Scottish and Welsh funding councils and the Northern Irish Department for the Economy indicates a desire to “change the emphasis of national assessment from the performance of individuals to the contribution institutions and disciplines make to healthy, dynamic and inclusive research environments.”
Significantly, the primary focus is no longer on the quality of research outputs – it now encompasses “the broader contributions to research and research processes.” This is intended to be a more inclusive exercise, giving “appropriate recognition to the people, culture and environments that underpin a vibrant and sustainable UK research system”, and “capturing the valuable contributions or a wider range research and research-enabling staff.” To this end, the previous components of outputs (60 per cent), impact (25 per cent) and environment (15 per cent) are replaced by “contribution to knowledge and understanding” (50 per cent), “engagement and impact” (25 per cent) and “people, culture and environment” (25 per cent).
There are several potential issues with this. Most importantly, it moves away from a focused assessment of research outputs, with its weighting now less than 50 per cent, placing greater stress on more nebulous considerations of environment and culture that, by their very nature, are difficult to measure. This represents a significant shift for the REF.
Without question, the research ecosystem has its faults and there is data to illustrate that components of it, such as the funding system, are not equitable. We need to create a fair and inclusive research environment, and to support, reward and celebrate all the varied contributions to research in the UK. But is the REF the most appropriate or best mechanism to achieve this?
Protect the environment
All institutions strive to foster a culture which produces fantastic, world-beating research outputs – drugs that cure diseases, new policies that address societal problems, strategies to solve the climate crisis and dramatic insights into the great diversity of human cultures. These are the goals of research at every institution, and it is the quality of this work that should be measured and assessed.
Instead, we are increasing the assessment inputs rather than outputs. It is a little like judging a world-famous artist not by the quality of their paintings or sculptures, but by how attractive their studio is.
The ever-increasing emphasis on environment threatens to undermine the drive for excellence which successive RAEs and REFs have helped to sustain. It represents a gradual dilution of the focus on competitive spirit which has made UK research the envy of the world. If we want to see enhancements in research environment and culture – and we certainly should – perhaps the best way to do this is to judge institutions by their research outputs. The rest will follow.
Counter-narrative
While these recent changes to the REF are significant, it is surprising that interdisciplinarity has received little attention. As we seek solutions to complex global social, economic, ecological and political challenges, interdisciplinary research is an increasingly important part of our research activities. Is this a missed opportunity? Or is it just too difficult for what is a discipline-centric assessment process?
The new proposals will likely place further weight on narratives rather than being driven by a genuine attempt at peer review of quality. It is impossible to measure one environment against another on the basis of a narrative. Moreover, there is a danger that the REF will drive a rather prescriptive and unimaginative approach to enhancing research environments, instead of encouraging institutions to put energy into those areas important to them.
By the same token, the assessment of impact will not be made easier by inviting institutions to produce “an accompanying statement to evidence engagement and impact.” If a university has been successfully engaging, this should be evident from the case studies. We should concentrate on the outputs, not the processes or systems which produce them – and we should try, wherever possible, to assess things that are actually measurable.
Perhaps these criticisms are too harsh. Previous research exercises have seen similar radical changes, such as the introduction of the assessment of impact, that have driven positive change. Subject sub-panels are populated by academics who will be able to navigate this new approach. But their jobs would be easier if we avoid creating a system which can be gamed to excess and maintain a laser-like focus on the things that really matter. The REF consultation period ends on 6 October – there is still time to let your views be known.
While I share the contention that REF will certainly get more nebulous and less quantifiable under the new regime, I would argue that brings it into greater correspondence with the reality of research.
I would be interested in any evidence that UK research has soared since 1986. Certainly there was no shortage of world-leading discoveries before 1986. In the absence of evidence to the contrary, I would contend that the conflation of research assessment with research quality is simply a collective loss of memory of Dorothy Hodgekin, John Gurdon, Peter Higgs et al.
Certainly the reputation of UK research has declined significantly over the last two REF cycles, as research’s reputation has globally with the perverse incentives and inappropriate quantification that have accompanied what is now termed the reproducibility crisis. The existence and expansion of RetractionWatch also speaks to this, as do the many and various moves towards more responsible publishing models, including preprints and post-publication review. Inappropriate quantification of research ‘outputs’ and resulting poor leadership have been part of the motivation underpinning the DORA and Leiden declarations. As the Director for the Wellcome Trust noted in 2019, in the UK this focus upon quantification has led to destructive hyper-competition, toxic power dynamics and poor leadership behaviour (https://www.nature.com/articles/d41586-019-02951-4). This was abundantly in evidence across UK research-intensive institutions in relation to REF2014 (https://www.theguardian.com/higher-education-network/blog/2014/jun/23/kings-college-london-redundancies-bad-management) and REF2021 (https://www.nature.com/articles/d41586-021-00793-7).
The focus upon competitive spirit is not what has made UK research the envy of the world. It is what has driven disabled people, carers, and other disadvantaged groups, particularly mothers, out of research. The focus upon actual research problems and the freedom to pursue them with curiosity and room for repeated failure is what has made UK research the envy of the world. And, correspondingly, that is how Dorothy Hodgekin, John Gurdon, and Peter Higgs described what they achieved.
Yes, lack of precise goal posts and over-reliance on narratives are unhelpful in REF2028, but the focus on people and culture, as a strategic position, is spot on. The unit of analysis in REF / RAE has never been the individual ‘artist’, but indeed the ‘workshop’. When the workshop is toxic, it’s likely that their only inhabitants are those with high tolerance to toxicity. But are they necessarily the best artists?
Moving on with the metaphor, to attract government funding, ‘workshops’ need to meet hygiene conditions that make them safe and, crucially, stimulating to work in for more than just the usual suspects (who may not be the most talented but the only ones who would do it given the circumstances).
Wouldn’t it be great if UK research was to be excellent not because of a handful of lonely geniuses, but because UK is home to enviable research environments valuing (and paying) people well, being inviting and inclusive of more diverse researchers than anywhere else, being in essence a greenhouse of great minds and keeping them in the UK where opportunities are equal to none?