Wonkhe has collected the views and insights of leaders and scholars across the higher education sector on the Review of the Research Excellence Framework by Lord Stern.
Maddalaine Ansell, Chief Executive, University Alliance
“It would be a shame if the new framework discouraged researchers from engaging in a mix of beneficial activities”
Overall, the previous REF worked reasonably well. It increased research quality, encouraged dynamism and, within institutions, helped managers to make strategic decisions. We were therefore pleased that Lord Stern left the bones of it – including assessment predominantly by peer review – in place. It also makes sense to align the REF with the TEF by recognising the positive impact that research can have on teaching.
We do have concerns, however, about the new requirement to submit all research active staff. There can be significant variation in how much time faculty members spend on research that leads to returnable outputs. Many researchers also spend time teaching and carrying out contract work for business. It would be a real shame if the new framework discouraged researchers from engaging in a mix of beneficial activities. However, enabling institutions to return more outputs for some research active staff and fewer for others may allow for some flexibility.
In the end, what matters is that research excellence is funded wherever it is found – including within those universities that currently don’t receive high volumes of research funding. As long as the REF is used to identify this, it will be serving its purpose.
Martin McQuillan, Pro-Vice Chancellor of Research and Innovation, Kingston University
“We should think long and hard before deciding to unpick the current dynamism in the academic job market”
On the face of it things could have been much worse. Last year in the Green Paper it looked as if the government was pushing for a metric-based REF. Elsevier had been awarded a tender to pave the way for just such an outcome. Instead by July 2016, with the publication of Lord Stern’s review, it seems like we are being offered a kinder, gentler REF.
As a REF-sceptic, Stern accepted the challenge of looking at the efficiency of the exercise but he was more concerned with the ways in which the REF skewed institutional and individual research priorities. His suggestion of all-staff inclusion with weighted outputs attempts to address that, and will probably be welcomed by many as a more inclusive approach to research.
However, one suspects that this might not work out as intended. As a whole the proposals on research environment are more likely to benefit established research universities over, shall we say, ‘challenger institutions’. Placing new emphasis on institutional level environment scores can only have that outcome and is now in tension with the basic REF principle of supporting excellence wherever it is found. Just because a university has excellent research in bioscience does not mean that therefore it’s research in the social sciences must by default also be excellent.
The recommendation on non-portable outputs is also likely to be contentious. In case we have all forgotten one of the drivers for the original ‘Research Assessment Exercise’ was to create dynamism in the academic job market. This and an increase in both student numbers and institutions brought about the relatively fluid arrangement we have at the moment. We should think long and hard before deciding to unpick that eco-system at a time of considerable change. Any one who doubts this should look to the US where research production may be more measured but where the tenure system has created stasis for a generation of adjuncts and a top-heavy, geriatric academy.
In the technical review that now follows, these are the two aspects of the Stern report that are most likely to be put to the test. However, by resisting the machine-like computation of metrics, Stern has left the game in place, with new rules and still with plenty of scope for game playing. In a choice between the Machine and the Game, academics can now look forward to having their surplus value extracted in a nice way rather than a nasty way.
Professor Martin Paul Eve, Professor of Literature, Technology and Publishing, Birkbeck, University of London
“The new recommendations pave the way for scholarship in the humanities, which often takes longer to produce, to come to fruition”
The Stern review of the REF is, on the whole, an extremely positive and well-thought-through set of recommendations, focusing on eliminating gaming where possible, creating a hiring environment that should stop the “poaching” of researchers at the last minute, and avoiding the dangerous false economy of metrics for the sake of metrics. Of course, there are details of implementation that must be considered, particularly around early-career and independent researchers and the need for them to transfer their outputs to their first jobs. It is also important that, alongside TEF, the proposed submission inclusivity does not drive a wedge between teaching and research, with universities flocking to hire staff on teaching-only contracts. If this were to occur, then the point of gaming would simply have been shifted to the institutional gaming of contracts.
It is also a positive outcome of the review that, with the average number of outputs expected to fall, we might see a reversal of the “fear of the monograph” of the last REF. Indeed, the new recommendations pave the way for scholarship in the humanities, which often takes longer to produce, to come to fruition. I was also pleased to see that the plans for open access in the REF, which will lead to a better environment where anyone can read work funded by QR, are confirmed in the review. The fact that peer review remains central and the “gold standard” of research is also a welcome statement.
Overall, I welcome the Stern recommendations. I am not wholly convinced that the efficiency drive that is wanted will materialise; after all, there is still quite a lot of institutional selectivity at play in the choice of outputs. (And REF is cheap compared to the RCs as an allocation percentage anyway.) But I look forward to these matters being cleared up in the implementation phase.
Pam Tatlow, Chief Executive, MillionPlus
“The proposals risk driving another wedge between teaching and research that may well limit rather than enhance opportunities for early career researchers”
MillionPlus warmly welcomes Stern’s restatement of the principle that excellent research should be funded wherever it is found. The panel has also worked hard to balance the diversity of sector views about the current REF system. While we have questioned some of the funding outcomes in the past, we remain convinced that the REF itself is worthwhile and not the burden that some imply.
One of Stern’s more contentious recommendations relates to the idea that all research active staff should be submitted to the REF. This opens up a Pandora’s box of potential definitions and will result in academic staff contracts being refocused on either teaching or research. Given departmental changes in Whitehall and the HE and Research Bill’s proposals to create the OfS and UKRI with neither having responsibility for holistic oversight of the sector in England, this proposal risks driving another wedge between teaching and research that may well limit rather than enhance opportunities for early career researchers.
The attempt to minimise ‘gaming’ in the REF transfer market by linking research with institutions where it was undertaken, is admirable although management of this may prove challenging. Widening the definition of impact is welcome but there is a strong case to significantly increase its weighting.
Going forward, Jo Johnson was right to acknowledge that Stern has UK-wide implications which need discussion with the devolved administrations and the funding bodies prior to any UK government response.
When this is known, the REF consultation must include an opportunity for the sector to respond to the principles that the government wishes to apply to future REFs. A consultation focused only on technical implementation would undermine the chance to improve the overarching REF framework that the Stern review has helpfully provided.
James Wilsdon, Professor of Research Policy, University of Sheffield
“University managers need to call time on the arms race of gaming that has attached itself to the exercise in recent cycles”
As I’ve written in a longer piece for the Guardian, the overall direction of travel is sensible. The big question now is whether Stern will succeed in his aim of reducing the overall burden of the exercise. This hinges on how universities incorporate the new rules into their management processes.
If we compare the challenge here to that of the REF itself, the design of the exercise is rather like the environment section: crucial, but only 15% of the total task. Lord Stern deserves a four star rating for his contribution here. But the remaining 85% is about implementation, engagement and communication. And this is a task that we in the sector have to undertake collectively.
Within Stern’s framework, the potential to reduce burden is clear: moving the emphasis away from individual outputs, and towards departments and institutions, should enable a more flexible approach, able to support a greater diversity of research types and researcher career paths, and encourage more patient investment in people and the wider research environment. The changes that Stern proposes to the definitions of impact, and to support for interdisciplinary research, are also a definite step in the right direction.
But if we see a fresh wave of gaming to secure tiny advantages under the new model, Stern’s efforts will be in vain. Greater flexibility in the number of outputs per person, complex rules around portability, and possible exceptions for ECRs, could all increase the scale of the selection task at the unit level.
To realise the potential of Stern’s framework, university managers need to call time on the arms race of gaming that has attached itself to the exercise in recent cycles. A lot rests on getting the detail right in HEFCE’s technical consultation that will follow in November.
Ultimately, we need to shift the terms of the debate, so that the REF is no longer seen as something done to us by a faceless, managerial bureaucracy, but a task that we undertake together, as a sector, every six or seven years, to renew a social license with government and our fellow taxpayers that brings almost £2 billion of essential funding to our universities each year. Lord Stern has taken us halfway there: the rest is now down to us…
Jamie Arrowsmith, Programme Manager (Research Policy), Universities UK
“Including all staff risks increasing the burden on the evaluation panels if not managed sensitively”
Lord Stern has delivered a considered reflection on the strengths and weaknesses of the REF. His review group have tried to balance the many different interests at play, proposing an evolution of the current approach (albeit with some significant highlights) rather than major reconstructive surgery. The important role played by the REF and QR funding in supporting the excellence of UK research is recognised alongside the costs of the process – to people as well as university finances.
The approach sets out in the review is broadly in line with what Universities UK called for in our response to the call for evidence in April, with one significant exception. On balance, we concluded that universities should retain the ability to select staff to include in the exercise, whereas Lord Stern proposes that all research staff should be returned. This, he argues, should help mitigate some of the less desirable consequences of the exercise while retaining its valuable features. This will no doubt create challenges – not least the potential to hugely increase the number of staff and therefore outputs being evaluated, which risks increasing the burden on the evaluation panels if not managed sensitively – and work will need to be done to fully understand the impact of this proposal.
Yet even here, in this more contentious area, the language is nuanced, and the review leaves many questions unanswered – which is no bad thing. There is significant scope for the community to work with government and funders to help shape implementation of Lord Stern’s vision, which should be welcomed. The university sector is facing a challenging time, and it is important that our institutions are given clarity and certainty wherever possible. Hopefully, through the engagement of the whole sector, the framework set out by Lord Stern can help provide this.
We would like to ensure that a statement from Martin McQuillan, who said, “Last year in the Green Paper it looked as if the government was pushing for a metric-based REF. Elsevier had been awarded a tender to pave the way for just such an outcome,” isn’t misinterpreted to suggest we believe in a metrics-only approach. We think it’s important for stakeholders to know that while Elsevier endorses the use of metrics in all decision-making processes including the REF, we also believe that metrics alone will never convey the complete picture of research quality. Metrics should always be used as a complement to the qualitative approaches of peer review, expert opinion and narrative and not as a substitute for human judgment.
Our response to HEFCE’s call for evidence can be read in its entirety at: https://www.elsevier.com/research-intelligence/resource-library/response-to-hefces. And we wrote further about the use of metrics in research assessment at Elsevier Connect, here: https://www.elsevier.com/connect/impact-of-science-the-need-to-measure.