Reading Amanda Solloway on the REF

If you take Michelle Donelan’s “bureaucracy” kick as an attempt to operationalise the letters page of the old THES circa 1998, action on REF (or RAE, as it was then) was clearly the other shoe that was waiting to drop.

I was expecting something akin to Gordon Brown’s metrics-led shakeup in 2008, without the consciousness that such approaches have been tried and found wanting for anything other than supplementary information in a handful of science units of assessment. What we got today was something less certain, but much more interesting.

Amanda Solloway - who has thus far had the same impact on science policy as Jason Newsted had on Metallica - has been listening to some academic critiques of REF. The framework is a complex and much-mythologised beast, to the extent that many commonly held beliefs about REF are absolute nonsense.

Researchers tell me they feel pressure to publish in particular venues in order to gain the respect of their peers, which wrongly suggests that where you publish something is more important than what you say. That just can’t be right.”

It is not right. I’ve said this so often that I’m tired of saying it but, here goes.

REF doesn’t care where you publish

REF doesn’t care where you publish. Your choice of journal matters not one iota to the REF. Journal Impact Factors are not a thing. Whatever “citation impact” may be, it does not play a part in REF. REF is a system based on the peer review of academic outputs - doesn’t matter what the outputs are, or where they are published. For the biggest chunk of the assessment process (glossing over the “impact” and “environment” measures) It only matters what other academics make of it. Nobody cares about your h-index either, put it away, boys.

Amanda Solloway has fallen a long way down the REF rabbit hole:

People talk to me about “REF-able publications” – a total distortion of the value of research and a constraint on the diversity of research objectives.”

There is no limit to what is “REF-able” - excepting the very reasonable expectation that research funded by the public is readable by the public for the most part (even here, there are numerous get-out clauses). You can return a symphony to REF, or a public event, or a dataset. Whatever works. Academics do tend to like articles and monographs as these are good ways of conveying complex information accurately and quickly - once you have learned a few basic rules reading an academic paper is not a complex endeavor.

Men made of straw

Far be it from me to offer wider political critique, but people in positions of power inventing a pretend bad thing (or riffing of a bad thing someone else has invented) for political gain has been a feature of recent political life in the anglosphere. As delighted as I am that these lines suggest that we’re not going to get Gordon Brown’s fully automated luxury RAE any time soon, I’d rather keep research policy conversations among the “fact-based” community.

The meat of the announcement could have been contained in a single tweet:

I have today written to Research England to ask them to start working with their counterparts in the devolved administrations on a plan for reforming the REF after the current exercise is complete.”

It is customary to review REF/RAE after each iteration - and new ideas are taken on board as the system is tweaked - note for example the impact of Stern on REF2021. I was pleased to note that the Forum for Responsible Metrics had cut through, as had wider international initiatives on research assessment. The way in which some providers use the mythologised REF to promote anxiety and competition is long overdue reform.

There is a consultation coming. Based on the speech it is difficult to see much more than tweaks around the edges being on the cards. But stranger things have happened. I'd wait for the letter.