#REF2014 Results Day

Thank you for following us today through #REF2014. It has been Wonkhe’s biggest ever day by far (any way you cut the data) showing the intense level of interest in this crucial process. Whilst today’s coverage is finished, we’ll be analysing all the data in lots more ways in the days, weeks and months to come. But to recap some of our coverage so far:

Wednesday night:

1. Data: The first full sector results – a ranking of institutions by grade point average – our calculated scores based on HEFCE’s REF data.

2. Analysis: REF results marred by fears of grade inflation – our first story to accompany the data on Wednesday night.

3. Data: The hidden-bang-for-buck heroes of UK research – our first bit of new analysis, cutting the REF data in an interesting new way looking at efficiency.

4. Comment: The Research Excellence Framework: fascinating, flawed and essential – Mark Leach’s comment on the REF day and whole process.

Thursday:

5. Data: The full sector results – building on the HESA contextual data which came later, and showed some very different results. The first place to make this data publicly accessible.

6. Analysis: Rankings, data, tables and spin looking at all the coverage so far in light of our new intensity table (5.) from John O’Leary.

7. This live blog will stay here and has plenty of unique pieces of analysis such as REF vs VC pay and our early assessment of impact.

And from earlier this week:

1. Confessions of a concentratist from Edward Peck

2. Enjoy your REF, while you can from Martin McQuillan

Both of those pieces still being read and shared widely.

And thank you’s for various different bits of help with our coverage of the 2014 REF:

David Morris

Graeme Wise

Alan Roberts

John O’Leary

Emily Lupton

UoE Consulting

Martin McQuillan

See you in 2020. Possibly.

Mark 

In recent years much has been made within the sector about the rise of vice chancellor’s pay packages. Paying top dollar for the best leaders – so the conventional wisdom states – is an investment that will lead to better performance.

So do the REF results suggest that ‘super vice chancellors’ attracted by competitive salaries deliver results? At a first glance, maybe. There is a positive linear relationship between vice chancellor’s total annual remuneration (as of 2012-13) and the percentage of 4* and 3* research achieved by an institution. Some of the highest achieving institutions in the REF pay their senior leaders’ very handsomely indeed, including Imperial (£330k), LSE (£480k), Oxford (£434k) and Bath (£384k). Excluding small and specialist institutions, we get a correlation coefficient of 0.52 when putting the datasets together.

Of course, correlation is by no means equal to causation. Many of the top paying institutions are established REF and RAE ‘performers’ with large budgets, and the table of vice chancellor salaries appears to equate with these institutions’ reputations and ability-to-pay as much as any noticeable improvement in research performance. Informally controlling for these factors makes the relationship much less significant and appears to confirm the argument of many opponents of soaring executive pay: that research and university performance is a collaborative effort, and not an individual one.

Thus the data perhaps demonstrates how possible it is to achieve relatively strong results without paying the biggest salaries. Vice chancellors at the University of Bradford (£198k) and the University of Ulster (£204k) both appear to be relatively good ‘value for money’ compared to some of their peers, with both of their institutions coming ‘ahead of the line’ in terms of relative performance. Top of that list appears to be Colin Riordan (£252k) at Cardiff University, already noted as one of the REF’s top achievers this year, having not been paid a highest-end salary.

It remains to be seen what practical effect the REF results will have for the fate of vice chancellors and their pay packets, which even at their lowest appear to the majority of students and staff to be extremely high. Remuneration committees at many institutions may wish to take a long look at the data before rushing into their respective assessments.

Thanks to David Morris for data help

Our REF 2014 intensity measure table is being used by many users to show the ‘game playing’ in the results. Lots of examples on Twitter already.

Fantastic work by @wonkhe and associates on #REF2014. Suddenly the data makes sense and the extent of 'gaming' is rightfully exposed

— Steve Woodfield (@sjwku) December 18, 2014

This @wonkhe intensity measure makes short shrift of @cardiffmet claims to be leading post-92 in #REF2014 http://t.co/hDdSfSsvY0

— Leonardo Loredan (@LeonardoLoredan) December 18, 2014

Some universities played the REF game by entering only a small % of staff. This link explains the spin http://t.co/Ofbs8VQ76e via @deevybee

— Essi Viding (@EssiViding) December 18, 2014

RT @deevybee: Ah, everything you saw about #REF2014 looks different if you factor in %staff entered http://t.co/4sbVBgq63g surprise surprise

— Sean Fielding (@Fielding20) December 18, 2014

New table from @wonkhe places @lborouniversity 11th in England and 14th in UK for research intensity http://t.co/6bX4Mutuso #REF2014

— Richard Taylor (@RichardTaylor_) December 18, 2014

Of course the reverse is true for the universities that did better in this ranking:

"@wonkhe: NEW: Sector REF 2014 results including latest HESA data showing intensity: http://t.co/NBsjLjeHSU #REF2014"@unilncoln g8 result!

— Mary Stuart1 (@lol1090507) December 18, 2014

Research intensity rankings place Bristol joint fifth with Oxford #REF2014 http://t.co/ADvDCN5FVG

— Bristol University (@BristolUni) December 18, 2014

John O’Leary looks at the latest data we’ve been able to publish this afternoon, and what it tells us about all the REF game-playing:

Oxford says that it has the “largest volume of world-leading research”. UCL says that it is “Number 1 for research strength”- multiplying its GPA by its number of entrants. And Imperial claims the highest proportion of world-leading or internationally excellent (4* and 3*) work, as well as the “greatest concentration” of high impact research.

A must read, and bang up to date – find it here.

Following on from this morning’s first results of the HEFCE REF data which you can find here, we now have a new table which includes the all-important measure of ‘intensity’ which is drawn from the HESA data released this morning and which shows the total full-time equivalents (FTE) of staff identified as eligible to be submitted to the REF 2014 by HE provider and REF Unit of Assessment. Find the table here.

We also have new analysis of the new data from John O’Leary here – Rankings, data, tables and spin.

We will shortly be publishing data and analysis of the REF results taking in to account the latest HESA data.

You may have read the piece earlier in the week from Edward Peck, vice chancellor of Nottingham Trent University – Confessions of a concentratist in which he comes clean:

I have a confession to make. During my days at Russell Group institutions I favoured the research concentration tendency; that is, the view that it would be better if fewer universities were funded by central government to undertake research. I now understand that I was wrong. There, I’ve said it. And I’m sorry.

We came up with the term ‘concentratist’ to describe those that favour the concentration of research funding in to a limited set of institutions. We posited that the opposite then would be a ‘distributist’. Although it’s possible the former would prefer the term ‘qualitists’, and the latter ‘recognists’.

New dividing lines in the HE factional political economy?

It has been revealed that the Government is again considering primary legislation to give new powers to HEFCE.

In the guidance issued alongside the Counter Terrorism & Security Bill for consultation, the Government reveals they may give HEFCE new powers to ensure universities comply with their duty to stop any ‘terrorism’ on their watch.

73. In addition to the central monitoring (section D, above) BIS will use the regional higher and further education Prevent co-ordinators and other local providers to determine how the duty is being implemented and complied with in institutions (this will include evaluation of action on complaints and whistle-blowing).

74. In order to ensure that higher education institutions are complying with this duty, the government is considering expanding the current powers of the Higher Education Funding Council for England (HEFCE) to allow it to become the body which monitors compliance with this duty for higher education institutions and private HE providers. It is the government’s view that HEFCE is best placed to undertake this role because they already have a role in monitoring some HE institutions in relation to governance and finance. Any extension to HEFCE’s current role would require changes to primary legislation.”

Political barriers still exist to passing new HE legislation, but this Bill passing will further strengthen the case for the Government of the day to bring forward some HE-related legislation.

Read our coverage so far our the Counter Terrorism & Security Bill and its relationship to universities in Analysis and Policy Watch.

We are continuing to monitor mainstream media for its treatment of the REF results today. HEFCE’s David Sweeney, clearly enjoying his final outing as the top REF official, has appeared on the BBC News Channel this afternoon.

Research Fortnight also has an excellent profile of Sweeney in their special REF supplement which is well worth a read.

Our first main story today is by John O’Leary, the veteran higher education reporter. He found that admits the noise, there were no shortage of vice chancellors that would admit there was a level of grade inflation in today’s results, with virtually everyone improving by so much.

One vice chancellor, speaking on condition of anonymity, described the results as a fiasco. “There has been massive grade inflation and I doubt that anyone would pretend otherwise,” he said.

Another former vice chancellor said that some grade inflation was built into the system. But the scale of the increase at a time of level or declining funding might lead ministers to question whether another REF would represent value for money.

The last point here is really key – the Treasury will need to be convinced of the value of quality-related research funding and the REF exercise itself as it seeks to cut billions out of BIS in the next Parliament. Let’s hope the grade inflation today doesn’t undermine the critical negotiations, as next year the sector will need to have a watertight case about the importance of this pot and the mechanism that distributes it.

Read John’s full piece here.

What do social policy at UCL, civil engineering at Cardiff and communications at LSE, have in common with psychology at the University of East London, history at Hertfordshire and English at Bedfordshire? Answer: they are all making a big difference to our lives, according to the 2014 REF. These are among the forty university departments judged to be 4* for impact across 100% of their submissions. This factoid hints at a long-standing concerns about the impact factor – can we really say the difference being made here is comparable across such very different fields?

Perhaps more pertinently, let’s consider what is being claimed here – that everything these forty departments does achieves (and we quote): ‘outstanding impacts in terms of their reach and significance’. No small feat, if we really believe it.

The reality is that what is being tested is not everything they do, but rather everything they submit. The other thing that unites the six departments above is that they all put fewer than 15 FTEs into the REF. This may tell us more about the benefits that can be garnered by making judicious (and no doubt internally difficult) choices at submission point. It also makes achievements like that of the medicine and public health departments at Bristol (both rated 100% 4* with 84 FTEs and 75 FTEs submitted respectively) look genuinely stunning.

Spare a thought for the five departments (we won’t name them, but you can look) that were rated 80% or more unclassified for impact (i.e. ‘the impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research’) and all put fewer than ten FTEs in for submission. What on earth were they thinking?

Today’s #REF2014 comment piece is from Wonkhe’s Editor in Chief Mark Leach:

HEFCE officials said yesterday that they had already started work on the tentative 2020 exercise. But the looming fight is the one to keep control over this pot of money at all (and given the scale of cuts announced in the autumn statement, perhaps even the pot itself), and so the principle of quality-related funding needs visible champions – from the outstanding small research groups that would not get a look in any other way, to the institutions that may fall out of political fashion and favor – and yes the big players that have the volume of research and are richly rewarded for it.

Read the full article here.

HESA have now published their contextual data.

The table shows the total full-time equivalents (FTE) of staff identified as eligible to be submitted to the REF 2014 by HE provider and REF Unit of Assessment.

Download it here.

We’ll analyse on Wonkhe as soon as we can.

We have completed fixing that buggy 4* column in our REF rankings by GPA – find it here.

It is the first Twitter REF, with 2008 RAE coming out before the popular social media platform really got started.

And #REF2014 is trending in the UK on Twitter, and has been most of the night. This shows the great interest in today’s results and the amount of people who the exercise touches – and a small fraction of them are likely to even be on Twitter.

Universities have taken to Twitter in large numbers to claim victory and highlight their successes (you won’t hear them talking about their disappointments) which has led to much humour and scepticism eg:

#REF2014 results morning shows that UK universities' ability to spin with numbers remains world-leading. #chooseyourownmeasure

— Stephen Curry (@Stephen_Curry) December 18, 2014

But there are plenty of people who are genuinely pleased for how they’ve done institutionally, or in a UoA – either for themselves or their colleagues and friends. I’ll round up some of the best stuff later on.

How did the mainstream media treat the REF results today?

It’s a tricky calculation for news editors – on the one hand a big exercise that matters to lots of people, but on the other – difficult to find stories that are newsworthy for a non-HE audience. Additionally, the data is big and complicated and there were only a few hours to turn it around before print deadlines yesterday.

From what I can see, the REF didn’t make it on to any front pages. The common theme has been the regional variation and London/Golden Triangle beating the rest. Newspapers like writing about the top players and people like reading about them – and with limited space it seems like a logical direction to go in – and the sector press picks up the heavy-duty analysis and commentary. So business as usual!

The BBC’s online story by Sean Couglan goes with the London v. the rest of UK story as London institutions have pulled themselves up at the expense of others.

The Guardian focus on the success of Oxford and London.

The Telegraph’s headline is ‘UCL breaks Oxbridge stranglehold on UK research’

The Independent leads with the North losing out or ‘taking a hammering’ in their words

The Times has ‘UCL beats Cambridge in research race’

The Scotsman’s headline is ‘How Scottish universities challenge world’s best’

A warm congratulations to City University who by our reckoning provided the best value for money in research over the last five years.

City University pulled out a 75.7% 4*+3* result from a 377 FTE submission, after getting just £37,000 per FTE submitted over the five year period in public research funding.

Read our whole story and analysis – The hidden bang-for-buck heroes of UK research.

As has been pointed out, our REF results has a flaw in the 4* table which appeared in the software after the GPA calculation was made and so does not affect the results at all. We’re just updating that column now so that it displays correctly.

The Higher Education Statistics Agency release their data at 10am today – the  Staff Record 2013/14 which “are intended to provide contextual information to the Research Excellence Framework 2014”.

This will bring a whole raft of important extra data that will allow us to measure intensity of research at different universities and which will add to the results again. And so the debate and analysis will continue!

Good morning, we’ll be live blogging the big REF 2014 results day all of Thursday. In the mean time find our first pieces of coverage for this morning:

1. Data: The full sector results – a ranking of institutions by grade point average

2. Analysis: REF results marred by fears of grade inflation

3. Data: The hidden-bang-for-buck heroes of UK research

4. Comment: The Research Excellence Framework: fascinating, flawed and essential

And from earlier this week:

Confessions of a concentratist 

Enjoy your REF, while you can