You’d have to know what you were looking for.
The second-to-last paragraph of a pdf copy of an email sent to accountable officers on 14 April 2020 isn’t quite Douglas Adams’ legendary bottom of a locked filing cabinet in a disused lavatory with a sign on the door reading “beware of the leopard” – but I can perhaps be forgiven the comparison.
It seems that the Teaching Excellence and Student Outcomes Framework (the TEF) is now without a date for 2021. The consultation on the future of TEF (which will follow the publication of the Pearce Review, which was initially slated for the start of summer 2019) is still pending. And in these times of Covid-19 the Office for Students will be delaying the publication of the results of the subject TEF pilot.
Jo Johnson’s flagship league table analogue is facing its very own “long dark teatime of the soul”. Wildly popular with Jo and his former SpAD Iain Mansfield, and of intermittent interest to the kind of vice chancellors who like to plaster their campus signage in gold sparkle-paint, the ratings have proven less interesting to prospective students than was expected. Academics hate it by default, data analysts see it as another job, and the rest of us see it as a vague annoyance that makes for a good Wonkhe article (the total stands at just over 200).
From the looks of the scant paragraph we get it would seem that this is a temporary measure, and that there would be every expectation that we would get a new date in time. But it is not hard to imagine this indefinite pause as a quiet death for a basket of metrics that has failed to capture the imagination of the audience it sought.
TEF after Covid-19
I’ve covered the long lasting and differential impact of the pandemic on the sectors data elsewhere – in short we’ll be seeing 2019-20 and 2020-21 data in TEF for the next 10 years or so, and it will be weird – not the compensatable kind of weird either. The sheer logistics of running a TEF that is reliable even in its own terms are not to be underestimated.
Who knows what the 2020 National Student Survey will look like (yes, it is still going on), and how it will compare to previous or subsequent years. We don’t even yet know what Graduate Outcomes is meant to look like, and the impact of Covid-19 on student continuation rates is unclear.
But to focus on the practicalities is to ignore the main point, which is precisely that – what is the point of TEF?
Really, now?
TEF costs money to administer centrally – something in the region of £3m. It also has a cost and capacity impact on institutions and individuals. For this cost we would expect a concurrent benefit – the TEF rating itself as an output, and the positive impact on student recruitment as an outcome.
There’s no comparison with the league table industry – there the outcome is direct and quantifiable (clicks, sales, advertising spend, expensive consultancy opportunities, subscriptions) and is private. TEF, being a use of public funds, should really have a public impact.
And, as best we can tell, there is not one. Research by DfE and UCAS independently found that applicants are not making widespread use of TEF – with nearly half of all applicants not even having heard of TEF before applying. To be fair, applicants say (as applicants always do) that more information is more useful than less – the DfE found applicants are keen on the idea of a subject level TEF. But in terms of the costs of TEF, you would hope to see reliable evidence that the existing TEF is of use. There is not any.
Subject to consideration
The subject TEF, would – of course – require even more work. The review of the first set of pilots put the cost to the sector at a cool £14.8m when overheads are included (thankfully, the proposed teaching intensity component of subject TEF has been scrapped). We don’t get an estimate of the cost to the regulator, but I’d argue that the administration of 37 separate subject panels plus an overall panel would increase existing costs by an order of magnitude.
In the grand scheme of sector finances, these are not huge sums of money. Certainly there is no comparison to the costs of the REF. But to spend this amount of money without any path to public benefit, or even any administrative use case, would feel – in a time of austerity as the world returns to normal after Covid-19 – very strange indeed.
Is this it?
It could, of course, be argued that the current situation suits everyone involved perfectly well. The Government seems in no hurry to publish a review of TEF that is likely to have been less than glowing, the OfS doesn’t need to respond to it or consult on it (making it easier to integrate TEF into the mainstream of regulation), and TEF remains on pause forever. Nobody loses face, the decision to cancel TEF is never explicitly taken (so the government never goes back on a manifesto promise) but it is quietly understood that no future work will be done on an indicator that signally failed to indicate anything.
But, at least, people might use significance flags properly in future.
