TEF: The Incredible Machine remixed

In music the idea of a “remix” can cover a multitude of sins – everything from the regular song with more cowbell and an extra chorus, through the guts of the vocal backed by the fashionable beat and bass of the day, all the way up to an entirely new (and often preferable) recording which somehow has the same name despite sharing none of the same components.

TEF3 (The TEaSOF Mix) stands somewhere in the middle of this – with new and modish nods to contemporary concerns like grade inflation and whatever it is we are currently pretending LEO tells us – but is still struggling to win mass affection and chart success. The guts of TEF2 are there under the new frills, and we are still using the same sources to assess the same things, but has the competition changed to the extent that it no longer comparable with the previous iteration?

We would argue that it has.

The change in the weighting of the National Student Survey components mean that, ironically, the voice of those being taught is substantially weaker this time around. Each NSS metric – there are three – is now worth half that of the other three metrics, which use data from the Destination of Leavers in Higher Education (DLHE) survey and a HESA/ILR data on non-completion.

You can download the diagram as an image here or as a pdf here.

I got new rules

There’s a clear benefit to the TEF performance of universities that have a history of poor NSS results, and of graduates getting good jobs. Several of these institutions performed less well than they might have hoped in TEF2, and it is hardly donning a tin-foil hat to suggest that this change may well have been put in place to improve their reported performance.

Other institutions – those delivering high-quality and well-received teaching to students from non-traditional backgrounds – will find it harder. The argument that a good student experience is enough to get a good student experience grade has become harder to make, as the remorseless power of social background continues to have an enormous effect on employment.

Of course – not all institutions will re-enter TEF in this year – one of the features of the competition is that awards are multi-year. In TEF2 these were three-year awards, this remains in TEF3, but later iterations (should we get there) will extend this further. Quite how far is a matter currently under consultation. Future TEFs are likely to be underpinned by subject area TEF ratings, changing the terms and meanings of the ratings even further.

Review to a kill?

We are awaiting, of course, the establishment of the statutory review of TEF, as set out in sections 25 and 26 of the Higher Education and Research Act. A significant concession to concerns raised during the passage of the Act – particularly in the House of Lords – this review will have the power to determine whether TEF is valid and in the public interest.

The review will produce an independent, and likely high-profile, report which – importantly – will be released after the post-18 review. As TEF was originally (before a significant change in parliamentary arithmetic) supposed to be linked to fee levels, it will be interesting to see the options for making this link reviewed.

As of April 1st, fee cap rises have to be voted on in both houses of Parliament. Regular recent attempts by the opposition to force related votes demonstrate the depth of feeling that exists on the issue after the snap election; neither house is currently likely to vote to raise student fees.

But it seems more likely the review will take a more fundamental look at the functioning of the scheme.

A solution in search of a problem

So what – honestly – is TEF now for? It doesn’t adequately capture the student experience or the quality of teaching. It does not confer any benefit – other than a questionable marketing boost – to providers, and there is no evidence that students are making serious use of it to choose courses, universities, or colleges.

Internationally, concerns have already been raised that the three-level ratings are confusing – it’s been widely reported that “Bronze” institutions are often not considered to meet the UK’s laudably stringent teaching quality thresholds.

And it is not even a reliable time series – a TEF3 Gold is now achievable by an institution that would not have passed the test under TEF2 rules. Later iterations may well be built “ground up” from subject TEF assessments, once again changing the rules fundamentally. Let’s not even mention TEF1 (it’s OK, no-one ever does) in this context.

Murder on the dancefloor

The TEaSOF mix of TEF is not one the producers should be proud of. Where it is effective, it is cynically chasing headlines. Where it is useful, it is reinforcing existing sector and socio-cultural hierarchies that have nothing at all to do with teaching qualities. The source material was never that great, but a judicious ear and a talented hand could have produced something that would fill dancefloors at meetings of senior institutional staff. As things stand – we’d have to say we preferred the early stuff.

You can download the diagram as an image here or as a pdf here.