TEF results – the chair’s post-match analysis

After a couple of years in which my predictions have had a rough time (EU referendum, US presidential election, UK General Election), I am delighted to have been proved right about one thing. I said that the TEF results would provide rich pickings for higher education analysts; the TEF has obviously not disappointed Wonkhe and nor has Wonkhe disappointed the TEF.

Over the course of 24 hours, the Wonkhe team have cut and diced the results, with, no doubt, more to come. It has been fascinating for me, as chair of the TEF panel to look at the analyses and comments which have emerged, so here is my own contribution to the post-match analysis.

There is still a line of criticism of the TEF which begins by arguing that it is not a measure of teaching and that the metric bases are inappropriate. In an obvious sense this is true: it is not a direct measure of teaching, being underpinned by a range of survey data. But neither I, nor, I think, anyone else has ever argued that the TEF is a direct measure of teaching. It is a measure based on some of the outcomes of teaching.

Those who argue that the exercise is invalidated by not including direct measures of lecture or seminar room teaching would appear to be arguing, at least implicitly, for an inspection based teaching assessment on the Ofsted schools model. This would be infinitely more intrusive and costly, and it would detract from the TEF’s focus on the relationships between institutional practices and institutional outcomes.

Others have argued in response to the data that whilst the principles are generally sound, the metrics are simply looking at the wrong things. As I’ve argued elsewhere, no social statistics are ever straightforward. If critics argue that metrics for the TEF are wrong, then there is some obligation on them to suggest alternative proxies. As yet, no-one has suggested alternatives.

We may need to refine and to develop the metric basis, but the metrics do focus on aspects which – time and again, as Edward Peck from Nottingham Trent has argued – reflect student concerns: whether the degree programme prepares students well for graduate employment, retention and completion rates, assessment and feedback, learning resources.

Metric led, not metric determined

There is a second set of issues which arise from analyses of the metrics. I yield to no-one in my admiration for the Wonkhe analysts, who have tried to use the results to throw light on the process.  But some of the data they have presented is simply baseless. The derivation of a league table from core metrics alone is a flawed exercise. There was no ‘initial award’. It would be rather like scoring a football league based on the scores after thirty minutes.

The TEF specification was clear: initial hypotheses were derived from core metrics; the hypotheses were refined based on the split metrics exploring institutional performance in relation to defined sub-populations; contextual data on student composition and on regional graduate labour markets was analysed; all this was tensioned against provider submissions which were analysed for internal coherence, evidential underpinning and measures of impact on outcomes.

The TEF assessors and panel worked intensively on both quantitative and qualitative data. There should be no surprise that institutions with similar core metrics might emerge with different TEF ratings. The TEF is metric-led, not metric determined – this is why one institution was able to move from Bronze through to Gold. In a higher education system which retains its student assessment based on firsts, 2:1s, 2:2s and thirds, there should be no surprises that there are boundary issues in making borderline judgements.

Refining benchmarks

Of more interest are the patterns which Wonkhe has unearthed on relationships to other datasets. The panel took no cognisance of mission group membership – which is, after all, simply an arbitrary collection of clubs, nor of REF scores, nor of any of the extraneous measures that Wonkhe have set against the TEF.

There appear, in some of the commentaries, to be two different critiques. There is one which is that the TEF has paid insufficient attention to the reputation and facilities in universities with high absolute scores. This critique misses the point about the focus on outcomes. It argues that the TEF has been too disruptive of established hierarchies.

There is another critique which is that the TEF has not been disruptive enough, and that not enough cognisance has been taken of context. There is a case for re-examining some of the technical issues here, and notably on whether POLAR is helpful in terms of employment. The TEF has gone some way to putting in place a relative, rather than absolute measure of university and college performance, but there is more to do, and reviewing benchmarking approaches is one of those.

None of the wonkery should detract from some important issues. We can benchmark for intake characteristics, but disadvantage is complex. The intersectionalities – of one disadvantage measure with provider type, other provider characteristics, as well as with other student characteristics – are complex but play an important role in interpreting the profile of outcomes. TEF has helped to factor out some of this, but inequalities are embedded in university intakes and in university student cohorts.

One of the most important lessons of the TEF for me, which has not yet been picked up sufficiently, is that whilst universities have been impressive at widening participation they have been less assiduous in combatting the impact of disadvantage after students enroll.

Most important of all, is the fact that the TEF has raised the profile of teaching. I’ve been struck by some comments I’ve received from universities in all three categories about how the TEF has focused attention on things which need to be done better, and that the TEF itself, and its focus on outcomes, have shifted perceptions and attention in universities. Surely, that is something worth doing.