TEF results – the chair’s post-match analysis

Image: Shutterstock

After a couple of years in which my predictions have had a rough time (EU referendum, US presidential election, UK General Election), I am delighted to have been proved right about one thing. I said that the TEF results would provide rich pickings for higher education analysts; the TEF has obviously not disappointed Wonkhe and nor has Wonkhe disappointed the TEF.

Over the course of 24 hours, the Wonkhe team have cut and diced the results, with, no doubt, more to come. It has been fascinating for me, as chair of the TEF panel to look at the analyses and comments which have emerged, so here is my own contribution to the post-match analysis.

There is still a line of criticism of the TEF which begins by arguing that it is not a measure of teaching and that the metric bases are inappropriate. In an obvious sense this is true: it is not a direct measure of teaching, being underpinned by a range of survey data. But neither I, nor, I think, anyone else has ever argued that the TEF is a direct measure of teaching. It is a measure based on some of the outcomes of teaching.

Those who argue that the exercise is invalidated by not including direct measures of lecture or seminar room teaching would appear to be arguing, at least implicitly, for an inspection based teaching assessment on the Ofsted schools model. This would be infinitely more intrusive and costly, and it would detract from the TEF’s focus on the relationships between institutional practices and institutional outcomes.

Others have argued in response to the data that whilst the principles are generally sound, the metrics are simply looking at the wrong things. As I’ve argued elsewhere, no social statistics are ever straightforward. If critics argue that metrics for the TEF are wrong, then there is some obligation on them to suggest alternative proxies. As yet, no-one has suggested alternatives.

We may need to refine and to develop the metric basis, but the metrics do focus on aspects which – time and again, as Edward Peck from Nottingham Trent has argued – reflect student concerns: whether the degree programme prepares students well for graduate employment, retention and completion rates, assessment and feedback, learning resources.

Metric led, not metric determined

There is a second set of issues which arise from analyses of the metrics. I yield to no-one in my admiration for the Wonkhe analysts, who have tried to use the results to throw light on the process.  But some of the data they have presented is simply baseless. The derivation of a league table from core metrics alone is a flawed exercise. There was no ‘initial award’. It would be rather like scoring a football league based on the scores after thirty minutes.

The TEF specification was clear: initial hypotheses were derived from core metrics; the hypotheses were refined based on the split metrics exploring institutional performance in relation to defined sub-populations; contextual data on student composition and on regional graduate labour markets was analysed; all this was tensioned against provider submissions which were analysed for internal coherence, evidential underpinning and measures of impact on outcomes.

The TEF assessors and panel worked intensively on both quantitative and qualitative data. There should be no surprise that institutions with similar core metrics might emerge with different TEF ratings. The TEF is metric-led, not metric determined – this is why one institution was able to move from Bronze through to Gold. In a higher education system which retains its student assessment based on firsts, 2:1s, 2:2s and thirds, there should be no surprises that there are boundary issues in making borderline judgements.

Refining benchmarks

Of more interest are the patterns which Wonkhe has unearthed on relationships to other datasets. The panel took no cognisance of mission group membership – which is, after all, simply an arbitrary collection of clubs, nor of REF scores, nor of any of the extraneous measures that Wonkhe have set against the TEF.

There appear, in some of the commentaries, to be two different critiques. There is one which is that the TEF has paid insufficient attention to the reputation and facilities in universities with high absolute scores. This critique misses the point about the focus on outcomes. It argues that the TEF has been too disruptive of established hierarchies.

There is another critique which is that the TEF has not been disruptive enough, and that not enough cognisance has been taken of context. There is a case for re-examining some of the technical issues here, and notably on whether POLAR is helpful in terms of employment. The TEF has gone some way to putting in place a relative, rather than absolute measure of university and college performance, but there is more to do, and reviewing benchmarking approaches is one of those.

None of the wonkery should detract from some important issues. We can benchmark for intake characteristics, but disadvantage is complex. The intersectionalities – of one disadvantage measure with provider type, other provider characteristics, as well as with other student characteristics – are complex but play an important role in interpreting the profile of outcomes. TEF has helped to factor out some of this, but inequalities are embedded in university intakes and in university student cohorts.

One of the most important lessons of the TEF for me, which has not yet been picked up sufficiently, is that whilst universities have been impressive at widening participation they have been less assiduous in combatting the impact of disadvantage after students enroll.

Most important of all, is the fact that the TEF has raised the profile of teaching. I’ve been struck by some comments I’ve received from universities in all three categories about how the TEF has focused attention on things which need to be done better, and that the TEF itself, and its focus on outcomes, have shifted perceptions and attention in universities. Surely, that is something worth doing.

9 thoughts on “TEF results – the chair’s post-match analysis”

  1. Adam Smith says:

    ‘This would be infinitely more intrusive and costly, and it would detract from the TEF’s focus on the relationships between institutional practices and institutional outcomes’.

    Intrusive and costly? That sounds awful. I’d prefer my work to be judged by a few tangentially related statistics gathered on the cheap, by people who have never visited my institution, never mind my classroom. Otherwise people may get an impression of what it’s actually like to be taught here, rather than a hazy picture of ‘institutional practices and outcomes’.

  2. Matthew says:

    Chris Husbands is obviously right to say that an Ofsted-style TEF would be awful, but I think he’s dead wrong to say that you can’t legitimately criticise the validity of a measure without proposing a better alternative. My reading of the education literature is that no one has successfully developed a way of evaluating teaching quality with high levels of validity and reliability. If I’m right about this (happy to receive pointers to counter evidence), then Husbands’s argument falls apart. If we don’t know how to do something properly, it doesn’t necessarily mean that we should do it badly. Maybe we simply shouldn’t do it at all, or at least not until we’ve worked out how to.

  3. Matt McCallum says:

    “But neither I, nor, I think, anyone else has ever argued that the TEF is a direct measure of teaching. It is a measure based on some of the outcomes of teaching.”

    Then why not call the exercise the “Some of the Possible Outcomes of Teaching Framework” rather than allowing every media outlet (and crucially potential students) to infer from the name that this bears some direct relationship to the quality or otherwise of teaching delivery?

  4. dpattie says:

    You’ll be aware by now that the TEF results have been widely reported as a direct reflection of the state of teaching in UK institutions. You might wish to argue that, as you put it, neither ‘I , nor, I think, anyone else has ever argued that the TEF is a direct measure of teaching’; but in the real world, that is precisely the assumption that has been made.

  5. CE says:

    Or better yet, call it ‘Teaching Outcome Framework’. Makes for an interesting acronym at the very least.

  6. deevybee says:

    The argument put forward by Chris Husbands amounts to saying that an unsatisfactory TEF is better than no TEF. Thus the default assumption is that we must have a TEF come what may, and nobody has come up with anything better, so we must go along with what we have, warts and all.

    I regard this as highly irresponsible, given that income-raising powers and institutional reputations stand or fall according to the TEF rankings. The question we should be asking is not ‘is this the best way of ranking institutions that we can achieve’, but ‘are the potential harms associated with introducing the TEF outweighed by its advantages’. I don’t think that case has been made, and until it is, we should continue as we have done for many years, relying on the QAA to ensure courses meet adequate standards, and ensuring that institutions continue to publish information on course content, completion rates and, yes, even NSS, to provide students with information about the institutions they are considering applying to.

  7. Daniel Katz says:

    Well said, Matt.

  8. dpattie says:

    This article answers all the points you make; and provides an appropriately damning verdict on the whole exercise.

    http://cdbu.org.uk/tef-and-the-reputation-of-uk-higher-education/

  9. Martin says:

    TEF is a clumsy attempt to provide administrators with a box to tick. As often with such misguided ranking systems based on flimsy criteria, it will quite quickly be open to blatant manipulation as Universities focus on playing the system rather than actually teaching.

Leave a Reply

Comments 9 See all

  1. Martin View
  2. dpattie View
  3. Daniel Katz View
  4. deevybee View