A league (table) of its own: why TEF must be more than a numbers game

Today the Times Higher Education published a league table predicting the results of the Teaching Excellence Framework based on data sources already available. This was a natural next step following the announcement of the planned TEF 2 metrics in the Technical Consultation including NSS scales, retention rates (collected for the UK Performance Indicators) and graduate employment destinations six months after graduation/DLHE results.

Anyone can cobble together a league table of their own based on publicly available data. However THE has tried to match its methodology as closely as possible to that set out in the TEF Technical Consultation, though its use of Q22 of NSS as a correlate of the NSS teaching quality scale (Q1-4) may be perceived as (ideologically and methodologically) problematic. Efforts have been made to weight the results according to student demographic, allowing for some levelling of the playing field between different types of institution.

The top ten from THE's analysis:

[table id=52 /]

Source: Times Higher Education

The big splash - that research-intensive institutions lose some of their lustre when research performance is stripped out of the equation - is already clear to any vice chancellor that has combed the various subject and teaching-only league tables already out there in hopes of finding some key feature to show their university in the best light.

The real issue here is whether the data will ultimately end up being the whole story for the TEF. The TEF must be more than a league table if it is going to be the spur to action for the improvement of the sector’s teaching quality, whatever you may feel about the robustness of the evidence base for the argument that improvement is required. There are several key features of TEF that differentiate it from a simple league table: the institutional commentary; the access and widening participation element; and the judgement of the independent TEF panels.

If the hairs on the back of your neck are already standing to attention, it’s time to apply to be a member of said panel, for which the deadline is 1 July 2016.

Cynics will argue that the institutional statements are likely to be of lower importance than metrics. One would indeed hope that the TEF will not become the prize for the best short-story fiction. The institutional submissions will, as in the equivalent REF submissions, give an opportunity for the provider to produce an account of its intentions in learning and teaching, put its data in context and submit its own additional evidence outside the limited metric framework.

The recent UCAS data release shows just how tricky it is to shape access and admissions data into league table form. Institutional averages tend to mask the intra-institutional patterns that often hold the key to the more important story and point the way to making a change. Small numbers lead to big inductive leaps. And only the strategic planners understand how average offer rates are calculated.

We can reasonably infer from the proposals to assemble TEF panels that there will not be a simple, uniform, way of assessing the data, but that there will be room for nuance and debate. The panels are meant to be independent, expert and deliberative. If their role is simply to sign off the data, then it is a very expensive way of designing a system. By introducing the qualitative element and putting providers into broad categories rather than splitting the hairs of a tenth of a percentage point here or there, the panels will have some latitude to consider the characteristics of the classification rather than simply to line up providers in numerical order. Once assessment at subject discipline level gets underway, we can expect the picture to be even more pleasingly complex and varied.

Expect the latest THE table to set the cat amongst the pigeons. Anticipate joy from universities at the top of the table, and claims of methodological error or data deficiency from those further down. But don’t miss the fact that TEF can be, and should be, in a whole different league.