I’d very much like to buy whoever it was in the Office for Students who decided that TEF4 metrics data should be released to the public in early January a drink.
You’ll recall last year, and the year before, Wonkhe has spent “TEF day” (late June) diving deep into the underpinning data to make sense of what has happened and why. Last year I had a go at running TEF3 under TEF2 rules the night before, to give people an idea of what to expect – but it was a lot of hard work just to be called out for supposedly making a “schoolboy error” by Chris Husbands. (I didn’t, for the record – I just didn’t have all the data I needed to model the latter stages of assessment).
Now – with vast swathes of data made available earlier in the year – we get to do a large chunk of analysis in early January. It’s not perfect – the data covers English entrants only, and all eligible institutions not those who will definitely enter – but it allows us to run at least the very first part of the process and take a sneaky look as to what their results might look like.
In all, of 320 eligible institutions, 31 have a Gold initial hypothesis (based on part 1a of the process), 198 a Silver, and 91 a Bronze – a far lower proportion of Gold awards than previous TEF iterations. Please not that these are not “predictions”, this is just the starting point on which the TEF panel will work their magic.
First up, here’s the overall change between the new initial hypotheses and the TEF awards that institutions currently hold.
Your first thought may well be “Wow, that’s a lot fewer Gold TEF awards”, and you’d be right. Of the English institutions eligible to become involved in TEF4, the initial hypotheses suggest a lot less Gold awards would be handed out. Notably not one single Russell Group institution sees a Gold award.
On the visualisation you can use the highlighter box to look for any institution you are interested in, and or use the group and regional filters to take a look across your preferred cut of the sector.
I’ve also build a more detailed look-up on an institutional basis.
Use the box at the bottom to choose your institution.
Yet more fun with flags
The way TEF4 awards will been generated doesn’t differ vastly from the TEF3 methodology. This makes the two comparable – TEF2 and TEF3 were not. The first stage sees the calculation of an initial hypothesis, based entirely on the core metrics for the dominant mode of provision (full- or part-time). Each of the core metrics is compared to a benchmark, and a significance flag is generated, showing the importance of the difference between the two.
These flags are displayed in the data as double positive (++) through to double negative (- -). The use of this gradation is frequently criticised as it did not cut the statistical mustard – a single significance flag in the sector key performance indicators is equivalent to a double flag for TEF – which makes the TEF single significance flag not especially, erm, significant.
These days single and double flags are treated the same for the purposes of calculating the hypothesis – but the six core metrics are not. The NSS derived metrics – for teaching, academic support, and assessment/feedback – are weighted at half that of the other metrics.
So the formula (as set out by DfE, paragraph 7.11) goes as follows
Positive flags on metrics worth 2.5 or more, and no negative flags – Gold
Negative flags on metrics worth 1.5 or more – Bronze
Otherwise – Silver
This gets us to our initial hypothesis – step 1a of the TEF assessment process. And I’ll add one little caveat – institutions with similar numbers of full and part time students will have a 1a calculated both modes of delivery, whereas I’ve just done the one indicated as the primary mode by OfS.
What happens next
In some cases nothing – if the initial hypothesis is looking clear cut at this stage it is unlikely the award will change (paragraph 7.13 in the document above). Otherwise we move onto step 1b, which looks at these initial hypothesis in terms of absolute values and split metrics.
Beyond this, step 2 brings in factors that could affect performances against core and split metrics – the supplementary metrics and anything relevant in the provider statement. The three supplementary metrics are grade inflation (which I looked at last month) and two metrics derived from LEO, which I plot here.
These two graphs work in the same way as the overall change graph – but I’ve included the two DLHE-derived measures in the tooltip as these are the ones most likely to be affected.
I’ve always been suspicious of the utility of the provider statements – but a failure to submit any substantive additional evidence in the statement now means metrics are assessed much more harshly. So even if your metrics are looking great, an insubstantial statement could see you awarded Bronze for as much as one negative flag in any core, split, or LEO supplementary metric. It looks like this is being done to address the issue of people submitting statements that say “Our metrics look like a Gold, so we’ll have that – thank you” as memorably happened in one instance during TEF2.
Finally, Step 3 is the holistic judgement – a final check that the awards look right against the descriptions of each award level (paragraph 7.71).
This data will be a huge help to institutions in planning their statements – for the entire of the week they have left to submit before the 17 January deadline. They’ve had their own data since October – but this will be the first sight they’ll have of the metrics for comparator institutions. But after then, assuming no verification queries arise, it’s the long wait til early June.
If they choose to enter the competition, that is. Existing awards will be rolled over for an extra year to allow time to build a subject TEF driven system, and/or respond to the conclusions of the statutory review of TEF. Institutions holding Gold awards may choose to stick rather than twist at this point.
There’s a lot of other data there – and a lot to play with over the coming weeks. A little taste of June busting out all over the cold days of January.