Rewrite those TEF tables! – the University of East Anglia has succeeded in appealing their award level and has turned silver into gold. As the only successful appellant in the year 2 competition, what could have made their case compelling?
From the TEF guidance, we know that “providers will be able to appeal their TEF outcome on the basis of a significant procedural irregularity in the consideration of their TEF application. A provider will not be able to appeal or challenge the academic judgement of the Panel or any founding principle of the TEF.”
So TEF appeals are permitted only on a very narrow set of grounds – the nature of the rules and processes are not up for grabs, only the way they have been applied. Rather than claiming they had received the wrong award, UEA would have to show that the process had not been correctly followed – a requirement that makes it harder to appeal but could mean that any successful appeal could have a much wider applicability. If the process had been followed incorrectly, it must have been followed incorrectly for all institutions under similar conditions.
The TEF guidance suggests that appeals can be made under only three sets of circumstances – if the question concerns:
- whether to accept a data amendment request
- whether the provider is eligible for a TEF Year Two award
- the panel’s judgement on the rating awarded to the provider.
As we know UEA are eligible for the TEF (they did get a Silver initially), the issue must have been with the one of the other two.
More fun with flags
Looking first at UEA’s core significance flags, we see double-positive (++) for teaching and academic support, reflecting continued success in the National Student Survey. But there are no flags for assessment and feedback, non-continuation or either of the DLHE-derived metrics – and two positive flags isn’t enough to tip them into automatic gold.
Diving into the splits we see a similar pattern, the only hint of a negative flag for most students comes from a slight issue with highly skilled employment in year 3 of the assessed UEA statistics.
But there does appear to be an issue with part-time students and non-continuation – though a row of single negative flags appears to be the only findings that has been presented for part time, with NSS and DLHE measures not reportable due to a low response rate (less than 59.5% for part time for the DLHE measures, less than 50% for the NSS measures).
Non-continuation is derived from HESA data, so there is no response rate to worry about, but it is notable that we see a very low cohort size – just 695 part-time undergraduate students covered from UEA (that’s 6.8% of the total student body).
The only argument to be made here concerning award level would be that this small number of part-time students:
- Had circumstances that made the non-continuation designation invalid
- Which was significant enough that it lowered the main non-continuation flag
- And thus prevented UEA from achieving the three positive core flags (or equivalent panel judgement) that would suggest a Gold award should be made.
Going to the panel
The initial panel statement suggested that “Retention of part-time students is below the provider’s benchmark, and the Panel deemed this to be partially addressed in the provider submission”. In statement-ese this suggests that it was felt by the panel that the argument made by the UEA concerning this group of students was not sufficient to alter a flag.
That last point is important – this was a panel decision not to take into account an argument made in an institutional submission. The decision to award a Silver (as with all other TEF decisions) was a panel decision based on their interpretation of the metrics.
UEA has confirmed that their appeal “focused on misinterpretation of information relating to part-time students”. So what did UEA say about the issue and why didn’t it (initially) convince the panel?
Part-time students are only addressed once in the provider statement, and an argument is made based on the classification of the students in question as part-time undergraduate degree students.
To quote in full from the submission:
Our data set for part-time students was too small to generate metrics in most areas, because the University’s UG provision is overwhelmingly full-time. One exception to this is the part-time non-continuation rate, where we have a negative flag. This apparent under-performance is overwhelmingly confined to one School (Health Sciences). Importantly, a number of students in this School were registered on full awards, but always intended to complete only a certain number of modules as part of their professional development as nurses. Our reading of this flag is that it may well be a matter of an inappropriate mode of registration rather than a lack of support leading to a change of intention.
So students were registered on full awards, but only intended to complete certain modules – in other words, these students should have been coded as “other UG” rather than “first degree” as they were not studying towards the qualification in question.
Significance?
Handily, HESA releases data on part time non-completions within the non-continuation UKPIs – part-time students are handled on the “two year non-completion” table (T3e). These show that in 2015/16 there were 15 part time entrants under 30, and a further 15 aged 30 or over. With person-based statistics like this, HESA round to the nearest 5 on publication to avoid identifying individual students – which is laudable, but unhelpful for these purposes.
For all PT first degree entrants, it is reported that of the 30 that entered UEA in 2013-14, 15 were either still studying or had achieved their award by 2015-16. This isn’t a great retention rate by any standards, but we need to bear in mind that the numbers are tiny – and were equally tiny in previous years.
So tiny, in fact, that it would be very difficult to make the argument that this rate would make any difference to the main flag.
So what might have happened?
The fact that we can see these students on HESA suggests that the students were incorrectly coded at the point the return was made by UEA – which doesn’t feel like a problem in the TEF process. Institutions are, of course, able to correct their HESA data whenever an issue becomes apparent – when data affects a TEF Award this becomes far more urgent.
For TEF this would also need a data amendment request, which you’ll recall is one of the possible grounds for appeal if this was not accepted. But para 5.46a suggests it is required that “All the requested amendments are necessary due to widespread and significant errors in the underlying data, rather than reinterpretation of the data (for example, recategorisation).” and 5.46c confirms requests would only be accepted if “The amended data makes a material difference to the TEF metrics for one or more of the providers that have agreed to the data amendments”.
It is difficult to see how the classification of this tiny set of students would have affected the Award. The University of East Anglia had a very good set of metrics, and a very good provider statement. But, in the judgement of the panel, they narrowly missed out on gold – even on significance flags alone they missed out. The revised panel comments remove the section on PT students entirely, but no change has been made to any flags.
The opacity of the appeals process means that we can only speculate (as above) as to what might have happened – in this instance our best guess is that the panel’s judgement has been brought into question. It is hard to see how not sharing the full details makes for a more robust TEF, or does anything other than cast doubt over the confidence that we can place in the judgement of the panel.
