Nearly half of all first class honours degrees are “unexplained”.
Does this mean that the students holding these qualifications haven’t done the work? Does this mean that the universities and colleges in question have been marking their work wrong?
Or does this mean students have, with the support of their institution, transcended the expectations placed upon them by their pre-university background?
This finding has a surprising amount in common with the pandemic-era “mutant algorithm” that ruined A level results for many – basing expectations about the performance of today’s students on the historic impact of disadvantage.
The word “unexplained” strives for neutrality – OfS experimented with “unwarranted” a couple of years back and were rightly called out for it. “Unexplained” should simply mean “we don’t have anything in our model that explains what we are seeing”: and in any proper science subject (cover your ears, economists) fifty per cent of observations being unexplained by a model would point to the need for a better model.
By model I mean that your likelihood of getting a first or 2:1 is explained by the provider you are at and the subject area you are studying in (based on historic data), the year you graduated in (based on the performance of your wider cohort), plus the qualifications that you held – and your age – when you entered higher education (if you didn’t do as well at A level, you are deemed less likely to get a first or 2:1 – but if you are a mature student you are more likely to get a first or 2:1).
Modelling
OfS has done the maths so we don’t have to – here’s what that looks like:
What you are looking at are coefficient estimates – big words that mean “this is what OfS has calculated as the impact these factors have on your chances of getting a first or 2:1”. The reference values – a young student, three As at A level or above, business studies, 2010-11 – are at zero – larger than zero means that a characteristic means you have more chance of getting a first or 2:1, below zero means less chance.
We can immediately see that every year since 2010-11 show students as having more chance of getting a first or 2:1. Any qualifications other than three As at A level have a huge negative impact on your chances, whereas subjects have small effects in both directions.
To get the estimate for a given year for a provider, OfS takes all these factors into account, based on the make up of the cohort at that provider in that year. If you’ve got a different entry tariff, a different mix of subjects, or more mature students (and you well might) the estimate will change.
Reality
The upshot of all this is that if you are the kind of provider who actively looks to recruit among people who may not have great A levels, you are expected to hand out less firsts and upper seconds than someone who asks for three As. Why? – because, in the past, people with three As have been awarded better degrees.
OfS is not all that concerned that at, say, University College London 93 per cent of undergraduates got either a first or upper second (it is a little bit concerned, it would have expected 86.5 per cent). It is a lot more concerned that the University of Chichester awarded 77.2 per cent of graduates a first or upper second – it expected this number to be 60.9 per cent.
My go to example is the University of Leicester. It has (commendably) shifted from a quasi-Russell group high tariff approach to recruitment, to a greater focus on its local area. And it has done so while maintaining a similar proportion of graduates getting firsts and upper seconds – 80.3 per cent in 2011-12, 79.9 per cent last year now. However the OfS model would have expected 76.7 per cent in 2011-12, and just 67.1 now.
To me, the fact that Leicester has become more inclusive in recruitment while maintaining similar levels of achievement is cause for celebration – this is exactly what a good civic university should be doing.
Instead – we get the “unexplained” narrative, which instead hints that something untoward may be happening, and that regulatory action may be forthcoming (whatever happened to those investigations launched in 2022?)
The animating idea here is that students would be more attracted to providers that are more likely to offer them a “good degree” that they don’t deserve. Quite how applicants are meant to identify these shady organisations (it’s been a long time since the proportion of firsts and upper seconds featured in a league table, even if they are aware of Discover Uni it isn’t on there) is unclear.
Consider, also, Buckinghamshire New University. It has always been focused on supporting local graduates who may not otherwise have succeeded in higher education, and it is pretty good at that. But according to the OfS’ own modelling, it awarded nearly 6 percentage points too few firsts and upper seconds last year. Does this mean that BNU is under-rewarding some very hard work against difficult odds? Is OfS suggesting that graduates can use this data to appeal their result?
There is, and should be, a role for regulation in ensuring that higher education providers are maintaining academic standards. I am pretty sure that this data, and this model, are not the best tools for that particular job.
Update
Interesting question from Rose below – we do get given this data, so let’s plot it! The actual proportion for the number of people who got three DDDs at A level who got a first is 24.4 per cent. I can only assume that The Times has made a data error – the figure is 72 per cent for a first or 2:1.
On this chart the degree classification menu lets you see firsts or firsts and 2:1s via the drop down, and you click on the entry qualification of interest to highlight the line. It looks like the awarding gap has narrowed over time – which is good if you happen to like universities and believe they are doing valuable social mobility work, or bad if you want to scream “grade inflation”.
The article makes the point that only explaining 50% of the variation observed indicates serious flaws in the model – maybe what actually happens during their studies has some impact on students’ outcomes, but who knows, eh?
Rather than acknowledge that the model may be wrong though, the OfS have gone with the bold approach of insisting that it must actually be reality that is the problem.
This is a misreading of the article, half of the variation between years is unexplained, the proportion of variation explained in the base year will be much higher, you’d have to check back to find it. The point of the work is to highlight how a model which precited outcomes well no longer does so there must be something else going on that the available data doesn’t include that may be better teaching, more motivated students or it may be something negative such as grade inflation. What the model tells you is that something the OfS cannot measure has changed and as a responsible awarding body you’d expect a University to have pretty good handle on what it was and be able to explain it.
The model only tells you that if it’s a good model in the first place.
In it’s original form, the predictors of the model included demographic characteristics, including ethnicity. They have not been used for the last two years.
One consequence of their inclusion meant that a greater proportion of good degrees awarded to BAME students were found to be unexplained when compared to White students, other factors being held equal.
That’s not a good model.
DK, you’ll have seen that the Times and Telegraph have run articles on the quite specific findings that three-quarters of students who achieved three Ds at A-Level get a first. My instinctive reaction to this is ‘good for them’. However, I wondered if you had any more specific insights into this stat? Does it play out across the grades? And is there evidence the HE can be a ‘leveller’?
The Telegraph article I read says “nearly one quarter” which is correct?
Is there any analysis of the difference between students educated in the state sector or the independent sector?
Not in this release, no. But we know private education generally correlates with better A level grades.
This “unexplained” behaviour is rather straightforward to explain if you worked in academia for any length of time and marked student work as you would probably have experienced any of the following:
1. You are heavily discouraged to mark fairly (and fail) bottom lot of students (targets on how many have to pass is like 85-95%), thus by passing them on D’s you compensate and push everyone else’s grades higher, plus you have to hit internal targets of how many 1sts and 2.1s you meant to have
2. A-level work from one university might be a C/B equivalent of work from other university as some teaching teams add to grades students journey and development (personal growth) rather than just focusing on face value of the submission, which is what industry will look at, thus doing disservice to students as they have unrealistic expectations and understanding of quality of their work, and then cannot find work post graduation as they are simply not good enough.
3. Students pay, students get – not getting 1st often ends up with student complains and challenges towards grades – we do not have time nor energy for this so overcompensate grades
4. Academic services have improved their work and support over last two decades tremendously and do amazing work to bring disadvantage students to a C/B level.
5. Cheaply (or even freely) and instantly available materials (Youtube, udemy, Khan Academy etc.) really help many students to engage with their courses differently than just through books – thus helping neurodivergent bunch that was left behind at pre-universiyy level.
I don’t recognise 1-3 here at all
Maybe more universities are finally using the full range of marks across many disciplines? Categorical marking is becoming more popular. Why should a mathematician for example be able to achieve 100% but this is not possible for humanities subjects for example? I have seen it throughout my entire career where these disciplines are disadvantaged.