The Institute for Fiscal Studies (IFS) has, perhaps, been the greatest beneficiary of the Department for Education’s decade-long experiment with Longitudinal Education Outcomes (LEO) data.
It conducted the first analysis, and has been the most prominent user of this dataset ever since.
LEO, over this period, has been a fascinating (if flawed) dataset that has delighted both researchers and people who like to read articles about graduate salaries in broadsheet newspapers – but despite much anticipation it has never really made it (bar a brief period in the TEF orbit and a mystifying appearance on Discover Uni) into the gilded echelons of regulatory metrics.
All this is despite a substantial body of ministerial and commentariat opinion suggesting that graduate salaries were both a useful and reliable indicator of the quality of teaching offered on a given course at a given university. Clearly something had to give, and this being the last government the “I reckon” contingent was always going to win out over people who actually understood the available numbers. And you thought “vibes-based policy” was a Labour innovation.
B3 plus one
To this end, in the dying days of the last government, we learned from Kemi Badenoch (of all people) that DfE commissioned IFS to examine ways to “accurately and fairly” incorporate an improved earnings metric into the way OfS identifies “poor quality courses” (which at the time was understood to mean the B3 conditions).
Crucially, it will control for students’ backgrounds to avoid unfairly penalising providers that admit students from disadvantaged backgrounds who may achieve poorer salary outcomes in the future for reasons beyond a provider’s control, which may include geographic, financial and cultural factors.
Just over a year later, during recess, the IFS report was published. And guess what?
A key outcome from the work is that we do on balance recommend using an earnings metric calculated from administrative data in regulation. We believe that the best option would be to integrate this new metric into the current OfS regulatory framework in such a way that it complements – rather than replaces – the existing B3 progression metric.
Devil in the detail
Now, this isn’t your grandparents’ LEO metric. Previous presentations of everyone’s favourite (tax and social security derived) administrative data on salaries have focused on the median (and quartiles), and – at their best – have controlled for some combination of sex, provider, (level 2 CAH) subject area, background characteristics. But never all of them – don’t make me tap the “Kernohan’s Law” sign.
We know we need to control for these factors because each iteration of LEO data has demonstrated statistically significant differences in graduate outcomes along these axes. Even Kemi was wise enough to specify that a salary metric would need to control for graduates’ background (including “geographic, financial and cultural factors”) in order to “avoid unfairly penalising providers that admit students from disadvantaged backgrounds”.
The IFS baseline proposal is to take the highest earnings of graduates three to five years after graduation (pooling two cohorts to ensure statistical robustness) does recommend an institution-specific benchmark – as a means of examining the impact of choosing a course at a particular institution as against a course in a similar area elsewhere. This is at odds with the OfS’ ideological commitment not to use benchmarks in B3, but would control for prior attainment and demographics in a similar way to the existing TEF metrics. Indeed, as we will see, IFS has identified a strong correlation between prior attainment and earnings.
However, the report recommends against controlling for the location (region or travel-to-work area) of graduates during the sample period – the reason given is that “enabling geographic mobility is one channel through which a course may impact graduate earnings”. To unpack that a little, what IFS is saying is that graduates may be able to move to (for example) London because of what they learned on their course and the opportunities it unlocked – which is probably true as it goes, but has the side effect of painting any decision not to move in order to maximise earnings (for example for family, cultural belonging, affordability, or career goal reasons) as a failing of the institution and course.
And let’s not leave that word “course” dangling. For IFS (and most regulatory purposes) a “course” is a combination of subject area (in this case CAH2) and provider. This is not the same thing as the course that an applicant actually chooses – in that a pharmacy graduate will see salary data lumped in with pharmacology, with no attempt to disaggregate an actual regulated profession from a vague interest in the chemicals people put into their bodies to make them feel better. Likewise the myriad species of engineer (mechanical? civil? bioengineering?) are lumped together unhelpfully.
The recommendations holds only for full-time first degree provision – the amount of graduate salary data collected would be too small to be useful (both because of the small numbers of students on these routes, and the preponderance of of mature students for whom school and prior attainment records may not be available)
Salaries and earnings
I am enthused by the idea of removing the impact of lower salaries (the suggestion is below £3,000 in each year of the sample). Those who have used the LEO dataset will confirm that the focus on earnings rather than salary is unhelpful, in that without proper caveats the earnings of someone working part-time (perhaps for family reasons, or to support artistic practice, or to support a start-up initiative, or because of a disability) look very low and can pull down median values.
I would actually go further – £3,000 is too low, and I think it would be fair to assume that earnings below full-time earnings national minimum wage for the year suggest part-time work or an incomplete year of work. IFS has also considered this and suggests that if this decision is made the share of graduates excluded on this basis should be published, which makes sense.
The three to five years after graduation sample makes sense in that those difficult years immediately after graduation would be over by that point, and we could feel more confident in assuming that a graduate – if able – has secured a role that they are enthusiastic about as a career. After all, measuring the earnings of graduates temporarily doing non-graduate work while applying for graduate roles seems unhelpful. We are combining multiple years of data, but we will still have an issue with cohort effects (for example, the cohorts who graduated around the time of the pandemic will likely see a medium-term salary impact because of their formative experiences).
Maximum’s the word
The actual metric under discussion is the natural logarithm of the highest earnings recorded in the third, fourth, and fifth years after graduation.
The logarithm is there for nerdy economics reasons far too dull to go into here, but I’ll cut through the maths and suggest that the log difference between two values is a way of approximating the percentage difference between them – and that might make it less tempting to compare raw salary numbers across subject areas as a million “which are the worst degrees?” articles tend to do.
As well as the low earnings cut off detailing above, we’re also “winsorising” – fancy talk for assuming anyone earning above the 99th percentile is earning exactly at the 99th percentile and thus taking Coldplay’s Chris Martin out of the UCL classics data.
Beaux selector
It’s unlikely that Chris Martin’s early immersion in classics led directly to his decade-spanning career in music, but it is very likely that – had he chosen another path in life – his choice of UCL rather than any other provider would have secured him a (comparatively) good salary. But what is not clear is whether this would be because classics at UCL is “better” than, say classics at Lincoln, or because UCL selects people with excellent A level results (who are more likely to secure higher earning employment, both via their own academic skills and the socio-cultural background that allowed them to develop these skills).
IFS found a strong correlation between earnings and attainment, and between provider selectivity and provider ranking within subject specific plots using earnings metrics. So it is perhaps surprising that the proposal does not include key stage 5 (A level and other qualifications taken at a similar time) – the reasons given are the interaction between attainment and admissions practice, and the diversity of KS5 attainment types.
For the interested, the correlation between the current B3 progression metric and the proposed salary metric that would sit alongside it was 0.62 – with a lot of variability by subject (history got up to 0.84!).
Yes, but will it actually happen?
The detail is impressive, and this is a solid piece of work as you may well expect. But it will be a stretch to see this implemented in the near future – not least because a change of government and a refocusing of the Office for Students under new leadership and a tweaked remit makes it less likely that the will or the capacity is there to regulate in this way.
That said, I’d like to see this metric published – not least because it is better than the alternatives for featuring earnings data in university league tables.
