Data lag and ambition aggregation means APPs are fundamentally flawed

In her letter to the sector last November, Secretary of State Bridget Phillipson said that she expects universities to play a stronger role in expanding access and improving outcomes for disadvantaged students.

Her letter noted that the gap in outcomes from higher education between disadvantaged students and others is unacceptably large and is widening, with participation from disadvantaged students in decline for the first time in two decades.

She's referring to the Free School Meals (FSM) eligible HE progression rate – 29 percent in 2022–23, down for the first time in the series.

Of course in 2023–24, or this year, the numbers for FSM and any number of other factors could be much worse – but on the current schedule, we won’t be seeing an update to OfS’ access and participation data dashboard until “summer or autumn 2025”, and even then only for 2023–24.

If you’re prepared to brave the long loading times – which for me generate a similar level of frustration to that I used to experience watching Eurovision national finals 20 years ago – you can drill down into that dashboard by provider.

It’s a mixed picture, with a lot of splits to choose from. But what the data doesn’t tell us is how providers are doing when compared to their signed off targets in their (mainly 2020–21 to 2024–25) access and participation plans.

The last time OfS published any monitoring data was for the 2020–21 academic year – almost three years ago, in September 2022.

That means that we can’t see how well providers are doing against their targets, and nor do we have any sense of any action that OfS may (or may not) have taken to tackle underperformance.

So I decided to have a go. I restricted my analysis to the Russell Group, and extracted all of the targets from the 2020–21 to 2024–25 plans that were measurable via the dashboard.

I then compared the 2022–23 performance with the relevant milestone, and with the original baseline. Where the target was unclear on what type of student was in scope, I assumed FT, first degree students.

The results are pretty worrying.

[table id=395 /]

Milestones and baselines

If we start with access, of the 25 targets that can be analysed, 14 behind milestone – and 10 show a worse performance than the baseline.

On continuation, 11 of the 17 are behind milestone, and 9 are behind the baseline. And on attainment, 25 of the 38 are behind milestone, and 14 behind baseline.

Notwithstanding that some of the other targets might have been smashed, and that in all cases the performance may well have improved since then, that looks like pretty poor performance to me.

It’s the sort of thing that we might have expected to result in fines, or at least specific conditions of registration being imposed.

But as far as we know, nothing beyond enhanced monitoring has been applied – and even then, we don’t know who has been under enhanced monitoring.

And the results are a problem. When OfS launched this batch of plans, it noted that young people from the most advantaged areas of England were over six times as likely to attend one of the most selective universities – including Oxford, Cambridge and other members of the Russell Group – as those from the most disadvantaged areas, and that that gap had hardly changed despite a significant expansion in the number of university places available.

At the rates of progress forecast under those plans, the ratio was supposed to be less than 4:1 by 2025. It was still at 5.44 in the Russell Group in 2022–23.

It was supposed to mean around 6,500 extra students from the most disadvantaged areas attending those universities each year from 2024-25 onwards. The Russell Group isn’t the whole of “high tariff” – but it had only increased its total of POLAR1 students by 1350 by 2022/23.

OfS also said that nationally, the gap between the proportion of white and black students who are awarded a 1st or 2:1 degree would drop from 22 to 11.2 percentage points by this year. As we’ve noted before on the site, the apparent narrowing during Covid was more of a statistical trick than anything else. It was up at 22.4 in 2022–23.

And the gap in dropout rates between students from the most and least represented groups was supposed to fall from 4.6 to 2.9 percentage points – it was up at 5.3pp in 2022–23.

The aggregation of ambition into press-releasable targets appears to have suffered from a similar fate to the equivalent exercise over financial sustainability.

What a wonderful thing

Of course, much has happened since January 2020. To the extent to which there were challenges over the student life cycle, they were likely exacerbated by the pandemic and a subsequent cost of living crisis.

But when you’re approving four year plans, changes in the external risk environment ought to mean that it revises what it now calls an Equality of Opportunity Risk Register to reflect that – and either allows providers to revise targets down, or requires more action/investment to meet the targets agreed.

Neither of those things seem to have happened.

It’s also the case that OfS has radically changed how it regulates in this area. Back then, the director for fair access and participation was Chris Millward. It’s now John Blake. And the guidance, nature of the plans expected and monitoring regimes have all been revamped.

But when we’re dealing with long-term plans, a changing of the guard does run the risk that the expectations and targets agreed under any old regime get sidelined and forgotten about – letting poor performers off the hook.

It certainly feels like that’s the case. And while John Blake is widely respected, it’s hard to believe that he’ll still be the director for fair access and participation by the end of the latest round of plans – 2029.

Hindsight is a wonderful thing, of course, but notwithstanding the external environment changes, few anticipated that any of the gaps, percentages or ratios would worsen for any of the targets set back in 2019.

That matters because of that OfS aggregation issue. It’s not just that some providers can drag down the performance of the sector as a whole. It’s that no provider was set the target of not getting any worse on the myriad of measures that it didn’t pick for its plan.

For all we know, while a certain number of providers might have set and agreed a target, say, on POLAR1 access or IMD attainment, performance could have worsened in all of those that didn’t – and that poses a major problem for the regulator and the design of the thing.

It remains the case that we’re lacking clarity on the way in which the explosion of franchised, urban area business provision has impacted the stats of both the providers that have lit that blue touch paper, and the sector’s scores overall. For me, improvements in access via that method look like cheating – and declines in continuation, completion or progression ought to mean serious questions over funding policy within the Department for Education.

We don't really know – but need to know – the impact of other providers’ behaviour on an individual provider’s external environment. If, for example, high tariff universities scoop up more disadvantaged students (without necessarily actually narrowing the gap), that could end up widening the gap elsewhere too. There’s only so many moles to whack when you’re looking at access.

We still can’t see A&P performance by subject area – which has always been an issue when we think about access to the professions, but is an even bigger issue now that whole subject areas are being culled in the face of financial problems.

And the size and shape question lingers too. UCAS figures at the close of clearing suggested that high tariff providers were set to balance the books by expanding in ways they claimed were impossible when the “mutant algorithm" hit in 2020.

Much of continuation, completion and progression appears to be about the overall mix of students at a provider – something that’s made much more challenging in medium and lower tariff providers if high-tariff ones lower theirs.

In the forthcoming skills white paper, we should expect exhortations from ministers that the sector improves its performance on access and participation. It will have choices on provider type, subject area, the types of disadvantage to focus on, and the mix of measures between things inside its control in the external environment, and things within providers’ control (or at least influence) that OfS should expect.

Whatever it chooses, on the evidence available, it will have real problems judging either its own performance, its regulator’s, groups of providers or even individuals’. If you think the sector still has some distance to go on fairness, that just won’t do.