This article is more than 1 year old
There’s three pillars to the Office for Students’ outcomes (B3) monitoring – the most visible of the ways the regulator attempts to assure quality and value on behalf of higher education students.
Continuation and completion are derived from HESA Student data, while progression is based on the Graduate Outcomes survey.
But each of these underlying data sources – and thus the validity of B3 itself – is facing questions about reliability and validity next year.
The most startling reversal in fortune is the most recent -next year’s Graduate Outcomes data release will be the first based on less than half of eligible UK graduates submitting responses. Just 46.3 per cent of home students responded to the survey this time round – in overall terms that works out to a response rate for all domiciles of just 39 per cent.
This is by far the lowest response rate on record, and continues a multi-year pattern of declining rates – though it is important to relate that these rates are not a final assessment of responses, the completed figure is unlikely to be above 50 per cent.
A part of this year’s poor showing can be laid at the door of a cyberattack. We understand that a direct denial of service (DDoS) experienced by the survey system provider early in October meant “over a week of telephone surveying at much reduced capacity” and the loss of “an estimated 9,000 interviews”.
But this unfortunate event affected only the cohort D collection – the smallest of four cohorts, comprising primarily students who began (and/or completed) their studies at a non-traditional time.
Fifty per cent is an important psychological boundary for surveys that claim (as Graduate Outcomes does) to be a population level (or census) data collection.
With more full-time UK students not answering the survey than answering it, we need to answer difficult questions about the representative value of what is, basically, a self-selecting sample. Previous HESA work had not identified a notable skew around responding graduates, but such research is by nature limited – and referred to a much higher response rate.
There were gasps when this news was related to Jisc’s Student Data Forum – with many sector representatives asking difficult questions about the future… of data futures.
Jisc’s Head of Surveys, Gosia Tuner, told us that:
Although this year’s response rates don’t match our targets, they are still exceptional for a national survey of this kind at over 40% for most groups of graduates. Every year we analyse the survey responses to ensure that the data we have is representative of the year’s graduates in general, and of each sub-population (such as by subject or level of qualification).
While the sector’s designated data body will make the best of a disappointing outcome, providers investigated on the basis of a low progression rate may rightly question how less than half of their graduates can form the basis of a determination of teaching quality.
If it was just one of OfS’ three B3 measures that was in question, you might expect that the regulator might choose to spend a year focusing on the other two.
Unfortunately, future continuation and completion metrics will both be based on data collected during the flawed and heavily caveated 2022-23 Student data process, and on a 2023-24 collection that shows some signs of being worse.
If you have a tangential awareness of the travails of Data Futures, you may be labouring under the impression that the problems were addressed during the last cycle, and the data currently being collected with respect to the 2023-24 academic year will be a return to form. You would be wrong.
We are aware of 17 providers that have not yet submitted their HESA return – the deadline has been quietly extended. This includes 10-12 large, well-known, providers covering nearly 10 per cent of the entire student population – there are discussions afoot as to whether these providers would be named officially.
A combination of a failure to address aspects of the collection (at both provider and sector level) that did not work well last year, staff attrition, and poor staff morale have all left the sector struggling to cope with current demands. The 2022-23 data bears several large asterisks – 2023-24 may be similarly stellar when we get the open data.
Again, when OfS comes knocking (or even before) will your provider be content to be judged on flawed data?