The funny thing about the story about today’s intervention by the Office for Students is that it is not really about grade inflation, or degree algorithms.
I mean, it is on one level: we get three investigation reports on providers related to registration condition B4, and an accompanying “lessons learned” report that focuses on degree algorithms.
But the central question is about academic standards – how they are upheld, and what role an arm of the government has in upholding them.
And it is about whether OfS has the ability to state that three providers are at “increased risk” of breaching a condition of registration on the scant evidence of grade inflation presented.
And it is certainly about whether OfS is actually able to dictate (or even strongly hint at its revealed preferences on) the way degrees are awarded at individual providers, or the way academic standards are upheld.
If you are looking for the rule book
Paragraph 335N(b) of the OfS Regulatory Framework is the sum total of the advice it has offered before today to the sector on degree algorithms.
The design of the calculations that take in a collection of module marks (each assessed carefully against criteria set out in the module handbook, and cross-checked against the understanding of what expectations of students should be offered by an academic from another university) into an award of a degree at a given classification is a potential area of concern:
where a provider has changed its degree classification algorithm, or other aspects of its academic regulations, such that students are likely to receive a higher classification than previous students without an increase in their level of achievement.
These circumstances could potentially be a breach of condition of registration B4, which relates to “Assessment and Awards” – specifically condition B4.2(c), which requires that:
academic regulations are designed to ensure that relevant awards are credible;
Or B4.2(e), which requires that:
relevant awards granted to students are credible at the point of being granted and when compared to those granted previously
The current version of condition B4 came into force in May 2022.
In the mighty list of things that OfS needs to have regard to that we know and love (section 2 of the 2017 Higher Education and Research Act), we learn that OfS has to pay mind to “the need to protect the institutional autonomy of English higher education providers” – and, in the way it regulates that it should be:
Transparent, accountable, proportionate, and consistent and […] targeted only at cases where action is needed
Mutant algorithms
With all this in mind, we look at the way the regulator has acted on this latest intervention on grade inflation.
Historically the approach has been one of assessing “unexplained” (even once, horrifyingly, “unwarranted”) good honours (1 or 2:1) degrees. There’s much more elsewhere on Wonkhe, but in essence OfS came up with its own algorithm – taking into account the degrees awarded in 2010-11 and the varying proportions students in given subject areas, with given A levels and of a given age – that starts from the position that non-traditional students shouldn’t be getting as many good grades as their (three good A level straight from school) peers, and if they did then this was potentially evidence of a problem.
To quote from annex B (“statistical modelling”) of last year’s release:
We interact subject of study, entry qualifications and age with year of graduation to account for changes in awarding […] our model allows us to statistically predict the proportion of graduates awarded a first or an upper second class degree, or a first class degree, accounting for the effects of these explanatory variables.
When I wrote this up last year I did a plot of the impact each of these variables is expected to have on – the fixed effect coefficient estimates show the increase (or decrease) in the likelihood of a person getting a first or upper second class degree.
One is tempted to wonder whether the bit of OfS that deals with this issue ever speaks to the bit that is determined to drive out awarding gaps based on socio-economic background (which, as we know, very closely correlates with A level results). This is certainly one way of explaining why – if you look at the raw numbers – the people who award more first class and 2:1 degrees are the Russell Group, and at small selective specialist providers.
Based on this model (which for 2023-24 failed to accurately predict fully fifty per cent of the grades awarded) OfS selected – back in 2022(!) – three providers where it felt that the “unexplained” awards had risen surprisingly quickly over a single year.
What OfS found (and didn’t find)
Teesside University was not found to have ever been in breach of condition B4 – OfS was unable to identify statistically significant differences in the proportion of “good” honours awarded to a single cohort of students if it applied each of the three algorithms Teesside has used over the past decade or so. There has been – we can unequivocally say – no evidence of artificial grade inflation at Teesside University.
St Mary’s University, Twickenham and the University of West London were found to have historically been in breach of condition B4. The St Mary’s issue related to an approach that was introduced in 2016-17 and was replaced in 2021-22, in West London the offending practice was introduced in 2015-16 and replaced in 2021-22. In both cases, the replacement was made because of an identified risk of grade inflation. And for each provider a small number of students may have had their final award calculated using the old approach since 2021-22, based on a need to not arbitrarily change an approach that students had already been told about.
To be clear – there is no evidence that either university has breached condition B4 (not least because condition B4 came into force after the offending algorithms had been replaced). In each instance the provider in question has made changes based on the evidence it has seen that an aspect of the algorithm is not having the desired effect, exactly the way in which assurance processes should (and generally do) work.
Despite none of the providers in question currently being in breach of B4 all three are now judged to be at an increased risk of breaching condition B4.
No evidence has been provided as to why these three particular institutions are at an “increased risk” of a breach while others who may use substantially identical approaches to calculating final degree awards (but have not been lucky enough to undergo an OfS inspection on grade inflation) are not. Each is required to conduct a “calibration exercise” – basically a review of their approach to awarding undergraduate degrees of the sort each has already completed (and made changes based on) in recent years.
Vibes-based regulation
Alongside these three combined investigation/regulatory decision publications comes a report on Batchelors’ degree classification algorithms. This purports to set out the “lessons learned” from the three reports, but it actually sets up what amounts to a revision to condition B4.
We recognise that we have not previously published our views relating to the use of algorithms in the awarding of degrees. We look forward to positive engagement with the sector about the contents of this report. Once the providers we have investigated have completed the actions they have agreed to undertake, we may update it to reflect the findings from those exercises.
The important word here is “views”. OfS expresses some views on the design of degree algorithms, but it is not the first to do so and there are other equally valid views held by professional bodies, providers, and others – there is a live debate and a substantial academic literature on the topic. Academia is the natural home of this kind of exchange of views, and in the crucible of scholarly debate evidence and logical consistency are winning moves. Having looked at every algorithm he could find, Jim Dickinson covers the debates over algorithm characteristics elsewhere on the site.
It does feel like these might be views expressed ahead of a change to condition B4 – something that OfS does have the power to do, but would most likely (in terms of good regulatory practice, and the sensitive nature of work related to academic standards managed elsewhere in the UK by providers themselves) be subject to a full consultation. OfS is suggesting that it is likely to find certain practices incompatible with the current B4 requirements – something which amounts to a de facto change in the rules even if it has been done under the guise of guidance.
Providers are reminded that (as they are already expected to do) they must monitor the accuracy and reliability of current and future degree algorithms – and there is a new reportable event: providers need to tell OfS if they change their algorithm that may result in an increase of “good” honours degrees awarded.
And – this is the kicker – when they do make these changes, the external calibration they do cannot relate to external examiner judgements. The belief here is that external examiners only ever work at a module level, and don’t have a view over an entire course.
There is even a caveat – a provider might ask a current or former external examiner to take an external look at their algorithm in a calibration exercise, but the provider shouldn’t rely solely on their views as a “fresh perspective” is needed. This reads back to that rather confusing section of the recent white paper about “assessing the merits of the sector continuing to use the external examiner system” while apparently ignoring the bit around “building the evidence base” and “seeking employers views”.
Academic judgement
Historically, all this has been a matter for the sector – academic standards in the UK’s world-leading higher education sector have been set and maintained by academics. As long ago as 2019 the UK Standing Committee for Quality Assessment (now known as the Quality Council for UK Higher Education) published a Statement of Intent on fairness in degree classification.
It is short, clear and to the point: as was then the fashion in quality assurance circles. Right now we are concerned with paragraph b, which commits providers to protecting the value of their degrees by:
reviewing and explaining how their process for calculating final classifications, fully reflect student attainment against learning criteria, protect the integrity of classification boundary conventions, and maintain comparability of qualifications in the sector and over time
That’s pretty uncontroversial, as is the recommended implementation pathway in England: a published “degree outcomes statement” articulating the results of an internal institutional review.
The idea was that these statements would show the kind of quantitative trends that OfS get interested in, some assurance that these institutional assessment processes meet the reference points, and reflect the expertise and experience of external examiners, and provide a clear and publicly accessible rationale for the degree algorithm. As Jim sets out elsewhere, in the main this has happened – though it hasn’t been an unqualified success.
To be continued
The release of this documentation prompts a number of questions, both on the specifics of what is being done and more widely on the way in which this approach does (or does not) constitute good regulatory practice.
It is fair to ask, for instance, whether OfS has the power to decide that it has concerns about particular degree awarding practices, even where it is unable to point to evidence that these practices are currently having a significant impact on degrees awarded, and to promote a de facto change in interpretation of regulation that will discourage their use.
Likewise, it seems problematic that OfS believes it has the power to declare that the three providers it investigated are at risk of breaching a condition of registration because they have an approach to awarding degrees that it has decided that it doesn’t like.
It is concerning that these three providers have been announced as being at higher risk of a breach when other providers with similar practices have not. It is worth asking whether this outcome meets the criteria for transparent, accountable, proportionate, and consistent regulatory practice – and whether it represents action being targeted only at cases where it is demonstrably needed.
More widely, the power to determine or limit the role and purpose of external examiners in upholding academic standards has not historically been one held by a regulator acting on behalf of the government. The external examiner system is a “sector recognised standard” (in the traditional sense) and generally commands the confidence of registered higher education providers. And it is clearly a matter of institutional autonomy – remember in HERA OfS needs to “have regard to” institutional autonomy over assessment, and it is difficult to square this intervention with that duty.
And there is the worry about the value and impact of sector consultation – an issue picked up in the Industry and Regulators Committee review of OfS. Should a regulator really be initiating a “dialogue with the sector” when its preferences on the external examiner system are already so clearly stated? And it isn’t just the sector – a consultation needs to ensure that the the views of employers (and other stakeholders, including professional bodies) are reflected in whatever becomes the final decision.
Much of this may become clear over time – there is surely more to follow in the wider overhaul of assurance, quality, and standards regulation that was heralded in the post-16 white paper. A full consultation will help centre the views of employers, course leaders, graduates, and professional bodies – and the parallel work on bringing the OfS quality functions back into alignment with international standards will clearly also have an impact.
OfS is essentially an unaccountable organisation, as it has not justified changes to its own interpretations and why providers should change their practices. How can the OfS be formally disputed and over-ruled except through litigation? Universities UK should make representations to DfE about the conduct of the OfS. In order to make these judgements, OfS must have reference to an objective testable standard of what it thinks is right, a universal standard which fits all academics, all providers, and all subjects and no such standard exists because academic judgements are non-justiciable (technical and professional qualifications contain some knowledge which can… Read more »
OFS is clearly guilty of over-reach in this, and other areas. It as clearly lacks independence, because its actions are essentially responses to political and media agendas. Whenever I read OFS reports, especially, relating to quality, I am curious about the identity and expertise of those individuals who have made these judgments. In the interests of transparency they should always to named. In the case of this report we need to be told the names of those who have made these assessments, and then we can judge their real-world experience of academic standards and exam grading systems. Too often they… Read more »
That’s an interesting point about authorship Peter. It’s consistent with the failure to name reviewers when OfS quality assessment reports of individual providers are published, which broke with previous sector practice; which is to say this is consistently bad practice, and clearly a conscious decision.
Exactly.
I agree. I particularly want to know if they even are a group, or whether this is essentially one person’s vendetta.
The idea that it is simply impossible to do better at a later assessment than you have in a previous one (particularly if you are from a non-traditional background) is the sort of regulatory nonsense we get when the people making these decisions were all traditional students who spent much of their compulsory education in private schools, grammar schools, or high performing state schools and were told over and over again that their GCSE and A Level exams were the single most important thing they would ever do because it would set them up to win at life (by getting… Read more »
I was hoping the sudden quietening of previously forthright OfS senior leaders might be signalling a move away from this kind of over-interventionist whim-based regulation, whereby the OfS conducts some analysis and makes a range of contentious decisions without consulting anyone (low points being direct responses to editorials in Sunday newspapers). And the consultation on Integrated Quality really was feeling like the OfS were consulting (rather than just waiting to explain why we were wrong). On the other hand, you’d think after almost a decade of this I’d have learnt to stop hoping. I don’t like either of the degree… Read more »
I had assumed that this had more with OfS wishing to grab headlines than actual quality of degrees. Presumably somebody, somewhere has put pressure on them to actually define what good and bad quality provision is.
What worries me most is the way they’ve used the UKSCQA principles document, which I’d always understood to be a sector guidance or good practice document. Suddenly saying that not following it exactly makes you at risk of regulatory non-compliance changes its status a lot. Will the sector want to keep producing and collaborating on best practice guidance – things that are aimed at enhancement – if they become regulatory sticks for the OfS?
Late to the party on this, but I want to back up DK’s point about the contradiction between this guidance and the OfS’ encouragement of efforts to close award gaps for those from more deprived backgrounds. As Jim Dickinson notes in the other article, we know from COVID-19 no-detriment policies and the work of people like David Allen that degree classification rules can make a radical difference to the final award. The rules on uplifting ‘borderline’ candidates alone can shift outcomes. Consider two students, both with the same Year 2 average of 59, and both with a weighting of 40:60… Read more »
Paired cases like these are part of the reason why my institution doesn’t do rounding. Both of these students would be in the 2.2 classification by the numbers, and both would be in the “zone of discretion” which would allow the board of examiners to look at trajectory (do their marks go upwards from year 2 to year 3, and is their final year, if considered by itself, in the higher category?), preponderance (are a majority of their credits in the higher category?), and the mark for the dissertation/major final year project, to determine if the classification should be uplifted.… Read more »
The whole issue could be easily solved by adopting a National Degree Mark (NDM): the weighted average of all module marks that make up the 360 credits of the typical degree, where universities could use their current algorithms but would be obliged to publish the NDM also. Besides being simple to calculate and understand (its intuitive after all), the NDM offers a whole raft of advantages to those interested in tracking the students academic journey.
Though it won’t be entirely comparable unless you standardise progression variations which will impact average module marks (number of resits, number of times an assessment can be resat, whether resits are capped, whether resits are capped at the assessment or module levels, whether passing grades can be resat, whether assessment marks are altered in light of personal circumstances/as a reasonable adjustment or not, whether repeating modules or years is also permitted etc).
And it does risk undoing some of the actions taken in light of a students mitigating circumstances (if assessment marks are not changed, but degree classes are).
A fundamental point being missed in this discussion is that these decisions shouldn’t be reliant on an algorithm at all. The decision should be made on whether the programme outcomes have been met (assuming these are at an appropriate standard, and comply with the subject benchmark statements) and the degree to which they have been met as judged by academics who have calibrated their standards across their subject/disciplinary community. The problem is that the sector (UUK, OFS, Advance HE, etc) refuses to address this issue seriously, but instead fiddles around with changes like this, achieving nothing.
Of course, many institutions who do not discount grades DO allow discretion at classification boundaries (some as much as 5% below defined boundaries), and this has a far greater effect on grade inflation! OfS has not considered this, largely because its effect can’t be measured as an algorithm can.
Come on OfS, talk to those who have experience across several institutions (through external Committee membership, external examining, etc.) so you can understand the full picture. By focussing on the discounting of modules and multiple algorithms you only have half the picture.