New TEF, new data, same issues

If you are currently writing your provider’s submission to the second round Office for Students consultation on quality, I am sorry to tell you that there is now even more to read.

While more information and data is always preferable, it is difficult to piece together a clear picture on how the new TEF will work as quality assurance and quality enhancement at both a baseline and high quality level for every kind of institution. A suite of new publications fill in some of the gaps in the existing documents.

Perhaps the most substantial additions to what has already been published are the proposed guidance documents for providers (and assessors) and for students, which both offer different levels of detail covering similar ground to Debbie McVitty’s piece for Wonkhe a couple of weeks ago.

Alongside this, we get a first taste of what the proposed indicators will look like at a provider level – via a new TEF dashboard and associated data downloads (of which more later). There’s the text of the proposed new condition B6, plus an explanatory document that helps us understand what has been changed and why.

Finally there’s two new data publications that underpin the way benchmarks are constructed for TEF measures. The NSS Associations Between Characteristics of students (ABCs) allow for consideration of how likely students in characteristics groups are likely to answer positively in the NSS (there’s a full time and part-time variant). And a new geography of earnings dataset (four years on from OfS saying it was unlikely to produce this in the future) will be used to benchmark earnings by locality, using travel to work areas (TTWA).

An overview of TEF

Here’s a quick diagram that gives a simple and high-level overview of the way TEF will work to replace both the existing TEF (which is ostensibly concerned with quality enhancement) and B condition (which deal with performance below minimum thresholds).

Risk-based?

The striking points for me are what is missing. There has been an explosion of validated and subcontracted provision, some of which has been revealed to be (not least via OfS’ own investigations, but also in the national press) of low quality. Validated provision (where the awarding body does not register the students) does not count towards the ratings that underpin main TEF award, and while subcontractual provision does this is hamstrung by most of the available data referring to partnerships that may have ended or changed several years ago. That OfS’ flagship quality assurance system does not include a mechanism to deal with the widely understood risks presented by current franchise and partnership activity is surprising to say the least.

Furthermore – thinking about next year’s headlines rather than last year – In an age where just about every provider is concerned with developing and growing transnational education (TNE) provision it is worthy of note that this scheme explicitly does not look at such provision (which is almost completely unregulated as things stand). And even if it did the proposed approach is not ESG (standards and guidelines for quality assurance in the European Higher Education Area) compliant – so would not count as actual quality assurance in many countries, thus making it harder to set up TNE.

OfS is clearly very keen to convince itself that the new approach is ESG compliant – it is after all a cyclical review process that involves students, two major criticisms levied at the old model. But ESG is based on the assessment of a provider’s own quality assurance mechanisms by an agency independent of government: OfS is technically independent of government (though it has blotted its copybook on that over the past few years), but it still appears very reluctant to examine the ways in which providers regulate and assure quality and academic standards.

The other thing worth noting is the enhanced consequences from TEF performance. While in the past an unexpected bronze has been (at worst) a slight reputational hit and a loss of confidence in the way your vice chancellor enters Universities UK meetings, under the new arrangements bronze could in future mean the loss of inflationary fee-cap increases, a lack of eligibility for OfS funding streams, problems applying for degree awarding powers or university title, or even (from round 2) a recruitment freeze. And if you end up with a “requires improvement” in either aspect, you could even see a recruitment cut. Scary stuff – even though, given OfS’ current track record, anyone affected might want to think about bringing a judicial review of a decision like that.

Everyone forgets about the LLE

Another missing aspect is any attempt to define an approach for lifelong learning entitlement regulation. A letter published alongside the other materials described here tells us that OfS plans to:

develop outcome measures for [LLE funded] modules, starting with a completion measure, which we would not expect to be available until 2028-29 at the earliest. We will also consider whether we can develop other outcomes measures, including progression, that we could use from 2030.

So provision that you can apply to now and start in January will not be regulated until the year after next, which feels like OfS having a perfectly normal one. I mean, there seems to be no great urgency to regulate year-long HE courses or postgraduate taught provision (both coming 2031) either but I’m sure nothing will go wrong there in the meantime.

Condition B6

Our new revised general ongoing condition in relation to quality assessment makes it clear that either TEF or a baseline assessment is required from all registered providers, previously TEF was optional for providers with less than 500 undergraduate students. The publication of TEF or baseline results (the “scorecard”) will be mandatory on all relevant course webpages (a definitional minefield right there).

Other new clauses relate to data and information requirements, and to the regulatory consequences of a poor result: the “requires improvement” or “unsatisfactory” rating will lead to a recruitment reduction from the first time the scheme is run, with the cap at current levels for bronze or “satisfactory” coming from 2031. Somewhat menacingly, we are also told that a provider that holds a “requires improvement” or “unsatisfactory” rating in either aspect “must not” get one next time round – we are not told the consequences, but this would be a breach of the ongoing condition of registration: usually not a good thing.

The text of B6 is up for change during the consultation process.

DK complaining about the state of the dashboard and data

Sadly, neither the dashboard itself or the data that purports to underpin it (it does not, as we will see) shed much light on the implications of this system. For anyone at a provider looking to undertake any serious analysis of the TEF indicators and the way they look for various splits will need to download two semi-compatible files – one dealing with continuation and completion (the two indicators that still involve a benchmark) and one dealing with everything else.

After manually adding the subject area names and partner provider names from the dashboard that have been omitted – along with actual provider names, for some reason – from each dataset, you then need to negotiate a join on six dimensions and account for various inconsistencies in labelling. To be clear, this isn’t impossible, and it would be doable at a push for a larger provider with a reasonably well resourced data analysis team, but for the hundreds of very small providers that have a registry and regulatory compliance function with a headcount somewhere in the region of 1.9 it will present a serious problem.

I hope to publish something to help these people very soon, but in the meantime we should reflect that OfS has form on making unreasonable assumptions about sector capacity when it comes to data analysis and seems institutionally incapable of making the kind of data design decisions that make such things possible for mere mortals. As a provider of Official Statistics, it needs to do better.

Even the existing dashboard data looks easier in comparison.

NSS ABCs

OfS has a peculiar relationship with benchmarking. It started off with a kind of absolutist position that we could not make excuses for poor quality student outcomes even if these did not stem from any failings of a provider, to a more nuanced understanding of systemic risks. The NSS ABCs are an attempt to bring what we know about groups of students who are particularly likely to report a poor quality student experience into the construction of benchmarks at provider and split levels.

I’ve put this into a very simple dashboard for you as OfS doesn’t seem to have been bothered to:

[Full screen]

You can use the filters on the left to examine the number and proportion of people in a given intersection fall into each quintile of historic responses to the National Student Survey (yes, this looks at all responses, something that OfS was adamant it would never need to do when it scrapped the “overall” NSS question…). And the NSS ABCs quintile that a student in that category would be put in for TEF benchmarking purposes is shown at the top left.

The characteristics on offer cover sexual orientation, age group, FSM eligibility, sex, religious belief, ethnicity, estrangement status, and disability (covering an overall marker and specific disability types) – and can be combined freely. Sadly the “sex” marker is of the post Anne Sullivan review variety, so the notably poor student experiences among people with a transgender or non-binary identification are not visible.

Geography of earnings

OfS swore it didn’t see the need to do a geography of earnings back in 2022, but life – it turns out – comes at you pretty fast. This new version shows the proportion of graduates either one, two, or three years after graduation that earn more than the national median salary for people aged 25-29 in a given travel to work area (TTWA). It assigns a quintile to each area in England, Scotland, and Wales – and benchmarks will be constructed based on the number of your graduates that end up working in each area.

[Full screen]

It’s neat but it isn’t perfect – for me the question would be why it didn’t compare using the median salary for 25-29 in the TTWA under examination. Comparing graduates who work in Hartlepool with a UK average tells us more about the poor availability of well-paid jobs in Hartlepool than any failings at providers who may have graduates that live and work there. As this is a benchmarking tool not an indicator this is a moot point, but it doesn’t really give me confidence in the benchmarking process.

Difficult work, higher expectations

The construction of this system is astonishingly complex – and it demonstrates the limitations of a reactive, data-driven approach to regulation. It is difficult to be truly “risk-based” if the risks are observed via a set of assumptions that are often questionable, and based on data that it is surprising old using mechanisms that are very difficult to recreate accurately.

Regulating in this way is a choice (coming up with ways to be able to trust and monitor providers who could then to assure quality themselves in documented and visible ways would be another) and it is one predicated on getting the data absolutely right, first-time, every time. OfS is not there yet, and in pushing this approach forward it is – apart from going directly against the recommendations of an independent review commissioned by parliament – very likely to find itself at the high court again in the near future.