Wales picks benchmarks over bright lines to judge tertiary quality

The whole point of creating a single tertiary regulator in Wales, on the long road from the Hazelkorn review to the Tertiary Education and Research (Wales) Act 2022, was to look at universities, colleges, sixth forms, apprenticeship providers and adult learning through one lens rather than five.

Since Medr took over from HEFCW in August 2024 – and especially since its regulatory framework went live in August 2026 – the open question has been what that lens would mean for judging quality.

Now we have a proposal, sort of. Medr’s consultation on provider quality performance measures sets out a single suite of published, benchmarked indicators across all five parts of the tertiary sector – responses are due by 9 December 2026, with the measures phased in from May 2027.

Chief executive James Owen calls it “a pivotal point in our regulatory journey”. Most of the measures will be familiar to anyone who remembers HESA’s old UK performance indicators – what matters is how Medr proposes to judge them, which is relative rather than absolute, conversational rather than trigger-based, and both quite public and quite private.

And while Medr is very much an arm’s length body, it’s not at all clear that the data fits the wider direction of political travel.

What’s on the table

The quality and continuous improvement condition of registration and funding requires providers to have “acceptable performance data”, but the consultation never says what acceptable – or unacceptable – looks like. The phrase appears once, in paragraph 3, and is never defined. The nearest thing to a line is a “−” flag that Medr insists only starts a conversation, and the Provider Risk Assessment that would decide what happens next isn’t due for consultation until spring 2027.

There are 28 measures in all, counting sub-parts – five for apprenticeships, ten for FE colleges and sixth forms, four for adult community learning and nine for HE.

• H1 counts enrolments by mode, level, domicile and subject
• H2 measures progression
• H3a and H3b cover one-year retention for full-time entrants and two-year retention for part-time ones
• H4 is degree classifications
• H5 is the share of undergraduate leavers gaining the qualification they aimed for (success)
• H6a and H6b draw on Graduate Outcomes for employment or study and professional or managerial jobs at 15 months, and
• H7 is NSS overall satisfaction

Recruitment measures in every sector won’t be benchmarked, and paragraph 18 says they won’t be used for quality interventions because they “are not an indicator of the quality of provision”. Footnote 2 says the existing National Measures will be phased out as strategic plan measures are developed.

The practical news for universities is at paragraph 20 – Medr proposes to calculate enrolments, progression, degree classifications and success through HESA’s IRIS system during the student record submission process, phased over the 2026/27 and 2027/28 returns.

Benchmarks over bright lines

The biggest design choice is benchmarking. Annex C sets out direct standardisation “similar to that which was used by HESA” in the UK performance indicators – the sector-wide outcome for each category of student is applied to a provider’s own mix, giving the outcome you’d expect if its learners did as well as similar learners elsewhere.

In the worked example, provider X, with 3,500 entrants, gets a continuation benchmark of 88.6 per cent. Under paragraph 11, a “+” or “−” marker is only applied when the gap between performance and benchmark is both more than three percentage points and more than three standard deviations.

That double test is deliberately conservative – Medr wants to catch differences that are “substantial both statistically and practically” rather than raise a flag every time a small cohort has a bad year.

England works very differently of course. OfS sets absolute numerical thresholds for continuation, completion and progression under condition B3 and expects every registered provider to clear them, with limited room for context. Medr judges providers against what’s expected given who they recruit, and paragraph 13 of Annex C says benchmarks “should not be interpreted as targets”.

For universities that recruit lots of students from disadvantaged backgrounds, mature learners or those with non-traditional entry qualifications, that’s a far friendlier regime than the one their English counterparts live under. But friendlier isn’t the same as clear – a regime that compares providers with what’s expected of them still has to decide how far below expectation is too far, and on that the consultation is silent.

The comparator group differs by sector, mind. Under footnote 1 and paragraph 15 of Annex C, HE providers will be benchmarked against UK-wide data, with Medr “exploring the possibility” of HESA calculating the benchmarks, while colleges, sixth forms, apprenticeship providers and ACL partnerships are compared with other Welsh providers in their own sector.

It means a Welsh university’s graduate employment figures will be judged against a UK average, yet none of the proposed HE benchmark factors – entry qualifications, subject, mode, level, age, ethnicity, sex, and deprivation via HESA’s SEISA measure – accounts for graduates who stay in Wales entering a very different labour market from those in London and the south east.

Disability isn’t among the HE benchmark factors either, even though paragraph 191 of Annex B confirms OfS uses it to benchmark the NSS results that Medr imports wholesale for H7. No reason is given, which makes it look like an omission rather than a decision.

Graduate Outcomes compounds the labour market problem. It uses “professional or managerial” jobs – SOC 2020 major groups 1 to 3 – as its proxy for a graduate job, a long-contested definition that bites harder in Wales. Census 2021 found 18.2 per cent of workers in Wales in professional occupations, against 20.3 per cent in England, 25.8 per cent in London and 21.2 per cent in the south east.

A university whose graduates stay to work in Welsh communities risks looking worse on H6b than one whose graduates head east – an odd incentive in a country whose politicians worry about keeping graduate talent at home.

Flags by another name

Medr insists this isn’t a league table, and its press release says it “does not propose that these measures enable a threshold or trigger point approach”. Under the principles at paragraph 10, outcomes will “normally trigger a conversation in the first instance”.

The escalation route in Diagram 1 runs from a granular look at the data, through a conversation with the provider, to triangulation with external quality assessment and other evidence, and only then to “promotion of good practice, or regulatory follow-up, as appropriate”.

Stuck record alert, but missing from that route is anyone representing students. Learners Students appear only as survey respondents, with no role for SUs or learner representatives in contextualising a flag – the provider gets a private conversation in which to explain its numbers, and the people whose outcomes they are don’t.

Whatever the regulator does next, a published “−” applied by a statistical rule is a threshold in all but name. Paragraphs 28 and 29 propose public provider-level dashboards with benchmarks, +/− markers and subject-level analysis, with measures also going on to StatsWales by learner characteristics and language of study.

Once a “−” sits next to a provider’s name on a public dashboard, governing bodies will understandably treat the benchmark as a target the moment their institution falls below it.

Nor will the flags stay on Medr’s dashboard. Annex A makes the measures part of the evidence base for QAA reviews, and paragraph 33 wants both working from the same dataset, so a “−” is likely to shape the lines of enquiry in a university’s next review well before any regulatory follow-up arrives.

Half-built methodology

Annex D shows what this would look like in practice – one row per measure for each provider, with the count, rate, benchmark, flag and a three-year trend arrow. There’s no confidence interval or standard deviation on show, and no split by subject, learner characteristics or language of study, so the richer analysis promised in paragraphs 28 and 29 will presumably sit elsewhere.

Q4 asks whether the extraction criteria and methodology in Annex B are appropriate, but a fair amount of Annex B isn’t settled. Paragraph 106 says the H2 progression methodology is “being developed”, with more work needed “to incorporate intercalating and dormant students correctly” and to work out whether students who don’t appear to progress have transferred or are resitting.

H3b depends on HESA providing data, and paragraph 144 warns the specification “may change”. Paragraph 56 then concedes that for H2, H3a, H3b, H5, H6a and H6b, Medr is still discussing benchmark production with HESA, so “the publication timetable is not yet defined”.

That’s six of the seven benchmarked HE measures without a date, which makes the May 2027 start look optimistic for universities. The only HE measures with a slot in the indicative cycle are H1 and H4 in February, neither of them benchmarked, and H7 in July, which OfS benchmarks rather than Medr.

The retention measures also have a transparency problem. H3a and H3b use HESA’s fuzzy matching to find students who’ve left one provider and turned up at another, which improves accuracy – but paragraphs 138 and 161 of Annex B say data protection rules mean “it is not possible to share all of the underlying data used in the calculation with providers”.

A university could therefore be shown a “−” on retention that it can’t fully reconstruct from its own records – hardly ideal for a regime built on contextualising conversations.

H2, meanwhile, tracks students only within the same provider and course, so a student who moves elsewhere without a recorded transfer looks like a non-progression on H2 but a success on H3a – an odd result for a tertiary system that’s supposed to make moving between providers easier.

The populations don’t line up either. H2 counts full-time first degree students of all domiciles, H3a full-time UK undergraduates, H3b part-time UK first degree students, H5 undergraduate leavers of any domicile, H6a UK graduates and H6b UK and EU graduates. Students studying mainly overseas are excluded from H1 and H3a, so a university’s transnational provision gets no measure at all.

None of that is necessarily wrong, but the dashboard will present measures that look related while counting different groups of students. A university with large international numbers could look solid on progression and shaky on retention, or the other way round, for reasons that have nothing to do with quality.

H5 has its own problem. Paragraph 174 counts a leaver as a success only if they gained the qualification they were aiming for or something higher, so a student who steps off a degree with a CertHE or DipHE – the flexibility lifelong learning policy is supposed to value – counts as a failure.

And while H5 is described as covering undergraduate leavers, the coding in paragraph 173 appears to select first degree students only, which would leave foundation degrees and HNDs out.

Two satisfaction regimes

H7 imports NSS overall satisfaction and OfS’s own benchmarks rather than using Medr’s method. Because OfS expresses its benchmark comparison as a confidence percentage rather than a significance test, paragraphs 192 and 193 of Annex B add a “Medr NSS notable marker”.

A result counts as “a lot below” if it’s more than 2.5 percentage points under benchmark with over 70 per cent confidence that it’s materially below, and “a lot above” in the mirror case. Medr is candid that “this is not a statistical method”, but it means H7 will flag on a looser rule than every other benchmarked measure on the same dashboard.

The press release calls learner voice “a key measure”, but for the regime’s first couple of years the dashboards will be dominated by outcomes data.

It’s also a narrow reading of the NSS. H7 uses only Q28 on overall satisfaction, ignoring the three questions in the survey’s student voice theme – on opportunities to give feedback, whether staff value students’ views and whether it’s clear feedback is acted on – and Q25 on how well the SU represents students’ academic interests.

Postgraduates, who don’t take the NSS, get no satisfaction measure at all, and the consultation doesn’t say whether the Learner Voice Survey might fill the gap.

Who should worry

For the larger Welsh universities the regime looks manageable. UK-wide benchmarks for a research-intensive university with high-tariff entrants will set expectations it already broadly meets, and the double test means few will see a “−” except for a persistent problem.

Universities with small cohorts – in particular subjects, modes or campuses – are more exposed, because paragraph 14 of Annex C warns that results for small populations “should be treated with caution” as they swing from year to year.

Franchising is the obvious gap. The quality condition covers provision “delivered on their behalf”, and several Welsh universities have run substantial franchised partnerships, some of them over the border in England. Yet paragraph 61 concedes that “the visibility of franchised delivery varies across providers” and that a more consistent approach is needed.

Nothing in the proposed measures separates out franchised students, so a provider’s dashboard could blend very different delivery models into a single number – and anyone who’s followed the franchising saga in England will know where the risk sits.

That includes Welsh FE colleges delivering degrees franchised from a university. Their students sit in the university’s HESA return, so their continuation and completion feed straight into its H measures – and nothing on the dashboard would tell a reader whether a “−” originated on the main campus or at a college partner thirty miles away.

What goes unmeasured

Paragraph 16 lists “regulatory concerns raised with Medr” among the evidence Medr will consider, but no measure is built on them, and there’s nothing on complaints, assessment or teaching quality beyond the single NSS question.

On equality, the consultation leans on the Act’s duty to promote equality of opportunity. But benchmarking on ethnicity, sex and deprivation has a well-known side effect – if outcomes for a group are consistently worse across the sector, the benchmark expects that gap, and a provider replicating it won’t be flagged.

Those hoping for more on widening access will have to wait. Paragraph 5 says the measures will support wider monitoring of equalities, participation and access, but that Medr will do separate work on progress against its long-term ambitions – so fair access is coming in another vehicle.

The impact assessment’s own list of weaknesses – inconsistent collection of protected characteristics and socio-economic data, poor capture of intersectional disadvantage, and a lack of robust geographic breakdowns – sits at the very end of the document as “areas for review” rather than as a limitation on the measures being consulted on.

Annex A rules out a value added measure because it works poorly for vocational and adult learning, can’t be produced in time from verified data, and is hard for users to understand. That’s a reasonable call, but it leaves the benchmark as the only adjustment for where students start from.

Degree apprentices are another awkward fit. Paragraph 38 says Medr will consider whether the apprenticeship measures need updating for the new programme, including degree apprenticeships – which for now fall between an A set built around LLWR apprenticeship frameworks and an H set built around HESA first degree students, with no clarity on which they’ll count towards.

Paragraph 48 says the measures “are not intended to measure progress against” the National Plan for the Welsh Language in Tertiary Education, due by April 2027, though paragraph 47 does commit to analysing them by language of study “where the availability and robustness of the data allow”. Given the small cohorts in much Welsh-medium HE provision, that caveat may end up doing a lot of work.

What happens now

Parts of the timetable are tight. HE providers must nominate a contact to receive their own data by 31 October 2026, eleven days after launch, and the data will land in a “Performance Measures” folder on their secure file sharing sites.

Under paragraph 31 each provider is being sent its own data privately during the consultation, so universities will soon know whether they would carry a “−”. That gives providers a head start on everyone else – and it’s hard to see why the students whose outcomes are being measured and responses invited should have to wait for a public dashboard to find out what their university’s data says.