We’re still waiting for proper OfS performance measures

Sometimes I really don’t understand the Office for Students.

In the teeth of an oncoming financial squeeze, with a new government looking for efficiency savings and with a documented interest in getting rid of arms length bodies, the performance of our regulators is under intense scrutiny.

Problem state

The National Audit Office and the Public Accounts Committee have already laid into the state of OfS performance measures this year, noting that:

Out of 26 indicators, eight are still in development or have incomplete performance information, and a further 11 indicators do not yet have associated targets

and, importantly:

The OfS does not routinely ask providers and sector stakeholders for feedback on its own performance as a regulator

There’s no strong measure on value for money because they don’t define it – the whole thing is such a mess that NEO recommended that DfE should:

review, improve and agree with the OfS the key performance measures and other indicators it uses to hold the OfS to account, to include measures of the impact of the regulatory regime, rather than measures outside the OfS’s control

Solutions focus

So today sees the launch of a new set of OfS KPMs. Obviously they take all of these recommendations into account and the new KPMs are presented as a complete set of easy to understand regulatory information that would allow Kit Malthouse to glance over a dashboard and click away satisfied.

Alas, no.

Of eleven new KPMs, seven are not present – with due dates ranging from October to “later this academic year”. A whole section on operational measures is due later this month.

And this isn’t just missing data – we don’t even get a sense of what will be presented outside of broad categories like “access to higher education” or “extent of student outcomes”. Is there anything on provider feedback (the old, never used, KPM21)? There is not.

Compare and contrast

As it happens, I had a list of the old KPMs from January (as usually happens, OfS has blitzed all documentation on the old approach). Then there were 26KPMs – of which eight were listed as being in development and three had defined targets linked to them. Just six have survived the cull as things stand:

• Scary sounding new KPM1 on poor quality provision looks similar to old and never used KPM13, though we don’t have the detail on that yet.
• Gaps between degree outcomes by ethnicity (old KPM4 – though tweaked to look at just firsts rather than firsts and 2:1s)
• Students responding positively on NSS (Old KPM10, though now a basket of scales – teaching on my course, assessment and feedback, learning resources, academic support, and student voice for new KPM4, just the first three for KPM9B)
• The proportion of full-time undergraduate students achieving first class degrees (old KPM18)
• That strange student survey on value for money (old KPM19)
• And the efficient regulation measures – number of data collections, enhanced monitoring, fee levels – shift from old KPM26 to new KPM11.

Clearly more will move across in some form as decisions are made at OfS, but it is an odd decision not to make this clear at this stage.

Those KPMs in full

Here’s the new ones (note this is paginated):

[table id=283 /]

And here’s the old ones:

[table id=284 /]

It’s clear that many of the changes reflect changes to the OfS strategy.

What to make of it

Quis regit regulatores, as a former Prime Minister may have put it. There are standards for regulation in public life – as well as scrutiny from the NAO and PAC we have a Regulator’s Code which is managed for some reason by the Office for Product and Safety Standards.

There is also a UK regulators’ network – a membership organisation with most of the big names as members. This works as a self-policing uber-regulator, benchmarking, setting and maintaining standards, and using scorecards for self-assessment. Notably, the Office for Students is not a member.

But even outside of all this, there is a sense that a regulator should be accountable to those it regulates and those it regulates on behalf of, to the taxpayer, to sponsors and to government officials. I was hoping this release was the OfS finally getting its house in order. I was mistaken.

The OfS has the habit of holding the sector to account via the analysis of data. The increasing vigour of investigations and tough talk does not represent a decline of sector standards, it represents a failure of regulations. Quality rules were set on establishment in 2018 – investigations and concerns now represent a failure of regulation as much as they do deficiencies in the sector.