The 2011 White Paper Students at the Heart of the System promised to deliver “a genuinely risk-based approach” to quality assessment. It was light on detail and it did not justify the assertion of authenticity. We were told, simply, that quality assessment would in future be “proportionate” to the “track record” of individual providers.
Until recently, the sector bodies, including QAA and HEFCE, have accepted this characterisation of a risk-based approach. However, the funding bodies’ consultation document in June 2015 appeared to signal a different approach. While the new Quality Assessment Framework would remain “risk-based and proportionate”, the consultation document made no reference to a provider’s track record as being part of that approach.
HEFCE and its partners in Wales and Northern Ireland will shortly be announcing their next steps. Now is, therefore, the time to ask whether a “genuinely” risk-based approach to the regulation of higher education might be in the offing.
Risk after the banking crisis
Following the banking crisis, financial regulators, parliamentary committees and the National Audit Office (NAO) have all looked carefully at the risk-based regulation of the financial sector and whether and how regulation itself had contributed to the financial crisis.
The findings of two of these autopsies are relevant to today’s attempts to develop a risk-based approach to the regulation of HE. The first was undertaken by the NAO for the House of Commons Regulatory Reform Committee (RRC) and characterised a risk-based approach as one which is “informed by good evidence and risk-based assessment so that regulation is proportionate and focused on the risks which matter most”.
The second is the 2009 report of the RRC itself which concluded that the validity of a risk-based approach could be tested by establishing whether it featured: “diligence in understanding risk”; “an awareness that risk assessments … should be subject to appropriate challenge’; a “willingness to be intrusive rather than light-touch when appropriate”; and, crucially, a willingness and ability on the part of the regulator “to use [its] powers more effectively”.
Echoing the NAO, the RRC emphasised the importance of a regulator’s work being “informed by good evidence” or, as the RRC called it elsewhere, “the right intelligence”.
Risk and higher education
So what is risk in relation to HE and its regulation? In 2001, in its Guide to Good Practice in Risk Management, HEFCE defined a risk as “the threat that an action or event will adversely affect an organisation’s ability to achieve its objectives”: note the use of the future tense. Likewise, the 2009 RRC report urged regulators to “focus more on assessing possible future risks” but also to recognise that risks may be “systemic” in origin as well as operating at the individual (or provider) level.
This understanding of risk is also a feature of the Regulatory Risk Framework developed by the Australian Tertiary Education Quality and Standards Agency (TEQSA). The Framework distinguishes between risk indicators (lead indicators) and performance indicators (lag indicators), stating that “lag provides a view of the actual history and … record of the provider” whereas “lead indicators assist in identifying potential emerging risks through consideration of activity that may cause a risk event”.
The distinction is fundamental: lag indicators show whether a provider or its provision is already at risk. Since a provider’s outcomes are, by definition, lag indicators, they can only point to past performance and are not themselves a secure guide to the future.
In 2009, the RRC concluded that the validity of a risk-based approach to regulation ultimately depends on whether the regulator is willing when necessary to be “intrusive rather than light touch”, and to act in a “proportionate” manner, focusing on “the risks that matter most”.
Although the RRC offered little by way of guidance on how a regulator should act, the advice that it does give prompts two basic questions for today’s HE regulators: to what should their actions be proportionate and, once they have found that a provider is exposed to significant risks, just how intrusive should their actions be?
In our view, following the RRC, any judgement about the intensity with which an HE provider is to be scrutinised must be proportionate to its exposure to risks (which may or may not be realised in the future, and which could be located within or outside the provider).
The judgement should also entail an evaluation of the provider’s own competence in identifying, assessing and managing these risks, and it should be made by those with the experience, expertise and training to ensure that the right intelligence is being used.
If this is what the funding bodies have in mind for their new Quality Assessment Framework, it will represent a radical and timely break with what has gone before. The Competition and Markets Authority, the Russell Group and (until recently) QAA have separately advocated approaches that are avowedly risk-based even though they are exclusively provider-focused, retrospective and ignore the dictum that ‘past performance is not an indicator of future success’.
Towards a genuinely risk-based approach to HE regulation?
In the case of QAA, there are signs of a growing awareness of the need for reviews to be more forward-looking. Back in 2013, a QAA publication suggested that a risk-based analysis of transnational education might usefully place less emphasis on an institution’s track record and focus more on ‘how and where’ its collaborative activity is being undertaken. It should also focus more on ‘the future’.
Then, at QAA’s 2015 Annual Conference, the former chief executive made a passing reference to the need for the Agency also to consider how institutions “manage their autonomy and self-assure their own quality”.
More recently, QAA’s response to the Green Paper acknowledged that Higher Education Review had “centred on a provider’s current ability to meet expectations, [which] does not provide assurance about its ability to maintain quality and standards in the future”.
QAA’s response tacitly recognises that its earlier audit-based methods had been more closely aligned with the principles of a risk-based approach because they had “incorporated assurances about a provider’s ability to secure quality and standards both at that time and in the future”. The response concluded that “a truly risk-based approach should reintroduce these [previously] forward-looking elements” together with statements of “future confidence in providers”.
Next steps
The funding bodies’ June 2015 consultation paper suggested that reviews of HE providers should be proportionate to the maturity and capability of their governance arrangements and the robustness of their methodologies for internal review. If these principles can indeed be shown to be at the heart of the new Quality Assessment Framework, they will take us much closer to a genuinely risk-based approach – even though their proposed reliance on outcomes data will give insufficient insight into potential future risks.
The publication of the funding bodies’ next steps will also be significant for what it tells us about how they will reconcile public accountability with provider autonomy when dealing with the diversity of the HE sector and how they will judge whether and in what circumstances they should intervene or hold back.
I found this evidence submitted to the BIS select committee by Alex Griffiths (Kings) to be quite convincing on whether a risk-based approach would work.
Having compared possible metrics to the outcomes of QAA reviews he notes that “No effective model could be developed for identifying ‘unsatisfactory’ providers and prioritising them for review”. He goes onto conclude: “in the case of quality assurance in higher education, a risk-based approach using available data is not feasible”.
http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/business-innovation-and-skills-committee/assessing-quality-in-higher-education/written/23795.html
Thanks for the comment, Mike.
Alex Griffiths’ summary of his findings is that his ‘analysis could not identify any meaningful relationship between the vast array of higher education data and the subsequent outcome of QAA reviews.’
Quite so.
However, it is not clear from his submission whether Alex Griffiths explored the possibility that inbuilt limitations in the method used by QAA to produce HER reports might account for their unsuitability for identifying risks; something QAA itself has more recently acknowledged.
Mike’s comment gives us a welcome chance to clarify our position, and I would like to add to David’s comment.
In paragraph 12 of his submission, Alex Griffiths says that ‘if it proved impossible to identify high-risk providers … any future risk-based approach is unlikely to succeed’. This, and the research that he describes, begs some key questions: what does Alex think a ‘risk-based’ approach entails, what does he mean by the term ‘high risk provider’ and, fundamentally, how does he define ‘risk’?
In the absence of answers to these questions, I assume that Griffiths’ research reflects the received (but contestable) view that a risk-based approach is one in which a regulator’s engagement with a provider is proportionate to the latter’s ‘track record’. (This would be consistent with the fact that most of the items in his data set can be described as ‘lag’ rather than ‘lead’ – or, in TEQSA’s terms, ‘risk’ – indicators).
In our short piece for Wonkhe we question whether such an approach can be described as ‘genuinely’ risk-based. This, of course, leads us to also question whether the research described by Griffiths justifies the conclusion that ‘a risk-based approach…is not feasible’. It all depends on what you mean by a ‘risk-based’ approach!