High-rise challenge: the new quality assessment model

High-rise, a recent movie based on J G Ballard’s dystopian novel, envisages a society’s decline in what seemed like a beautifully designed and organised tower block. HEFCE recently published its Revised operating model for quality assessment which has echoes of this in promising a perfectly crafted structure for a future quality framework but with features which indicate that there are reasons for significant concern about how things are going to turn out.

There are many troubling aspects of this very far-reaching operating model; it is difficult to see substantiation for the claims that are made that the already significant burden of regulation will be reduced. The language too is challenging and the document has the style and tone of an abstract theoretical industrial quality assurance text and, therefore, feels quite removed from academic realities and the student experience.

A remarkable document

The operating model is presented in a long document, over 50 pages, and it is problematic in many ways. As others, including Mark Leach and Gordon McKenzie here on Wonkhe, have observed there are lots of issues for the sector in what is proposed. It is a big and very far-reaching operating model. So broad is its scope, in fact, I’d suggest it is one of the most remarkable documents in the history of UK academic quality assurance.

The starting point here is HEFCE’s responsibility for quality which was expressed succinctly in the 1992 Act which established all of the funding councils:

The Further and Higher Education Act (1992)

70.—( I) Each council shall——
(a) secure that provision is made for assessing the quality of education provided in institutions for whose activities they provide, or are considering providing, financial support under this Part of this Act, and
(b) establish a committee, to be known as the “Quality Assessment Committee”, with the function of giving them advice on the discharge of their duty under paragraph (a) above and such other functions as may be conferred on the committee by the council.

Given this starting point the HEFCE document presents some real challenges for the sector and there really are many significant problems with the operating model as set out here.

Above and beyond

As noted above, HEFCE’s obligation is to “secure that provision is made for assessing the quality of education” in funded institutions. This document though goes well beyond this statutory responsibility to provide for the assurance of quality. In doing so it offers a major challenge to institutional autonomy, unhelpfully combines quality assurance and financial regulation and assurance, inappropriately draws governing bodies into the detail of quality matters and digs deeply into academic standards.

At the same time as extending dramatically HEFCE’s reach in these areas, the document asserts that “this tailored approach will significantly reduce the regulatory cost and burden for many providers”.

It really does not seem wholly credible though that the proposals contained in this paper, which entail substantial quality assurance changes for universities as well as extended regulatory interventions, can result in a significantly reduced burden.

Outsourcing quality assurance

Not all of this activity though is proposed to be undertaken by HEFCE or other funding bodies:

190….Some activities will be undertaken by the relevant funding body as part of its broader regulatory approach. Other elements are more appropriately delivered under contract by one or more external organisations with relevant expertise and experience.

Table 1 which follows this paragraph highlights six areas in which will be offered under contract to external organisations.

Gateway

• independent peer review visits to test quality-related aspects of the baseline regulatory requirements

Review for established providers

• one-off verification of a provider’s approach to its own review processes
• support for governing bodies
• Stage 2 when investigation and intervention are necessary

Degree standards

• external examining and calibration

International activities

It is not at all clear why these activities have been broken up into six different packages. But more significantly is the question about which body is capable of bidding for this work? There is currently one organisation which could carry out these functions at present, the Quality Assurance Agency, but it would be by no means certain of being in this position after a tendering process. According to HEFCE though there are others:

192. …We do, however, recognise the expertise and experience of those organisations that have been working in this area in the UK for many years, and can provide reassurance that the quality thresholds that any successful bidder will be required to meet will at least be commensurate with the levels currently provided.

Given the activities expected of such organisations, it is far from clear which ones might be capable of delivering all or any of these.

The challenge to autonomy

There are some significant challenges to institutional autonomy in the proposals. Whilst paragraph 43 states that the proposals will “ensure that the autonomy of providers is preserved” this is a rather limited list and does rather beg the question why should we need this reassurance? Autonomy should not be under threat in any way and by listing only some areas – strategy, admissions, determining the student experience and how to improve it, setting standards and external examining – this manages to imply that autonomy is under threat in those institutional activities not specifically identified. Overall, therefore, it is not a terribly reassuring prospectus.

Historically the QAA has sought to cement its position through various means including positioning itself as the recognised quality body required under European Standards and Guidelines which set out institutional quality assurance arrangements and those for QA bodies. Part of the issue here is about ensuring there is some regulatory distance between government and those overseeing quality assurance (so quality does not become something which is prone to political interference). It could be argued that to have HEFCE leading on QA doesn’t meet that requirement and you really need a body like QAA to fulfil this role. However, paragraph 177 makes it clear this will not be a problem:

177. Our expectation is that the funding body in England will seek recognition as the named quality body required by the ESG. We are advised that the statutory basis for the funding body’s quality assessment duty requires it to operate this duty independently from both Government and the sector. It is expected that each funding body will continue to specify its requirements through one or more contracts with third-party organisations but, as now, will retain responsibility for the effective discharge of the statutory duty as a whole. For England, HEFCE’s Quality, Accountability and Regulation Strategic Advisory Committee already confirms annually that it has indeed discharged these responsibilities independently.

Overall therefore the sector is, it seems, to be reassured about the independence of its regulator and the preservation of institutional autonomy.

Combining quality assurance with financial regulation and assurance

HEFCE already spends a lot of time regulating universities. Why not then combine that with quality assurance? It’s all basically in the same ball park:

9. More broadly, the approach is underpinned by the view that ‘quality’ is an inescapable part of an overall approach to regulation that cannot in future be considered separately from the broader context in which an individual provider operates. In particular, the refocusing of regulation around the student interest demands ‘intelligent’ regulation that provides the assurances that matter to students – on degree standards, student outcomes and the academic experience – and makes it impossible to separate out scrutiny of ‘quality’ from other tailored regulatory activity and intervention.

This is a key paragraph. So key in fact it is repeated word for word later in the document as paragraph 42. It is also an extremely challenging proposition. Leaving aside the inverted commas around ‘quality’ (twice) and ‘intelligent’ which only serve to confuse matters the idea that it is impossible to distinguish academic quality assurance – dealing with courses, learning, assessment, etc. – from other non-academic regulatory requirements – such as audited financial statements, financial forecasts, audit reports, data assurance, risk management, value for money, etc – is surely not defensible. Especially given that this is the current arrangement and has been since the establishment of the QAA in 1997. These are very different forms of regulatory activity.

Dictating to the governing body

University governing bodies do, of course, carry overall responsibility for every aspect of institutional activity including academic quality assurance. However, this has in practice been delegated to the most senior academic body, Senate or Academic Board, with appropriate reporting upwards on key issues. This is now not enough it seems as this paragraph notes there will be a new process:

21c. A re-focused periodic Assurance Review visit, conducted by the relevant funding body, to test the basis on which a governing body can provide assurances about the provider’s activities in this area. We will put in place support and guidance for governing bodies as they undertake this role.

Later on paragraphs 111-17 set out how HEFCE does not expect governing bodies to be actively involved in QA. Just much more involved than they are at the moment. This is not something which autonomous institutions will be free to determine their own approach to though:

118. The funding bodies intend to contract a third-party organisation to evaluate any gaps in the capabilities of a range of governing bodies in this area, so as to design and pilot different approaches to additional support for governors.

A third party is therefore going to inspect (autonomous universities’) governing bodies’ approach to addressing QA and then offer support. Three key questions arise from this:

1. Is there any organisation in the sector remotely qualified to do this?
2. is there any way this cannot be seen as both increasing the regulatory burden and fundamentally challenging the autonomy of institutions?
3. And perhaps most importantly, is there any governing body which will accept such intervention in its operations?

(Two minor asides: first, paragraph 126 identifies “meets requirements” as the best possible outcome from this intervention which does not really give institutions much in the way of additional motivation; secondly, paragraph 128 has another strange use of inverted commas where it asserts that a quality assessment regime needs to have ‘teeth’.)

Striking at the heart of institutional responsibility for Standards

Despite the statutory brief that ‘provision is made for assessing the quality of education’ HEFCE has decided that academic standards are very much its business. It is therefore proposed to cover:

concerns about the integrity of standards
“calibration of standards”
comparability of standards
external examining
degree classification

Paragraphs 139 onwards offer justification (such as it is) for these developments including this:

144. The findings of the HEA’s research, combined with the views expressed by students, have confirmed for us that the funding bodies have a legitimate interest in the security of degree standards and their broad comparability, and that we need to address this area. But we wish to do so in a way that is credible to the academic community and respects the autonomy of providers. We have discussed our ‘next steps’ with the sector representative bodies and have asked that they take the lead on this work. The next steps will also involve careful piloting and evaluation before any sector-wide implementation. The funding bodies will, however, wish to satisfy themselves that the sector’s work in this important area does indeed result in the strengthening of current arrangements. We do not propose, at this stage, to take forward the creation of a ‘register’ of trained external examiners. However, we do expect that the sector-wide group convened to oversee the other aspects of this work will wish to consider the most appropriate way to identify those external examiners who have been trained.

While it is pleasing that the original proposal for a register has been dropped there is still a very wide range of interventions here. And there’s more:

151. We will therefore support the sector representative bodies to explore different approaches to the calibration of standards. The intention is not to seek to create common marking criteria for all providers, but rather to establish a simple mechanism to bring together examiners from a subject community (however best described) to compare their students’ work and to judge student achievement against the standards set in order to improve comparability and consistency. We would expect to see this activity result in increased capacity and capability to reach robust judgements about the comparability of standards.

It is surely though not for a funding body to get involved in leading on academic standards. Standards are set by academics in universities who do not do so at the behest of HEFCE. It is difficult to see how making ‘provision for assessing the quality of education’ includes requiring institutions to behave in this way.

And there’s even more:

152. The four UK funding bodies have agreed with the sector bodies a tender specification for an expert body to explore approaches to flexible (online) training for external examiners and to the calibration of marking practices.

Which “expert body” (other than, conceivably, the QAA) is likely to be equipped to do this?

And yet more:

155. The majority of respondents to the June 2015 consultation agreed with the proposal to develop guidance for providers on a sensible range of degree classification algorithms. Support was particularly strong from further education colleges, PSRBs and student organisations.
156. The funding bodies in England, Wales and Northern Ireland believe that there is a clear student interest in this issue and that action also provides an opportunity to strengthen the reputation of the UK higher education sector. We will therefore support the sector representative bodies to undertake detailed research into the range of classification algorithms currently in use and to determine a sensible range of possible algorithms according to the desired pedagogic and other outcomes.

It is difficult to imagine any HEIs requesting guidance and support on degree classification. While organisations without a history, culture, expertise or experience of degree classification may want guidance, it is not at all clear why this would result in such an imposition on universities.

And then there is this paragraph which it is hard to make any sense of:

157. As part of its approach to assuring the integrity of degree standards, we would expect each provider with degree awarding powers to engage at an institutional level with the academic standards against which the achievement of its students are assessed.

The funding bodies should not be involved in academic standards. Standards are set by academics, acting in an academic context, and within a framework for assuring standards determined by a university. As well as adding yet more regulatory burdens all of the activities set out here represent a significant challenge to the autonomy of universities in setting, defining and assuring standards.

Three slightly surprising interventions: students, data and enhancement

There are several interventions described in the document which represent rather surprising additional steps. The first of these relates to students where it is noted that the funding bodies’ visits and engagements with universities do not, unsurprisingly, capture students’ views. Paragraph 99 therefore says:

We wish to plug this gap in our understanding of a provider’s activities, and will therefore establish effective ways to capture the views of a provider’s students.

So the funding council wants to find out what our students think directly. It is not clear why or why it is thought that this would be an appropriate thing to do. Especially given the many ways in which institutions do this themselves through surveys, feedback, course/module/teacher evaluations, staff: student liaison committees, student representation on decision-making bodies, National Student Survey and so on.

Secondly, concerning data, the document notes that universities now have the capacity to analyse a much wider range of data than before and use the findings to improve the student experience. It does seem though that HEFCE wishes to see more of the workings here and not just the outputs as paragraph 85 notes:

We intend to recognise the importance of these existing institutional activities and build them more formally into quality assessment arrangements. Each funding body will expect to see individual providers making effective use of a range of data relevant to their mission, provision and students. This routine analysis will provide the basis for internal reflection with students on the quality of student outcomes and the learning and teaching experience that is delivering them, as well as a framework within which continuous improvement activities can be planned and implemented.

So it looks like funders will examine the way in which universities use data to ensure they are making the most of it. This does seem quite intrusive.

Thirdly, innovation is promised. It is intended that the new quality assessment arrangements:

should seek to encourage innovation in learning and teaching rather than driving providers towards risk-averse activities and homogenised provision  (paragraph 40)

But this form of quality assessment cannot deliver enhancement in the way that it is optimistically supposed here and in paragraph 87. This kind of approach demands compliance which is not compatible with innovation. You don’t fatten a pig by weighing it; you can’t inspect quality into a course.

Three rather surprising approaches, therefore.

The new arrangements are unlikely to reduce the burden of regulation

10. We believe that this tailored approach will significantly reduce the regulatory cost and burden for many providers…

Given the scale of the activities set out in this highly detailed document, it is tough to conceive how this could be the case.

The starting point for the new assessment regime is a ‘one-off scrutiny process’:

86. To ensure that all established providers are able to meet this set of expectations, we will verify the approach each takes to its own periodic review processes through a one-off scrutiny process. The purpose of this external verification is to ensure that the chosen approach is sufficiently focused on improving student outcomes and has sufficient strong external scrutiny built into it.

As an opener, this does seem surprising given the track record all institutions already have with the Quality Assurance Agency. It effectively discounts all previous external quality interventions and the additional one-off scrutiny cannot be seen as anything other than an additional burden.

Beyond this the new model for periodic inspection is claimed to be flexible and sensitive to each institution’s context and position:

87. The funding bodies are not looking for a single model for internal periodic review. Indeed, the reverse is the case: the approach chosen by a provider should be informed by its context, provision and students. The funding bodies intend to contract a third-party organisation to undertake the detailed design and operation of this verification process.

This all sounds fine in principle but the details are nevertheless at least partly prescriptive and universities will have to change to adapt to this new regulatory regime.

Then it all sounds like it will be simple. As paragraph 90 sets out this will all be tested through an annual scrutiny process which will combine quality and other regulatory assurance. Again this does not sound like a set of activities which will “significantly reduce” the burden on universities. Moreover, it is claimed that there will be a risk-based approach here:

47. This model ensures that enhanced regulatory scrutiny can be targeted on the providers that represent greater risk to the student interest and to the reputation of the sector as a whole

How this targeting will happen is not clear. Although there is greater scrutiny for new entrants to the sector and there is the promise of flexibility the periodic assessment process does look like a rather blunt instrument. Moreover, while it is stated in paragraph 187 in relation to the Green Paper proposals that:

HEFCE will use its central role in these activities to ensure that quality assessment and the TEF are designed and can be delivered as one coherent system

No matter how coherent the incorporation of the as yet still under-specified TEF into the new quality assessment regime it really does not seem credible to suggest that this is anything other than an additional regulatory requirement which cannot be anticipated to reduce the burden and cost of regulation for universities.

An integrated and seamless model for quality assessment?

Paragraph 189 expresses confidence that the new model can deal with any of the alternatives set out in the Government’s Green Paper:

We have tested the arrangements for delivering the revised model for quality assessment against these proposals, and are confident that our approach to implementation can transition smoothly into any of the range of options set out in the Green Paper, particularly those that relate to potential roles for HEFCE, any successor body, and other sector bodies.

Given the disjointedness of the Green Paper proposals (see for example this comment on the AHUA blog and this post on Wonkhe) it does not seem wholly credible that the two new regimes can be seamlessly integrated as promised here.

There remain a number of troubling features of the operating model. The model goes significantly beyond the statutory responsibility of the funding council to “secure that provision is made for assessing the quality of education” by, among other things, digging deeply into the heart of the institutional responsibility for academic standards. HEFCE anticipates outsourcing most of this regulation but there remains the question about which body, other than the QAA, is capable of undertaking any of this work.

Combining academic quality assurance with financial and other regulatory assurance and the challenges to institutional autonomy through the new detailed engagement with governing bodies are also major challenges. And it really is hard to envisage how the cost and burden of regulation can be reduced under these new arrangements.

Given all these issues with the operating model it would not seem unreasonable to hope that HEFCE would be approaching its implementation in a measured and pragmatic way and taking the time to ensure that it will actually work and deliver the promised significant reduction in the regulatory burden. However, paragraph 39 indicates that, despite these concerns and the huge uncertainties in terms of the implementation of the Green Paper proposals, HEFCE believes that this all needs to happen now:

We have concluded that it is important to begin implementing the revised operating model now, to ensure that transition arrangements are secure and that each funding body is able to fully discharge its statutory responsibilities during 2016-17 and beyond.

This does not seem to me to be the right approach. Although this operating model may not lead to an academic regulatory dystopia it is extremely difficult to imagine it either improving the regulatory landscape or reducing the cost and burden of regulation on universities.

It’s time to think again.