The strategic imperative of the TEF was to achieve long-term, systemic, cultural change of the higher education system, providing a permanent counter-balance to the distorting impact of the REF and its predecessors.
This means creating a TEF that is embedded in a sustainable, consistent form, that does not change on an annual basis, focused upon the fundamental priorities of high quality teaching and strong student outcomes. My submission to the Independent Review of the TEF calls for the review to focus on delivering the essential elements of the TEF needed to achieve this goal.
Goals and essential elements
The two principal goals – driving up the standard of teaching and informing students – are fundamentally linked. The REF is responsible for allocating billions of pounds of research funding. If the TEF is to be an effective counter-balance to the REF, it must have a similar impact on matters of direct, material impact to providers. In the current higher education system of minimal grant and no student number caps, this can only be done by impacting reputation and thereby student choice. To ask which goal is more important is a false dichotomy: removing the reputational and informing students element of the TEF would simultaneously destroy its ability to drive up the standard of teaching.
Though the details of TEF may change, there are a number of essential elements that are fundamental to achieving its goals. These are:
- Clear and meaningful differentiation with at least three different ratings. Only clear differentiation can deliver reputational impact and thus the incentive to improve teaching.
- Outcomes, not process, focused. This ensures that the results reflect and measure real-world outcomes.
- Metrics informed but not metrics determined. The strength of the TEF is in the interaction between the metrics and the panel’s judgement to deliver a balanced outcome.
- Benchmarking. Only by benchmarking can high-tariff and low-tariff providers be judged on a level playing field.
Restoring simplicity
The TEF has become overcomplicated. The Independent Review should aim to strip out unnecessary complexity and restore simplicity to the TEF, reducing the number of metrics to 5-6, eliminating supplementary guidance and removing little-used parts of the assessor guidance and contextual evidence. The strength of the TEF is in the panel and they should be trusted and empowered to deal with nuance, borderline cases and exceptions.
Although it is essential to retain clear, differentiated ratings, the names of these ratings are less important. If a decision was taken to move away from Gold, Silver and Bronze, it would be most conducive to public understanding if the rating names were aligned with those used by Ofsted.
The use of robust outcome-focused metrics is a fundamental element of TEF. The wider public rightly see completion rates and progression to highly skilled employment as important measures of university quality and it is entirely appropriate that they should form a major element of any performance framework.
The precise combination and weighting of metrics used is of secondary importance to maintaining the essential elements set out above, provided they are robust, outcomes-focused and suitably benchmarked. However, some recommendations based on the current methodology are:
- There should be no more than five or six core metrics and no supplementary metrics.
- The NSS should ideally account for between a quarter and a third of the total metric weighting.
- All forms of further study, including progression to professional study, should be classed as positive outcomes.
- The time-period assessed by LEO should be moved to align more closely with that assessed by Graduate Outcomes.
- The grade inflation metric should be removed and other non-essential metrics, such the proposed metric on attainment gaps, should not be added.
- Absolute continuation rates lower than 90% and rates of progression to highly skilled employment or further study lower than 60% should attract an automatic negative flag.
- The differing ways in which providers relate to their regions mean there is no easy way to benchmark by region without introducing new distortions.
Subject TEF must not be “big, bossy and bureaucratic”
The principle behind implementing subject TEF remains valid. Performance by subject varies significantly within individual providers. Repeated research has shown that choice of subject is of high importance to applicants when choosing where to study and over 80% of applicants said that subject TEF would be useful to them in choosing where to study.
Nevertheless, the cost/benefit analysis of moving to subject TEF was predicated upon a methodology being found that did not require the assessment of all subjects. The pilots have found that neither model A nor model B were suitable robust. However, the cost of implementing the currently proposed model, in which all subjects are assessed, is a matter of concern and does not meet Jo Johnson’s commitment that the TEF would be “proportionate and light touch, not big, bossy and bureaucratic”.
I recommend that a minimum threshold be applied to subject TEF, such that only subjects with more students than that threshold are assessed. The aim should be to ensure that (a) at least 80% of students in the sector should be in assessed subjects and (b) the number of submissions should be reduced to between 1000–1500. This would also address the concern that the reliability of the metrics and therefore judgements diminishes for small subjects.
Such a change could reduce the cost of assessment by 65%-80%, a saving of up to £28m, while still providing subject information to over 80% of students. The remaining students would continue to be able to rely on provider ratings. Such a change would make the cost significantly more affordable for both the taxpayer and the sector, whilst preserving the majority of the benefits of subject TEF.
If you want metrics to be “robust and outcomes-focused” then why say the NSS should count for up to a third of them? It is not robust, because it is subject to (e.g.) campaigns to lower response rates and because different groups of students will assess their experience against different expectations; it is not outcomes-focused because of the nature of its questions and because it is administered before students have left HE.
However, much of the rest of your article I agree with.
Interesting article Iain, but I disagree wholeheartedly with your reliance and on metrics.
Teaching quality cannot be determined through metrics alone and suggesting it can be undermines Higher Education and learning activities. Making judgements based on metrics that are student-dependant (such as LEO) does not identify good practise or good teaching. It merely identifies early career pathways (which are often determined through personal choice).
Metrics like LEO skim the surface of employment outcomes and are not dependant on quality of education. Students apply for jobs independently and these judgements can be based on things not related to education at all such as location, salary, availability, personal networks, change of career focus etc.
Degrees typically last between 3-4 years, and assessment of those years cannot and should not be metrics driven – particularly given that NSS is a satisfaction survey, not a benchmark of quality.
The QAA used to spend a week assessing an institutions portfolio – this was robust. face-to-face and meaningful.
TEF has not been remotely close to that level of quality assurance because the hasty use of metrics that are to hand was deemed sufficient. It’s like judging a film’s quality by interviewing the audience when they exit the cinema. Some will like it, others will not. But unless you actually watch the film yourself, you will never know it’s strengths and weaknesses.