TEF results must be carefully communicated to improve students’ choices

Publication of the results of the TEF trial year are now just two weeks away, and can be expected to generate plenty of media coverage. Coming just after the General Election, the results could even be the first major announcement for a new Universities minister.

Unfortunately, we can also expect that much of the coverage will fail to explain what the ratings mean, and instead focus on simply highlighting unexpectedly good or bad outcomes for a selection of institutions. Alongside the media coverage, potential applicants will be made aware of the results via the UCAS system and through Unistats. Results will also appear on the HEFCE register and officials are considering how to deal with queries about TEF from agents, students and parents overseas.

In presenting the TEF ratings, it will be important to provide a proper explanation of what TEF actually measures so that applicants can use the ratings (alongside many other existing sources of information) to help them make informed decisions. Indeed, providing clearer information to students about teaching quality was the original stated aim of the exercise in the first place.

Explaining benchmarking

Rather than simply publishing the TEF results and leaving applicants to come to their own conclusions, the government, UCAS and HEFCE should provide supplementary guidance for applicants explaining what the TEF results mean in practice and how they can be used alongside the range of information about courses and institutions which is already available.

For one thing, applicants are unlikely to be aware that, unlike many other sources of information which students will be used to using such KIS and traditional league tables, TEF does not focus on absolute scores of institutions’ performance against the core metrics. So rather than reflecting how institutions across the sector have performed in absolute terms, the TEF can more accurately be described as a tool to identify the institutions performing well against expectations for their particular student intake.

For example, University A has a relatively high benchmark for non-continuation (say, 11%), and can gain a positive significance flag (making them more likely to receive a silver or gold outcome) with a non-continuation rate of 8.5%, as they will have beaten their benchmark. However, University B has a much lower initial benchmark of 2% of its students dropping out, and meeting this benchmark would not be enough for a positive significance flag. Indeed, University B would not get a positive flag if it exceeded this benchmark unless it meets the tests for statistical significance and “materiality”. This means that an institution where close to 1 in 10 students drop out after the first year gains a positive flag, whilst one where the number of drop outs is closer to 1 in 100 does not. Understanding that the TEF does not measure absolute performance will thus be extremely important to convey to applicants.

Exceptions to the rules

The TEF metrics cover student satisfaction, retention rates and employment and further study outcomes, all of which are very helpful sources of information for applicants. But, it will be important to ensure that applicants know that these metrics are proxies for teaching quality itself and that many other elements of the teaching and student experience they can expect to receive are not covered (at least by the metrics).

Perhaps one of the trickiest issues to address will be in explaining why some institutions have chosen to participate and others have not. Applicants need to know that TEF is voluntary, and that there are good reasons why some high quality institutions may have choosen not to participate. For example, there are concerns that TEF may not complement or add significant value to the enhancement-led approach to quality assurance in Scotland, which is why a number of Scottish institutions have chosen not to participate.

A recent survey by Hobsons suggests international students are likely to be very interested in the outcomes of the TEF; a gold outcome in the TEF was ranked highly in a list of indicators when overseas applicants were asked “what metric tells you most about the quality of a university?”.

International applicants should also be made aware that the TEF does not actually provide much information about international students’ experiences at UK institutions. The majority of metrics in the TEF do not cover international students. NSS data is the only one to include international students’ experiences, and only around a third of Tier 4 sponsoring institutions are eligible to participate in the TEF. Furthermore, almost half of international students in the UK are undertaking postgraduate study and so will be entirely excluded from the TEF.

How the TEF outcomes are communicated at home and abroad is critical. There are a number of risks which must be avoided including a perceived decline in the standing of UK higher education overall if only a small proportion of universities receive gold awards. This is a particular risk as there are no international equivalents to TEF.

The government has made it clear that this is a trial year, and the announcement that there will be an independent review of the TEF’s use of statistics before 2020 is very welcome. Russell Group members will be working closely with government to develop the assessment exercise further with the aim of ensuring that it is truly reflective of teaching quality. Until then, applicants need clear guidance about what TEF results mean and how they should be interpreted to aid decision-making.