Finding new ways to measure graduate success

The release today of this year’s Destination of Leavers of Higher Education (DLHE) data will spark the annual debate about the usefulness of this self-reported snapshot of what students are doing six months after they graduate.

We’ve also recently learnt that the Higher Education Statistics Agency will review the data it collects on destinations and outcomes for leavers in higher education.

The DHLE is self-reported in two respects, first by students and then by universities. This is probably relatively harmless in relation to the overall employment rate of former students. However, it becomes more of a concern when used as the basis for the calculation of those in graduate level employment. Descriptions of their jobs need to be accurately reported by students and then accurately coded by universities. In some cases this is straightforward but in others – in particular for those students who are entering various forms of self-employment – it becomes reliant on local interpretation. This makes the data inevitably prone to error and potentially liable to gaming.

There are two other significant problems with DLHE: the omission of certain occupations in the Standard Occupational Classification codes on which it is based, most notoriously veterinary nursing, and the fact that many students choose not to pursue their chosen career – including entering a graduate-entry training scheme – immediately after graduating. Admittedly, it does have one virtue; the immediacy that comes with providing data on a cohort which is only three or four years ahead of the potential applicant. In my view this is not enough to offer redemption.

Admittedly, none of these concerns are new. However, the need to find a better alternative to DLHE is given added current impetus by three factors.

The first is that, as vice chancellor, my top priority for Nottingham Trent University is to deliver on the promise to our students that we will enable them to transform their lives by gaining the knowledge, skills, and attributes to pursue the career of their choice. I am sure that I am not alone in this aspiration. It seems to me that we – and they – deserve a more robust measure on which to base the judgement on whether we deliver on that promise than the DLHE provides.

Secondly, the forthcoming availability of HMRC tax data to HESA and the Student Loans Company means that we could use a robust measure where we can select the census point at which we present data on average earnings by university and/or by course. This would not be dissimilar to the approach some rankings take to MBA programmes. With secondary education performance data also being brought into the mix, we have the hope of finding a much needed way to measure added value or learning gain. Other input measures could include: the postcodes of origin of students derived from UCAS; the % of professional accreditations held by eligible courses; and further development of the National Student Survey. These would all provide important factors to set alongside pure earnings data.

The third is that employment rates surely will feature in some form in the Teaching Excellence Framework (TEF); in so doing they will impact directly on the financial well-being of universities whereas currently, whether standalone or contained within leagues tables, this effect is at best indirect.

These issues were explored at a University Alliance roundtable that I chaired last week which brought together colleagues from the civil service, employers, the HE sector with league table compilers.

We acknowledged that the emerging connection between the TEF and the level of fee that universities can charge creates an entirely new environment in which to measure teaching success in higher education. There are major challenges to overcome when determining the metrics we might use. First of all, they will be contested in ways that current league table metrics are not, given they will result in winners and losers in the race to generate income. Indeed, it would be unwise to start with most of the current league table measures. For instance, many of the metrics they deploy would be perverse in the TEF – such as staff-student ratios and spend per student which reward inefficiency – or do not adequately reflect the significant government policy objective of widening participation.

Furthermore, this new context means that the measures must possess political and public confidence. Integrity is one of the great strengths of higher education in the UK. In the age of the TEF this will be best protected by putting the chosen metrics as far as possible beyond the reach of either error or gaming by individual institutions. Given the record in other sectors over the last two decades it would show extraordinary confidence in universities not to do so if that choice is available. This would surely represent a triumph of faith over experience, to assume that higher education would be entirely immune as the financial stakes rise.

Another output measure that TEF may lead us to question is the comparability of degree outcomes across institutions. Some form of national test to assess ‘cognitive gain’ may develop here as it is in the US, albeit we heard at the roundtable that the results to date are not particularly useful for cross-institution comparisons. Notwithstanding the HEFCE pilots in this area and the OECD’s AHELO project, looking at whether it would be feasible to assess what students in HE know and can do upon graduation, such a test will take time to design and deliver.

It may also struggle to gain the support of the sector, which may see it as a step too far. In truth, the evidence from the use of degree classifications in league tables over many years suggests that academics make judgements about individual student performance without any reference to overall institutional standings. Perhaps HEFCE’s proposal of a national register of external examiners to moderate standards more effectively across universities is a practical and palatable way forward here, giving a beefed-up level of verification to the process. In any event, the first iteration of the TEF will need to proceed in the absence of such a national test if it is to facilitate the rise in fees that is becoming pressing in many universities.

However, and to return to where we started, the TEF should not be launched using the DLHE in its current form. In a period when opinion will differ on most aspects of the TEF, I hope we can all agree on that.