Timing is everything. Just last Friday, the Office for Students published details of its information, advice and guidance strategy – with research and its resultant approach stressing the need for information for students to be as personalised as possible.
And now we have the Department for Education’s response to the consultation on the TEF at subject level. The question is – do the two hang together?
Sources of information
In the OfS work – which references the multitude of information sources available to students – TEF is posited as just one of many sources of rating information about providers and courses. There is naturally no direct criticism of the TEF (although we do get a retread of both Students’ Union Research Group/Trendence research that demonstrated that the metrics in the TEF are not necessarily those that students think are about teaching quality).
But there’s also a recognition that aggregation is an issue and a sense that big data and clever tools ought to be able to produce something more meaningful for individuals. So it is worth indulging in a thought experiment to see whether a TEF with metrics decided by DfE and then dumped on OfS match OfS’s own vision for personalised information.
Right now, the metrics in TEF are in three categories. Student satisfaction looks at how positive students are with their course, as measured by teaching quality and assessment and feedback responses to the NSS. Continuation includes the proportion of students that continue their studies from year to year, as measured by data collected by the Higher Education Statistics Agency (HESA). And employment outcomes measures what students do (and then earn) after they graduate, as measured by responses to the Destination of Leavers from Higher Education survey – which will soon morph into Graduate Outcomes.
Let’s imagine for a minute that one accepts that putting metrics into a mixer, adding benchmarking, contextual narrative and then boiling into a medal is OK. Let’s also imagine that doing so at institutional level involves so many different courses and student experiences that making these gongs available at course level is at least more meaningful in the student information space. And let’s ignore – just for a minute – the extra metric changes being put into the next round of subject-level pilots.
You may be a student for whom assessment and feedback and the views of your peers are bang on when assessing teaching quality. And you may care about employment outcomes more (doubly as much in fact), although you care about salary and graduate level jobs equally. In this scenario, the algorithm built into the design of TEF is perfect for you as an individual student. As long as you also like benchmarking.
It would then be even more perfect if the metrics were to morph to suit your desire to choose a university on the basis of other students’ perceptions of feedback responsiveness.
But imagine that graduate outcomes only matter half as much to you as assessment and feedback and the views of your peers on teaching. Or imagine that you care much more about a career where getting a graduate level job with a long ladder to salary prosperity than getting on the graduate level rung in the first place. Or imagine that you’re pretty self-sufficient when it comes to assessment and feedback, and that it’s the views of peers on teaching quality that matter more to you. In these scenarios the weighting involved in the creation of the algorithm inside the TEF is inherently faulty. What matters to you might produce different results.
Let your mind wander
Then let your mind wander further. Imagine that you’re a pretty self-sufficient learner and that resources in the library matter more to you than teaching quality. Imagine that you’re the type of learner that needs extra academic support, and that matters much more than the performance at the front of the lecture theatre. Or imagine that other types of outcome – wellbeing, broader satisfaction with the higher education experience all matter more for you. In these scenarios the aggregated metrics selected to be involved in the creation of the algorithm inside the TEF are faulty.
You can imagine further, of course. If you accept that data on outputs matters (not least because outcomes normally involve a shared responsibility), you could be an applicant who wants to judge providers only their side of the bargain. You might be an applicant that wants to see outcomes “raw”, not benchmarked against a particular set of groupings. You might be an application that wants to judge quality on all sorts of factors (some of which are measured now and some of which aren’t). On the subject you’re interested in, some factors in the NSS might make much more sense than others in judging teaching excellence. In fact, your personal preferences, situation and subject choice might mean that any pre-designed set of metrics and weighting don’t work. And if your course is being judged on a set of metrics, you’re certainly not going to want your provider overall to be judged on the same metrics. That would just confuse you, wouldn’t it?
Listening to students
On one level, adding in metrics on learning resources and student voice to the TEF subject level pilots is good news and demonstrates that the department is listening to students and reading the research into what they want. But that still involves tipping averages into an algorithm.
OfS is right. If done right, big data and personalisation – and a new Unistats – offer the ability for students to create their own versions of the TEF – with their own algorithms, their own weightings and their own medals, if they want them; all personalised to the sort of learner that you actually are, rather than the sort of student imagined by a team of DfE officials off the back of aggregated survey research data and press coverage.
Isn’t it therefore odd that the regulator’s sponsor department seems determined to press ahead with an approach which seems to offer precisely the opposite?