Nudge emails don’t work but humans do

The first commissioned three randomised controlled trials (RCTs) of the analytics-to-nudge where providers use engagement data to identify students with apparent wellbeing problems and send them emails or app notifications pointing them to support services.

The second evaluated two small, relational interventions – gym and art sessions at St Mary’s Twickenham, and one-to-one coaching for global majority students at Leeds Trinity. One report tests whether the cheap, scalable thing works. The other asks whether the expensive, human thing works.

Three RCTs at the University of East Anglia, UEA, Northumbria and Staffordshire tested whether light-touch nudge interventions prompted by analytics data had any effect on students’ subsequent academic engagement.

The answer across all three is unambiguous – no measurable impact on attendance at timetabled classes, no impact on VLE logins, and no evidence of a causal link between the nudges and uptake of wellbeing support services.

At UEA, 3,631 undergraduates with less than 60 per cent attendance by week five were sent either a 100 word or 700 word email nudging them towards wellbeing services. The shorter email got more click-throughs – 28 per cent acknowledged receipt versus eight per cent for the longer email – but this made no difference to any of the outcomes that matter.

At Staffordshire, nudges went via the institutional app Beacon, and some students found the display of their engagement data “confusing and unclear.” One student was angry about being watched. At both, staff and students questioned whether engagement and wellbeing were really related to each other in the way the system assumes.

The Northumbria trial matters most. Students were randomly assigned to be identified as “at risk” using either a WHO-5 wellbeing survey at enrolment or the university’s analytics continuation prediction score – and the implementation and process evaluation revealed something striking. Of 3,093 students flagged as at risk by either method, only 261 – about eight per cent – were flagged by both.

Oh dear. The assumed chain is poor wellbeing causes poor engagement, so spotting poor engagement through data means finding students with poor wellbeing. If those two populations barely overlap, the chain is broken.

TASO proposes an alternative model in which external circumstances – financial instability, academic struggles – independently drive both poor wellbeing and poor engagement, rather than one causing the other. If that’s right, a student who stops turning up doesn’t need an email about counselling – they might need money advice, or academic support, or a change to whatever in the institutional environment is making things worse.

And how about who the system actually reaches? At Northumbria, only 54 per cent of enrolled students consented to participate, and consent was gendered – 59 per cent of female students opting in versus 51 per cent of male students. At Staffordshire, Black students were least likely to receive a notification – 23 per cent compared to 40 per cent for students with mixed heritage.

TASO’s key finding:

The assumed link between light-touch communication based on analytics data, wellbeing and academic engagement is not supported by the evidence from these trials.

This wasn’t, by the way, an obscure pilot that happened to disappoint. The analytics-to-nudge approach was one of three approaches championed by the DfE’s HE Mental Health Implementation Taskforce, drawing heavily on an earlier OfS-funded Northumbria pilot. The Student Support Champion and Jisc issued a model specification for providers to follow.

With a shortage of cash around now, it now very much doesn’t look like something you’d throw cash at.

The other problem at Northumbria was capacity. At-risk students did open nudge emails at notably higher rates than the general population – 48 per cent compared to 18 per cent overall – but staff in the evaluation acknowledged “ongoing capacity issues” and said they wouldn’t be able to meet the needs of students the system surfaced without increased resources. The pipeline has assumed the support would be there at the other end. It wasn’t.

It’s worth noting that the Northumbria trial design partially disentangles a question the other two trials can’t answer – whether the problem is the identification or the intervention. Because it randomly assigned students to be flagged by either the WHO-5 survey or the analytics system, and neither group saw improved outcomes, the answer appears to be both. The analytics are finding the wrong people, and even when you find the right ones, emailing them a link doesn’t change what happens next.

What humans can do

Meanwhile at St Mary’s Twickenham, the Boundary Spanner project ran twice-weekly gym sessions and weekly art drop-ins facilitated by academic and wellbeing staff. The sessions were open to all, drop-in, and held on campus at times when students were already in for teaching. University staff were invited to join too.

Conversations during sessions were dictated by students, but often landed on the things that were actually affecting their wellbeing – academic overwhelm, time management, adapting to university life, and making friends.

The evaluation found these sessions gave students a social space to make connections and develop friendships, leading to an increased sense of belonging at university. Facilitators played a central role in providing informal academic and wellbeing support, and the activities themselves were good for students – painting and exercise had value in their own right. For social and wellbeing outcomes, the intervention was the most frequently reported driver of change.

The more telling finding is what happened when staff joined in. Students experienced a different kind of relationship with lecturers present in these informal settings – conversations between equals rather than within teaching hierarchies. That broke down barriers that might otherwise prevent students from asking for help.

The evaluation validated a causal pathway from these new relationships through improved help-seeking behaviour to better engagement with studies. It’s a small thing – a room, some paint, someone who shows up – but what it unlocks matters. Students who feel they belong somewhere are students who stick around and engage.

At Leeds Trinity, the Coaching the Gap project offered one-to-one online coaching to global majority students at Levels 5 and 6 who were considered at risk of disengagement. The coaching was delivered by an external provider, with global majority coaches holding nationally accredited qualifications. Students were entitled to six sessions, set their own goals, and worked at their own pace. An associate professor ran introductory workshops, made proactive contact with all students, and checked in when they missed or rescheduled sessions.

The findings are specific and encouraging. Coaching supported students to achieve their goals through linked processes – breaking goals into manageable steps, having someone to whom they were accountable, developing supportive skills and habits, and “surfacing and reframing limiting beliefs”.

The strength of the relationship between coach and student was what made self-awareness possible. Students engaged more with their degree programmes – submitting work earlier, attending more lectures, proactively seeking academic help from lecturers and tutors. The proactive outreach helped students engage with coaching in the first place, “particularly those living in difficult circumstances.”

Both evaluations also involved peer researchers – students contributing to the research process. The evaluators considered their perspective:

A valuable addition to the research, providing insights and sensitivities to the students’ particular contexts that staff and evaluators could not have.”

That’s a completely different model of the student’s role from the analytics report – not a data point to be nudged, but a co-producer of knowledge about what works. But can you scale it?

The St Mary’s intervention relied heavily on particular staff, and the evaluation notes that this “risks the sustainability of the intervention” in the current financial climate. The Leeds Trinity coaching depended substantially on a single associate professor who did everything from design to delivery to pastoral follow-up. The coaching was restricted to 30 students because of funding. The St Mary’s evaluation ended up with a smaller, more self-selecting sample than planned because the targeting didn’t work as intended.

And both evaluations found the same hard truth – students’ personal circumstances, including finances, health, bereavement, and caring responsibilities, were independent determinants of academic engagement that no wellbeing intervention, however well designed, can fix on its own. The Leeds Trinity evaluation:

“For students with difficult life circumstances, other support systems such as financial support may be needed before engaging with coaching.

The missing measure

Both reports, from completely different directions, arrive at the same finding – belonging, the sense of being part of a community of staff and students with trusted relationships and spaces for connection, is the mechanism that improves wellbeing and engagement. It’s what the analytics approach can’t create with an email, and it’s what the relational approaches activate when they work.

Reading all of this, I’m still annoyed by the decision by OfS to remove the NSS question “I feel part of a community of staff and students”. The one national measure that corresponded to the mechanism both TASO reports identify as central is gone. In its place, providers have no data for the thing the evidence repeatedly says matters most.

The challenge now is making interventions sustainable, extending them beyond self-selecting participants, and embedding them in institutional cultures rather than leaving them dependent on individual champions.

And as such I do keep coming back to the sort of student led departmental/academic society style scaffolding that appears to trundle on without specific support from universities on the continent. SUs have a role to play in establishing it, and all sorts of professional services teams would do well to support it, rather than running unscalable projects that collapse when the pet project manager moves on.

Both reports land on the same uncomfortable truth at the end – the students with the most difficult circumstances are simultaneously the most likely to need support and the least likely to be reached by either approach. The analytics system can’t find them because they don’t consent or don’t fit the engagement proxies. The drop-in sessions can’t reach them because they’re not on campus. The coaching can’t help them because they need financial support before they can engage with anything else. Your mates will, if you have any.