Universities aren’t short of data on student experience – they are lacking meaningful analysis and evaluation of what works

Universities spend enormous time and energy introducing new initiatives, technologies, campaigns and projects designed to improve student outcomes and satisfaction. Yet we rarely know whether they work.

While the sector has become very good at measuring performance and producing dashboards, it is much less good at evaluating causality. If we want to improve student experience and outcomes, we need to move from a culture of activity to a culture of evidence. As an example, every university can tell you its National Student Survey (NSS) scores. Many can identify areas of concern. Most can list dozens of interventions intended to improve student experience. Far fewer can answer a simple question: which of those interventions worked?

One of the prevalent challenges in higher education is that it often rewards activity more than impact. Action plans, projects, programmes and initiatives proliferate, each well-intentioned but rarely subjected to rigorous scrutiny. Too often, interventions are adopted because they feel intuitively right, are supported by anecdotal evidence, or reflect the latest sector trend, rather than because there is robust evidence that they improve outcomes.

Institutions are in the business of monitoring and reporting, yet comparatively poor at evaluation: dashboards can tell us what has happened, but they rarely help us understand why it happened or whether a particular intervention caused the change. As a result, universities can find themselves mistaking activity for progress, investing significant effort in delivering activity without generating the evidence needed to determine whether that activity has made a meaningful difference to students’ experiences or success. At a time when public sympathy with universities is shaky and the new Teaching Excellence Framework (TEF) is likely to have more teeth, the need for a different approach is clear.

An institutional research laboratory

Universities need capacity to test ideas rather than simply implement them. One such approach is to build what is essentially an institutional research laboratory, bringing together disciplinary expertise, analytical capability and student insight to answer the questions that matter most. UCL’s What Works programme provides an example of this model in practice. It combines academic and professional services experts with PhD-level analysts and student researchers to investigate specific institutional challenges, from teaching and assessment and feedback to student voice.

Analysis and impact evaluation workstreams are commissioned around real strategic decisions, designed to generate actionable evidence. The aim is not simply to understand problems but to provide robust, timely evidence about what is likely to work, for whom, and under what conditions, creating a stronger foundation for educational enhancement and investment decisions.

The value of this approach becomes clear when it is applied to questions that institutions often assume they already understand. Through the programme, we have combined large-scale analysis of NSS data with focused evaluations of specific interventions, moving beyond description to explanation. By examining the relationships between different NSS domains and modelling how institutional factors influence students’ perceptions, the programme has helped identify the drivers of satisfaction.

In doing so, it has often challenged established assumptions. For example, while cohort size is frequently cited as a determinant of student satisfaction, analysis found that module size was a much stronger predictor, suggesting that the learning environments students experience directly may matter more than the size of their programme overall.

We are using the same evidence-led approach to evaluate interventions such as the introduction of a feedback tracker and centrally supported mid-year assessments, providing insight not only into whether they were associated with improved satisfaction and outcomes, but also through econometric modelling, into the mechanisms through which they impacted these metrics. Once you start linking institutional datasets and applying robust statistical methods, some of the things everybody “knows” turn out not to be true.

Moving from performance monitoring to evaluation

Better decisions come from understanding change, alongside sectoral comparisons and raw scores. Many institutions overreact to low scores and underappreciate improvement trajectories. One example of how our evidence-led approach can change decision-making is our new strengths, risks and opportunities framework developed through the What Works programme. Rather than focusing solely on absolute NSS scores, the analysis examines each question through two lenses simultaneously: UCL’s raw positivity score and its distance from the relevant sector benchmark, tracking how both change over time. This provides a more nuanced understanding of performance.

Strengths are areas where the university achieves both high scores and a positive gap relative to the sector. Risks are areas where performance is low and the gap below benchmark remains stubbornly flat or increasing, indicating that the university is neither performing strongly nor demonstrating meaningful improvement. Most interesting, however, are the opportunities: areas where scores remain below benchmark but where the gap is closing significantly year on year. These are often the places where improvement activity is beginning to gain traction and where further investment may yield the greatest return.

By distinguishing between static underperformance and genuine momentum, the framework moves the conversation beyond simplistic traffic-light reporting and towards a more strategic assessment of where intervention is most needed and where it is most likely to succeed. In doing so, it demonstrates how better evidence can lead to better decisions, helping institutions focus not simply on where they are today, but on where change is already underway and can be accelerated.

An infrastructure for evidence

If universities are serious about evidence-based improvement, they need the infrastructure to enable analysis and evaluation at scale. One of the biggest barriers is that the data needed to understand student experience and outcomes are often highly sensitive, limiting access to a small number of specialists. Through the What Works programme, we have begun addressing this challenge by developing synthetic datasets that preserve the statistical properties and relationships within institutional data while removing the risks associated with using identifiable student records.

This creates opportunities for much broader participation in analysis, enabling academic colleagues, PhD researchers and students to engage directly with institutional questions without compromising privacy or data protection obligations. It also aligns with open science principles by making methods more transparent, analyses more reproducible, and findings more open to scrutiny and challenge. The underlying goal is cultural as much as technical: to move evidence generation from a specialist activity undertaken by a handful of analysts to a routine institutional capability that can be mobilised by a much wider community in support of continuous improvement.

The challenge facing universities is not the absence of data: it is the absence of evaluation. We have spent the last decade learning how to measure student experience. The next decade needs to focus on understanding what changes it.

The author ‘s What Works programme co-leads are Daniel Farrell (Director of Student Operations) and Karen Barnard (Director of UCL’s Higher Education Development and Support Institute), and its members and analysts are Elinor Jones, Gill Wyness, Nikki Shure, Susan Smith, Robbie Maris, German Pulido Parra and Haiming Wu.