Why we can’t get no dissatisfaction

“I don’t want to see threes,” said the Head of Department to the glum-looking all staff meeting. “Our efforts need to be focused on moving the threes to fours or fives”.

It’s an annual conversation familiar to academics and support staff all over the country – and a neat case study of how turning a metric into a target can create perverse incentives. Every year, most reporting of student satisfaction (measured by the National Student Survey (NSS)) is reported using a synthetic measure called “Agree” – produced by simply adding the percentages of students who ticked boxes four and five.

Likert a rolling stone

If you look at the NSS questionnaire, each question is answered on a five-point Likert scale as below:

• 1 – Definitely disagree
• 2 – Mostly disagree
• 3 – Neither agree, nor disagree
• 4 – Mostly agree
• 5 – Definitely agree

There’s also an “N/A” box to tick if the question doesn’t apply – for instance if you are asked about a library and your provider doesn’t have a library or you’ve never used it.

Mainstream reporting (and the all-important OfS press release) will focus on the percentage of students who answered four or five on question 27. It will also focus on whole institutions rather than subject areas. The same applies to the use of the measure in the Teaching Excellence Framework and for other quasi-regulatory purposes, though these will look at other questions.

If we go along with the original purpose of the survey – to flag issues with courses to help prospective students make an informed decision, and to help institutions address issues within their provision – then this is the worst possible way to analyse the survey. It also rewards institutions who influence undecided (three) students to agree (four), and does not reward institutions who work to address the concerns of clearly dissatisfied students (one or two).

Here’s a more useful one – I’ve plotted the percentage disagree (percentage for one plus the percentage for two) against the percentage agree (percentage for four plus the percentage for five) for top-level CAH subjects in institutions. You can filter or highlight by provider, subject area, mission group or region. I’ve defaulted to looking at Q27 (overall satisfaction) and at areas in institutions where the number of students responding is greater than 50 – you can use the controls at the top to change these defaults.

[Full screen]

Three is the magic number

The central value in a Likert scale does a lot of work. An answer of three can mean “I don’t know”, or “I have no opinion”, or “I have no strong feelings either way”, or “I’m bored of answering questions now”, or even “not applicable” (even though there is a dedicated field for the latter). This default is not useful for understanding student satisfaction – it can have too many possible meanings.

But three is also bad news for providers – it does not count towards the all important “% agree” field. Moving a small number of ambivalent or confused students from three to four is one of the most reputationally beneficial interventions a provider can make. It doesn’t take much concerted action (as addressing ones or twos would), and in the popular imagination it is accomplished by making students feel a bit more special during that all important period in term two.

Cake, or pizza, is frequently applied. Meetings where departments listen to students and explain their processes may appear. I’m not against feeding or listening to students, but I do feel that this may not address underlying problems.

Students giving answers of one or two are flagging major concerns with aspects of their course. These are likely to be basic “hygiene” factors – fundamental issues with assessment and feedback, or timetables, or IT resources. Often these are addressable at an institutional level, rather than by department. But as long as enough threes can be moved up to four, these numbers (unless they are truly exceptional) are rarely examined outside of institutional planning offices.

If the NSS is to fulfil a formative role – and plans to add a comparable survey for all years of study suggests that this idea is making a return – students need to feel free to use the whole range to flag areas where their institution needs to perform better. If we really are going to rescue the NSS from being a manipulable reputational signal we need to make changes to the way it is used and presented.