Graduate Outcomes gets tested

Graduates, eh? They don’t understand anything.

Specifically, in fact, I have heard it argued that graduates do not entirely understand the Graduate Outcomes survey. And because a poor showing on that metric can lead to the Office for Students turning up on campus in the regulatory concern helicopter, the way in which graduates answer questions about their outcomes has become an issue of significant concern.

HESA, who just occasionally produce reports so delightful that I literally stand up and applaud, commissioned IFF research to do what is called “cognitive testing” on the current Graduate Outcomes survey. The report (out right now) suggests that, in the main, graduates do understand what is being asked of them and answer accordingly. Which is very good news for regulators and anyone else who wants to understand graduate destinations.

Cognitive testing is pretty much that: a way of ensuring that the questions you are asking are understood in the way that you intend and answered accordingly. IFF contacted just under a 100 students across seven key “activity” groups (in work, in study, developing a portfolio, and so on) over the summer. In the interviews that followed, research was concerned with testing changes and potential new questions alongside existing wordings to see how graduates understood them and how they might respond.

The areas of inquiry came from concerns raised by statutory customers and institutions. In particular, concerns were around reporting activities (particularly around understanding the concept of “portfolio building”), the job quality questions,  and defining reasons for unemployment – plus approaches around what qualifications are required for current work, and the location of employment.

I’ll cut to the chase early on – the review did not find any massively substantive issues, and the recommendations made as a result are correspondingly minor. It’s very much what you would expect from testing a survey that had been through a thorough regime of assessment before it was first used, and has had experienced eyes over it the whole time it ran.

The closest we got to controversy is on qualification requirements. This is reckoned to be (in part) because graduates themselves don’t often know what qualifications are required: as the report puts it “reliability is inherently limited by graduates’ knowledge and perceptions of their employer and their qualification(s)”. It’s difficult, for example to parse whether a job that “requires” a degree in a job advert actually requires either a degree or a specific degree. A broader discussion is needed to decide whether the benefits of this question outweigh the drawbacks.

As the report put it:

While the degree may have ‘felt’ necessary to them, this is not necessarily the case universally. This highlights that the perception of whether a qualification was required for the job is subjective for the graduate and may also have been a subjective decision made by the hiring manager.

There’ll also be a bit of clarity brought to the location of work questions – the reality of home and hybrid working patterns means that sometimes respondents are not clear whether they should give their home address or the nominal address of the company the work for. Here, the recommendation is for a new location question on working patterns, and building this information into the existing questions on working location.