Life is full of surprises.
Sometimes the things we thought we knew as simple unassailable truths turn out to be wrong or, perhaps more often, complex, in ways we couldn’t imagine.
We all know that students go to university and study on courses. The whole narrative of post-compulsory education is based on the idea of a student doing a course, and government policy is increasingly obsessed with them: good courses, bad courses and helping students choose the right course. But what is a course?
A bit of history
In 2011 I celebrated two decades working in HE data. At that time the big news in sector data was the Key Information Set (later rebranded UniStats) which sought to improve the provision of course-level information to applicants. It turned out to be a huge challenge since the policy behind KIS was based on that simple unassailable truth of students studying on courses.
Knitting together course-level data around entry criteria (UCAS), student satisfaction (NSS) and graduate destinations (DLHE) was a massively complex issue because there was no common understanding of what a course is. This had never been a big problem prior to the KIS but with the increasing emphasis on course-level information the question needed to be unpacked and better understood.
A small project ensued, looking at what a course is at different institutions and how the concept changes at different stages of the student lifecycle. A brief report – aimed at policy-makers rather than data geeks – was drafted and published in December 2011.
Why the question is hard to answer
“Course” is defined by providers. We concluded that:
there is a broadly common model of a student registering on a thing that is made up of components (which might include sub-components) and which is defined by some sort of outcome
So the definition of a course within an institution will depend largely on the academic regulations and structures of that institution and is often influenced by regulatory bodies linked to specific disciplines and professions. It often varies significantly within an institution as different disciplines take different approaches to pedagogy and to the type and amount of flexibility that students have. In some cases it is normal for the qualification a student is aiming for not to be established until the student has completed two (or three in Scotland) years of study.
Across the student lifecycle the differences can be even more stark. An institution might have a large number of courses in the UCAS and UniStats data that define specific entry routes to what is actually a single course structure within the institution. This could be reflected differently again in the SLC course data (where data is optimised for the administration of tuition fee payments) and the HESA data (where the definition of a course depends on a complex and esoteric set of factors, largely in the hands of individual institutions).
The data professionals who link and analyse these datasets understand this – so work that links back to the HESA student record (like NSS results, destinations data and LEO) uses broad subject groups rather than having any pretence to analysis by actual courses. But a course can have many (previously 3, now 5) different subject codes – so data for a course can be split across many subject groups – and a subject group can contain data from many different courses delivered in various parts of the provider.
This many-to-many relationship is why the Subject TEF could never say anything useful to an applicant about a specific course. And while we might breathe a collective sigh of relief that the Subject TEF appears to be lifeless – possibly dead – the worrying thing for me is that it ever got as far as it did.
Up to date
Now, nearly a decade after we unpacked “What is a Course”, the paradigm of students on courses continues to dominate policy, as attention turns to the LEO data and the dogmatic attack on low value courses, as demonstrated by this babbling nonsense from one of our recent Secretaries of State.
Much has been written about why graduate salaries say nothing about teaching quality. But beneath this there is an awkward truth about the debate around low quality courses; given the diversity of the sector and the complexity of the student journey it is impossible to build a data model that provides true course-level information.
Life is full of surprises, isn’t it?
Best endeavours of superb teachers cannot plug the gaps in the knowledge and training of students with very low A-level results. Low entry requirements (irrespective of discipline) is, in my opinion, a sufficient condition for a course ( regardless of how we choose to define it) to be defined as ‘low quality’. There will be some who will deny this truism. They would have to explain however, assuming for simplicity uniformity in the quality of teaching, by which process two cohorts with, say, 2 D’s and 3 A’s average entry will attain, by the end of the third year of study, the same level of achievement.
There’s substantial evidence that courses with lower entry requirements often produce better salary outcomes than those with higher entry requirements. Explain – as you say – that?
A-level performance leads to better salary outcomes:
https://londoneconomics.co.uk/wp-content/uploads/2015/03/London-Economics-Report-Returns-to-GCE-A-Levels-Final-12-02-2015.pdf
A-level attainment leads to better university outcomes:
https://dera.ioe.ac.uk/id/eprint/19811/1/HEFCE2014_03.pdf
Highest earners come from the higher A-level performing cohorts:
https://wonkhe.com/blogs/the-terrifying-return-of-the-uks-worst-higher-education-courses/
(Hate to use your own article against you, table 3, this is a weak trend but it cannot be ignored)
In your own articles on the topic, the comparison of courses can seem like there is no difference between courses and universities. However, your general trend of direct comparison is one that takes a low salary outcome course from a “high-status” university and a high salary outcome course from a “low-status” university and use that to demonstrate how the university and entry requirements don’t matter. Bristol Pharmacy students salary after three years is the same as Oxford PPE students about £37,000. But you fail to do a like-for-like comparison Bristol PPE students earn only £19000 after three years.
Oxford English courses want AAA for entry despite the fact that salary outcomes for all English students are low, this is also the case for classics, philosophy, psychology, and sociology, not to mention the creative arts. All of which have high entry requirements at what we shall dub “high-status” universities despite the fact that the whole field has low salary outcomes.
However, STEM (and computing), business, finance and construction courses all lead to relatively high salary outcomes compared to the courses outlined above regardless of entry requirements in many cases. This is not to mention vets and doctors who have the highest earning outcomes after finishing study and have some of the highest entry requirements (this is a specialist field with many years of study beyond the traditional UG course so is a bit of an outlier for general comparison)
The entry requirements for a course do not define outcomes between fields, some areas just don’t pay the same, however, when comparing “like-for-like” courses we see that those with higher A-levels have better university and salary outcomes. The difficult part here is determining what impact the university course actually had. “High-status” universities select for high performance students who then go on to perform well in university and in life, apparently, regardless of where they go to university. There is data on this but I have not read or conducted analysis.
David, you have an agenda that I actually agree with to a degree, but the way you oversimplify things undermines your positions in many cases. The rebuttal here is not one that agrees with the initial comment, in fact I disagree with it, but the simple idea of saying some low entry courses lead to better salary outcomes sometimes isn’t really enough.
Course focus and skills development are what seem to be important when it comes to university courses, accredited courses seem to be important, but we cannot ignore that in a like for like comparison higher achieving A-level students do better in life and university when it comes to salary.
Defining a low quality course will really come down to a combination of these factors, subject area, skills, salary outcomes and entry requirements.
Regardless of subject area a higher entry requirement will be regarded as better quality as a course, this however doesn’t mean that lower entry requirement courses should be disregarded. As ever, nuance and complexity is the answer (how boring).
I enjoyed this article greatly. My favourite bit was trying to guess which bit of “babbling nonsense from one of our recent Secretaries of State” you were linked to without clicking the link, because there’s just so f***ing much.
Subject TEF is a car crash. It was always going to be more misleading (for students) than actual TEF, as it pretend to tell stories about courses while never having a hope of doing so – we’d have swapped an institutional rating (which, even if accurate, would not be accurate for every course, but was at least *called* an institutional award) for false subject ratings (which, even if accurate, would not be accurate for every course, but would absolutely pretend to be).
Having said that, one of the few positives of TEF has been access to benchmarked performance data, as it gives HEIs an insight (albeit one with caveats given the limitations of the data) into their performance relative to the sector (it is junk as assessment, but that doesn’t mean it isn’t a decent prompt for where you might want to look to address issues internally), and forces the regulator to pay attention to institutions which might otherwise coast on the basis of their student intake.
I’m not looking forward to the MKII approach as balloon-floated by the new universities minister, which reverts to just using the absolute data, and relegating anyone without the kind of well schooled (often not especially diverse, and largely independently educated) student body which tends to generate high rates of completion and graduate employability, to “low quality”.
I’d very much like to see the evidence. Thank you. Perhaps that: https://www.thecompleteuniversityguide.co.uk/league-tables/rankings
Try here to start with https://wonkhe.com/blogs/britains-worst-higher-education-courses/
Good piece, but you’re still a newcomer Andy and it’s actually got a lot simpler in the last 20 years than it used to be back in the Thatcher days in English FE/HE, thanks to the virtual wiping out of non-vocational and free standing part time professional “courses”.
I was working in finance originally and remember turning up to my first day release class in Economics at the Oxford College of FE, thinking I was preparing for the chartered secretaries qualification. It turned out the majority of students in the class were taking the exams of the “Chartered Institute of Bankers”. Similar syllabus, so the college combined the teaching to save money (Thatcher was Prime Minister after all!). Nice polite bunch of students of course, all on day release from sitting on counters in Oxford High Street handing the likes of a juvenile Boris Johnson wads of cash from daddy Stanley to buy the Bolly…
The class in Economics was excellent as well. I learnt all about monetary theory – I used to think M1 was how you got to Leeds. I also learned that Thatcher and her government were economic lunatics …
When I subsequently moved over into into registry work, I was responsible for classifying “course” data at the local Poly. I used the massive tome “College Administration: A Handbook” (1980 published by NATFHE and weighing in at 890 pages and 1.4kg) to try to understand the then system …
So I was delighted the other day to see I could buy a copy for 50p or so second hand online. When I opened the parcel I found it had been withdrawn and given away by the University of Kent at Canterbury University Library.
It has an entire chapter on “the grading of courses” and you will particularly enjoy this section:
“The grading of courses system has attracted, in recent years, a good deal of criticism from a number of quarters.
Its ways may seem mysterious …” !
followed by 17 pages of explanation of the mysteries! (pages 537-554).
Serious researchers into the history of classifying “courses” wanting to use Kent University library have no idea what important archival documents they have lost access to …
I like this very much 🙂