This article is more than 5 years old

A more refined employment metric

Stuart Johnson takes a detailed look at the benefits and drawbacks of recent refinements in employment metrics.
This article is more than 5 years old

Stuart Johnson is Director of the Careers Service at the University of Bristol.

Employment metrics are forming an increasingly important part of league tables and the Teaching Excellence and Student Outcomes Framework (TEF), and understandably so.

Current and prospective students have a huge stake in their education, and one of the key factors in evaluating whether or not their investment is worthwhile is what kind of employment outcomes they are likely to achieve. But in order for the metrics to be helpful, they need to be reliable.

Imagine you were considering studying Law at university because you held a lifelong ambition to join the legal profession (not everyone who studies Law wants to join the legal profession, but for the purposes of this example let’s imagine you do). You’d think that the employment metrics would be helpful in informing your decision about which university would be the best to study at. But a recent “refinement” to the employment metrics by the Office for Students (OfS) means that, for the subject-level TEF Year Four pilot at least, you’d be wrong.

The TEF Guide to subject-level pilot data, published by OfS in October contained the following, rather technical, information relating to professional qualifications (paragraph 55):

Job titles and descriptions of duties are coded into the Standard Occupational Classification (SOC). Highly skilled employment is categorised as those jobs matched to SOC groups 1-3 (managerial and professional).

So far so good

All good so far. However, it continues…

Higher study is categorised as study that is reported by the student to be a qualification that is at a level higher than the one that they have recently obtained before completing the DLHE survey. This is a refined version of the metric used in TEF Years Two and Three, which considered outcomes in highly skilled employment or any further study.

So, given that the metrics require the follow-on qualification to be “higher” than the student’s previous one means that if our aspiring lawyer wants to pursue the obvious next professional step, this would be classed as a negative outcome in the TEF. A Masters would be a positive outcome, as would a PGCE – but anything that’s not regarded as a step up (including studying Medicine after a BSc in Biosciences) wouldn’t be.

Half an eye on the metrics

Careers services at universities spend huge amounts of time and effort supporting students to figure out what they want to do, helping them develop relevant skills and experience, and enabling them to be successful in selection processes. They also have – more than – half an eye on the employment metrics.

However, there’s sometimes a tension between helping students get to where they want to be when they graduate, and achieving as many positive destinations as possible. That’s because what a student/graduate wants to do (eg. “I want to get into TV production”) won’t necessarily be a positive destination as measured by the DLHE at the six month census point, or indeed the Graduate Outcomes survey at the 15 month census point (are you sure you don’t want to become an accountant?).

Contradiction

The recent refinement, however, makes this not just a tension, but in the case of our student wanting to pursue a career in the legal profession, a direct contradiction. This is because none of the professional qualifications required for you to join the legal profession as a lawyer (Legal Practice Course), a barrister (Bar Professional Training Course), or to convert to Law from another subject (Graduate Diploma in Law) are higher than the one that they have recently obtained before completing the DLHE survey.

This has two significant implications, one for universities and another for prospective students. For universities (in our example, Law Schools in particular), those who have a high number of students going on to professional qualifications that are classified at the same level or lower than an undergraduate degree will be penalised; their subject-level TEF ratings will plummet. For prospective students, they are at risk of being misled if they draw the understandable conclusion that a particular subject isn’t up to much because it’s adrift from its TEF highly skilled employment benchmark, little knowing that this could be as a direct result of a subject producing lots of professionally qualified graduates.

Blunt instrument

I assume this refinement has been introduced to keep in check the number of universities (I think a small number) who have made strategic use of GradCert qualifications in order to artificially inflate their employment metrics – something that does need addressing. However, the refinement is a very blunt instrument that has huge, presumably unintended consequences, for those universities that offer (or encourage their students to progress onto at another institution) professional qualifications that form part of a genuine career plan.

The one thing that encourages me in all this is that subject-level TEF is a work in progress; it’s still in pilot phase. Indeed the TEF is undergoing an Independent Review, the consultation for which closes 1 March. And the employment metrics at this current point in pilot phase are as follows:

highly skilled and higher study – currently derived from the DLHE;
sustained employment – derived from Longitudinal Education Outcomes (LEO) – salary data from HMRC linked to education data from DfE;
above median earnings – also from LEO.

The “highly skilled employment and higher study” metric is an important measure, but this recent refinement has significant drawbacks. Anyone concerned with the integrity of the data and its reliability in informing decisions should raise their concerns with the Independent Review. There’s a real danger that if the integrity of the highly skilled employment metric is undermined it could be dropped, and then we’ll be left only with the Longitudinal Education Outcomes. And that really would be a blunt instrument.

10 responses to “A more refined employment metric

  1. I’m struggling to follow this. I’m looking at the TEF Year 4 guidance right now (https://www.officeforstudents.org.uk/media/c32f61aa-ceb7-4bbb-9f87-2cfc60f2bf89/tefyearfour_metricstechnicalalgorithmsandrebuildinstructions_revised20181219.pdf) and I can only see the statement:

    “Highly skilled employment:
    Denominator of the core metric: TEFEMPEXCL = 0 and TEFEMPINDPOP = 1

    Numerator of the core metric: TEFEMPEXCL = 0 and TEFEMPINDPOP = 1 and
    TEFEMPIND = WORK_HIGHSKILL, STUDY_DEG, STUDY_OPG, STUDY_PGT,
    STUDY_OTHER, STUDY_PHD”

    This says to me that STUDY_DEG is always positive regardless of the level in TEF Year 4.

    Are you saying the technical specification is wrong?

    Edit: oops, I was looking at provider-level guidance and not subject-level guidance!

  2. The general point is well made, but aren’t both the LPC and BPTC at postgraduate level either as PGDips or now wrapped into masters’ courses?

    If metrics are discounting graduate diplomas as ‘higher study’ then that should be changed. But clearly this:

    “the number of universities (I think a small number) who have made strategic use of GradCert qualifications in order to artificially inflate their employment metrics”

    is a concern!

  3. Hi Paul. I need to ask a colleague who knows more about the tech spec than me, but the guidance notes seem pretty clear, as per the article:

    Higher study is categorised as study that is reported by the student to be *a qualification that is at a level higher than the one that they have recently obtained before completing the DLHE survey*. This is a refined version of the metric used in TEF Years Two and Three, which considered outcomes in highly skilled employment or any further study.’

    I’ll try and get back to you on the tech spec…

  4. As someone working on a subject-TEF pilot I can confirm the article is accurate. This really isn’t, I think, all that clear from the technical guidance (which one could also say about many aspects of the metrics) – it only became apparent to us when we began to query the figures (which of course takes time and effort).

    I understand there’s a distinction between definitions applied to provider-level TEF and subject-level TEF, which may explain why colleagues cannot see it in the TEF4 guidance.

  5. Hi Paul. Update from a colleague more knowledgeable than me on the tech spec:
    I think the tech spec you are looking at must be on older one. On page 50 of the more recent subject level document (which can be found here https://www.officeforstudents.org.uk/media/1800f469-142e-429d-8d22-5073a09c33f7/tef_201819_subject_level_pilot_metrics_technical_algorithms_document.docx) it states that STUDY_DEG is only counted as further study at a higher level if the TEFEMPLEVEL (the qualification awarded to the student during the reporting year) = “OUG” (aka “other undergraduate”), and not if the TEFMPLEVEL is “DEG” (aka “first degree”).

    Hope that helps

  6. Thanks Dave. I think that’s exactly right – 1) we only realised the enormity of the technical change when we got our TEF 4 subject-level results and 2) the difference between the definitions applied to provider- and subject-level in TEF4

  7. Hi Mark. Just checked with a colleague who used to be very involved in the coding:
    “I would say it depends on what details your graduate has given about their course in the response to the survey as to how you would code LPC/BPTC. For any course titles we were given as ‘Legal Practice Course (LPC)’ we would code as 06 ‘Professional qualification (e.g. ACA)’. For any that specifically included a master’s qualification too (e.g. Legal Practice Course (LPC) with MA Commercial Legal Practice) we would code as 02 ‘Higher degree, mainly by taught course (e.g. MA, MScm MBA)’. All BPTC have been coded as professional qualifications as not a single person in response to the survey has stated anything about a PGDip element of the programme. I would argue if it has a PGDip element, that’s not common knowledge among those conducting the DLHE survey!”

  8. When investigating the issue we have found that it can be down to how the graduate has responded to the survey, and how the survey data collector has interpreted the information. Two students can be on the same legal practice (masters) course, but one is recorded as higher and the other as professional qualification depending on the wording given over the phone.
    My understanding is that the Graduate Outcomes survey will remove this ambiguity by taking study data directly from HESA returns.

    There are a few little sneaky tweaks that OfS have made to the metric methodology for the current TEF Subject Pilot (worth noting that there are changes to the Continuation measure also). It is interesting that they are trying out methods – I would have welcomed a little more openness/honesty from OfS about what changes have been made and the reasoning behind them. You really have to get deep into the technical algorithm files to find some changes.

Leave a Reply