The incredible machine? Our visual guide to the TEF

This page is now out of date following the publication of the Higher Education White Paper. Find our updated version here.

Some wonks may remember an addictive computer game called The Incredible Machine, in which the player has to construct increasingly convoluted machines to achieve specified goals. The idea of nudging a basketball into a rubbish bin using a set of bellows may stand as a metaphor for contemporary policy design.

Since the Green Paper was released under embargo yesterday, we at Wonkhe have found that one has to read it over and over again, with a set of multi-coloured post-it notes on hand, to really get to grips with the Teaching Excellence Framework proposals it contains. This is because the many different bits and pieces of TEF architecture it describes are scattered throughout the document without any logic, and means are jumbled with ends.

As the government hasn’t thought it beneficial to provide any visual plan of the overall design, we will make the first attempt. Our interpretation sees the TEF as a modular machine, with inputs, processing, and outputs – and each set of components is associated with different rules and conditions. The exercise has helped us to understand what is being proposed, and hopefully it will help others, but we are also happy for this to be ‘version one’ and eagerly invite crowdsourcing of improved diagrams, either based on ours, or completely different approaches. The process has also raised new questions about the TEF that haven’t emerged in the commentary thus far, and these can be fed into the consultation.

The main lesson is perhaps that the TEF may prove to be an ‘incredible’ machine in both senses of the word.

Click here to open the model in a new window. 

KEY (with references to the Green Paper in parentheses)

The inputs:

a. In 2016-17, a satisfactory quality assurance review from QAA, ISI for course designation or equivalent, in place by February 2016, will lead directly to a TEF Level 1 award (Chapter 1, Paras. 26,27). Candidacy for these awards will not go to the Independent Panel, as far as we can tell. Those who reach TEF Level 1 in 2016/17 will be allowed to raise their fees along with inflation in 2017/18.

b. Applications for higher TEF awards will be subject to three ‘pre-conditions’ that will be assessed by the Independent Panel:

• The provider will need an Access Agreement or similar device (C1, P19)
• The provider will need to show it is compliant with ‘market practice’ guidelines set out by the Competition and Markets Authority (C2, P3)
• The provider will need to state whether or not they use a Grade Point Average assessment system (C1, P40); but note the requirement is only to state their position, and actual use of such a system is not to be a ‘prerequisite’ for higher TEF awards

c. Applications for higher TEF awards will be informed by ‘common metrics’ initially drawn from a set of three (C3, P12) to include measures of employment and earnings (starting with DLHE but going on to use data from the HMRC data match), retention and continuation (from HESA’s performance indicators), and student satisfaction (derived from the NSS). These metrics will change over time; in particular, note the NSS is itself under review and several suggestions for others have been put forward (C3, P14). As metrics will presumably change every year, and providers will be on different assessment cycles (no ‘gathered field’ as in the REF; C2, P6) then they will routinely be judged using differently constructed common metrics depending on when they are assessed or re-assessed – they are therefore not really ‘common’ metrics at the point of use.

d. Providers will be able to supplement these metrics with additional evidence, both quantitative and qualitative (C3, Ps. 13, 17), of various types.

e. Reporting of metrics will be dis-aggregated by student background (C3, P4) to show performance in the context of student profile differences.

The processor:

f. Applications for higher TEF levels will be conducted by an Independent Panel, comprised of ‘academic experts in learning and teaching, student representatives, and employer/professional representatives’ (C2, P9); note, there is no proposal to include provider representatives on this panel. The assessment framework will include

• teaching quality;
• learning environment;
• student outcomes and learning gain;

…and various sub-factors are also sketched (C3, Ps. 5,7,8,9). Re-assessments are envisaged to take place on a 3-5 cycle, with trigger events for sooner re-assessment (C2, P5).

The outputs:

g. TEF award level one might best be described as ‘baseline quality assured’, as it is dependent only on the QA review input module. The Independent Panel will make higher TEF awards at either two or three additional levels. Levels 2 and 3 are not further defined, but are indicated to be ‘differentiation levels’ (C2, P15). Level 4 is further defined as ‘requiring performance significantly above expectations’ and/or ‘compelling evidence of excellence’ (C2, P15). This implies that providers can win the ultimate TEF prize by being “better than they really ought to be – if you know what we mean”, and that any provider who doesn’t get to Level 4 may be deemed “excellent alright, but not quite compellingly so – if you know what we mean”. The problem is that we don’t know what they mean.

h. Presumably, it will also be possible to fail a panel assessment and get pushed back out with a Level 1 award, though this isn’t explicitly stated.

i. At some point it is envisaged that these award levels may be given differentially for different subject areas within all providers and that these would then be aggregated to form an award for the provider as a whole (C1, P23); multiple independent panels would then be formed, presumably feeding into a ‘lead panel’ of some kind – needless to say, we haven’t even tried to put any of this in the diagram.

The question is – can you improve on any of this? If you’d like to make your own version, we’ll give a prize to the best and most innovative version, and publish all the submissions. Email mark@wonkhe.com.