The Knowledge Exchange Framework: year zero

I’ve got a theory about KEF – if it didn’t have such a flashy name it wouldn’t get half the attention it does.

The Knowledge Exchange Framework is not (like REF and TEF are) an “excellence framework”. It doesn’t make any judgement on the quality of business and community interaction, just on the proportional volume and likely output of a number of activities described in the HE-BCI survey data. Neither is it of use to professional or armchair rankers – it doesn’t offer named awards or simple stepped gradations that demonstrate one thing is unfailingly better than another.

It may eventually be used to support the allocation of the £200m Higher Education Innovation Fund (HEIF), which is currently allocated using similar data. But for the first year (2020-21) it is for entertainment and edification purposes only.

KEF under the bonnet

Though we’ll have the results of the first iteration of KEF in the summer, much of the data that underpins them has not yet been published. We’re expecting the next iteration of HE-BCI data from HESA in April – meaning currently we have only two of the three years of necessary data. What’s more, a small amount of data in the form of self-assessments and narrative statements still needs to be collected, and there is a co-authorship of research outputs strand where a suitable source needs to be identified.

Two or more metrics sit under seven perspectives as follows:

PerspectiveMetricsHistoric data available currently? 
Research partnerships (RP)Contribution to collaborative research (cash) as proportion of public funding Yes
Co-authorship with non-academic partners as a proportion of total outputs (data provider TBD)No
Working with business (WB)Innovate UK income (KTP and grant) as proportion of research income No (Innovate UK data not available)
HE-BCI Contract research income with non-SME business normalised for institution size by HEI IncomeYes
HE-BCI Contract research income with SME business normalised for institution size by HEI IncomeYes
HE-BCI Consultancy and facilities & equipment income with non-SME business normalised for institution size by HEI IncomeYes
HE-BCI Contract research income with the public and third sector normalised for institution size by HEI IncomeYes
Working with the public and third sector (WPT)HE-BCI Contract research income with the public and third sector normalised for institution size by HEI IncomeYes
HE-BCI Consultancy and facilities & equipment income with the public and third sector normalised for institution size by HEI IncomeYes
Skills, enterprise and entrepreneurship (SEE)HE-BCI CPD/CE income normalised for institution size by HEI IncomeYes
HE-BCI CPD/CE learner days delivered normalised for institution size by HEI IncomeYes
HE-BCI Graduate start-ups rate by student FTEYes
Local growth and regeneration (LGR)Regeneration and development income from all sources normalised for institution size by IncomeYes
Additional narrative/contextual informationNo
IP and Commercialisation (IPC)Estimated current turnover of all active firms per active spin-out Yes
Average external investment per formal spin-outYes
Licensing and other IP income as proportion of research incomeYes
Public and community engagement (PCE)Provisional score based on self-assessment developed with NCCPE. Optional submission to Research England as part of narrative template to be provided in February 2020.No
Additional narrative/contextual informationNo

The major change from the consultation is the use of a subset of institutional income (funding councils, tuition fees, research contracts) as a way to control for institutional size in place of staff numbers. Income from facilities and equipment are now also included when considering working with business. And a measure of academic time commitment to public and community engagement has been replaced with a provisional self-assessment score based on an instrument co-developed with the National Co-ordinating Centre for Public Engagement (NCCPE).

Metrics are a three-year average, mostly (as can be seen from the table) as ratios, which are converted at perspective level into deciles. This reduces a great deal of data and analysis into what amounts to a set of marks out of 10, which are compared to an average mark from comparable institutions (the infamous clusters) – externalising the strategy office job of identifying sensible comparators via a superbly rigorous report. The composition of these clusters is mostly as in the consultation, there are a few changes to some of the smaller groups that will be made in consultation with the institutions in questions

But what will it look like?

Research England has a grand plan to use spider graphs to show institutional scores alongside cluster averages, with an option to drill down into more detailed data on each metric. I’m not as struck by this as they are – the exercise is designed to support comparisons and spider diagrams are an unwieldy way to do this. I also feel like the individual metrics are still fairly abstract, you have to go quite a long way back down the methodology to get something that the mind can easily take hold of.

But I wanted to give you some sense of what the results would feel like, so – using the last three years of available data – I’ve plotted the top level metrics as two simple bar charts, one each for a provider and their appropriate clusters (using the originally consulted-on groupings plus extras for Wales, Scotland, Northern Ireland and not otherwise grouped, which I’ve included on a separate tab). Think of it as KEF year zero, if you will.

[Full screen]

On the main dashboard you’ll need to manually select the appropriate cluster to compare – I’ve done this to highlight the fact that there is not much difference in terms of these metrics between the clusters, something you can see a little more clearly via the “cluster comparison” tab.

Update: I’ve made some changes to the presentation of the data based on feedback – from today (17 January) high scores are “better” (a higher value in the underlying calculation) than lower ones. In the actual KEF my understanding is that a score of 1 will be the highest, but on reflection I don’t think this makes for a readable set of indicators.

Closing thoughts

I’ve hung around a lot of league table compilers to know about the sniff test – a ranking needs to look right (with the right kinds of provider at each end) as well as have a defensible methodology. However we try to deal with our prejudices, all of us carry around an idea of which providers we would expect to find where in any given instrument – which is what makes this first glimpse of KEF all the more fascinating. Though this isn’t a new set of data or a new area of analysis, Research England has succeeded in developing a refreshing and challenging new tool to make sense of what happens in the sector. I’m fascinated to see it run for real in the summer.

Caveat: Though (as with all these things) I’ve run KEF year 0 to the best of my ability, it is entirely possible that I may have got something wrong. If I have the error is mine – so please complain to me and not Research England. I’ll endeavour to fix issues that people spot.

8 responses to “The Knowledge Exchange Framework: year zero

  1. Looking at the clusters tab, you could conceive a ranking from the sum of decile scores. But, because the analysis is cluster based, it’s impossible to compare between clusters. So that may be step to confound the compilers. (Unless they just get the raw data, nearly all of which is open data, and use that…)

  2. As the comparison is based on deciles, surely all the cluster averages will tend to 5 – so is cluster comparison on this basis valid?

  3. Anne M – the method described by RE is that the decile position is across the whole sector, so when put into clusters you can get a mean average for the cluster that is not 5. (This didn’t have any clarity in the initial consultation on KEF, but the published method is quite explicit).

  4. Hi David, thanks for this – very interesting. Are the tableau charts based on the final metrics, including the HEI income normalisation, or the previously consulted ones?

    C

  5. Great stuff. What data are the public and community engagement scores derived from? the KEF identifes self assessment as the methodology which hasnt been undertaken yet?

Leave a Reply