Differential offer rates do not necessarily indicate bias in admissions

The publication of the second tranche of UCAS equality datasets is another step on the path for greater transparency in university admissions. The release of data is surely to be welcomed, but caution must be applied before being certain about any interpretation of it.

In his Wonkhe blogs (part one and part two) on last year’s datasets, David Morris asked what he termed the ‘million dollar question’: “are UK universities biased in their admissions’ standards towards less privileged applicants?” Morris has suggested a number of institutions, Nottingham Trent University (NTU) included, gave a statistically significant lower number of offers than might be expected to Asian and Black applicants in at least two of the three application cycles between 2013 and 2015. A few other institutions (not including NTU) also gave apparently lower offers than expected to applicants from disadvantaged neighbourhoods, according to POLAR3 data.

The data, as UCAS reported, does not attempt to control for other factors that may influence admissions decisions, such as the subject of the qualification studied, and the exact profile of grades predicted. Nevertheless, such lists are not ones we wanted to be on and the unexplained differences in offer rates between Asian, Black and White applicants prompted us to investigate further. If there is any evidence of unfair admissions policies, this requires serious action to correct it. Therefore, we obtained the raw, anonymised data behind the initial publications and liaised with UCAS data scientists with regards to the exact methodology they had used.

Nottingham Trent: 2015 average and expected offer rates by ethnic group (excluding 'Unknown')

[table id=88 /]

So what did we find? Firstly, we were able to reproduce the figures published by UCAS using their shared methodology in each of the last three years. The above table shows that NTU’s Asian and Black applicants’ offer rates were a statistically significant 2.0 and 3.2 percentage points respectively lower than might be expected, when taking account of subject applied to, the pre-entry tariff, and pre-entry route. Moreover, there remain unexplained, statistically significant differences no matter how we cut the data (e.g. analysing A-Level and BTEC qualification routes separately; breaking down to JACS subject group level and even individual course level). The possibility of conscious or unconscious bias in our admissions decision processes could not be eliminated using statistical techniques alone, using the data available to us. A more holistic approach looking at our specific admissions criteria for different courses was therefore required.

Our more in-depth investigations found that, for the majority of NTU’s courses with statistically significant lower offer rates than expected for Asian and/or Black applicants, limitations with the available data prevented any firm conclusion that there is and admission bias.  As UCAS themselves have noted, their methodology cannot control for other known factors influencing the selection of applicants. For NTU, these factors were primarily:

• The UCAS methodology treats applicants with a BTEC qualification only exactly the same as applicants with a BTEC qualification plus A-Level. In effect, they are all treated as solely BTEC applicants for the purposes of calculating average (expected) offer rates. However, for a number of NTU’s courses highlighted as having unexplained differences in offer rates, one of the pre-requisites is to have an A-Level in a specific subject. Importantly, amongst these courses, Asian and/or Black applicants are disproportionately less likely to hold an A-Level in addition to their BTEC qualification. One would therefore expect lower offer rates for these applicants.
• Amongst applicants with A-Level qualifications only, Asian and/or black applicants were, again, more likely to not have met specific A-Level subject requirements.

By investigating specific courses at the granular level, and despite NTU’s aggregated data suggesting that there are unexplained differences in expected and actual offer rates for differing ethnic groups, we were able to eliminate most of the apparent concerns due to data anomalies. More importantly, we did not find evidence of any systematic bias in our centralised admissions procedures towards any different groups of applicants.

So what should we conclude by all of this? Well, a large handful of salt should perhaps be grabbed when headlines suggest that there may be evidence of admissions bias against certain groups of students, either as a sector or in specific universities. UCAS's public data can only give us a starting picture, when more in-depth analysis may tell us a very different story.

Whilst we should welcome greater transparency of data about admissions, we mustn’t be mislead into thinking we have a definitive problem with unfair processes. Institutions should conduct their own investigations into their data, and in doing so, may well find perfectly explicable for differential offer rates.