There has arguably never been as much focus as there is today on the evaluation of widening participation outreach interventions.
In case you hadn’t noticed, the most recent DfE widening participation figures reveal that the HE participation gap in England between free school meals (FSM) eligible and non-FSM eligible students is the widest it has ever been since the DfE first started compiling the data 16 years ago.
Within this context, the renewed emphasis on evaluation that has emerged in the John Blake era of access and participation regulation should be welcomed by the sector. It has never been more important for us to know which WP interventions tend to ‘work’, and also which ones tend not to.
What applicants actually do
Evaluating WP outreach interventions is no straightforward task, even when you have really good quality data in front of you. To support a piece of academic research on WP outreach that I’ve recently published, I was lucky enough that Realising Opportunities (RO) kindly supplied me with comprehensive anonymised data concerning participants in their WP outreach programme. RO is a large national outreach programme for 16-18 year olds delivered across a consortium of research-intensive universities, with over 1,500 young people being recruited to take part in the programme in 2023. The aim of the game is to support more young people from disadvantaged backgrounds to progress to more selective universities in particular.
Given that RO staff had diligently tracked (using HESA administrative data) the destinations of RO programme participants, I was able to observe the proportion of RO students who do indeed eventually enrol at a research intensive university. Of the 769 students who took part in RO between 2015 and 2017, 248 of them ended up at a university which is in the DfE’s “top third” of universities as ranked by average tariff upon entry. This was 32.2 per cent of the participants – a figure which might seem a little on the low side until you consider that, of the whole cohort of young people the same age, only 10.2 per cent ended up at a top third university.
Obviously, such a crude comparison of percentages reveals very little given that the RO participants were certainly not representative of the wider cohort of all young people. Not only did they tend to be more disadvantaged, they also attained much higher grades on average in their GCSE exams. But a more meaningful comparison can be made possible with a little statistical modelling.
Diving deep into the data
In England we are extremely lucky to have at our disposal both the DfE’s mammoth National Pupil Database (NPD) and the HESA student record, both of which can be made available to researchers in the right circumstances and both of which can be linked together for the purposes of statistical analysis. These datasets are a goldmine for education researchers like me, but so far they have been exploited far too little by those who evaluate WP interventions.
By linking the NPD and HESA datasets together, I was able to use an entire cohort of data concerning over half a million young people in England to produce a statistical model which could generate a percentage likelihood that a young person would end up at a top third university, based on both their school attainment (at age 16) and their personal characteristics such as their gender, ethnicity, FSM eligibility and neighbourhood of residence. Details of all of the 769 RO participants could then be entered into this model, which could then offer an overall prediction of the number of participants who might be expected to progress to top third universities based on their personal characteristics and school attainment. The modelling suggested that ordinarily 107 of the participants would be expected to progress to top third universities (which is 13.9 per cent of them), whereas as noted above the actual top third progression rate was 32.2 per cent.
This suggests then that participation in RO is indeed positively associated with an increased likelihood of progression to a top third university, though we can’t be sure that the effect is causal given that not all potential confounding variables can be accounted for in the modelling.
This data analysis revealed that the RO programme is very well targeted, and not just because it succeeds in recruiting students from a disadvantaged background (which it does). Crucial to the success of the programme is the fact that – as revealed by the statistical modelling – most students recruited do not appear to be on a pathway to a more selective university prior to their involvement in the intervention. This is good news for RO, as it gives them a large margin for improvement and leads to the possibility that, in many cases at least, their programme could be what tips the balance in terms of whether a given young person does or does not end up at a research intensive university. Whilst the data I used for my analysis is now a little old, more recent data shows that the RO programme continues to go from strength to strength, with 63 per cent of those completing the programme between 2019 and 2021 progressing to a research intensive university in the year after programme completion.
Preaching to the converted?
Appropriate targeting of WP outreach programmes is crucial, but unfortunately there may be some outreach programmes which, data would suggest, are not as well targeted as they might be. WP outreach programmes which have the aim of simply supporting young people to progress to HE per se, rather than supporting progression to more competitive universities or courses in particular, seem to be particularly difficult to target successfully.
Take, for example, the recent evaluations of WP summer schools published by TASO. These evaluations concerned both online and in-person summer schools delivered in 2021 and 2022 respectively. In an ideal world, these programmes would recruit cohorts of students who were not interested in going to university, but due to their participation in the summer schools, students would have their minds changed. However, it’s fairly clear from the evaluation of these interventions that this is unlikely to happen as things stand. In the case of the online summer school, 94 per cent of participants stated that they were either “likely” or “very likely” to apply to university when quizzed about this before taking part in the intervention. For the in-person summer schools, this figure was similar at 95 per cent.
For programmes such as these, there is a big risk that they simply preach to the converted. Yes, many participants will progress to HE after the intervention. But there are many signs to suggest that the very same outcome would have been achieved even in the absence of the intervention. This is not to say that such programmes are badly designed in terms of their content. The content of these programmes may well be such that, if they were delivered to young people who were initially disinterested in HE, these young people would emerge more enthusiastic about university by the end. But we may never know just how effective these programmes might be if they continue to be targeted in the way that they currently are.
The targeting challenge
In practice, recruiting suitable cohorts of students for programmes which focus on access to more selective universities is likely to be somewhat easier compared to recruiting to programmes which focus on access to HE in general. In the first case, students who are already likely to be interested in HE just need to be nudged towards a different university or course choice. But in the second case, outreach providers may find themselves with the difficult challenge of trying to recruit participants with no existing inclination towards HE to take part in an HE related programme.
There is an inherent issue here which we probably don’t discuss often enough – if a young person is not interested in going to university, why would they voluntarily choose to take part in a WP outreach intervention? There doesn’t seem to be an easy solution to this problem, other than perhaps making participation in outreach interventions a bit less voluntary. This is achievable with interventions catering to those under age 16, if school pupils are directed by their teachers to take part in an outreach intervention which is delivered during the school day. But the cost-benefit analysis of doing this always has to be carefully considered, given that withdrawing pupils from their lessons risks having a negative effect on attainment. We know from existing evidence that it is in fact attainment – not aspiration – which has the biggest bearing on the likelihood of a young person participating in HE.
We always need to remember that the success of an outreach programme is a function not just of the quality of the intervention itself but of the average character of those who are recruited to take part. Outreach providers put a great deal of time and effort into designing and delivering high quality interventions. It might be the case then that the single best way to improve outreach would be for the sector to take a more careful look at who is taking part.
If you are interested in learning more about RO, please contact Sarah Beech, Head of Realising Opportunities (sarah.beech@newcastle.ac.uk).
