Since the appointment of John Blake as the new Director of Fair Access and Participation in late 2021, evaluation has been much higher on the regulatory agenda.
In April 2022, the Evaluation Collective came together to publish a manifesto which challenged aspects of the new agenda which were perceived by some as unhelpful for the higher education sector.
At the end of March 2023, the Office for Students (OfS) published its response to their sector consultation on revisions to the access and participation regime.
Evaluation is still front and centre – some aspects and expectations have changed, and others remain the same.
The Evaluation Collective has now published its response to these revisions based on its manifesto principles.
Collective and transformative
In line with the principles in the manifesto, the collective welcomes OfS’ instruction to evaluate the “effectiveness and impact” of activity in the APP, and the focus on institutional learning from their “own and other providers’ work on equality of opportunity”.
Although this is likely to happen through existing sector networks, progress may be slow – and we would welcome any opportunity to accelerate this process through increased regulatory support and the explicit prioritisation of collaborative practices.
Furthermore, we continue to advocate against performative evaluation and note the risk that the new reporting requirements could lead to evaluation being treated as a time bound, box ticking exercise which discourages providers from ensuring that evaluation is embedded in a longer-term approach to practice development.
An empowering tool for social justice
We are pleased to see some development of intervention strategies and specifically how each strategy may address multiple risks to equality of opportunity. Hopefully, this will enable providers to discuss interventions holistically and recognise the complexity of intervention design and implementation in higher education spaces.
The continued emphasis on theories of change is also an important inclusion. However, while we acknowledge the importance of identifying clear quantitative targets in APP intervention strategies, we also recognise that the worth and value of interventions will often be broader than can be captured by a single measure.
We therefore encourage institutions to treat their headline measures as indicators of change, but equally consider broader or more difficult to measure outcomes in their theories of change. This should include a recognition of the value of provisional, partial, and productive forms of knowledge/ data, including qualitative understanding.
Visible, inclusive, and supportive spaces
We understand the pressure facing colleagues in institutions that have chosen to submit their APPs in Wave 1 and appreciate any learning that can be shared with the sector on evaluative expectations.
We hope that the support and guidance offered during this timeframe will recognise the different levels of experience of institutional teams and take steps to address any barriers to evaluation design and implementation, specifically by encouraging cross-provider learning.
Demystifying by rejecting labels and gatekeepers
We acknowledge and appreciate the change in language regarding evaluation and a refocus on inclusive and accessible communications.
Building confidence and capacity
We recognise the continued need to develop confidence and capacity building within institutional teams to rise to these new expectations. The Collective will continue to be led by the sector and make supportive connections to enable colleagues to submit an APP response.
Grounding evaluation in criticality and context
We appreciate the softening of the emphasis on independent evaluation and recognition that practitioner-led evaluation has a place in higher education.
We hope this includes notions of participatory evaluation at intervention level, which moves beyond the OfS expectation that providers encourage student involvement in “the design, implementation and evaluation of the plan”.
We note, however, this may be epistemologically at odds with the desire for “a robust, objective and credible evaluation plan” which suggests there could be judgements made about suitability.
All types of evidence used in evaluation can have value
We continue to believe that OfS’ standards of evidence provide a useful starting point for discussions about evaluation purpose and proportionality. Careful interrogation of the implicit assumptions underpinning these standards should still be conducted to ensure they do not discourage innovative and purposeful evaluation.
We also hope that OfS continues to recognise “that each provider will be at a different stage of developing its evaluation expertise” rather than using the standards of evidence as judgement criteria for intervention strategies in the APP.
Hierarchical assumptions about methods should be disrupted
We welcome the method-neutral stance of the APP guidance, whilst acknowledging the continued specialist remit of TASO for generating causal evidence. We continue to believe that there should be no hierarchy to evaluative evidence, and that pragmatic evaluators can and should make use of a range of methods to enable learning. We would like regulatory language to reflect this.
An inclusive approach to evaluation funding
We encourage providers to include funding for evaluation capacity building in the reported spend on evaluation and for OfS to acknowledge this as integral to APP returns.
In addition, the increased regulatory emphasis on evaluation should now be used as a further mandate within institutions to secure the resources to enable confidence and capacity building in teams across the student lifecycle.
A cultural shift to ensure evaluation leads to learning
Finally, we have some concerns that the timebound publishing requirement (note the regulatory impact of a “reportable event”) may inhibit meaningful reporting and learning.
From experience, evaluations can be “derailed” by many events in the complex world of higher education. We would ask OfS/TASO to review this aspect of their guidance as APPs begin to be rolled out.
The Evaluation Collective is a cross-sector group of like-minded evaluation advocates working to enhance higher education student outcomes. More information, including our mailing list, can be found here: https://evaluationcollective.wordpress.com/
