Evaluation is now core practice for higher education providers in the UK.
Whether driven by regulation, or just a desire for enhancement or understanding, providers are routinely, formally and informally, assessing their own practices, strategies and interventions to support students. Such evaluations can serve many purposes, often simultaneously supporting enhancement, fulfilling internal and external reporting requirements, contributing to accreditation or quality marks, and even institutional marketing.
It might seem obvious, particularly for institutions where research is a core function, that this institutional evaluation should be ethical. Indeed, the Office for Students (OfS) makes this very point when it comes to the statutory Access and Participation Plans (APPs), stating that “regardless of whether ethical approval is sought, well established ethical principles should be considered.” TASO has also produced research ethics guidance aimed at evaluators. However, getting from principles to practices can be complicated even where there is evaluation capacity, particularly without formal processes or clear oversight for evaluation ethics.
These aren’t niche issues. Evaluation ethics can relate to the OfS’ biggest priorities – how we apply principles of transparency relates to good governance and institutional integrity, considerations of what good we expect to come from evaluation affects equity, and consideration of harm can relate to student mental health and integrity of data. From our experiences – as evaluators working within four different providers in England – the complexity of institutional evaluation ethics is often overlooked or poorly understood, both within our organisations and in regulation. The challenges this presents aren’t insurmountable, but they do require appropriate scrutiny.
Regulatory and ethical tensions
Ethical evaluation considerations are especially evident in the regulatory environment in England.
Here providers are required by the OfS to show measurable impacts from the interventions in their APPs and to demonstrate evidence-based practices for the Teaching Evaluation Framework (TEF). But, as evaluators, we are pulled between internal and external expectations about the design of evaluations and how (and if) we might share findings.
Regulatory expectations may favour impact evaluations to identify “what works,” sharing knowledge with the sector and publishing in peer-reviewed outputs; even encouraging the use of particular methodologies such as randomised control trials. Internal reporting however may prioritise process evaluations or value for money, utilising methodologies suitable and practicable to small groups and local disadvantage.
Therefore, the role of the evaluation “commissioner” becomes blurred between many stakeholders. There can be a tendency to view evaluation as a “product” more than a “process,” focusing on a final report more than the journey to get there.
The understanding of what evaluation is can become increasingly transactional, meaning that internal reflection and learning may be lost as outputs steer the focus (and resourcing) of evaluations.
When there are different views on what produces the most “good” and for whom, implementing ethical principles around transparency and beneficence can be challenging.
Occupying liminal spaces
Evaluators occupy an uncomfortable liminal position within their institutions.
Whilst academic researchers typically examine phenomena beyond their organisation, we must apply methodological rigour and critical analysis of practices, strategies and interventions from within. Our position requires navigation of internal institutional politics and reputational sensitivities alongside externally imposed regulatory timelines and expectations.
Rarely are evaluators in sole control of all aspects of the evaluation process: decisions on data management and sharing, methodology, data collection or incentivisation may be taken by colleagues or external parties with limited expertise of evaluation ethics, and differing priorities. As such, we may find ourselves recognising the imperative of ethical processes but, as reluctant ethics “police,” we lack the authority to enforce them. Inevitably, colleagues will see different purposes and value from evaluation and may prioritise these over unknown ethical protocols.
Space for ethics of evaluation
But don’t universities’ research ethics committees help?
Besides being an OfS suggestion, ethical scrutiny of evaluation processes is invaluable, bringing welcome rigour, balanced review and acting as a kitemark of quality. Unfortunately, it isn’t available to every evaluator, and research ethics committees may not always appreciate the regulatory compliance timelines behind the collection and use of data and may lack experience of evaluation or educational research, weakening the relevance and depth of scrutiny applied.
Aligning ethics, evaluation, and regulatory scrutiny can be challenging as ethics processes are geared towards research timelines, not evaluation, causing delay. These delays may then shape what can be delivered to students and ultimately what we, and others, can learn.
More importantly, incorporating evaluation into research approval processes does not equip those delivering, commissioning or using evaluation with the expertise to recognise and manage ethical challenges in their specific contexts. Nor does it create the necessary authority and accountability for ethical considerations to be embedded throughout the evaluation lifecycle. Arguably, there are also meaningful differences between some types of evaluation and research, which research ethics committees are not supported to consider.
Some universities have developed bespoke solutions, including separate ethics committees, staff training, and specific governance bodies. Yet these are largely individual responses, often focused on evaluation linked to regulation, for what is a sector-wide issue. There may be more potential for shared approaches to support both smaller, specialist institutions and a sector facing significant resourcing challenges.
Towards a shared responsibility
The challenges of evaluation ethics are not limited to higher education but, as learning institutions, we are well-placed to address some of those challenges, if we accept that they matter.
Currently, the burden for managing ethical evaluation is often falling on just a few individuals within institutions, or no-one, and may be caught between competing institutional and role priorities. Concentrating responsibility for ethical evaluation in the hands of a few individuals risks undermining the centrality of ethics for universities, their accountability for their own practices, and the quality of knowledge they produce.
Taking a whole institution approach to evaluation is one way to ensure institutional alignment, enhance evaluation capacity, and establish ethics as an essential component of evaluation practice. Establishing shared understandings of evaluation, of the centrality of ethics, and processes of ethical scrutiny and governance may be others.
Institutional evaluators can hold the professional expertise to guide ethical decisions and practice, but they alone do not shape how evaluation is commissioned, carried out, and used. Sector and institutional cultures and practices play a significant role and, currently, can work against consistent ethical practice. Recognising this, and the shared responsibilities of evaluating ethically, can be an important first step to finding solutions.
