Deliver us from rankers

It feels churlish to respond with a weary snarl to the launch last week of THE’s University Impact Rankings. Sustainable development, wellbeing, decent work, gender equality – what’s not to like?

And of course, at one level, anything that incentivises, recognises and rewards universities for their diverse and myriad contributions to society is a good thing. As Jonathan Grant of King’s College London describes, participating in the pilot round of this exercise was, for them at least, a positive experience, which “challenged us to think even more broadly about how we considered our social impact.”

Yet I found myself struggling to applaud this latest beauty parade – or to take seriously the proposition that Auckland is top of the impact pops, and Canada is the best-performing nation. This has nothing to do with my own institution’s – eminently sensible – decision not to take part. Indeed, my reservations about university league tables are both evidence-based and longstanding.

But there’s something about the “impact rankings” that sticks in the craw.

Rankings: Infinity Wars

In part, it is the sheer boredom at the proliferation of increasingly meaningless rankings now being produced on a weekly basis. THE is the worst offender by far – I would list all their rankings here, but it would quickly exceed the word-count of this piece. We all recognise that, as THE itself acknowledges, it has “transitioned from being a primarily UK-focused media company” into a private-equity fuelled “global data and insights business”. But perhaps new owners Inflexion, who acquired THE last month, might recognise that, just occasionally, less can be more, and there is a fast-diminishing return – in attention, if not revenue – from bombarding us with a new league table once a week.

THE reminds me of Marvel – wringing every last twist from its rankings franchise, while its audience yawns and switches off – or prays silently for a higher education equivalent of Thanos to wipe out half of all rankings in the known universe.

More seriously, if THE really wants to encourage a stronger focus on the social, environmental and economic impacts of the university sector, by far the easiest way of achieving this would be to incorporate some of the positive indicators in these new rankings – around equality, employment practices, inclusive education, resource consumption – into its main world rankings. At a stroke, this would achieve what many critics of rankings have called for, by diversifying and pluralizing measures of global “excellence”, and spurring far more than the 500 universities which participated in the 2019 impact pilot to give these issues greater priority.

For all of its bureaucracy, this is what the UK’s REF has achieved, by giving impact a serious weighting alongside research outputs.

Methodology, please

Beyond this, a further way in which rankers could make a genuinely progressive contribution to a more diverse, equitable and sustainable higher education sector would be to treat the communities that it ranks like the data-literate grown-ups that they are. Across all league tables, we should see an acknowledgement of the uncertainties in the data, a concession that weightings are arbitrary and open to debate. A stop to organisations cross selling between rankings and consultancy arms would help, too.

At a time when the big four audit firms are facing growing calls for an enforced separation of their audit and consultancy services, we as a sector need to scale up the scrutiny we apply to the rankers, and demand greater transparency and accountability. Criticisms of their methodological flaws and pernicious effects are nothing new.

We restated these arguments four years ago in my Metric Tide review. But while there has been tangible progress by many universities and funders in their commitment to responsible metrics and evaluation methods, the pace of change among the rankers is painfully slow

One of the UN’s sustainable development goals – SDG 16 – encourages greater accountability, scrutiny and transparency of key institutions. Across our sector, in little more than a decade – rankers have become increasingly visible and powerful actors. If they want to prove that they care about sustainability and social impact, it’s time to put their own house in order.

Otherwise, I expect more vocal and creative grassroots efforts to resist rankings. And I for one will be cheering them on.

4 responses to “Deliver us from rankers

  1. Hello James
    While I share some of your reservations about rankings and league tables, I found your comment about the University of Auckland rather offensive. I live and work in the UK but closely follow my alma mater(graduated 1981). The impact of the research and teaching both nationally and internationally is extraordinary by any standards. The work of University of Auckland scholars is making a remarkable contribution to the lives of people across the globe in many fields, and I was delighted and very proud to see this recognised.
    Kind regards Kath

  2. Not sure the Marvel comparison is valid – the last venture (Captain Marvel) recently passed $1bn in global ticket sales

  3. No offence intended Kath. Like you, I’m a great admirer of University of Auckland, having visited it on several occasions, and planning to do so again later this year. One of my closest collaborators – Peter Gluckman – is an Auckland academic!

    My critique is not directed at Auckland – it’s at the absurdity of anyone claiming that of all the universities in the world, this or any other can be singled out as having greatest social impact! If you can persuade me of the methodological and statistical robustness of this claim, please go ahead. As academics, shouldn’t we be setting higher standards for the claims we make about our institutions – particularly when they are used competitively for marketing purposes – and (implicitly at least) to trumpet our superiority over others? To do this in the name of ‘sustainable development’ is doubly perverse….

  4. The appalling thing about this latest sustainability ranking is that no account whatsoever is taken for the anti-sustainability impact of universities.

    So a university near where I live features in the top 10.

    Yet compared to other universities in our metropolitan area, it excludes students from ethnic minority and working class backgrounds; has and incredibly high proportion of faculty on precarious contracts. It has had (anecdotally) huge problems with bullying; and is infamous for nudging out well performing faculty who live locally, and hiring ‘fly-in’ (literally) international stars at huge salaries which are thereby siphoned out of the local economy

    This is all action which actively diminishes the achievement of UN Sustainable Development Goals, on which the Times’ Higher Ranking outfit claims to base its latest ranking. But simply by pretending their calculators don’t have ‘minus’ buttons, Rankers are complicit in the inflicting of actual social harm.

Leave a Reply