Metascience is research into the art of research. As the UK Metascience Unit puts it

Our work starts from a simple idea: that the scientific method should be applied to the systems, policies and processes of science itself, so that we can improve them.

Given the enormous amount of money that is spent on research and its importance to everything that we do as a species it would seem that metascience is an obviously good idea. It is therefore surprising that the Metascience Unit has only existed for one year. It has £10m to spend up to 2027 with a remit to fund, experiment, and disseminate insights on the things that make science better and more impactful. Amongst other topics they are interested in funding, metrics, the replication crisis, AI, and the collaborations and incentives that make up the research system.

As we learn from their new report A Year in Metascience they have been busy. As the report rather prosaically puts

Science is a complex social system. And so metascience is a complex social science. The science system is made up of people, teams, institutions, policies, markets, and infrastructure. These elements are combined in multiple different ways to produce an output which is extremely hard to quantify: knowledge. Knowledge manifests in some tangible outputs, like academic papers and patents, but it also makes a vital but more intangible contribution to new technologies, policy, society and culture. To understand the science system, and how inputs best come together to create and use knowledge, we need to draw together multiple disciplines, from economics, sociology, philosophy, history, psychology, computer science, network science, data science, management, and more. Historically, efforts have been siloed across disciplinary lines. Metascience strives to be the unifying umbrella.

Given the enormous purpose, broad access to tools, and cross-disciplinary approach it is unsurprising to find the projects they have funded (23 in total) and the interventions they have tried are extremely varied.

Some of their more interesting interventions include trialling a Distributed Peer Review (DPR) model where funding applicants also act as funding reviewers (like Come Dine With me or Four in a Bed as the report puts it.) The result, on a single funding opportunity trialled, was that it saved significant administration time and turned around the funding bid much quicker. A reasonable proportion of survey respondents were however suspicious that this process will result in bad reviewers (66.2 per cent of survey participants strongly or very strongly disagreed with the statement “this process will result in appropriate reviewers for my application”.)

Elsewhere, there is not enormously strong evidence that partial randomisation- using a lottery for closely fundable applications where the remaining budget cannot cover them all- is highly impactful. Partially, this is because the process of arriving at the cut-off point for partial randomisation is itself a process laden with all kinds of inefficiencies and selection challenges, and in any case even where applications are very close say by 0.2 of a mark the fact they can be rank ordered suggests there is some difference between them. If 0.2 is not significant then why not argue one mark, or ten, or one hundred marks, is not significant either.

There is also emerging evidence that adding more peer reviewers improves the consistency of scoring. As the report points out “Using an Intraclass Correlation Coefficient measure (0-1 range where higher values indicate greater agreement), we saw a move from 0.4 at 3 reviewers to 0.7 at 9 reviewers.” The obvious problem, as the report also points out, is that in an era where peer review is already difficult, finding nine reviewers is exceptionally hard.

To the future then, there will be research on: whether application anonymisation makes things any fairer, whether desk rejection of bids where UKRI staff would take a first pass at proposals could be a viable route to saving capacity, what is going on with FEC, and much more besides

The meta lessons of the Metascience Unit is that small teams with agile funds can push the boundaries of how research works. Metascience has become an increasingly prominent part of the research landscape and it is already demonstrating its value. A government seeking to both cut bureaucracy and innovate in research has an obvious cause to champion in this work.