In an unprecedented effort, a coalition of more than 50 research teams has surveyed a swathe of Brazilian biomedical studies to double-check their findings — with dismaying results.

The teams were able to replicate the results of less than half of the tested experiments¹. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams.

The results provide an impetus to strengthen the country’s science, the study’s authors say. “We now have the material to start making changes from within — whether through public policies or within universities,” says Mariana Boechat de Abreu, a metascience researcher at the Federal University of Rio de Janeiro (UFRJ) in Brazil and one of the coordinators of the project.

The work was posted on 8 April to the bioRxiv preprint server and has not yet been peer reviewed.

Ambitious undertaking

The massive experiment was coordinated by the Brazilian Reproducibility Initiative, a collaborative effort launched in 2019 by researchers at the UFRJ. The scientists wanted to assess publications “based on methods, rather than research area, perceived importance or citation counts”, de Abreu says. And they wanted to do so on a large scale. Ultimately, 213 scientists at 56 laboratories in Brazil were involved in the work.

The project unfolded during the COVID-19 pandemic, which brought numerous logistical challenges. And teams disagreed about how closely to follow the tested protocols. “It was like trying to turn dozens of garage bands, each with its own way of playing, into an orchestra,” says project coordinator Olavo Bohrer Amaral, a physician at the UFRJ.

Reproducibility trial: 246 biologists get different results from same data sets

The authors began by reviewing a random sample of life-sciences articles to determine the most common biomedical research methods used in Brazil, ensuring that any biomedical lab interested in joining the project would be capable of reproducing the experiments.

They ended up selecting three of these methods: an assay of cell metabolism, a technique for amplifying genetic material and a type of maze test for rodents. Then the authors randomly selected biomedical papers that relied on those methods and were published from 1998 to 2017 by research teams in which at least half the contributors had a Brazilian affiliation.

The collaborators initially chose a subset of 60 papers for replication, guided by factors such as whether a paper included certain statistical information. Three labs tested each experiment, and an independent committee judged which of those tests was a valid replication. The coalition performed 97 valid replication attempts of 47 experiments.

Falling short

The authors judged a paper’s replicability by five criteria, including whether at least half of the replication attempts had statistically significant results in the same direction as the original paper. Only 21% of the experiments were replicable using at least half of the applicable criteria.

The authors also found that the effect size — the magnitude of the observed impact in the experiments — was, on average, 60% larger in the original papers than in the experimental follow-ups, indicating that published results tend to overestimate the effects of the interventions tested.