CAN BRAIN SCANS REVEAL
associations between MRI scans and various cognitive, behavioural and demographic traits, in samples ranging from 25 people to more than 32,000.
In simulated studies involving thousands of people, the researchers identified reliable correlations that linked brain structure and activity in particular regions with behavioural traits — associations that they could replicate in different subsets of the data. However, these links tended to be much weaker than those typ- ically reported by most other studies.
Researchers measure correlation strength using a metric called r, for which a value of 1 means a perfect correlation and 0 none at all. The strongest reliable correlations Marek and Dosenbach’s team found had an r of 0.16, and the median was 0.01. In published studies, r values above 0.2 are not uncommon.
To understand this disconnect, the research- ers simulated smaller studies and found that these identified much stronger associations, with high r values, but also that these findings were not replicated in other samples, large or small. Even associations identified in a study of 2,000 participants — large by current standards — had only a 25% chance of being replicated. More typical studies, with 500 or fewer partic- ipants, produced reliable associations around just 5% of the time.
Even larger studies
The study did not attempt to replicate other published brain-wide association studies. But it suggests that high r values common in the literature are almost certainly a fluke, and not likely to be replicated. Factors that hinder reproducibility in research, such as the ten- dency to publish only statistically significant results with large effect sizes, mean that these spurious brain–behaviour associations fill the literature, says Dosenbach. “People are only publishing things that have a strong enough effect size. You can find those, but those are the ones that are most wrong.”
To make such research more reliable, brain-imaging studies need to get much bigger, Marek, Dosenbach and their colleagues argue. They point out that genetics research was plagued by false positives until researchers, and their funders, started looking for associa- tions in very large numbers of people. The larg- est genome-wide association studies (GWAS) now involve millions of participants. The team coined the term brain-wide association study, or BWAS, to draw parallels with genetics.
For brain imaging, Marek says, “I don’t know if we need hundreds of thousands or millions. But thousands is a safe bet.”
“What the Marek paper suggests is that a lot of the time, if you don’t have these really large samples, you are most likely wrong or lucky in finding a good brain–behaviour cor- relation,” says Caterina Gratton, a cognitive neuroscientist at Northwestern University in
BEHAVIOUR? BOMBSHELL
STUDY SAYS NOT YET
Studies linking features in brain imaging to traits such as cognitive ability could be too small to be reliable.
IBy Ewen Callaway
n 2019, neuroscientist Scott Marek was asked to contribute a paper to a journal that focuses on child development. Previous studies had shown that differences in brain function between children were linked with
performance in intelligence tests. So Marek decided to examine this trend in 2,000 kids.
Brain-imaging data sets had been swelling in size. To see whether this growth was mak- ing studies more reliable, Marek, based at Washington University in St. Louis, Missouri (WashU), and his colleagues split the data in two and ran the same analysis on each subset, expecting the results to match. Instead, they found the opposite. “I was shocked. I thought it was going to look exactly the same in both sets,” says Marek. “I stared out of my apartment window in depression, taking in what it meant for the field.”
Now, in a bombshell Nature study, Marek and his colleagues show that even large brain-imaging studies, such as his 2019 effort, are still too small to reliably detect most links between brain function and behaviour (S. Marek et al. Nature 603, 654–660; 2022).
As a result, the conclusions of most published ‘brain-wide association studies’ — typically involving dozens to hundreds of participants — might be wrong. Such studies link variations in brain structure and activ- ity to differences in cognitive ability, mental health and behavioural traits. For instance, numerous studies have identified brain anat- omy or activity patterns that, they say, can distinguish people who have been diagnosed with depression from those who have not. Studies also often seek biomarkers for behav- ioural traits.
“There’s a lot of investigators who have committed their careers to doing the kind of science that this paper says is basically junk,” says Russell Poldrack, a cognitive neuroscien- tist at Stanford University in California, who was one of the paper’s peer reviewers. “It really forces a rethink.”
The authors emphasize that their critique applies only to the subset of research that seeks to explain differences in people’s behaviour through brain imaging. But some scientists think that it tars this field with too broad a brush. Smaller, more detailed studies
of brain–behaviour links can produce robust findings, they say.
After his botched replication, Marek set out to understand the reasons for the failure together with Nico Dosenbach, a neuroscientist at WashU, and their colleagues. That work resulted in the latest study, in which they ana- lysed magnetic resonance imaging (MRI) brain scans and behavioural data from 50,000 par- ticipants in several large brain-imaging efforts, such as the UK Biobank’s collection of brain scans.
Some of these scans gauged aspects of brain structure, for instance the size of a particular region. Others used a method called functional MRI (fMRI) — the measurement of brain activity while people do a task, such as memory recall, or while they are at rest — to reveal how brain regions communicate.
The researchers then used subsets drawn from these large databases to simulate billions of smaller studies. These analyses looked for
A scan using functional magnetic resonance imaging, or fMRI, shows areas of the brain that are active during speech.
“A lot of investigators have committed their careers to doing the kind of science that this paper says is junk.”
Nature | Vol 603 | 31 March 2022 | 777
ZEPHYR/SCIENCE PHOTO LIBRARY
.devresersthgirlA.detimiLerutaN
regnirpS
202 ©

News in focus
Evanston, Illinois. “This is an important paper for the field,” she adds.
But some researchers argue that smaller BWAS studies still have value. Peter Bandettini, a neuroscientist at the National Institute of Mental Health in Bethesda, Maryland, says that studies such as the ones Marek’s team simulated looked for correlations between crude measurements of behaviour or mental health (self-reported surveys, for example) and brain scans whose conditions might vary
from participant to participant, diluting bona fide associations.
By selecting participants carefully and analysing brain-imaging data using sophisti- cated approaches, it might be possible to find associations between brain scans and behav- iour that are stronger than those identified in the study, says Stephen Smith, a neuroscientist at the University of Oxford, UK, who leads the UK Biobank’s brain-imaging efforts. “I fear this paper may be overestimating unreliability.”
co-author Christopher Gill, a global-health specialist at Boston University in Massachu- setts. One of his colleagues in Zambia died from COVID-19 while working on the project.
“It’s not hypothetical to me,” says Gill.
The relatively low numbers of reported COVID-19 cases in sub-Saharan Africa led to the perception “that severe debilitation and deaths caused by COVID-19 were somehow less in Africa compared to other continents”, says Yakubu Lawal, an endocrinologist at the Federal Medical Centre Azare in Nigeria.
Lawal and other scientists speculated2 that the relative youth of Africa’s population might have helped to spare the continent, but also suspected that official numbers were under-reported. The question was by how much.
Missing COVID cases
Gill and his colleagues in Zambia tested bodies in one of Lusaka’s largest morgues for SARS- CoV-2 in 2020 and 2021. Test positivity was 32% overall — and reached around 90% during the peak of the waves caused by the Beta and Delta variants. Only 10% of the people whose bodies were found to contain the virus after death had tested positive while still alive.
Gill and his colleagues can’t confirm that all of these people died of COVID-19, but the results still contrast sharply with official numbers. So far, there have been fewer than 4,000 confirmed COVID-19 deaths in Zambia, home to around 19 million people. Separate findings published3 on 10 March suggest that Zambia’s ‘excess’ deaths — those above what would usually be expected — in 2020 and 2021 exceeded 80,000.
The Lusaka numbers mesh with statistics from South Africa, where a 2021 study found that only 4–6% of SARS-CoV-2 infections in two communities were officially documented4. Further study5 of the same communities showed that 62% of participants had been infected at least once from July 2020 to August 2021. Co-author Cheryl Cohen, an epidemiol- ogist at the University of the Witwatersrand in Johannesburg, South Africa, says that many of these infections were asymptomatic, but that people with symptoms might have gone undetected as well.
Gill suspects that a major reason for the gap between his results and official counts is that most people in Zambia who die of COVID-19 do so outside medical care. Four out of five people tested in the study were never admitted to a hospital.
But not everyone is convinced that the Lusaka findings invalidate the idea of the African paradox. In Ethiopia, for instance, “our experience is people get infected with the virus, are asymptomatic or have mild symptoms, and recover”, says Amare Abera Tareke, a physiologist at Wollo University in Dessie. “While it is difficult to ignore the
MORGUE DATA HINT
AT COVID’S TRUE
TOLL IN AFRICA
About 90% of bodies tested at a Lusaka facility during coronavirus surges were positive for SARS-CoV-2.
ABy Freda Kreier
lmost one-third of more than 1,000 bodies taken to a morgue in Lusaka in 2020 and 2021 tested positive for SARS-CoV-2, with much higher num- bers during viral surges, implying
that many more people died of COVID-19 in Zambia’s capital than official figures suggest1. Some scientists say that the findings further undermine the ‘African paradox’, a narrative that the pandemic was less severe in Africa than in other parts of the world.
This idea arose after health experts noticed that sub-Saharan nations were reporting lower case numbers and fewer COVID-19 deaths than might be expected. But research- ers say that the findings from Zambia could reflect a broader truth — that a deficit of testing and strained medical infrastructure have masked COVID-19’s true toll on the con- tinent. The findings have not yet been peer reviewed.
Ignoring the true extent of COVID-19 in Lusaka and beyond “is so wrong. People were ill. They’ve had their families destroyed,” says
A health-care worker in Lusaka is vaccinated against COVID-19.
778 | Nature | Vol 603 | 31 March 2022
XINHUA/SHUTTERSTOCK
.devresersthgirlA.detimiLerutaN
regnirpS
202 ©