Scientists rely on brain-wide association studies to measure brain structure and function—using MRI brain scans—and link them to complex characteristics such as personality, behavior, cognition, neurological conditions, and mental illness.
New research published on March 16, 2022, in Nature from the University of Minnesota and Washington University School of Medicine in St. Louis is highlighting a path forward for brain imaging use in diagnostics, prognostics, and treatment response in psychiatric, psychological, and neurological conditions. The research shows that most published brain-wide association studies are performed with too few participants to yield reliable findings.
The study used publicly available data sets—involving a total of nearly 50,000 participants—to analyze a range of sample sizes and found:
- Brain-wide association studies need thousands of individuals to achieve higher reproducibility. Typical brain-wide association studies enroll just a few dozen people.
- So-called “underpowered” studies are susceptible to uncovering strong but misleading associations by chance while missing real but weaker associations.
- Routinely underpowered brain-wide association studies result in a surplus of strong yet irreproducible findings.
- New parameters for brain-wide association studies provide direction for better utilizing findings to improve clinical practice and mental health care.
“For decades we’ve been highlighting the potential for MRI to assist in the clinical care—including the diagnosis, risk, response to treatment, etc.—for mental health disorders and neurologic conditions. However, the full potential has not been realized,” said Damien Fair, PA-C, PhD, senior author and Redleaf Endowed Director for the Masonic Institute for the Developing Brain (MIDB) at the University of Minnesota. “We now know our missteps and are redefining the required parameters, the so-called ‘special sauce,’ to move forward effectively.”
To identify problems with brain-wide association studies, the research team began by accessing the three largest neuroimaging data sets: the Adolescent Brain Cognitive Development Study (11,874 participants), the Human Connectome Project (1,200 participants) and the UK Biobank (35,375 participants). Then, they analyzed the data sets for correlations between brain features and a range of demographic, cognitive, mental health and behavioral measures, using subsets of various sizes. Using separate subsets, they attempted to replicate any identified correlations. In total, they ran billions of analyses, supported by the MIDB Informatics Group and the powerful computing resources of the Minnesota Supercomputing Institute.
The researchers found that brain-behavior correlations identified using a sample size of 25—the median sample size in published papers—usually failed to replicate in a separate sample. As the sample size grew into the thousands, correlations became more likely to be reproduced. Robust reproducibility is critical for today’s clinical research.
Scientists rely on brainwide association studies to measure brain structure and function — using brain scans — and link them to mental illness and other complex behaviors. But a study by researchers at Washington University School of Medicine in St. Louis and the University of Minnesota, published March 16 in Nature, shows that most published brainwide association studies are performed with too few participants to yield reliable findings. Credit: Alex Berdis
Senior author Nico Dosenbach, MD, PhD, an associate professor of neurology at Washington University, says the findings reflect a systemic, structural problem with studies that are designed to find correlations between two complex things, such as the brain and behavior.
“It’s not a problem with any individual researcher or study. It’s not even unique to neuroimaging,” said Dosenbach. “The field of genomics discovered a similar problem about a decade ago with genomic data and they took steps to address it. The NIH began funding larger data-collection efforts and mandating that data must be shared publicly, which reduces bias and as a result, genome science has gotten much better. Sometimes you just have to change the research paradigm. Genomics has shown us the way.”
Neuroimaging studies are expensive and time-consuming—just an hour on an MRI machine can cost $1,000. But if all of the data from multiple small studies were pooled and analyzed together, including statistically insignificant results and minuscule effect sizes, the result would likely approximate the correct answer, Dosenbach said.
“The future of the field is now bright and rests in open science, data sharing and resource sharing across institutions in order to make large data sets available to any scientist who wants to use them. This very paper is an amazing example of that,” said Fair. “Here at MIDB, we’re trying to provide the scientific community from all walks of life the resources necessary to do the work optimally.”
Through the MIDB Informatics group, other University of Minnesota departments involved in this study include the Neuroimaging Genomics Data Resource and the Minnesota Supercomputing Institute.
For more on this research, see Most Brain Studies Have Too Few Participants To Yield Reliable Findings.
Reference: “Reproducible brain-wide association studies require thousands of individuals” by Scott Marek, Brenden Tervo-Clemmens, Finnegan J. Calabro, David F. Montez, Benjamin P. Kay, Alexander S. Hatoum, Meghan Rose Donohue, William Foran, Ryland L. Miller, Timothy J. Hendrickson, Stephen M. Malone, Sridhar Kandala, Eric Feczko, Oscar Miranda-Dominguez, Alice M. Graham, Eric A. Earl, Anders J. Perrone, Michaela Cordova, Olivia Doyle, Lucille A. Moore, Gregory M. Conan, Johnny Uriarte, Kathy Snider, Benjamin J. Lynch, James C. Wilgenbusch, Thomas Pengo, Angela Tam, Jianzhong Chen, Dillan J. Newbold, Annie Zheng, Nicole A. Seider, Andrew N. Van, Athanasia Metoki, Roselyne J. Chauvin, Timothy O. Laumann, Deanna J. Greene, Steven E. Petersen, Hugh Garavan, Wesley K. Thompson, Thomas E. Nichols, B. T. Thomas Yeo, Deanna M. Barch, Beatriz Luna, Damien A. Fair and Nico U. F. Dosenbach, 16 March 2022, Nature.
DOI: 10.1038/s41586-022-04492-9
This work was supported by the National Institutes of Health (NIH), the Andrew Mellon Predoctoral Fellowship, the Lynne and Andrew Redleaf Foundation, the Kiwanis Neuroscience Research Foundation, and the Jacobs Foundation.
Masonic Institute for the Developing Brain
The Masonic Institute for the Developing Brain (MIDB) is a one-stop clinic, research, and outreach location specializing in children and youth with neurobehavioral conditions. By bringing together University of Minnesota experts in pediatric medicine, research, policy and community supports to understand, prevent, diagnose, and treat neurodevelopmental disorders in early childhood and adolescence, MIDB advances brain health from the earliest stages of development across the lifespan, supporting each person’s journey as a valued community member.
Be the first to comment on "Brain Studies of Complex Behavior Require Thousands of People for Accurate Results"