Bringing reliability to fMRI research – the key challenges for clinical utility

Michael Milham, Nathan S. Kline Institute Psychiatric Research, New York, NY, described the four key challenges, he believes, need to be overcome to remove the reliability bottleneck in functional magnetic resonance imaging (fMRI) research for it to attain clinical utility.¹

1. Confusion regarding the different reliability requirements for fMRI to be of scientific and clinical utility

There are subtle differences in the reliability required for fMRI to be of value in scientific research or the clinic. In scientific research, valid tools that measure intended constructs are an absolute requirement as are intra- and inter-scanner reliability. In the clinic, however, only intra-scanner reliability is an absolute must have. In this latter situation, valid tools and inter-scanner reliability are preferable, but not essential. Clinically meaningful outcomes can be gained if the scanner is being used consistently.

In scientific research, both intra- and inter-scanner reliability are required. In the clinic, however, only intra-scanner reliability is an absolute must have.

2. Assessing reliability – consensus is needed on multivariate approaches

Intraclass Correlate (ICC) is the most commonly used statistic to measure test-retest reliability of fMRI data. However, ICC is a univariate measure and for connectomics, for instance, multivariate analyses are preferred. Unfortunately, although multivariate statistical techniques are available, these approaches perform differently under different conditions. As Professor Milburn explained, what is needed is a multivariate approach that can be more broadly applicable.

A multivariate approach that can be more broadly applicable is needed

He also reminded delegates not to forget to take into consideration the pre-processing pipelines – all the steps taken when undertaking the data analysis – as it may be these rather than the data that are the sources of unreliability.

3. Limitations in the datasets available for quantifying and optimizing the reliability of functional neuroimaging

A key limitation is the need to optimize test-retest reliability both within and across sites.

Much work in seeking the attainment of intra-site reliability has already been done, however, what is lacking are large-scale, across sample data sets with a few dozen subjects.  These would allow researchers meaningfully to look at individual differences and their test-retest reliabilities to fully address the issue of cross-scanner, inter-site reliability.

A key limitation is the need to optimize test-retest reliability both within and across sites

What is the optimal scan state?

A number of further limitations also still remain to be addressed. What is the optimal scan state? Resting state might not be the optimal state – it has been suggested that movie watching could be a potentially powerful way for moving fMRI forward. It appears better suited for clinical populations and it allows the same intrinsic signals to be captured as resting state fMRI.

Movie watching could be a potentially powerful way for moving fMRI forward: It allows the same intrinsic signals to be captured as resting state fMRI

Optimal duration – midnight movies

What is the optimal duration for scanning data to be collected? In the past, 5 minutes of data was thought sufficient. Now, however, longer appears to be better. For instance, the CMI Healthy Brain Network Serial Scanning Initiative has collected 120 minutes of data from 13 participants in 12 repeat sessions of 10 minutes duration and in various states (resting, inscapes, movie watching, flanker task).² Similarly, The Midnight Scan Club also seeks to substantially increase the duration and scope of data collection – in this case with researchers scanning themselves outside of working hours.³

By increasing the amount of available data, reliability and, therefore, predictive accuracy is improved. Even multiple, different fMRI sets can be concatenated to increase the data available to investigators.

Artefacts do still present problems. Head motion has been called the ‘dirty little secret of neuroimaging’ in psychiatry and needs to be taken more seriously in structural imaging.⁴ Prospective motion correction software may prove useful in this regard.

Head motion - the ‘dirty little secret’ of neuroimaging in psychiatry

4. Reliance on behavior probes of psychopathology with unknown or suboptimal reliability

Even if all imaging measures achieved optimal reliability today, the job of attaining clinical utility is only half done. This is because most of the commonly used phenotyping tools, particularly those derived from the cognitive neurosciences, have not been assessed or optimized for reliability. So, relating something with a highly reliable signal with something that does not still means a block in the reliability challenge.

Even among instruments for which data on reliability and validity exist, few account for the developmental changes that take place across the [patient’s] lifespan. Fewer still are adequately normalized for under-represented minority populations.

Relating something with a highly reliable signal with something that does not still means a block in the reliability challenge

While much remains to be done, progress in this field is clearly expected to yield interesting results in the future.