Science measures depression badly and ruins attempts to understand it

This is a science fiction starring Stuart Ritchie, a newsletter for i. If you’d like to receive this straight to your inbox every week, sign up here.

If you break your leg, you can take an x-ray. You can see exactly where the broken bone is – we know exactly what is causing your problem.

This doesn’t just apply to “physical” symptoms: for example, if you’re suddenly having trouble expressing yourself, we can often use brain scans to pinpoint the location in your brain where you may have had a stroke or stroke. another type of brain injury.

And then depression. Scientists have spent decades trying to identify the specific brain difference that causes symptoms of depression — or, in fact, any brain difference between people with and without low mood, anhedonia, and other problems associated with the disorder. It didn’t go well.

In an ideal world, you would like to be able to classify every person who enters your office—or maybe your operating room if you’re a doctor—as “depressed” or “non-depressed.” Obviously, depression is much more complex, and it’s not just a binary on/off, but for our purposes let’s say you want to do the following: take a brain scan of a person and estimate the likelihood that they are depressed.

classification accuracy

We can measure our progress towards this goal by looking at the “classification accuracy” of our statistical models: plug in the brain data and ask how well our model can distinguish between depressed and non-depressed people. The worst accuracy would be 50 percent—no better than taking a person’s brain data and flipping a coin to see if they’re depressed or not. Numbers well above 50% tell us we’re on the right track, and our models contain a lot of useful information about the depressed brain.

A landmark 2016 study involving thousands of participants found that the size of the hippocampus — the part of the temporal lobe of the brain best known for its role in memory — is a potentially important predictor of depression. It was significantly different between cases and controls (the results showed an effect size – Cohen’s coefficient for stat buffs – 0.17, which is not small, but not huge).

What does this mean in terms of classification? Nothing particularly impressive. Subsequent analysis showed that the effect size found in the original study resulted in a classification accuracy of 52.6 percent, not much better than the 50-50 probability score.

By comparison, if you use the same type of classificatory analysis for the gender variable—by asking if the particular brain you’re scanning is male or female—you can get better than 90 percent accuracy. . This brain is really different and jumps right out of the model. This is not the case for depression, at least in the 2016 analysis.

But we’ve made a lot of progress since 2016, haven’t we? With all the new data coming from large brain imaging studies and advances in statistical methodology, such as when machine learning algorithms specialize in classification emerge, for example, we are confident that we will achieve over 52 percent accuracy. Right?

Not so much. Don’t take my word for it: look at the results of a 2022 study that scanned the brains of nearly 1,800 people and looked at classification accuracy. how easily water molecules can move through the white matter junctions of the brain and more, they found “a classification accuracy of 54 to 56 percent.”

Or look at a new preprint published late last month (and not yet peer-reviewed) that used the same data but this time used 2.4 million different machine learning models to try and classify cases of depression versus controls. using multiple brain variables simultaneously. In this case, the accuracy of the classification was better, but not by much: out of a variety of views on the data, the highest accuracy was 62 percent. Don’t get me wrong: 12 percent over chance isn’t such a terrible result, but it’s still surprisingly low given the sheer volume of data we feed into these models and our firm belief that we should be seeing signs of depression. somewhere in the brain.

What distinguishes the brain of a depressed person?

We have large high quality datasets. We have powerful complex statistical algorithms. So why do we still know so little about what defines a depressed person’s brain? Why are our models that try to classify depression so bad?

One possible reason is that our brain imaging data is not very good. Maybe we are looking in the wrong place or measuring the wrong variables. But even in the most recent studies, they covered a very wide range of measurements of the structure of the brain as well as its functions (meaning where the blood flow is the most and how well the different areas of the brain are connected). And while there is an endless list of different pieces of information you can get from a brain scan, depending on how you analyze it and depending on the type of scan, it’s hard to believe that something is so different from other variables that, if any , will nullify previous attempts at classification.

Maybe we should just keep improving our brain scanners: in the studies I mentioned, the resolution of the brain images was decent (for MRI lovers, it was a 3 Tesla scanner), but not as high as the best modern scanners. It remains possible that really sophisticated scanners — those with magnets so powerful that you feel dizzy the moment you approach them — could show more subtle signs of depression if they were given the chance.

But what about the statistical methods themselves? Is there something wrong with them? As mentioned, models work very well when it comes to something obvious, like sex, which you are trying to classify. There is no reason to expect that they will no longer be able to predict something like depression.

This is where it gets really interesting. What if the problem is measuring depression? The first thing you’ll notice is that we’re moving on diagnoses here: “depressed” someone or not. I mentioned above that this might not be the best way to measure depression for two reasons. First, different doctors can be inconsistent on whether they consider someone to be depressed or not (there is some evidence to support this), and of course a person’s own circumstances and personality predict whether or not they will even go to the doctor to have it diagnosed. diagnosis. Second, it would be better to measure depression as constantly variable and asks, “How depressed are you?” instead of “Are you depressed, yes or no?”.

Other researchers would say that we are focusing incorrectly. Instead of asking if someone has depression, they would say, but instead we should ask what their symptoms are: bad mood, insomnia, lack of interest in things they used to enjoy, and so on. This is due to the observation that two different people with depression can sometimes have very few symptoms in common. If so, how scientifically useful is a diagnosis of depression?

This may not sound right, but it’s a pretty radical take on it: it basically says that “depression” is a brain disorder that we think we know that causes symptoms of depression – doesn’t really exist. Instead, “depression” is just a generic word for those who have some of the many symptoms. And if that’s the case, it’s perhaps not surprising that we have such a hard time figuring out where “depression” resides in the brain.

It is not too far to go to take a really radical, essentially “anti-psychiatric” position and say that mental disorders are not “real” disorders of the brain. To be clear, this is not a step I am willing to take. I think the burden of proof is on the scientists. standardize– to conduct research knowing that depression was measured in the most similar way across all of their various participants – and to adopt new approaches that characterize depression as a “web” of symptoms rather than a single, monolithic cause and test. Also, make them as strict as possible.

At the same time, you can continue to work on these image processing technologies and machine learning algorithms. Understanding the biology of mental disorders—or at least the symptoms we associate with them—is a truly noble goal, and we haven’t made zero progress over the years. But if these new studies on depression and the brain are telling us one thing, it’s not the same as taking an X-ray of someone with a broken leg – when it comes to psychiatry, progress is incredibly difficult.

Other things I’ve written lately

Jeremy Hunt’s budget opens up a competition for physicists to design a small modular reactor that will help us meet our climate goals without waiting decades for large new nuclear power plants to be built. I wrote a short explanation about what these reactors are and what are their pros and cons.

Technically this is not what I wrote, but you can also hear me on I This week’s podcast is about the lab leak theory about the origin of the Covid-19 virus.

Science Link of the Week

If you’ll forgive me for using this section for self-promotion, you might be interested in my conversation with Helen Lewis on her BBC Radio 4 show. Spark. I talked about the many reasons why science can go wrong, about the open science movement that could fix at least some of them, and why being skeptical and critical of science doesn’t make you a denier.

This is a science fiction starring Stuart Ritchie, a newsletter for i. If you’d like to receive this straight to your inbox every week, sign up here.