Trials on trial: what do meta-analyses reveal about efficacy and bias?

Trials of psychotherapy versus some form of control arm, of pharmacotherapy versus placebo, of psychotherapy versus pharmacotherapy, and of combined treatment versus a single modality -- Pim Cuijpers and colleagues at the Free University of Amsterdam have been looking at the outcomes of studies using all of these designs. But comparing outcomes is difficult.

Many hundreds of psychotherapy trials have been undertaken. They differ in how long ago they were conducted, in the nature of the control groups (patients on the waiting list are often the only control), in the kind of psychotherapy considered, and in the nature of the patients treated.

With trials of drugs, there will again be differences in the nature of the patients treated. And some studies are with old compounds, while others are with newer agents. But both will generally have been compared with a “pill placebo” and hence be double-blind. 

No differences between type of psychotherapy

In the case of psychotherapy trials, there are no clear differences between forms of treatment. The overall finding is that the degree of efficacy is independent of the age of patients. But there is some indication effect size is less in patients with comorbid substance abuse, with chronic depression and in-patients rather than out-patients.

It does not seem to matter much whether the psychotherapy format is face-to-face, or in groups, or guided self-help – as long as there is real human contact with a therapist somewhere along the line.

Neither is there much influence on effect size of number of sessions (which ranged from 4 to 24), nor contact time: in both respects, long is as effective as short. And psychotherapy also seems effective in low and middle-income countries.

Psychotherapy trials with low risk of bias show reduced effect size

There is some evidence of an additive effect when drugs and psychotherapy are combined. Analysis of studies that compared their combination to a single treatment modality find a number-needed-to-treat (NNT) of 3.5, compared with 5.75 for pharmacotherapy alone and 7.1 for psychotherapy.

Additive sources of bias

But there is intriguing evidence of two potential sources of bias. When the 100 psychotherapy studies (out of 350) with designs judged to be at low risk of bias are compared with the rest, the effect size of the intervention is reduced. And the same effect is seen with potential publication bias. 

Of trials funded by the NIH, 13 of 55 were never published. Tracking these trials down and including them in the results reduced overall effect size by 25%.

Adjusting for both potential sources of bias increases NNT from 3 to 6; and the effect size falls from 0.71 to 0.31. Confining the analysis to trials that used CBT, the effect size falls from 0.71 to 0.34.

Professor Cuijpers emphasised that these trials – both those using drugs and those using psychotherapy -- all deal with relatively short-term outcomes, of around six months. Meta-analysis of trials with longer-term follow up and those that investigated relapse prevention may give a somewhat different picture.