Pros and Cons of real-word evidence in headache medicine: A debate

Editors View: Pro 

Professor Martelletti started his argument by discussing how randomised clinical trials (RCTs) may be seen as the gold standard for evidence-based medicine but that findings of false or ‘fatally flawed’ data in some RCTs means such studies may need backing from other types of data.¹ For example, while in 2019 results from a single RCT investigating a treatment for episodic cluster headaches was used for approval by the US Food & Drugs Administration,² in 2020, the European Medicines Agency (EMA) found this same trial lacking in enough evidence for them to approve the same medicine following analysis of individual participant data.³

Real-world evidence studies can complement data from randomised controlled trials

In cases such as the above, Professor Martelletti discussed, “real-world evidence (RWE) can complement evidence from other sources including RCTs.” Accordingly, the EMA recently published a review concerning such studies and how they are utilised in regulatory decision making.⁴ “RWE can support both pre-authorisation and post-approval assessment.” Professor Martelletti continued “however, more effort is needed to better anticipate the need for such studies.” He concluded that, given that most RCTs are robust, RWE that originated from registration studies and those caried out in primary care could support such trials.

Editors View: Con 

Dr Gelfand provided 10 common problems that, in her experience, could be wrong with RWE studies. These included:

• Non-standard outcome measures and timing of outcome assessments
• Lack of: control group; quality control on recording outcomes; systematic collection of adverse events
• Missing data, misclassification, and limited measures to assess treatment adherence
• Failure to register the analysis plan prior to study start; failure to report or assess co-interventions

Common problems with real-world evidence studies include lack of quality control, systematic data collection, and a clear analysis plan

With regard to what she considered the two most important points, missing data or non-standard outcome measures and timing, Dr Gelfand described how this includes information not being collected, or being collected at non-standardised timepoints following intervention; non-standardised inputting of information, and loss to follow-up. In many RWE studies, explained Dr Gelfand, “it’s simply what the patient reported to the clinician and what they wrote down. It’s an utterly subjective outcome.” She concluded that “RWE has simply too many problems to make it useful in headache medicine.”

Clinician's View: Pro 

While RCTs are thought to provide the highest level of evidence due to standardisation of methodology and outcomes, RWEs have lower credibility due to confounding factors and control issues.⁵ In headache research, Professor Torres-Ferrús explained, such confounding factors can include comorbidities; medication; psychological and sociodemographic differences; and a patient’s health system. If these are not controllable in an RCT, the patient will be excluded. In RWE though, these patients can be included and confounding factors examined as to how they affect outcomes. This should happen with caution however, and only following adjustment for bias using methodology developed by a statistician with knowledge in the field being studied.⁶

Real-world evidence studies often include a more heterogenous population than randomised controlled trials

Another criticism of RWE studies is regarding data quality. This can be addressed by having prospective studies with well-defined protocols said Professor Torres-Ferrús.⁵ In studies where data collected is not that originally intended for research, such as medication orders and hospital records,⁷ “it is imperative,” he explained, “that we pre-define clinically significant endpoints that are measurable, accurate and suitable for research purposes.” Endpoints for RWE studies can be adapted from those for RCTs, for example, using patient reported outcomes and standardised diaries to assess the onset of a drug’s effect on symptom occurrence and severity.⁷

Clinician's View: Con

“Migraine is a variable intensity condition,” said Professor Mitsikostas, “so you are not absolutely sure that the good outcome you are getting is due to the medication.” Evidence-based medicine should, he said, come from the best available current research. In which case, he argued, “RWE is not the best available research because placebo responses are not evaluated.” Such a response is not just an affect attributed to placebo mechanisms, including expectation and conditioning, but also to the regression of a medical condition to the mean in some patients.⁸

Placebo responses are often not evaluated in real-world evidence studies, making them invalid

While there are some modifiable placebo/nocebo factors, such as a patient’s expectations and cognitive traits, drug packaging, and verbal/nonverbal suggestions, there are also non-modifiable ones, such as the patient’s previous experiences and their personality, culture, gender, education, and use of social media.⁸ “Because life is much more complicated, the help RWE gives us is small,” concluded Professor Mitsikostas.

Conclusion

At the end of the debate, Professor Patricia Pozo-Rosich concluded that “I hope that we will be able to work as a society to be better at collecting RWE.” She added that “I think it makes you a better physician if you are aware of these things when practicing medicine.”