Thursday January 10 2013
A worrying number of breast cancer studies spin their results
The Daily Telegraph reports the alarming news "Breast cancer drug trials 'spun for impact'," saying that trial results are spun to make treatments appear more beneficial than they actually are.
The Telegraph reports on a valuable new review that identified all published randomised controlled trials (RCTs) investigating treatments for breast cancer. Well-designed RCTs are the best way of investigating whether new treatments are safe and effective – the so-called "gold standard" of evidence-based medicine. The review looked at how common it is for authors to bias reporting by overemphasising positive results, downplaying the negative results, or both.
The researchers argue that busy doctors often only have time to look at the abstract (summary of the aims, methods, results and conclusions) of new research papers. They wanted to see if abstracts clearly reported the results of the main (primary) stated outcome trials had been designed to investigate (for breast cancer, this may be overall survival or the period where the cancer does not get worse), and whether severe adverse effects were clearly reported.
A third of all the studies reviewed put a positive spin on the results, emphasising improvements in secondary outcomes that the study was not primarily designed to investigate.
It is also concerning that about two-thirds of the 164 studies did not clearly report severe adverse effects. Worse still, this was most common among studies that found positive results for their main outcome.
This study makes for depressing reading, as such spinning of results goes against the fundamental principles of evidence-based medicine. This serves to highlight the need for objective, clear reporting of the main results of clinical trials and any adverse effects associated with the treatments being studied.
Reporting bias is not just an issue relating to breast cancer trials, and similar reviews of trials in other medical conditions would be useful.
Where did the story come from?
The study was carried out by researchers from Princess Margaret Hospital and the University of Toronto, Canada. This study reports no sources of funding and the authors declare that they have no conflicts of interest.
The study was published in the peer-reviewed medical journal, Annals of Oncology.
Both The Daily Telegraph and The Independent report on this research well. However, this news should not be interpreted as there being a particular problem with how research on breast cancer therapy is presented. It just happens that breast cancer is the condition the researchers have chosen as the subject of their review of the scientific evidence.
For more information about bias in the reporting of science, see Half of medical reporting 'is subject to spin'.
What kind of research was this?
The researchers explain that RCTs are designed to tell whether a particular treatment performs differently from a control treatment (which could be either an existing treatment or a placebo).
A well-designed RCT should look at outcomes that are of meaningful importance to patients – in the case of breast cancer, this could be survival without disease progression or overall survival time. However, it is important that as well as looking at treatment benefits, RCTs should also include information on the harms (such as side effects and complications) of a treatment.
Well-designed RCTs have long been agreed as the best way to see whether new treatments are safe and effective before they are approved for use. However, as the researchers say, it is essential that the results of these trials are reported objectively so that professionals are reliably informed about the effectiveness and safety of new treatments.
This study is a systematic review aiming to identify all RCTs evaluating breast cancer treatments, see how well they were reported and whether there was any evidence of biased reporting, known as "spin".
Spin is defined by the review as the "use of reporting strategies to highlight that the experimental treatment is beneficial, despite a statistically non-significant difference in the primary outcome, or to distract the reader from statistically non-significant results."
The main outcome the researchers were interested in was to see if the summary abstract in studies reporting the results of individual trials accurately outlined:
- the result of the main outcome the RCT had been designed to investigate (called the primary outcome or the primary endpoint)
- any adverse (side) effects of treatment
Their reason for focusing on whether this was reported accurately in the study abstract (rather than only in the body of the article) is because many healthcare professionals (and journalists) may only read the abstract of a paper, rather than the whole report.
What did the research involve?
The researchers performed a search of a medical database (MEDLINE) to identify RCTs in adults with breast cancer published in the English language between January 1995 and August 2011. They only included larger trials with more than 200 participants, and excluded commentaries, review articles, observational studies, meta-analyses, ongoing studies and articles for which only the abstract was available.
For each identified study, two researchers extracted data on:
- the type of treatment – whether it was given as an adjuvant (after the main treatment for breast cancer, such as surgery, to try to prevent cancer spreading elsewhere in the body) or metastatic treatment (to try to improve outcomes after the cancer has already spread elsewhere in the body)
- sponsorship (industry versus non-industry funding, or not stated)
- year of study publication
- impact factor of the journal where the trial was published (a measure of how often a journal's articles are cited by other academic articles)
- the primary and secondary endpoints (overall survival, progression-free survival, disease-free survival, response rate, toxicity or quality of life)
- whether the primary endpoint listed in the clinical trial registry (ClinicalTrials.gov) differed from the primary endpoint reported in the trial publication – ClinicalTrials.gov is a US database that registers the details of medical trials
- whether the primary endpoint was defined in the abstract or in the whole paper
- whether the secondary endpoints were also reported in the abstract
The main aim was to look at how often there was spin or bias in the researchers' reporting of the RCT's primary outcome and adverse effects in the study abstract. Bias was defined as inappropriate reporting of either of these things in the abstract. Spin was defined as presenting findings in the concluding statement of the abstract in a way that suggested that a trial was positive due to benefits seen in one or more of the trial's secondary endpoints, even though the trial did not find a benefit in its primary outcome.
When looking at reporting of adverse effects, the researchers assessed this on a scale from excellent to poor, particularly looking at whether any severe (high grade) adverse effects were reported in the abstract and concluding statement or not.
The researchers also looked at whether they found any other factors associated with bias or spin, such as the funding source, the impact of the journal where the study was published, or the type of treatment being given.
What were the basic results?
The researchers indentified 164 relevant RCTs, which included 148 trials of systemic therapy (such as chemotherapy and other treatments given intravenously or by mouth), 11 trials of radiotherapy and five trials of surgical treatments. Around half of the trials (81) looked at adjuvant treatment and the other half investigated treatments for metastatic breast cancer. The majority of trials (91%) were published in high-impact journals.
Seventy-two (44%) of the studies had positive results, with significant improvements in the primary endpoint (outcome) using the intervention treatment compared with the control.
In the remaining 92 trials (56%), the intervention did not significantly improve the primary outcome.
Fifty-nine per cent of the 92 trials that found non-significant results for the primary endpoint gave biased reporting and put a spin on the results, reporting benefits when only positive results were obtained for secondary endpoints. Twenty-seven per cent of these 92 trials did not say anything about the primary outcome in the concluding statement of their abstract.
This means that one-third of all the breast cancer trials the authors identified (59/164) had biased reporting and had put a spin on their results. There was no association between bias and the type of treatment being given (adjuvant or metastatic).
When looking at the reporting of severe adverse effects of treatment in the abstract, 110 of the studies (68%) had biased reporting of severe adverse outcomes. The researchers found that there was a significant association between the trial having a beneficial improvement in its primary endpoint and having reporting bias for adverse effects.
This suggests that there was a tendency for researchers who conducted trials that genuinely found benefits in the primary outcome of their study to want to avoid downplaying this by emphasising any adverse effects that the treatment had caused.
Conversely, trials that did not find a beneficial effect of their primary endpoint, and had put a spin on their results to report positive improvements in secondary endpoints, were not found to be more likely to bias their reporting of adverse effects.
How did the researchers interpret the results?
The researchers conclude that bias in the reporting of outcome is common for trials that do not find that the treatment being investigated improves the primary outcome of interest.
They say that the reporting of severe adverse effects is also poor, especially in studies that found that the treatment improved the primary outcome of the study.
Well-designed RCTs are the best way of investigating the effectiveness and safety of a particular treatment being investigated compared with a control treatment.
This valuable new research highlights the need for objective and clear reporting of both the results for the main outcomes that the trial was set up to investigate, and of any adverse effects that have been associated with the treatments being tested.
This is essential to allow relevant healthcare professionals and healthcare policy makers to clearly see whether potential new treatments are safe and effective.
However, there are some points to bear in mind in relation to this review:
- The review has only evaluated how common reporting bias and "spin" is in published breast cancer trials. It cannot tell us how common other types of bias may be, most significantly publication bias itself, where only trials with positive results get published in the first place.
- The review has only captured breast cancer trials published in the English language and it would be useful to apply similar methods to other topic areas.
- It is important to bear in mind that this review does not suggest that this issue is limited to breast cancer research, this is just the condition the researchers have chosen to review. It is possible that had they chosen any other disease or condition, and looked for all RCTs for treatments for these conditions, they may have found a similar incidence of reporting bias and putting a positive "spin" on results.
Professionals making decisions about whether new treatments should be approved for use take full account of the primary results and any adverse outcomes of new treatments reported in all relevant trials.
People should be reassured that this review does not provide evidence that the current surgical, radiotherapy and medical treatments licensed for breast cancer are ineffective.
Analysis by Bazian. Edited by NHS Choices. Follow Behind the Headlines on Twitter.