Interpreting Research Outcomes with a Little Bathroom Humour
You have learned a method for how to read a research paper, and now you are faced with a bunch of outcomes. There are primary, secondary, composite, and safety outcomes. What are these outcomes? What should you focus on?
The researchers should have specified a clear research question or aim. Outcomes are measured, monitored, and recorded variables during the study to address the question or aim. These should be established and reported before publication.
You should check whether a published trial has been entered in a clinical trial registry such as clinicaltrials.gov, ANZCTR, or ISRCTN as it is not uncommon for published outcomes to differ from the proposed registered outcomes. This should make you wonder why they changed then. We know there is a bias towards reporting positive outcomes.
Primary Outcomes
The primary outcome is the variable the researchers believe to be the most important and best aligned with addressing the research question or study aim. Sometimes, this is not specified, or they include a bunch of variables. But ideally…
Secondary Outcomes
Secondary outcomes are additional outcomes that may help support the interpretation of the primary outcome. They may also include subgroup analysis. When a primary outcome is negative, the results from secondary outcomes may be hypothesis-generating and lead to further research. Be careful with over-emphasising secondary outcomes. The majority of them are NOT corroborated in later randomised clinical trials.
Safety Outcomes
These are often grouped into secondary outcomes and used to evaluate for possible harms of the intervention in a trial. This can include adverse effects like diarrhoea, rash, vomiting, nausea, etc. In the worst case, this can include mortality.
Composite Outcomes
Composite outcomes are groups of seemingly related variables. Be careful with interpreting this type of outcome, as the intervention in the trial may affect each variable to differing degrees.
For this next section, let’s look at the types of outcomes that are often reported and some examples from the literature. As you read through any research study’s primary and secondary outcomes, ask yourself, “What type of outcome is this?”
Types of Outcomes
Patient-Oriented Outcomes (POOs)
These are the outcomes that we should care most about. They are the variables that matter most to the patient. We love POOs.
In the BUCKLED trial that looked at whether ultrasound was non-inferior to X-ray in diagnosing forearm fractures, the primary outcome was physical function of the affected arm at four weeks. This is a fantastic example of a patient-oriented outcome. How well a patient can use their arm matters greatly to the patient!
Similarly, in the SCOUT-CAP trial that looked at short vs. standard duration antibiotic therapy for community-acquired pneumonia, the primary outcome was the RADAR/DOOR score, a more complex, composite outcome. However, it was still patient-oriented, considering the patient’s clinical response, symptom resolution, and adverse events.
Lab-Oriented Outcomes (LOOs)
In the same SCOUT-CAP study, one of the secondary outcomes was looking at something called resistomes. Some of these patients had throat swabs to assess for antibiotic resistance genes (ARGs) expressed as resistance genes per prokaryotic cell (RGPC). There were significantly lower RGPCs in the group with short-course therapy than standard therapy (1.17 vs. 1.33). What does this mean for the patient? It’s unclear. A difference based on a laboratory value may sound impressive, but this does not necessarily equate with clinical significance.
We don’t know whether a throat swab of flora actually correlates with the flora in the patient’s lungs. We don’t know if this change is lasting. We don’t know what this difference of 1.17 vs. 1.33 means clinically.
It also reinforces the concept:
A statistically significant finding does NOT necessarily mean that it is clinically significant, or vice versa.
Disease-Oriented Outcomes (DOOs)
The PARAMEDIC-2 trial looked at the effectiveness of epinephrine in adult patients who suffered an out-of-hospital cardiac arrest. They found that epinephrine administration was associated with higher survival at 30 days (3.2% vs. 2.4%). Wait, that sounds pretty good, right? However, this is a disease-oriented outcome.
When we consider the number of patients who survived until hospital discharge with favourable neurologic outcomes, there was no significant difference (2.2% vs. 1.9%). This outcome seems more patient-oriented. An intervention may impact the course of the disease, but this does not necessarily mean it is meaningful for the patient.
Monitor-Oriented Outcome (MOOs)
In a trial of pulse oximetry in infants with mild to moderate bronchiolitis, part of the group had accurately displayed pulse oximetry readings, and part of the group had pulse oximetry readings that were altered to show three percentage points above the true value. The primary outcome was hospitalisation within 72 hours. They found that infants with artificially elevated pulse oximetry ratings were more likely to be discharged. There was no difference between the two groups regarding unscheduled visits or hospitalisations within 72 hours.
This is a nice example of how pulse oximetry is a monitor-oriented outcome. It reinforces that we should not put too much emphasis on a monitor value as it does not necessarily correlate with clinically significant outcomes.
Treat the patient, not the number!
Acknowledgements
I want to thank Dr. Ken Milne for the inspiration behind this post. He is a tremendous mentor and teacher of evidence-based medicine. Much of the content from this post is drawn from his talks. Thanks for encouraging us all to be sceptical and kind.
Additional Resources
EBM Masterclass with Ken Milne
References
Chen T, Li C, Qin R, et al. Comparison of clinical trial changes in primary outcome and reported intervention effect size between trial registration and publication. JAMA Netw Open. 2019;2(7):e197242.
Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282.
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977-984.
Murad MH, Chu H, Lin L, Wang Z. The effect of publication bias magnitude and direction on the certainty in evidence. BMJ Evid-Based Med. 2018;23(3):84-86. doi:10.1136/bmjebm-2018-110891
Perkins GD, Ji C, Deakin CD, et al. A randomized trial of epinephrine in out-of-hospital cardiac arrest. N Engl J Med. 2018;379(8):711-721.
Schuh S, Freedman S, Coates A, et al. Effect of oximetry on hospitalization in bronchiolitis: a randomized clinical trial. JAMA. 2014;312(7):712-718.
Snelling PJ, Jones P, Bade D, et al. Ultrasonography or radiography for suspected pediatric distal forearm fractures. N Engl J Med. 2023;388(22):2049-2057.
Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Intern Med. 2017;177(4):554-560.
Williams DJ, Creech CB, Walter EB, et al. Short- vs standard-course outpatient antibiotic therapy for community-acquired pneumonia in children: the scout-cap randomized clinical trial. JAMA Pediatr. 2022;176(3):253-261.
