Vignette Set B: For developers of Clinical Practice Guidelines
Vignette B1.
Learning Objective: To understand the necessary steps in the process of performing a systematic literature review.
Primary Author: Roman Jaeschke, MD, MS
An ATS member wants to lead a project that will result in an official ATS document that makes recommendations for use of induced hypothermia in critically ill patients.
Which of the following is an essential step in the process of performing a systematic literature review?
A. Include only studies that are placebo-controlled, randomized trials
B. Include only published studies that are cited in MEDLINE
C. Blind reviewers of potentially eligible studies to information about authors, institutions and journal
D. Use a random effects model to quantitatively synthesize results across all included studies
E. Define the research question specifically, including a description of the target population, details about the intervention and control, and outcome(s) to be measured
Best answer: E
All systematic reviews should begin by specifying the research question in PICO (patient, intervention, comparator, outcome) format. Ideally, the search should include multiple electronic databases (not limited to MEDLINE) and should be supplemented by reviewing reference lists of retrieved articles and scanning abstracts from recent conference proceedings. While a well-designed and executed RCT provides the highest level of evidence, some research questions are not amenable to a randomized trial, and RCTs with an active control are preferred to those with a placebo control when the active comparator represents the standard of care. To minimize the possibility of publication bias, the reviewers should attempt to contact known investigators in the field and others to identify unpublished studies, which are more likely to be negative. While the possibility of reviewer bias can be addressed by blinding, there is no evidence that this has an effect on the results of a systematic review, and most experts believe that it is not worth the time, expense and effort. Finally, while many systematic reviews use statistical methods (meta-analysis) to pool study results, this is not always possible or desirable, especially if there is substantial heterogeneity in methods and design across studies.
Vignette B2.
Learning Objective: To understand concepts of baseline risk, relative risk and risk difference in summarizing evidence from randomized trials
Primary Author: Holger Schunemann, MD, PhD
A guideline panel is developing a recommendation for treating COPD with a novel anti-inflammatory agent. The panel commissions a systematic review of the evidence that is summarized in an evidence profile. Pooled results from 6 large, well-designed placebo-controlled trials showed that the relative reduction in the risk of death in patients who received the study drug was 20% and this relative risk reduction is quite consistent across risk groups and studies. The baseline (untreated) risk in a low risk population is approximately 5 %, and it is approximately 25% in high risk patients. Which of the following statements is true?
A. The risk difference for the low-risk group is 20%
B. The risk difference is the same for both low-risk and high-risk patients
C. The relative risk reduction is usually different for low-risk and high-risk patients
D. The risk difference for the low-risk treatment group is 1%
E. The risk difference for the high-risk treatment group is 2%
Best answer: D
An evidence profile is a table summarizing the information about the outcomes for a focused health care question [link to example of evidence profile]. It provides a simple and transparent summary of available evidence about a particular health care management option.
The results presented in an evidence profile are built around the assumption of a consistent relative effect. It is therefore important to consider the implications of this effect for populations at different baseline risks. An illustrative risk in the control group (i.e. baseline risk or assumed risk) is a measure of the typical burden of the outcomes in a certain population. The relative effect of the intervention for a dichotomous outcome from a single study or a meta-analysis will typically be expressed in the form of a risk ratio (relative risk), odds ratio, or occasionally a hazard ratio.
The relative risk reduction remains constant for many interventions, while the baseline risk differs, as it does in this case for different risk groups. The absolute measure of the effect of an intervention is the difference between the baseline risk of an outcome (e.g. in patients receiving control intervention or estimated in the observational studies) and the risk of outcome after the intervention is applied, i.e. the risk of an outcome in people who were exposed or received an intervention. The absolute effect is based on the relative magnitude of an effect and baseline risk. The effect measure for expressing an absolute effect is often called risk difference (RD), absolute risk reduction (ARR) or absolute risk increase (ARI).
In this case, the risk difference in the low-risk group is the product of the baseline risk and the relative riskreduction: (0.05*0.20)= 0.01 or 1%, while the risk difference in the high-risk group is 0.25*0.20= 0.05 or 5%, illustrating that the absolute effect varies according to baseline risk.
Vignette B3.
Learning Objective: To understand how to use GRADE to rate the quality of evidence.
Primary Author: Colin R. Cooke, MD
An ATS member assembles a group of experts to develop a clinical practice guideline for the outpatient treatment of pulmonary hypertension using GRADE methodology. The group seeks to make a recommendation about whether a newly developed class of drugs improves 6-minute walk distance to a greater extent than endothelin receptor antagonists. As a first step, the quality of evidence supporting this question is evaluated. A large well-designed and well-executed randomized controlled trial (RCT) showed that patients treated with the new drug walked 60 meters farther on 6-minute walk than placebo. Results were statistically significant. Previously published high quality RCTs in comparable populations demonstrate that patients treated with endothelin receptor antagonists walk 30 meters farther on 6-minute walk than those treated with placebo. There are no studies directly comparing the new drug to endothelin antagonists.
What is quality of the evidence that supports the superiority of the new drug over endothelin antagonists on 6-minute walk?
A. High quality
B. Moderate quality
C. Low quality
D. Very low quality
Best answer: B
High quality evidence exists that demonstrates that both drugs are superior to placebo and the magnitude of benefit favors the new drug; however, there are no studies directly comparing the two classes of drugs. Thus, comparisons between the two drugs are indirect. Indirectness of the evidence is one reason for high quality evidence to be downgraded to moderate or low quality evidence. Here, because the two studies have similar outcomes, populations, and controls, the quality is only downgraded by 1 point.
Correct answer can be found on slide 20 of “GRADE PowerPoint Presentation 1”, and slide 24 of “GRADE PowerPoint Presentation 2”, both available on the ATS Documents Development Web page at:
http://thoracic.org/statements/document-development/index.php
Vignette B4.
Learning Objective: To understand how to use GRADE to rate the strength of recommendations.
Primary Author: Colin R. Cooke, MD
An ATS member assembles a group of experts to develop a clinical practice guideline for the outpatient management of stable severe COPD using GRADE methodology. The group seeks to make a recommendation about whether providers should offer patients an inhaled corticosteroid to reduce the risk of an exacerbation. Based upon multiple well-designed RCTs, the group determines that the quality of evidence supporting a steroid-attributable reduction in exacerbations and for other critical outcomes is high. Compared with placebo, the relative risk of a COPD exacerbation for those taking inhaled steroids is 0.75 (95% CI 0.72-0.80), but the absolute risk reduction varies by baseline risk. The group also determines that the costs of treatment are minor in comparison to the benefits, but the treatment can have some side effects (thrush, pneumonia, fracture, burden of inhalers) that may not outweigh the benefits.
Based upon the above information what is most appropriate recommendation for the use of inhaled steroids in all patients with severe COPD?
A. Strong recommendation – the relative risk reduction is 25%
B. Strong recommendation – the quality of the evidence is high
C. Weak recommendation – the benefit depends on the baseline risk
D. Weak recommendation – the side effects outweigh the benefits
Best answer: C
The strength of a recommendation depends on multiple things. These include: the quality of the evidence, the balance of benefits and downsides to therapy, values and preferences, and costs. If there is considerable variability in the baseline risk of an outcome some individuals will benefit more from a given therapy than others. Here, the side effects of steroids may outweigh the benefits of treatment for patients at low risk of an exacerbation. When baseline risk varies to a significant degree, recommendations for a therapy should be classified as weak. An alternative is to make distinct recommendations for patients at low risk and those at higher risk depending on the available information for determining that baseline risk.
Correct answer can be found on slide 81 from “GRADE PowerPoint Presentation 1”, available on the ATS Documents Development Web page at:
http://thoracic.org/statements/document-development/index.php
Last reviewed: February 2015
