CHICAGO -- A study comparing different statistical methods used to remove the effects of selection bias in observational studies finds that results may vary and caution may be warranted when interpreting findings of studies using certain methods, according to an article in the January 17 issue of JAMA.
With financial, practical, and ethical challenges involved in undertaking randomized clinical trials (RCTs), investigators often use observational data to compare the outcomes of different therapies, to guide policy statements and clinical protocols, and in generalizing results to the community. However, these comparisons may be biased due to important baseline differences in prognostic factors among patients, often as a result of unobserved treatment selection biases, according to background information in the article.
Therese A. Stukel, Ph.D., of the Institute for Clinical Evaluative Sciences, Toronto, and colleagues compared four analytic methods applied to the same data to determine if the estimated benefit from invasive therapy depends on the statistical method used to adjust for overt (measured) and hidden (unmeasured) bias. Methods included multivariable model risk adjustment, propensity score risk adjustment, and propensity-based matching, which control for overt bias, and instrumental variable analysis, which is a method designed to control for hidden bias as well.
The study included 122,124 patients who were elderly (age 65-84 years), receiving Medicare, and hospitalized with acute myocardial infarction (AMI; heart attack) in 1994-1995, and who were eligible for cardiac catheterization. Patients who received cardiac catheterization (n = 73,238) were younger and had lower AMI severity than those who did not. Baseline chart reviews were taken from the Cooperative Cardiovascular Project and linked to Medicare health administrative data to provide a set of prognostic variables. Patients were followed up for 7 years through December 2001, to assess the association between long-term survival and cardiac catheterization within 30 days of hospital admission.
The researchers found that, even after accounting for prognostic variables, cardiac catheterization was associated with an approximate 50 percent relative decrease in death rate, using standard risk-adjustment methods such as multivariable model risk adjustment, propensity score risk adjustment, or propensity-based matching. Using regional catheterization rate as an instrument, the instrumental variable analysis showed a 16 percent relative decrease in the death rate. The survival benefits of routine invasive care from randomized clinical trials are between 8 percent and 21 percent.
"Within a large observational data set, the estimated association of invasive cardiac treatment with long-term mortality is sensitive to the analytic method used," the authors write.
"Randomized clinical trials cannot be undertaken in all situations in which evidence is needed to guide care. Well-designed observational studies are still needed to assess population effectiveness and to extend results to a general population setting. Our study serves as a cautionary note regarding their analysis and interpretation. First, propensity scores and propensity-based matching have the same limitations as multivariable risk adjustment model methods, and are no more likely to remove bias due to unmeasured confounding when strong selection bias exists. Second, instrumental variable analyses may remove both overt and hidden biases but are more suited to answer policy questions than to provide insight into a specific clinical question for a specific patient. Caution is advised regarding clinical protocols and policy statements for invasive care based on expected mortality benefits derived from traditional multivariable modeling and propensity score risk adjustment of observational studies," the researchers conclude.
(JAMA. 2007;297:278-285. Available pre-embargo to the media at www.jamamedia.org)
Editor's Note: This study was supported by grants from the Robert Wood Johnson Foundation, the U.S. National Institute of Aging, and a Canadian Institutes of Health Research Team Grant in Cardiovascular Outcomes Research. Please see the article for additional information, including other authors, author contributions and affiliations, financial disclosures, funding and support, etc.
Editorial: Estimating Treatment Effects Using Observational Data
In an accompanying editorial, Ralph B. D'Agostino, Jr., Ph.D., of Wake Forest University School of Medicine, Winston Salem, N.C., and Ralph B. D'Agostino, Sr., Ph.D., of Boston University and Harvard Clinical Research Institute, Boston, discuss the findings of the analysis of observational studies.
"… the article by Stukel et al is an important reminder of the need for careful and rigorous approaches to observational data analyses. Because the final inferences appear different depending on the method chosen, investigators must be cautious when conducting observational data analyses and must ensure that they have available what they consider to be the most important patient characteristics measured before treatment assignment. Furthermore, the analytic method for comparing treatments must be shown to properly balance these characteristics. In addition, sensitivity analyses also should be performed in much the same way as Stukel et al did. Moreover, external validation of results should be attempted, but always with caution.
"RCTs should not always be considered as the only source of valid scientific information. The data collected from such studies are strong only if it can be shown that in fact a truly random sample of eligible patients participate and complete the protocol as designed. When patients self-select to be included in observational studies, the findings may more accurately reflect 'real world' experience, but if and only if optimal, rigorous, and appropriate methods for dealing with selection bias and confounding are part of the analytic plan," they write.
Media Advisory: To contact Therese A. Stukel, Ph.D., call Julie Dowdie at 416-480-4780. To contact editorial co-author Ralph B. D'Agostino, Sr., Ph.D., call Kira Edler at 617-358-1240.
(JAMA. 2007;297:314-316. Available pre-embargo to the media at www.jamamedia.org)
Editor's Note: Financial disclosures - none reported.
For More Information: Contact the JAMA/Archives Media Relations Department at 312-464-JAMA or email: mediarelations@jama-archives.org.
Journal
JAMA