(SACRAMENTO, Calif.) -- The U.S. News & World Report graduate school rankings inspire discussion and marketing, but do they actually reflect quality differences between schools when it comes to medical education in primary care?
Researchers at the UC Davis School of Medicine are believed to be the first to take on that question by analyzing the primary care rankings from 2009 through 2012, reconstructing scores and proposing alternative measurements. Their findings are published online now, ahead of the August 13 print issue of the journal Academic Medicine.
Printed annually in the America's Best Graduate Schools guidebook, the U.S. News & World Report rankings compare medical school education programs in research, primary care and a few specialties. The goal is to help potential medical students evaluate their options by providing information on the relative quality of the training programs at various schools.
The publication's primary care rankings are based on weighted averages of two types of measures: statistical data, including students' MCAT scores and undergraduate GPAs, and expert opinions, including peer assessments from medical school deans and residency program directors.
One of the researchers' major findings was that the primary care rankings for most of the 89 schools below the top 20 shifted dramatically over the four-year study period. Half of these schools had at least 17 positions separating their best and worst annual ranking in that brief span.
"These huge shifts in such short timeframes cannot be plausibly attributed to actual changes in quality," said senior author Anthony Jerant, professor in the UC Davis Department of Family and Community Medicine. Jerant's collaborators on the research were his UC Davis colleagues Daniel Tancredi, assistant professor of pediatrics, and Klea Bertakis, chair of the Department of Family and Community Medicine.
The team found that small differences in calculations translated into big differences in primary care rankings between schools. In addition, the differences were reported without margins of error showing a school's published ranking and its potentially true position. The authors estimated a typical margin of error to be 22 positions.
"Rankings were much more stable for the schools that were consistently in the top 20 during the four-year study period," said Jerant. "In fact, the rankings for these schools were so stable it suggested an undue influence of long-standing reputation."
Also of concern were the scores based on peer surveys of residency program directors and medical school deans.
"It simply isn't possible for the typical respondent to be highly familiar with the education quality of all of the medical schools they are asked to judge," said Jerant.
Perhaps in part for this reason, relatively few deans and residency directors respond to U.S. News surveys. During the timeframe of the study, response rates from medical school deans were lower than 50 percent, and response rates from residency directors were below 20 percent. These two components, however, made up 40 percent of a school's final score.
The researchers were also troubled by the "graduates entering primary care" rate — or the percentage of graduates who select primary care residencies following medical school — that made up 30 percent of the overall score. U.S. News & World Report assumes that all medical school graduates who select residencies in family medicine, internal medicine or pediatrics end up practicing in one of these primary care fields. Most internal medicine residents and many pediatric residents go on to subspecialty training and never practice primary care, according to Jerant.
Another concern was a measure known as the "faculty/student ratio" or the ratio of science and clinical faculty to students. A school might have a large faculty, resulting in a relatively high faculty/student ratio, yet few of the faculty may be involved in primary care education. The researchers also questioned the relevance of the "selectivity rank" metric, which included mean MCAT scores, undergraduate GPAs and acceptance rates. Some evidence suggests that higher selectivity may work against students interested in primary care, who are more likely to be from under-represented groups and tend to have lower mean standardized testing scores.
The team then experimented with an alternative ranking methodology by removing some of the problematic elements of the U.S. News & World Report methodology, including the "faculty/student ratio" and "selectivity rank" measures. In a second alternate methodology, they also revised the "graduates entering primary care" measure, estimating conservatively that half of medical school graduates selecting residencies in internal medicine or pediatrics were destined to practice primary care. These changes produced major differences when compared to the published rankings, as this table from the article shows.
The revised ranking methodologies were not intended to supersede the U.S. News & World Report results but rather to highlight their arbitrariness and sensitivity to changes in key assumptions.
The research team's findings also highlight a profound problem — there are no other widely available statistical analyses related to the quality of medical schools.
"The U.S. News rankings are often considered 'gold standard' measures of how good a medical school is," said Jerant. "Yet there is nothing to compare the rankings against, and they are published without any sort of hedging or acknowledgment that there are limitations to the methodology."
"We hope at some point that credentialing organizations will gather and publicize reliable and valid quality information," Jerant added. "Given these organizations' access to data and history with the nation's medical schools, they have the best chances for providing accurate and objective comparisons. Until then, prospective medical students interested in careers in primary care should not let the rankings be their sole consideration when judging medical schools."
A copy of "Short-Term Stability and Spread of the U.S. News & World Report Primary Care Medical School Rankings" is available online or by e-mailing Mary Beth de Vilbiss at Academic Medicine at mbdevilbiss@aamc.org.
The study was funded in part by a UC Davis Department of Family and Community Medicine Research Grant.
The UC Davis School of Medicine is among the nation's leading medical schools, recognized for its research and primary-care programs. The school offers fully accredited master's degree programs in public health and in informatics, and its combined M.D.-Ph.D. program is training the next generation of physician-scientists to conduct high-impact research and translate discoveries into better clinical care. Along with being a recognized leader in medical research, the school is committed to serving underserved communities and advancing rural health. For more information, visit UC Davis School of Medicine.
Journal
Academic Medicine