Race in cardiac care: DMS researchers warn that statistics can lie
History's most vivid caution against the careless use of statistics probably came from the much-quoted British politician Benjamin Disraeli. "There are three kinds of lies," he said. "Lies, damned lies, and statistics." A trio of DMS researchers used much more restrained language but made basically the same point in commenting recently on a widely publicized study regarding the impact of race on the care recommended for patients with symptoms of cardiac disease.
The saga began with a February 25 article in the New England Journal of Medicine that reported a significant gap between the rates at which blacks and whites are referred for cardiac catheterization. The study—whose lead author was Kevin Schulman of Georgetown—asked internists to watch videotapes in which actors of both races portrayed chest pain and other cardiac conditions, and then assessed the frequency with which the doctors referred the "patients" for diagnostic catheterization.
Apparent bias: The article, titled "The Effect of Race and Sex on Physicians' Recommendations for Cardiac Catheterization," with its apparent findings of bias in cardiac care, was hot fodder for the nation's major newspapers and was also featured on ABC's Nightline. With headlines like "Heart Care Reflects Race and Sex, Not Symptoms" (USA Today), "Heart Study Points to Race, Sex Bias" (Los Angeles Times), and "Doctor Bias May Affect Heart Care, Study Finds" (New York Times), the Schulman paper attracted widespread attention.
It was the Nightline broadcast that brought the study to the attention of DMS faculty members Lisa Schwartz, M.D.; Steven Woloshin, M.D.; and H. Gilbert Welch, M.D. Their work focuses on outcomes research, and the Schulman study raised some questions in their minds.
|
|
Illustration by Suzanne DeJohn |
Misunderstandings: The DMS trio dug into the study's data and published a follow-up article— titled "Misunderstandings About the Effects of Race and Sex on Physicians' Referrals for Cardiac Catheterization"—in the July 22 New England Journal.
Welch says their article was more than simply a response to the Schulman paper, although that study "happened to have a number of very good examples of what we would say was really misleading." The DMS team wanted "to help reporters be more critical consumers of data and to encourage researchers to be more forthright in their display of information."
The differing rates of referral reported in the original article were widely interpreted in the media as demonstrating that blacks and women were "40% less likely" to be referred for cardiac testing than whites or men. In fact, blacks were referred only 7% less often than whites.
According to the DMS team, Schulman and his colleagues "chose to summarize the relative chance of referral for blacks as compared with whites using an odds ratio—literally, the ratio of the odds in favor of being referred for catheterization for blacks to the odds for whites." The media, in turn, "presented relative measures of difference when comparing the referral rates for blacks and whites, but failed to report the absolute rate of referral for each group."
The use of odds ratios "overstated the magnitude of the differences observed . . . and thus influenced the characterization in the media . . . that lower referral rates explain higher rates of cardiovascular-associated mortality." The DMS response emphasized that "exaggeration of the data does nothing to advance the fight against discrimination on the basis of race or sex; indeed, it arguably aggravates the problem."
Implication: Another problem with the original paper, pointed out the DMS trio, was that it aggregated the data by race and sex rather than breaking it down for black men, black women, white men, and white women—thus implying that all blacks were referred less often than all whites and that all women were referred less often than all men.
In fact, when the DMS team analyzed the data, it was clear that black and white men and white women were referred at the same rate (90.6%), and that only black women were referred at a lower rate (78.8%).
In addition, according to Woloshin, "the big difference between the black women and everyone else really boils down to the rate of referral for one actress —[an] older black woman. It turns out that she was referred less often than anyone else." Schwartz, Woloshin, and Welch do agree that the Schulman study "raises questions about differential treatment of black women and that the most reasonable approach is to take a second look."
Appropriateness: The DMS team also raised a third question. Among the findings of the original study was that "overall, almost 90% of the patients were referred for catheterization"—a rate that Schwartz and her colleagues found to be "implausibly high," given growing concern about the technique's side effects. The media, however, referred to catheterization as "the most sophisticated examination" and "the most aggressive treatment," without questioning whether this invasive diagnostic tool is, in fact, universally appropriate.
In other words, the media's implication was that because blacks and women were referred for catheterization less often, their care was inferior. But the DMS rebuttal noted that neither the original study nor the media indicated that catheterization involves real risks, and that outcomes research strongly suggests that "more treatment is not always better and can actually be harmful."
As Schwartz, Woloshin, and Welch point out, "If the research community does not get it right, we cannot expect the media to do better."
Coverage: After their follow-up article was published, the DMS trio was interviewed on National Public Radio, a story went out on the AP wire, a number of newspapers carried corrections, and Brill's Content, a media watchdog publication, ran an article on the matter.
Although ABC's Nightline declined to run a correction, Peter Jennings of ABC News briefly mentioned that "sometimes we get things wrong." A few reporters also called the DMS team in an effort to understand the problem better. Wrote USA Today's health and education editor: "Some of these statistical comparisons are pretty hard to convey. . . . Unfortunately, we have to take our cues from the experts."
Impact: It is actually these "experts" whom Schwartz, Woloshin, and Welch were hoping to reach. And in that, they appear to have succeeded. For example, in an unusual gesture, the executive and associate editors of the New England Journal of Medicine stated in the correspondence section of the Journal: "We take responsibility for the media's overinterpretation of the article by Schulman and colleagues. We should not have allowed the use of odds ratios in the abstract. . . . Although racism and sexism are prevalent in American life, the evidence of racism and sexism in this study was overstated."
According to Welch, "the biggest impact we've had is with the Journal, and, of course, that spills over to the rest of the world. The Journal is on the lookout for odds ratios and making sure they're not interpreted as relative risk. I've seen direct evidence of that in subsequent papers."
Although he and his colleagues feel their follow-up article got much less media attention than the original study did, they were pleased by the New England Journal's response. "They published our work, which certainly was not flattering to their work, and they went the extra step of acknowledging it themselves," concludes Welch.
"It has clearly changed their policies," he says, "and that's worth something."
Sara Fisher
If you would like to offer any feedback about this article, we would welcome getting your comments at DartMed@Dartmouth.edu.