From 1989-2002, annual screening mammography was recommended for women at low risk of breast cancer starting at age 50 in the US. In 2002, USPSTF changed that recommendation to recommend mammography every 1-2 years for women starting at age 40. This change was scientifically contentious – there were questions about the data – but endorsed by cancer organizations. This month, USPSTF changed the recommendation back to starting at age 50. You can read the published recommendation here. This change is less scientifically contentious, but has been even more upsetting to cancer organizations and, in many cases, to women.

I’m going to tackle just a few of the key decision making questions here to try to clarify what’s going on. For example, I’m going to set aside questions of cost and of insurance coverage, which are significant issues, to focus only on the health questions, and only on women at low risk for breast cancer.

Is there a threshold for screening?

Assumption: A false positive mammogram is harmful.

Why? A positive mammogram creates a decision: ignore the finding or confirm the finding? Ignoring the finding may mean ongoing anxiety, which can range from mild concern to debilitating worry. Confirming the finding requires a breast biopsy, which is uncomfortable, and, depending on the technique required, can involve small risks of breast damage, infection, anesthesia, etc. A positive biospy (which may also be false positive in rare cases) generally leads to treatment for breast cancer (lumpectomy, mastectomy, lymph node biopsy, radiation, chemotherapy, and drug therapies). Notably, some breast cancers can be very slow-growing, so that treating them at all is unnecessary (you are more likely to die with these cancers than of them), but, again, ignoring them may mean ongoing anxiety.

If you read that description and you don’t think that the possibilities of anxiety, unnecessary biopsy, and potentially unnecessary treatment of indolent cancer are harmful, then guidelines for starting ages for mammography are unnecessary — mammography should begin after puberty. After all, any chance of detecting cancer, however small, is worthwhile if there’s no harm in looking.

Most people, however, and USPSTF in particular, don’t discount those harms. Therefore, the question is: When do the benefits of screening mammography exceed the harms? As women get older, they are more likely to have breast cancer, so positive mammographies are less likely to be false positives. On the other hand, cancers found in younger women may be more beneficial to treat. (And as women get much older, they’re more likely to die of other causes, so it’s sometimes more harmful to find and treat breast cancer than to stop looking, which is why all these guidelines refer to women 75 and younger).

Put like this, the question isn’t “Is mammography a good thing?” Overall, it undoubtedly is. The question is “Knowing that there’s *some age* under which mammography is not worthwhile for the average woman, what age is that?” And, “How sure are we?”

It’s entirely possible, for example, that that age could be 45. But by convention, patients are usually studied in 10-year age ranges, so if the true “best age” for the average patient is 45, some studies will look like the right answer is 40 and some will look like the right answer is 50.

How sure are we?

So, how sure are we? In 2002, the USPSTF gave their recommendation (40-75) a certainty grade of “B”, which means (from their web site): “The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial.”

What is high certainty to USPSTF? “The available evidence usually includes consistent results from well-designed, well-conducted studies in representative primary care populations. These studies assess the effects of the preventive service on health outcomes. This conclusion is therefore unlikely to be strongly affected by the results of future studies.”

What’s moderate certainty? “The available evidence is sufficient to determine the effects of the preventive service on health outcomes, but confidence in the estimate is constrained [by concerns about the quality of the evidence]…As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion.”

Today, in 2009, USPSTF is effectively saying that they were overconfident in 2002. Or, if you like, they were moderately certain in 2002, and more information has become available that was large enough to alter the conclusion. The new 2009 recommendation (50-75) has a certainty grade of “C”:

“The USPSTF recommends against routinely providing the service. There may be considerations that support providing the service in an individual patient. There is at least moderate certainty that the net benefit is small.”

The new scientific evidence since 2002 hasn’t made us hugely more certain. But it has tipped USPSTF’s beliefs about the net benefit of screening mammography for 40-49 year old women far enough to cross back over their threshold for recommendations.

Why are people so upset about it?

New evidence changes recommendations all the time, and not just in health care. In my childhood, dinosaurs were depicted as brown and scaly. Evidence now suggests many were feathered, probably colorful, and the recent discovery of fossilized melanin-producing cells makes it likely that we will know what some of those colors were. This bothers no one. What’s different about mammography?

First, of course, health evidence affects the quality and length of our lives in a way that dinosaur colors don’t.

Second, because mammograms have been urged on women for years, they are considered a valuable health good. Extending a good (as in the 2002 recommendations) seems like a gain. Removing a good (as in the 2009 recommendations) seems like a loss, and losses are psychologically more painful than equivalent gains. Of course, the debate itself is over whether mammography really is a good in younger women, not about whether we should provide goods or not.

Third, there is evidence that people may actually think well of false positives. In a 2002 study of prostate cancer screening (which may have even less benefit than mammography for women 40-49), Cantor and colleagues found that several patients were willing to endure the anxiety of a hypothetical false positive PSA test and the pain of a biospy to be reassured that they did not have prostate cancer. On the other hand, Katz and colleagues surveyed patients who had actually had a (false) positive PSA test and negative biopsy, and found increased worry and decreased sexual function among these patients as compared to those with a negative PSA test. It may be that we underestimate the downside of a false positive. (Hat tip to Rob Hamm and Scott Cantor for the references.)

Fourth, the recommendations have changed and changed back within a short enough period that the USPSTF appears indecisive and inconsistent, and this leads to distrust of their recommendations. It is easier to see the recommendations change than the evidence behind them, and it’s certainly a lot harder for USPSTF to communicate the changing evidence, but we need to develop better strategies for doing so.

Finally, USPSTF recommendations are for the population — what I’ve called the average woman. There are women, however, who had true positive mammographies in their 40’s, and on the basis of their experience are vocal supporters of earlier screening. On the other hand, the many women who had (retrospectively) unnecessary mammographies from 40-49 and didn’t experience a false positive are not a loud voice in the discussion. This returns us to the threshold question — if a single 25-year old woman turns up to say that she had a mammogram that led to discovery and treatment of a dangerous cancer that saved her life, is that sufficient reason to begin screening mammography for women in their 20’s?

With respect to individual women in their 40’s, Diana Petitti, the Vice Chair of USPSTF, says “You should talk to your doctor and make an informed decision about whether a mammography is right for you based on your family history, general health, and personal values.” That, at least, is a recommendation that has always been true.

Videorecordings of the symposia are now available. You can find links at http://www.sjdm.org/content/video-recordings-2008-sjdmsmdm-symposia

My goal in intending the course was to hear about more advanced issues in the evaluation of diagnostic test evidence that I could use to enhance my own teaching as the residents and faculty I teach become increasingly more skilled in the basics. The course did a great job of doing this. After reviewing basic concepts that readers of our book should be familiar with (sensitivity, specificity, predictive values, LR+, LR-, ROC curves), Mike and Tom delved into detailed discussion of continuous and multi-valued tests, focusing particularly on the calculation of interval likelihood ratios.

I’ll illustrate with an example from Medical Decision Making: A Physician’s Guide. In chapter 10, figure 10-3 shows the ROC curve for the 13C-urea breath test at 30 minutes for H. Pylori infection, based on data from a paper by Herold and Becker (BMC Gastroenterology 2002; 2). The curve is based on test values called “delta-δ”; here’s a partial table of test characteristics based loosely on that figure:

In some papers (not Herold and Becker’s), tables like these are then used to compute the positive and negative likelihood ratio at different test values used as thresholds. Mike and Tom point out that this not only throws out a lot of useful information, but can even be misleading when test values are not normally distributed (for example, if both low and high values are problematic).

An interval likelihood ratio is a likelihood ratio for a test result between two test values. This is what you want, and here’s how it looks:

If your test result was 0.7, you’d use the LR for the 0.5-1 interval; if your result was 15, you’d use the LR for the 10-25 interval. By constructing reasonable intervals, you get to take advantage of much more information than creating dichotomous thresholds. (How do you compute those LRs? It’s easy. The interval LR for the interval 1-3 is the absolute difference in sensitivity between delta-δ=1 and delta-δ=3 (0.99-0.95 = 0.04), divided by the absolute difference in specificity between delta-δ=1 and delta-δ=3 (0.80-0.20=0.60).

They also presented some very nice techniques for teaching physicians how to understand ROC curves, and several important nuances around evaluating the validity of diagnostic test studies.

The course was based on their forthcoming book, Evidence-Based Diagnosis, due out in 2009. It’s now on my “must buy” list.

Dr. Roy Poses, a well-respected physician who has done a lot of work studying physician probability judgment (one example of which is mentioned in Chapter 7 of Medical Decision Making) has posted a rebuttal on the Health Care Renewal blog. Dr. Poses demolishes the misrepresentation of EBM that appears in the original article (as well as asking some on-point questions about the author’s interests in the matter), and does it quite effectively, so I won’t repeat his criticism here. Instead, I’ll focus on some other misunderstandings in the original commentary: that cutting health care costs is at odds with the medical imperative to improve care, that patients are so biologically unique that studies of patient groups has little value, and that EBM reduces physician autonomy.

Are considerations of costs at odds with the “medical imperative”?

The commentary invites us to “Consider an overweight man who is forced to take a cheaper, less effective anti-cholesterol drug. If he ends up in the emergency room because of undertreated cardiovascular disease, this could end up costing the health-care system significantly more money.” Setting aside that the author now seems concerned with health care costs in this situation (not only with effectiveness), we could equally consider a situation in which comparative effectiveness evidence helps ensure that the patient is prescribed one of a set of anti-cholesterol drugs that evidence suggests are equally effective and safe. The physician, knowing this, chooses which of these drugs to prescribe first on the basis of judgments about the needs of the patient, which might include which of the drugs the patient can afford, which offers the most convenient dosing, etc. A patient prescribed an equally effective but more costly drug who can not afford to fill the prescription may wind up in that same emergency room, when an equally effective and more affordable generic drug might have kept him healthy.

Here’s an example of a comparative effectiveness review from January 2007 by the US Agency for Healthcare Research and Quality (AHRQ): Comparative effectiveness of second generation antidepressants in the pharmacologic treatment of adult depression. Readers will note that this report makes no references to cost-effectiveness (it barely mentions cost at all), and focuses on reviewing the available evidence that compares the medical effectiveness of different second generation antidepressants on a variety of factors and for a variety of subgroups of patients. It points out the strength of evidence to answer different questions (which ranges from none to high), and, where evidence exists, what the evidence says (e.g. there is no difference in effectiveness between these drugs for major depressive disorder).

Is each patient biologically unique?

There is no question that individual human beings have unique genetic makeups that lead to unique biology. The question is whether this matters for medical treatment and medical research.

All medical research is based on the idea that biological similarity is as important as biological difference. When a new patient presents to a physician with the first strep throat of their lives, the physician considers prescribing an antibiotic, suggests the patient take an analgesic, but doesn’t offer an anticonvulsant. Why? Because the physicians knows the pathophysiology of strep throat, and understands that the infection can be eradicated and pain can be reduced with those medications. The physician knows this because we have studied enough people to enable us to generalize. Research on groups of people is fundamental to modern medicine.

Not every drug works as well for every person, but through well-designed research we can reduce the uncertainty and increase our confidence in how likely a drug is to work on average. We can, and do, also learn about how variable the drug’s effect is around that average, and for which patients it may perform better or worse.

As we learn more about genetic medicine, we may one day seek to tailor therapies to the specific genetic makeup of a patient. Of course, it will require extensive research on the application of these processes applied to large groups of patients before we can be confident of our ability to do this. If our knowledge is great enough that tailored therapies can complete reduce uncertainty in outcomes, the profession of medicine will experience fundamental changes. But people need health care now, too.

Does EBM reduce physician autonomy?

Physicians need the freedom to pursue effective treatments for a patient based on their knowledge — which should include available, credible research evidence — and judgment — which should include contextual factors that are individual to a patient (see, for example, this excellent article by my colleague, Dr. Saul Weiner). EBM enhances this freedom, by improving the knowledge base on which the physician relies.

Physician autonomy does not extend to prescribing ineffective cures when there is valid evidence of an effective cure. The same applies to diagnosis. No physician I know would prefer to measure fever by a hand on the forehead when a thermometer is available.

Physicians need evidence. Synthesizing evidence to review conclusions about comparative effectiveness allows the physician to reduce a major source of uncertainty, and to better inform his/her judgment in selecting a therapy. It also helps shield the physician and patient from spurious claims of effectiveness made by those with a vested interest in selling more profitable interventions — a financial imperative that can be much more at odds with the medical imperative than social imperatives to manage health care spending for the benefit of society as a whole.

The proverb captures important psychological nuances of choice. The same product – a bottle of wine or a risky medical procedure – may be perceived differently depending on its context, and it is often possible to arrange the context to influence a choice while still maintaining the decision maker’s autonomy.

The practice of structuring choices is called “choice architecture” in a brilliant and important new book, Nudge: Improving Decisions About Health, Wealth, and Happiness, by University of Chicago Distinguished Professors Richard Thaler (Business) and Cass Sunstein (Law). Nudge lays out the groundwork for the science of choice architecture in investing, insurance, health care delivery, and other areas, and argues for a “libertarian paternalism” in which choices are structured to make it more likely that a decision maker will select what is considered the most beneficial option, without impairing the ability to decision makers to select other options. For example, making enrollment in 401(k) plans automatic for new employees, with a form for opting out, is likely to result in greater retirement savings than an opt-in system, without limiting anyone’s freedom to choose.

Thaler and Sunstein apply the principles of choice architecture to a few problems in health care (How could Medicare part D be improved? How can organ donation rates be increased? Why shouldn’t patients be allowed to waive their right to sue for medical negligence in return for cheaper health care?) But the concepts in the book go beyond their specific examples and could prove very useful to practicing clinicians, who, they note, are often in the position of being choice architects for their patients.

Their principles of choice architecture (paraphrased by me and focused on physicians helping patients make decisions) are:

• Make sure incentives are aligned with desired outcomes
• Help patients map outcomes of different alternatives into formats they can understand (a major focus of Medical Decision Making as well)
• Arrange default options to favor better health. Pediatricians have done a good job of making vaccination a default option.
• Provide timely and relevant feedback about choices and outcomes. A patient seeking to lose weight needs to experience feedback in the form of measurable progress soon enough that they are not discouraged.
• Expect error and develop systems to prevent, detect, and minimize it. For example, pill cases and inhalers with dosage counters are simple and valuable ways to reduce the frequent errors people make in remembering medication. Psychological research provides direction as to what kinds of errors are to be expected when people are making decisions.
• Structure complex choices to reduce the difficulty of making good decisions. In many ways, that’s what medical decision making — and Medical Decision Making — is about.

I highly recommend Nudge. It’s a great read, and has the potential to change the way you think about clinical practice. Here’s a link to the Nudge web site and blog for more information.

The first is journalistic. National Public Radio’s program Talk of the Nation did a segment on a new blood test that can diagnose early stages of Alzheimer’s diease.

The second is literary, as the science fiction podcast Escape Pod presented the story Results by Kristine Kathryn Rusch, originally written in 2000.

Very different formats, very similar ideas about patient-focused decision making. Each is well worth a listen.

Suppose that we want to know if someone has an active H. pylori infection without forcing him to undergo invasive endoscopic testing. Instead, we perform a ¹³C-urea breath test (UBT) that measures change in the ratio of ¹³CO­₂ to ¹²CO₂ (denoted by δ) exhaled by the patient 30 minutes after ingestion of ¹³C-urea compared with the ratio before ingestion. Because H. pylori hydrolyzes ¹³C-urea to ¹³CO₂, the resulting change value, Δδ, tends to be higher in patients with H. pylori infection than the average uninfected patient, but there’s natural variation both groups, and it’s not impossible that a healthy patient could have a high Δδ. The figure below illustrates the situation.

Figure: Distribution of Δδ for infected (bell curve on the right of the figure) and uninfected (bell curve on the left of the figure) patients. Both infected and uninfected patients may have Δδ as low as 3 or as high as 15. Adapted from Figure 1 of Herold and Becker (2002), BMC Gastroenterology 2:12, with permission of the BioMed Central Open Access license agreement (http://www.biomedcentral.com/info/authors/license).

The UBT is thus an imperfect test, because there is the potential for error. A test which defines the disease (in the way that, for example, bacterial growth on a culture defines bacterial infection) and thus has, in principle, perfect discrimination, is often referred to as a reference standard or gold standard test.

Test thresholds

As the Δδ gets higher, however, the person is more likely to be infected, and as it gets lower, they’re more likely to be healthy. And our goal is to treat the sick differently than the healthy; for example, to prescribe a proton pump inhibitor and antibiotics if we are sufficiently convinced of the likelihood that the patient is, in fact, suffering from H. pylori infection. This implies that we need a criterion, or threshold Δδ, above which we will act in one way (e.g., start drug therapy), and below which we will act in a different way (e.g., watch and wait). A threshold would be graphically represented by drawing a vertical line at the Δδ threshold score; patients whose Δδ is higher than the threshold are treated as infected, and those whose count is lower are treated as healthy.

Because of the overlap between the distributions of Δδ (that is, because of the imperfect discriminative power of Δδ in this example), no threshold can accurately classify every patient. Whatever criterion we set for calling someone infected or healthy based on this test, there will be some people rightly classified as infected or healthy, and some people wrongly classified as sick or healthy.

The threshold determines the kind of error we are likely to make. The higher the Δδ threshold we require in order to call someone infected (graphically, the farther to the right we draw the vertical line), the more we’ll wrongly classify infected people as healthy; these errors are called false negatives. For example, if we set the threshold at Δδ=18, we will almost never wrongly classify a healthy person, but about half of those infected will be misclassified as false negatives (and potentially remain untreated for their infection).

Conversely, the lower the Δδ we require, the more we’ll wrongly classify healthy people as infected; these are false positives. For example, if we set the threshold at Δδ=2, we will almost never wrongly classify a infected person, but almost half of those not infected will be misclassified as false positives (and potentially undergo unnecessary treatment).

Changing the threshold will always either increase false positives and decrease false negatives or vice versa. Only improving the discriminative power can lower both false positives and false negatives.

Choosing thresholds

Unfortunately, we generally can’t improve the discriminative power of a given test; we have to develop new and more discriminative tests (or variations of tests). But the choice of the threshold is arbitrary. A threshold may be recommended by the test developer or by guidelines on the use of the test. These thresholds should be chosen on the basis of the purpose of the test, and the consequences of false positives and false negatives.

For example, consider a rapid strep antibody test for strep throat. A false positive on this test results in a patient receiving an unnecessary dose of antibiotics for a few days; a false negative results in a patient with an untreated bacterial infection for a few days (until the results of the throat culture, a gold standard test, are available). The general consensus among physicians has been that a few days of unnecessary antibiotics is generally preferable to missing a bacterial infection for a few days, but not so preferable that antibiotics should be routinely started in all patients. Accordingly, the kits are developed to have a relatively low, but not very low, threshold for positive results. When the cost of a false negative is much greater, tests may have a very low threshold. A 17-year-old with unknown vaccination history who presents to the emergency department with high fever and possible neck stiffness is very likely to receive immediate presumptive treatment for meningitis. Although the probability of bacterial meningitis is quite low, the consequences of missing a case are so high that a marginally positive finding on a test with low discrimination (neck stiffness) is sufficient to warrant the relatively benign treatment.

In general, when noninvasive and inexpensive tests are used to screen a population for a serious condition, the goal of testing is to broadly identify individuals who may be at higher risk for the condition and refer them for confirmatory testing or other evaluation. Screening tests, therefore, are usually designed to have very few false negatives, and are willing to accept a larger number of false positives in order to assure that high-risk cases are not missed.

On the other hand, when the treatment is invasive and the cost of the disease is low or when the primary aim of the test is to provide reassurance that a patient does not have a serious condition, false positives may be a much greater concern than false negatives. A high threshold is required of a test for carpal tunnel syndrome if the treatment contemplated is open carpal tunnel release surgery.

The primary source of new clinical information is medical research. Most clinical research is explicitly motivated by a need to reduce the uncertainties around etiology, prevention, diagnosis, treatment, or prognosis.

Although research can extend our knowledge, the effective use of research for reducing clinical uncertainty is not always straightforward. For example, research conducted on the development of new drugs often results in adding a new treatment to the set of available choices that is not universally or unequivocally better than existing treatments. Consequently, this kind of research may actually increase uncertainty about the most appropriate treatment for a particular patient, while simultaneously decreasing uncertainty about the availability of a suitable treatment.

Moreover, the translation of clinical research to clinical practice faces several obstacles. First, the amount of new clinical research published each year is astounding and shows no sign of decreasing. As a result, merely keeping up with one’s field has become a more and more difficult process for physicians, particularly those who do not have ready access to academic medical libraries or online article databases.

In addition, not all research is equal. Many studies are beset with serious, and sometimes fatal flaws in methods or measurement, or greatly overgeneralize their conclusions. Conflict of interest among study researchers is not uncommon.

Evidence-based medicine, an approach advocated in academic medicine since the 1980’s, seeks to address these concerns by providing tools and training to permit physicians to focus their limited reading time on research that is methodologically strong and thus likely to yield reliable insights. The approach involves formulating a well-structured clinical question focused on such matters as the diagnostic value of a particular test or the expected outcomes of alternative treatments for well-defined conditions. Answers to these questions are sought in the medical literature. Individual studies are rigorously evaluated to determine how well the study responds to the clinical question that prompted the inquiry. This assessment considers soundness of research design, whether the findings apply to the patient of concern, trustworthiness of the conclusions, and limitations of the evidence. When multiple studies bearing on a question exist, EBM advocates the use of meta-analysis and systematic review to form overall conclusions.

Even when high-quality research has been performed, identified, and even codified in practice guidelines by professional societies, the results may not be widely adopted by the medical community. There are three primary obstacles to the adoption of new approaches that promise to reduce clinical uncertainty: lack of effective dissemination to practicing physicians, lack of comfort learning an unfamiliar regimen among physicians, and health care system impediments such as cost of care.

At the same time that adoption of proven improvements in medical care is slowed by the aforementioned difficulties, it has become easier and easier for Internet-savvy patients to obtain information about conditions and treatments – from both trustworthy and untrustworthy sources. Indeed, both the decreased time available for appointments under many health plans and the increased emphasis on patient autonomy in medical decision making exert pressure for patients to “do their homework”, often with limited guidance as to how to evaluate the information they develop.

Although pharmaceutical advertising is regulated in many countries, and physicians are familiar with marketing strategies used by drug companies, patients may not be as prepared to evaluate advertising claims for pharmaceutical products, much less for the larger group of unregulated supplements and alternative medicine products. Patients who are less well-educated may be at particular risk, and face a double jeopardy: less likely to have access to optimal care and less likely to be able to discriminate between safe and risky treatments.

The combination of slow dissemination of rigorously-evaluated medical advances among physicians and rapid dissemination of unscientific advice and claims among patients is a recipe for disaster. Public policy initiatives to improve health literacy among patients are an important strategy to avert catastrophe, but can not substitute for the impact of effective communication between well-informed physicians and their patients. Physicians can help their patients by taking the lead in explaining how medical research is conducted, suggesting sources of credible information that patients can use, and addressing the strengths and weaknesses of information from whatever source. Patients can help their physicians by learning how to seek out reliable information and bringing it to the clinical encounter with a critical eye.