In Nassim Taleb's outstanding book, The Black Swan, he discusses how mathematical models are used in predicting the stock market. His general contention is that while there is a lot of thought and brainpower used in designing these models, they are still at best approximations of real financial markets, and can be catastrophically wrong. He discussed the 1987 stock market crash as an example. The book was published before this year's market collapse, but the 2008 crash helped illustrate just how important his point was.
I would term the phenomenon he is discussing "false precision." Because the statistical models work 70%, 80%, or 90% of the time, there is a false assumption that they will work 100% of the time. That is not true, but the precision that we see under "ordinary" circumstances can fool us into thinking that we will see that precision all of the time.
Many people have seen a similar phenomenon when they play fantasy sports. I have played both fantasy football and baseball in the past, but I'll use football as an example, since it is the more popular game (I don't play either anymore, since while fun, they are major time sinks). Fantasy football is what some will call a "mirror game"- it is based on football, uses many of the same principles of football, but it is not the same game. Back in era when I played, Daunte Cullpepper was probably the best QB in fantasy football, since he would throw a lot of touchdowns (thanks to Randy Moss and Cris Carter) and run for a decent amount of yards.
After playing for a few years, it was easy to start thinking that Daunte Cullpepper was the best QB in the NFL. He wasn't- Peyton Manning and Tom Brady were, even that wasn't reflected in their fantasy football numbers. What happened was that it became very easy to confuse that while fantasy football is a reasonable approximation of football, it is NOT the same thing as football. It is a mirror, a reflection. But the output data we received from playing fantasy football made us think that we knew more about the game than we really did. This false precision.
The data for the stock market and the data for sports are much, much cleaner than they are for medicine. It is not even close. The data for educational outcomes (used for things like No Child Left Behind) is much cleaner than it is for medicine, and No Child Left Behind, for all of it's strengths, is still dealing with a messy data set that only mirrors reality. Medicine is much messier than that.
Don't get me wrong- I think medicine needs to more toward using evidence to inform our judgments- just like you need to prep for your fantasy draft by reading and evaluating, just like you should pick you mutual funds by reading their prospecti, just like you want your kids to be educated using the most evidence-based techniques, when it comes to treating patients, I think we need to look to the evidence. We shouldn't be just making stuff crap up- we should make our best effort to actually use data.
We just can't be too arrogant about it. We can't assume that just because we are using the best data available that we are unfailable. We can't assume that because we are using evidence-based practices that our precision is any better than 50-70%.
I will be absolutely shocked if, within my lifetime, the predictive models we use for health care even approach the precision of the predictive models we currently use for the stock market, baseball, and the weather. And as we've all experienced, those models aren't particularly precise, and when they are wrong, they can be wrong with disastrous consequences.
The main reason medical models will never be as good is that the populations we are looking at are too heterogeneous. For example, the largest medical study ever conducted was the Women's Health Initiative. I am having trouble locating a specific cost for the WHI, but my memory was that it was a $12 billion study.
The study was about as well designed as a medical study could be. For all that said, it was largely inconclusive, and there have been literally hundreds of papers that have tried to parse the data to figure out what it means. The underlying problem is that the group of patients they studied- older women- was too heterogenous. It turns out that older women is too broad a category, and so the conslusions that would be true with one subgroup don't really apply to another subgroup.
Within my own field of specialty, the largest study ever dedicated to outcomes for patients with low back pain was the SPORT trial. Same problem- the group studied was too heterogeneous, and the study could not account for human behavior sufficiently, so we don't really have much more insight on how well back surgery works for low back pain then we did before the study was conducted.
Just to beat this dead horse some more, I am going to pick the first study in this month's New England Journal of Medicine. It is about using oral steroids for kids with wheezing, trying to determine whether steroids make a difference or not. The answer is "we can't tell," although I suspect it will be reported as "steroids don't make a difference." I just checked- the first story in Google News under the search terms (steroids, wheezing) is "Oral Steroids Ineffective in Treatment of Preschool Virus-Induced Wheezing." Why is it reported this way? I suspect that headlines of "we just spent several million dollars studying something, and we know only marginally more than we did before we did the study" doesn't sell a lot of copy.
If we look at the actual data, the kids who got steroids did spend less time in the hospital than kids who didn't (medians were 13.9 hours v 11 hours), but that this difference did not achieve statistical significance. Reporting this as "steroids don't make a difference" is a simplification, because the real picture is more complicated. This is not a criticism of the study authors- they acknowledge the study's limitations in their discussion. But most people don't have time for shades of gray.
The way I would interpret this study is that there probably is a sub-group of kids who would benefit from steroids, and we haven't figured out what that subgroup is yet. We just don't know enough yet. Even if we studied this with the same resources we use or analyzing the stock market, we are going to be wrong at least 5-10% of the time. Sometimes catastrophically wrong.
Going back to something closer to my scope of practice- low back pain. Low back pain is the most common chief complaint I see, and I think that I am very good at managing it. I am not perfect, and nobody else is either. If you have a failure rate of 10% in treating low back pain, you are outstanding.
One of the most frequent questions I am asked is whether a patient is best managed by surgery, spine injections, physical therapy, accupuncture, massage, chiropractic manipulation, nutritional supplementation, etc. The short answer is "I have no idea, really."
That is just being appropriately humble- in truth, I am easily in the top 1% in my competence to answer the question, but I am aware of the limitations of my and all of medicine's knowledge on the topic. Low back pain is a very complex topic, far more complex than the stock market, and there is an upper limit to how precisely we can answer thesse questions. I am just aware of our limitations.
What I believe is that all these different approaches, ranging for traditional approaches like physical therapy, surgery, and injections to more non-traditional approaches like accupuncture and nutritional supplements, they all work, but we have not yet perfected how to stratify which approach will work for which patients.
Does this mean that "well, since we can't be perfect, we should throw away this information?" Absolutely not! But it does mean that we need to be humble enough to realize that we are going to be wrong 15-20% of the time, at least.
Giving specific examples, with patients with low back pain radiating leg pain, the evidence for use of a neuropathic agent like Neurontin is significantly stornger than a muscle relaxant like Flexeril (which, in truth, is more of an anxietiolic than it is a muscle relaxant), so therefore I will use Neurontin more often.
As another example, the data is much stronger in support of using a transforaminal approach than an interlaminar approach for epidural steroid injections in the low back, so that is the approach I use the vast majority of the time.
As yet another example, if a patient has a directional preference (e.g., pain worse with bending forward), then a physical therapy program that takes that directional preference into account is far more likely to be succesful.
So, of course, I should adhere to these evidence-based guidelines. I just can't be so arrogant as to assume that just because I am adhering to evidence-based guidelines that I will always be right.
Coming back now to health policy .... there is a strong movement underway to make physicians more accountable to evidence-based guidelines. I think this is a good thing. Patients who have diabetes should have their hemoglobin A1c checked regularly, patients should have their blood pressure checked regularly and placed on appropriate agents, etc.
We just can't get arrogant about it. These standards we are moving toward will better approximate good health care- they do not equal good health care.
Just my 2 cents.
Sunday, January 25, 2009
Sunday, January 4, 2009
The Last Lecture
I just finished reading Randy Pausch's "The Last Lecture." Wow- what an inspiring book. For those who haven't read it (or seen the actual lecture on Youtube, which is worth looking up), Randy Pausch was a computer science professor at Carnegie Mellon with pancreatic cancer. He was given the opportunity to deliver one last lecture to Carnegie Mellon, and more importantly to his children. His lecture is about living out your childhood dreams, and just being a better person.
One thing that struck me in the book (it's not mentioned in the lecture itself) is an anecdote about when he was a professor at the University of Virginia. He was teaching a class on user interface, and would start the semester by bringing in a functional VCR to the front of the room, and then smashing it with a sledgehammer. His point was designers cannot lose site of the end user- the people who actually use the product. VCRs can do a lot of cool things, but they are frustrating to use.
This applies to a lot of consumer products. As an example, this is the year I finally made the switch over to using a Mac. So far, I've been very happy using the Mac. It is striking how much more work and play I have now that I don't waste 30-60 minutes daily dealing with anti-virus and anti-spam software. It's just an easier, cleaner, and more fun interface.
Anyway, reading Randy Pausch's book, I was struck by how often in medicine we lose site of the end user- the patient. Just like computer and VCR designers, we can do so many amazing things in medicine. But none of that really makes a difference unless it affects the end product, and improves the patient's experience.
And I really don't want to be smashing patients with sledgehammers.
One thing that struck me in the book (it's not mentioned in the lecture itself) is an anecdote about when he was a professor at the University of Virginia. He was teaching a class on user interface, and would start the semester by bringing in a functional VCR to the front of the room, and then smashing it with a sledgehammer. His point was designers cannot lose site of the end user- the people who actually use the product. VCRs can do a lot of cool things, but they are frustrating to use.
This applies to a lot of consumer products. As an example, this is the year I finally made the switch over to using a Mac. So far, I've been very happy using the Mac. It is striking how much more work and play I have now that I don't waste 30-60 minutes daily dealing with anti-virus and anti-spam software. It's just an easier, cleaner, and more fun interface.
Anyway, reading Randy Pausch's book, I was struck by how often in medicine we lose site of the end user- the patient. Just like computer and VCR designers, we can do so many amazing things in medicine. But none of that really makes a difference unless it affects the end product, and improves the patient's experience.
And I really don't want to be smashing patients with sledgehammers.
Bluefish
My favorite fish, growing up as a child in New Jersey, was bluefish. When I went away for college in Wisconsin, it didn't occur to me that they didn't have bluefish there, and I didn't rediscover bluefish until medical school.
In medical school, my love affair with bluefish really began. I loved the same things that I loved as a kid- how it was a big, powerfully flavored fish, but in medical school there was another, more important virtue. It was cheap. Very cheap. Another benefit was that most other people weren't particularly familiar with it, so when it was a great fish to prepare for dinner to impress a date (they didn't have to know it was $1.99/lb).
Now that I am older and have more perspective, I realize that many people don't like bluefish. Because it is so oily, it does not travel well, so it is very hard to get outside of the east coast. For example, I never ate bluefish at all in the years I've lived in Wisconsin, Illinois, or Arkansas- you just couldn't find it out there.
Another thing I realize about bluefish is that it is really a perfect metaphor for my homestate of New Jersey- unlike other fish that try to pretend that they aren't a fish ("try the flounder, it's not particularly fishy"), there is no mistaking bluefish- it very much tastes like a fish. Like NJ, when bluefish is at its worst, it's oily, rancid, and particularly smelly. But when it is at it's best, there is nothing better- it's powerful, colorful, flavorful, and completely in your face with how fishy it is. I just read on Wikipedia that the bluefish is cannabilistic. That fits too.
It's going to be a real stretch for me to try and extend this discussion of bluefish into something remotely relevant for a Kinemedics blog. In all honesty, I am just excited about two beautiful fish I picked up at Wholey's fish market in the Strip District, which is to me the most Pittsburghian location in Pittsburgh. But overly extended metaphors is what I do, so here goes ....
In a few weeks, I will be attending the Association of Academic Physiatrists (AAP) meeting. This is the first academic organization with which I became involved, and is still the meeting for which I have the closest ties.
If I have gained one thing from academic meetings, it is the value of meeting other people at other locations. I went to a very good residency program in New Jersey, but as good as the program was, the things I learned there were still just a subset of a much larger picture.
My favorite experience from the AAP was one of my earliest meeting's there. Somehow or another, I ended up sitting at the table for the AAP's Education Committee. In retrospect, I don't think I was supposed to be there, and I was the only resident sitting at a table with Residency Program Directors. I was too naive (and probably too brash) to keep quiet, so as points were discussed, I offered my opinions. I think the committee appreciated that I cared enough to want to help, and before I knew it, I was an actual member of the committee. I think this is how a lot of opportunities in life present themselves- they are accidents initially, but the world always needs people who give a crap.
It has been tremendously beneficial to my career to meet people from other organizations, and learn from them. While the fundamentals of who I am, both as a person and a clinician, are deeply rooted in my upbringing and training in New Jersey, I am glad that I have branched out and explored other parts of the world and expanded my experiences.
All that said, I am just really looking forward to that bluefish tonight.
Saturday, January 3, 2009
"It's a very odd experience, how this is starting to feel normal"
Over the past few days, I spent some time with a friend's family in the ICU waiting room.
It was an interesting experience for me- it certainly useful every once in a while to remember how things feel from the perspective of patients and there family.
One of the things I gained from the experience is the perspective that overall, modern medicine is a wonderful thing. There are a lot of complaints with the flaws in medicine, and we have all experienced our shares of frustrations. But overall, modern medicine can do some amazing things. This is particularly true at the University of Pittsburgh- I have been thoroughly impressed since I have come here with the overall quality of care, both technically, and just the general sense of humanity. This was over a holiday week (New Years), and the number of people working at UPMC who tirelessly extended themselves to take care of the patients and their families impressed me. I was proud to be part of the organization.
I was also impressed by the love family can bring to a patient. In the waiting room of the ICU, a few families were camped out. Some had not left the side of their loved ones for weeks at a time. As a physician, I was vaguely aware that families stay at the hospital to be with their loved ones, but I had never actually seen it first hand.
One thing was striking is how disorienting the experience is in terms of sense of time. Staying bunkered in the waiting rooms completely removes you from those external cues that separate morning from night. One day just blends into the next.
Another thing that was evident to me was the toll the human body takes from the lack of movement. This is true of the patients, but you also see it in the family members who are staying with the patients. Between the lack of exercise and the ample amounts of caffeine most family members consume, plus the intense emotional toll of their loved ones being sick, nearly everyone in the ICU waiting room was shaking their legs vigorously while they sat.
One of the family members said to me "it's a very odd experience, how this is starting to feel normal." I know what he meant, and I agree it is odd.
I was fortunate enough to make it through medical school without any patients dying on me. I remember the first time a patient of mine died- I was an intern in Cooperstown, NY. I wasn't expecting it at all- of the hundreds of patients I had seen that year, the one who died was by no means the sickest. But all of a sudden, there she was- just moments earlier she had been breathing and had a pulse, and now she lay motionless.
I had worked closely with a medical student that month, and the two of us just went into a back room, and I cried. For a long time- probably an hour. I had seen dead people before, but I had never seen someone die- actually go through the process of dying to the point where they were dead. And having lived through it, the experience was definitely not normal.
I can't say that I am used to the experience now- my line of work does not lend itself to patients dying very often. But I don't know that I would cry for an hour anymore. I don't think I am particularly jaded, but the process of death is just, well, more normal for me. And as my friend said to me the other day in the ICU, when I reflect on it, is very odd when the process starts feeling normal.
It was an interesting experience for me- it certainly useful every once in a while to remember how things feel from the perspective of patients and there family.
One of the things I gained from the experience is the perspective that overall, modern medicine is a wonderful thing. There are a lot of complaints with the flaws in medicine, and we have all experienced our shares of frustrations. But overall, modern medicine can do some amazing things. This is particularly true at the University of Pittsburgh- I have been thoroughly impressed since I have come here with the overall quality of care, both technically, and just the general sense of humanity. This was over a holiday week (New Years), and the number of people working at UPMC who tirelessly extended themselves to take care of the patients and their families impressed me. I was proud to be part of the organization.
I was also impressed by the love family can bring to a patient. In the waiting room of the ICU, a few families were camped out. Some had not left the side of their loved ones for weeks at a time. As a physician, I was vaguely aware that families stay at the hospital to be with their loved ones, but I had never actually seen it first hand.
One thing was striking is how disorienting the experience is in terms of sense of time. Staying bunkered in the waiting rooms completely removes you from those external cues that separate morning from night. One day just blends into the next.
Another thing that was evident to me was the toll the human body takes from the lack of movement. This is true of the patients, but you also see it in the family members who are staying with the patients. Between the lack of exercise and the ample amounts of caffeine most family members consume, plus the intense emotional toll of their loved ones being sick, nearly everyone in the ICU waiting room was shaking their legs vigorously while they sat.
One of the family members said to me "it's a very odd experience, how this is starting to feel normal." I know what he meant, and I agree it is odd.
I was fortunate enough to make it through medical school without any patients dying on me. I remember the first time a patient of mine died- I was an intern in Cooperstown, NY. I wasn't expecting it at all- of the hundreds of patients I had seen that year, the one who died was by no means the sickest. But all of a sudden, there she was- just moments earlier she had been breathing and had a pulse, and now she lay motionless.
I had worked closely with a medical student that month, and the two of us just went into a back room, and I cried. For a long time- probably an hour. I had seen dead people before, but I had never seen someone die- actually go through the process of dying to the point where they were dead. And having lived through it, the experience was definitely not normal.
I can't say that I am used to the experience now- my line of work does not lend itself to patients dying very often. But I don't know that I would cry for an hour anymore. I don't think I am particularly jaded, but the process of death is just, well, more normal for me. And as my friend said to me the other day in the ICU, when I reflect on it, is very odd when the process starts feeling normal.
Brazilian Jiu-Jitsu, Hammer and Nail, Moo Ridge, and the need to keep learning
I am in the process of learning Brazilian Jiu-Jitsu. I love it so far. I think BJJ is a great sport in of itself, but I also love the process of learning something where you are a complete novice.
Some parts of the sport are easier for me to pick up than others. The moves make a lot of sense to me- partly because I have some background in wrestling, partly because I have a very strong academic background in biomechanics, and partly because I've watched enough mixed martial arts (MMA) over the years to have some familiarity with basic terminology.
By far the hardest part for me is learning how to not rely on my size advantage. I am probably the biggest guy in the dojo, and in most cases outweigh the people I am practicing with by at least 50 pounds. Because of this, the temptation is to simply overpower people or muscle through. When I have sparred before, even with very skilled combatants, I have been able to succesfully use my size to overpower opponents. What I am trying to learn in BJJ class is that you can't simply do that with a skilled opponent. Today, every time I lunged forward, the people I was rolling with were able to grab the sleeve of my gi and start working a wrist lock or arm bar. At first, I was able to fight them off, but eventually I just gassed out.
This reminds me of one of my favorite medicine cliches- to a hammer, everything looks like a nail. What this means is that physicians tend to know that the things they know, and oftentimes we don't think outside of the box enough to consider options outside of our base skill set.
One of my favorite anecdotes related to this point dates back to my grad school days. Even though my PhD research was related to sports biomechanics in female athletes, many other people in my department did research on the anatomy of fossils. So one summer I was part of an expedition in western Colorado, looking for fossils that were part of the initial mammalian radiation.
When I tell people I was on a fossil dig, they find that incredibly interesting. It wasn't for me- it is probably the singular task that I have the least talent in the world. Also, these weren't big fossils- these were very small mammals the size of a mouse, so the bones we were looking at were smaller than my thumbnail.
I remember one day in particular that highlighted how bad I was at fossil collecting. I was looking at a patch of pebbles that was about 1 yard by 1 yard for what must have been several hours. I kept staring at the pebbles, and I couldn't see anything- they just looked like pebbles to me.
Then our expedition leader, Maureen, came by and looked at the same 1 yard patch I had stared at for the past several hours, and within 30 seconds picked out 3 or 4 fossils. It was humiliating, but it was also one of the greatest lessons I have ever had in my life- you can't find something if you don't know what you are looking for.
My lone contribution to the 6 week expedition is that I came up with a clever name for the fossil site. The whole time we were looking for fossils, hundreds of cows would walk over and stare at our team, so I called the fossil site Moo Ridge. I think Moo Ridge may even be referenced in some Paleontology journal somewhere.
The reason I bring this up- when relate this story to medical students and residents, I make sure they understand that everyone has their Moo Ridge- we all have weakspots where we can't see things that are right in front of us, because we don't know what we are looking for. Sometimes we learn to compensate for our blind spots by relying on some other strengths to compensate. But just like my experience in Brazilian Jiu-Jitsu class taught me today, eventually your strengths aren't enough, and you need to learn new skills.
I am reminded of this every week when I attend the Sports Medicine faculty's weekly case series. The great Freddie Fu, the chairman of Pitt's Sports Medicine team, likes to hammer home the point that being good is not good enough- that we always have to ask the question- "what can I do better?"
It's a great question, and it's one that I, and every physician, needs to be asking themself every day. And not just when they are getting choked out on the BJJ mat.
Some parts of the sport are easier for me to pick up than others. The moves make a lot of sense to me- partly because I have some background in wrestling, partly because I have a very strong academic background in biomechanics, and partly because I've watched enough mixed martial arts (MMA) over the years to have some familiarity with basic terminology.
By far the hardest part for me is learning how to not rely on my size advantage. I am probably the biggest guy in the dojo, and in most cases outweigh the people I am practicing with by at least 50 pounds. Because of this, the temptation is to simply overpower people or muscle through. When I have sparred before, even with very skilled combatants, I have been able to succesfully use my size to overpower opponents. What I am trying to learn in BJJ class is that you can't simply do that with a skilled opponent. Today, every time I lunged forward, the people I was rolling with were able to grab the sleeve of my gi and start working a wrist lock or arm bar. At first, I was able to fight them off, but eventually I just gassed out.
This reminds me of one of my favorite medicine cliches- to a hammer, everything looks like a nail. What this means is that physicians tend to know that the things they know, and oftentimes we don't think outside of the box enough to consider options outside of our base skill set.
One of my favorite anecdotes related to this point dates back to my grad school days. Even though my PhD research was related to sports biomechanics in female athletes, many other people in my department did research on the anatomy of fossils. So one summer I was part of an expedition in western Colorado, looking for fossils that were part of the initial mammalian radiation.
When I tell people I was on a fossil dig, they find that incredibly interesting. It wasn't for me- it is probably the singular task that I have the least talent in the world. Also, these weren't big fossils- these were very small mammals the size of a mouse, so the bones we were looking at were smaller than my thumbnail.
I remember one day in particular that highlighted how bad I was at fossil collecting. I was looking at a patch of pebbles that was about 1 yard by 1 yard for what must have been several hours. I kept staring at the pebbles, and I couldn't see anything- they just looked like pebbles to me.
Then our expedition leader, Maureen, came by and looked at the same 1 yard patch I had stared at for the past several hours, and within 30 seconds picked out 3 or 4 fossils. It was humiliating, but it was also one of the greatest lessons I have ever had in my life- you can't find something if you don't know what you are looking for.
My lone contribution to the 6 week expedition is that I came up with a clever name for the fossil site. The whole time we were looking for fossils, hundreds of cows would walk over and stare at our team, so I called the fossil site Moo Ridge. I think Moo Ridge may even be referenced in some Paleontology journal somewhere.
The reason I bring this up- when relate this story to medical students and residents, I make sure they understand that everyone has their Moo Ridge- we all have weakspots where we can't see things that are right in front of us, because we don't know what we are looking for. Sometimes we learn to compensate for our blind spots by relying on some other strengths to compensate. But just like my experience in Brazilian Jiu-Jitsu class taught me today, eventually your strengths aren't enough, and you need to learn new skills.
I am reminded of this every week when I attend the Sports Medicine faculty's weekly case series. The great Freddie Fu, the chairman of Pitt's Sports Medicine team, likes to hammer home the point that being good is not good enough- that we always have to ask the question- "what can I do better?"
It's a great question, and it's one that I, and every physician, needs to be asking themself every day. And not just when they are getting choked out on the BJJ mat.
Subscribe to:
Posts (Atom)