Australian ED EHR Study: An End to the Line "Your Evidence Is Anecdotal, Thus Worthless?"

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I lamented that while my early mentor in biomedicine Victor P. Satinsky MD taught the wise credo "critical thinking always, or your patient's dead" in the 1970's, our culture had become so perverse that this credo had been largely supplanted with:

"Critical thinking anytime, and your career's dead."

That post was in reaction to continued heckling on a professional mailing list, the American Medical Informatics Association (AMIA) Clinical Information Systems Working Group (cis-wg). On this list, Oregon Health Sciences University professor William Hersh expounded on how the evidence of health IT dangers was largely "anecdotal" therefore to be discounted, and how I, specifically, "didn't know the literature on health IT."



In that post I put the lie to the latter figment. On the former prevarication Dr. Jon Patrick, author of the recent thorough dissection of problems with the ED EHR system being rolled out in public hospitals in New South Wales, Australia (see my Mar. 5, 2011 post "On an EMR Forensic Evaluation by Professor Jon Patrick from Down Under: More Thoughts"), hit the ball out of the park:

Prof. Patrick to AMIA cis-wg:



I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.



Discounting Anecdotes:



1. Is a perfidious and specious act.



2. It denies early warning signs of problems.



3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.



4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.



5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.



6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

Prof. Patrick had to once again put the lie to this refrain at a comment on the HISTalk blog yesterday, where the anonymous proprietor had written in a review of Patrick's EHR study:

... On the other hand, I wouldn’t say it’s [Prof. Patrick's Cerner FirstNet study] necessarily unbiased, it focuses on implementation of a single department application that didn’t go well for a variety of reasons (despite many successful FirstNet implementations elsewhere), it uses the unchallenged anecdotal comments of unhappy users who make it clear they liked their previous EDIS better, and it nitpicks (I wasn’t moved to find a pitchfork when I learned that the primary keys in the Millennium database aren’t named consistently).



[Considering the complexity and changeability of healthcare and the corresponding software lifecycle, I duly note that that latter attitude about 'failing to find pitchforks' regarding breaches of sound software engineering practices seems to be a symptom of the larger health IT disease that Prof. Patrick writes about - ed.]

Dr. Patrick then knocked the ball out of the Southern Hemisphere with a comment of his own about anecdotalism and a link to an expansion of the aforementioned ideas he'd shared on "discounting anecdotes":

Prof. Patrick to HISTalk owner:



Your Comment “it uses the unchallenged anecdotal comments of unhappy users ” is not only unfair but unreasonably inaccurate. The comments made by the users are the Directors of 7 EDs and so they have a right to carry authority by virtue of the experience but also the number of 6 out 7 presenting a view of Firstnet as unfit for purpose has numeric validity, which they justify with about 20+ pages of their comments – see Part 2 Appendix 2.



I would also point readers to my editorial about the role of personal experiences being the most useful information to understand the nature of socio-technical failures. http://aci.schattauer.de/en/contents/archive/issue/1124/manuscript/15463/show.html

The essay at that link, "The Validity of Personal Experiences in Evaluating HIT", is an editorial in Johns Hopkins informaticist Chris Lehmann's brilliant new journal "Applied Clinical Informatics."



The editorial is available free, and is a must-read for anyone in a decision-making or managerial role in mission critical domains, including our elected representatives.



In the editorial Dr. Patrick concludes. similarly to his earlier AMIA mailing list opinion:

... the denial of recounted personal experiences in discussion and analysis of HIT is biased and specious and has the effect of:



1. Denying early warning signs of problems.



2. Denying a voice for the working clinical community who have to operationalise decisions made by others and thus disempowers them.



3. Denying process improvement within an institution – which is most important for Evidence Based Medicine and incremental review of local processes.



4. Discourages staff from engaging in any form of process improvement hence worsening the sense of disenchantment.



Every legitimate personal experience of a HIT deserves to be considered on its merits lest we wish to retreat from process and product improvement. Mechanisms of censorship both implicit due to contrived processes of disinformation and disempowerment or explicit due to contractual specifications will lead to more waste, lost productivity, contempt for the providers, and distress among frontline staff rather than increased productivity and improved patient health and safety as we all desire.

In my view, the drivers or motivators for the "anecdotalist" accusation are these, singly or in combination:

• Too much "education" to see the nose on one's face, as in, to think zebras and unicorns instead of horses when hearing hoofbeats outside one's midwest U.S. abode (eggheads);
  
  
• Too little common sense (fools), as in Scott Adams' example: "IGNORING ALL ANECDOTAL EVIDENCE - Example: I always get hives immediately after eating strawberries. But without a scientifically controlled experiment, it’s not reliable data. So I continue to eat strawberries every day, since I can’t tell if they cause hives";
  
  
• Too much concern for the possible interruption of flow of money or power in one's direction (gonifs).
  
  

In conclusion, the anecdotalist refrain of "your evidence is anecdotal" [therefore of little or no value] when used repetitively against competent observers is the refrain of eggheads, fools and gonifs.



In healthcare, the end result is "your patient's dead."



My "anecdotal relative" injured in a mid-2010 HIT mishap is sadly an example.



[June 2011 addendum: my relative, after much suffering, has now died of complications of the "anecdotal HIT mishap" - ed.]



As for myself, I am a Markopolist (see my Sept. 2010 post "Health IT: On Anecdotalism and Totalitarianism").



-- SS

The Dangers of Critical Thinking in A Politicized, Irrational Culture

My early mentor in biomedicine Victor P. Satinsky MD lived by the credo "critical thinking always, or your patient's dead."

Unfortunately, the motto of today's degraded culture in biomedicine (and other domains) might well be "critical thinking, and your career is dead."

At "Health IT: On Anecdotalism and Totalitarianism" I posted these thoughts:

At the article Blumenthal on EMRs: Debate "raging" over competition vs. standards, ONC czar David Blumenthal is cited as saying several interesting things:

... EMRs make him a better physician, he said, recounting personal anecdotes of discovering patients' allergies through automated EMR alerts and using stored image date to more quickly get a diagnosis for a patient without subjecting them to more radiation and toxic radiation agents ...

It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

Yet anecdotal reports of EMR "saves" are used by a czar to justify tens of billions of dollars of expenditures?

To the anecdotalists, I say: you can't have it both ways.

I also posted nearly the same complete Healthcare Renewal post to several mailing lists of the American Medical Informatics Association including the Clinical Information Systems working group (CIS-WG). CIS-WG is a mailing list read by something over 1000 healthcare informatics professionals at last time I had access to the statistics a few years ago.

I received some supportive replies from colleagues, including collaborators on the AHIMA (not AMIA) book we co-authored in 2009 entitled "H.I.T. or Miss: Lessons Learned from Health Information Technology Implementations" - itself not exactly a popular exercise among the strictly positivist informatics leadership class.

Now, I thought my posting on the double standard regarding "anecdotes" highly straightforward. From a high ranking academic leader of a major national informatics program, Bill Hersh at OHSU, however, the following reply was posted:

Scot,

For someone who is a faculty in informatics, I am surprised at how unfamiliar you are with the literature. There is solid evidence, much more than anecdotes, on the efficacy of health IT. Even Dr. Blumenthal himself has posted on that. (I think you are taking this quote out of context.

I am then served a platter of literature I must be "unfamiliar with" such as:

Goldzweig, C., Towfigh, A., et al. (2009). Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28: w282-w293.

[Note - I had commented on and linked to this very article at this Aug. 29, 2010 post - ed.]

Garg, A., Adhikari, N., et al. (2005). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 293: 1223-1238.

Amarasingham, R., Plantinga, L., et al. (2009). Clinical information technologies and inpatient outcomes: a multiple hospital study. Archives of Internal Medicine, 169: 108-114.

Longhurst, C., Parast, L., et al. (2010). Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics, 126: 14-21.

Now, aside from the serious breach of academic etiquette of attacking the competence of your colleagues in a public forum, I seem to be hearing that it's OK to purvey positive anecdotes about health IT (usually based on weak retrospective observational studies alone, not randomized clinical trials), but not anecdotes of HIT malfunctions or of HIT-related adverse outcomes, since there's 'solid evidence' of the efficacy of health IT.

In plain English, an ad hominem fallacy is followed by an appeal to authority of sorts ("the literature") to justify public Pollyanna attitudes towards HIT by high ranking officials. And since the literature is so glowing, negative anecdotes must be of low worth.

[Jan. 2011 addendum - perhaps the literature's not so glowing - ed.]

Actually, the response in its entirety was a non sequitur to my post.

Others in cis-wg took affront. One of my book co-authors responded that:

I didn't read Scot's comment as saying that there is no data in support of EHRs .... despite a body of evidence, Dr. Blumenthal made a statement only about personal experience in what Scot quoted.

At the same time, ONC has asked for EHR users to share their positive experiences, but has not (as far as I have seen) asked for their failures. Quite frankly, the failures would be more instructive and would constitute a very valuable repository. ONC has also not shared the studies on the dangers and failures of EHR implementations with nearly the same passion as the successes. My point is that there is data and there is anecdote for both sides and ONC has not presented a balanced picture so that we can adequately address the real risks.

Another CIS-WG reader shared valuable observations:

Regarding Goldzweig:

Regarding studies conducted by the HIT leaders (e.g. Partners Vandy, Regenstrief, IHC, etc...): "Many of the new studies report modest or even no benefits of the new applications or changed functionalities."

Regarding studies of commercial HIT systems: "These study results were similar to those reported by the health IT leaders—most studies demonstrated modest benefits, some demonstrated no benefits, and a few demonstrated marked benefits."

Regarding Adhikari:

The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome,

52 trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements.

And so on.

In other words, the literature's mixed.

Finally, the knock-the-ball-out-of-the-park response came from a Medical Informatics researcher Down Under:

I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.

Discounting Anecdotes:

1. Is a perfidious and specious act.

2. It denies early warning signs of problems.

3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.

4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.

5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.

6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

I, of course, added that ignoring "anecdotes" of HIT problems was even more cavalier if one recognized the context of the stories, that is, that they arise in an environment hostile to diffusion through contractual arrangements, poorly recognized reporting resources, fear, etc. Understood in context, they should be receiving more research attention than otherwise, and certainly not ignored.

However, the comment about my purported lack of knowledge of the literature was sent out to 1000+ people by a nationally-recognized informatics leader, people who may or may not read the followup in detail.

This is unfortunate and perhaps reflects the ethos of our day.

-- SS

Addendum:

Also see the Aug. 2011 post "From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things" for a truly stunning takedown of the "anecdotes" canard, which amounts to conflating risk management with scientific discovery.