"A Study of an Enterprise Health information System" - Finally, an Informatics Scientist Does A Rigorous Review of a Commercial EHR System, by Cerner

Australian professor Jon Patrick, who I wrote about at this blog in the past such as at my Nov. 2009 post "Academic Freedom and ED EHR's Down Under: An Update", has written what I consider the first truly serious, rigorous evaluation of a commercial EHR system.

He has evaluated the Cerner FirstNet ED EHR, being 'forced' on hospitals in the Australian state of New South Wales by their government.

His evaluation is entitled "A study of an Enterprise Health information System" and was released on March 4, 2011.

In my view, all current EHR's should undergo this level of scrutiny and critique.

His multi-part analysis, down the the level of the data schema, speaks for itself.

The report is at: http://sydney.edu.au/engineering/it/~hitru/index.php?option=com_content&task=view&id=91&Itemid=146

or download directly from: http://www.it.usyd.edu.au/~hitru/essays/Study%20of%20Enterprise%20Health%20IS%20-%20Parts%200-9%20PDF-ARCHIVE.zip [Note: this is a 12 Mb .zip file].

Summary:

This is a study into the roll-out of Cerner FirstNet into EDs in NSW. The original study was issued in Dec 2009 (Part 3.1). This has been added to with a new study in 2010 consisting of discussions with 7 ED Directors (Part 3.2), discussions with software experts who do performance evaluations on Cerner sites (Part 3.3), and reviews of Entity-Relationship Diagrams (Part 3.4), Schema descriptions and data tables from customer installations (Part 3.5 & 3.6). All this information is coalesced to establish a much more detailed picture of a Cerner installation (Part 3.7). A number of weaknesses are identified in the design and implementation and risk assessments are recommended for organisations using this software or intending to use it. Regulations that might minimise the risks to users of health software are recommended (Part 3.8). An alternative architecture and method for constructing clinical information systems is presented (Part 3.9).

The .zip file contains the following files:

3.0 Part 0 - Executive Summary
3.1 Part 1 - A Critical Essay on the Deployment of an ED Clinical Information System ‐ Systemic Failure or Bad Luck? First published here in Oct 2009, revised Dec 2009.
3.2 Part 2 - Discussions with ED Directors: Are we on the right track?
3.3 Part 3 - Discussions with Software Performance Experts.
3.4 Part 4 - Conceptual Data Modelling.
3.5 Part 5 - Database Relational Schema and Data Tables.
3.6 Part 6 - Coalescing the Analyses of the ER Diagrams, Relational Schemata and Data Tables.
3.7 Part 7 - The Integrated Assessment.
3.8 Part 8 - Future HIT Regulation Proposals.
3.9 Part 9 - Ockham's Razor of Design. Published at the IHI conference, Nov 2010 Washington.

A non-technical but revealing summary from Part 2:

Discussions were held with the Directors of 7 Emergency Departments in New South Wales (NSW) public hospitals assessing the impact of the introduction of the FirstNet information system into their Departments. All but one of the Directors has found that the system has had a deleterious impact on their department’s clinical work. The range of problems reported indicate that whilst the software is not fit-for-purpose, many of the problems are created or exacerbated by attitudes of the NSW Health IT support, Health Support Services (HSS). In most departments it was reported that staff have developed significant strategies for minimising and circumventing the use of the system. The Directors are frustrated by the lack of a reporting functionality that disables their ability to monitor their own department’s performance. Most Directors report an increase in the time required to deal with patients and therefore a deterioration in access block times. This has been masked by changes in the way this time has been redefined by NSW Health. Overall, most perceive that in moving from their previous information system EDIS to FirstNet they and their patients have suffered. Most Directors are resigned to the fact that no improvements will be made to the current performance of the system due to its inherent inadequacies and the attitude of HSS. A consequence of the ED Directors critique leads inevitably to the debate on the merits of enterprise wide systems versus best-of-breed systems. Emerging from these issues are criteria for a new technology for creating clinical information systems.

I will have more to say on this study later. I have to take an injured relative to the doctor. (Note - the culpable system was not by this vendor).

-- SS

HIMSS and Healthcare IT: We Don't Need A "Usability Maturity Model." We Need - USABILITY - and Less of Cold-Blooded Calculus

Health IT industry consortium HIMSS has followed up its Master of the Obvious, 50-years-too-late paper "Defining and Testing EMR Usability: Principles and Proposed Methods of EMR Usability Evaluation and Rating" (June 2009) that I wrote about in a Dec. 2010 post "Unintended errors with EHR-based result management: a case series, and a special pleading for health IT" with a new report:

Promoting Usability in Health Organizations: Usability Maturity Model (PDF, Feb. 2011)

I knew I was in for a heaping helping of gobbledygook after reading the title itself:

“Promoting Usability in Health Organizations: Initial Steps and Progress Toward a Healthcare Usability Maturity Model”

A “Healthcare Usability Maturity Model” is a clue that what follows will take simplicity, and expend considerable ink to tangle it into a mess of process mysticism, buzzwords, paradigms, very pretty charts, and other Master-of-the-Obvious-made-to-look-sophisticated, destined-for-the-dusty-shelf consultant report.

That is largely what follows. See for yourself.

This passage in the Executive Summary is pathognomonic of the amoral, money-over-blood, patients-as-guinea-pigs mentality that inhabits the commercial health IT sector. I find it particularly revealing and revolting:

... Leaders may be reluctant to incorporate usability principles and practices [in a safety-critical technology increasingly mediating all aspects of medical care delivery - ed.] because of perceptions that these methods may slow development and product fielding. However, data exist on usability return on investment (ROI), outlining the value of adopting usability in health organizations.

Excuse me? It takes proof of "ROI" to convince these companies to produce usable (which subsumes the concept of "safe") health IT?

This is as cold-blooded a calculus as it gets. This is Ford Pinto material:

... Critics alleged that the vehicle's lack of reinforcing structure between the rear panel and the tank meant the tank would be pushed forward and punctured by the protruding bolts of the differential^[15] [by a rear end collision of only 20 mph/32 km/h, causing a fire - ed.]— making the car less safe than its contemporaries.

According to a 1977 Mother Jones article, Ford allegedly was aware of the design flaw, refused to pay for a redesign, and decided it would be cheaper to pay off possible lawsuits for resulting deaths. The magazine obtained a cost-benefit analysis that it said Ford had used to compare the cost of an $11 repair against the monetary value of a human life—what became known as the Ford Pinto Memo.^[13][16][17]

An example of a Pinto rear-end accident that led to a lawsuit was the 1972 accident that killed Lilly Gray and severely burned 13-year old Richard Grimshaw. The accident resulted in the court case Grimshaw v. Ford Motor Co.,^[18] in which the California Court of Appeal for the Fourth Appellate District upheld compensatory damages of $2.5 million and punitive damages of $3.5 million against Ford, partially because Ford had been aware of the design defects before production but had decided against changing the design.

[In more recent years, others said the Ford Pinto scandal was not clear-cut, but health IT unusability increasing odds for medical errors is clear-cut - ed.]

Why are these companies and their leadership getting to play God - in a field such as Medicine?

Usability? Those Luddite doctors and crummy patients will get usability over my - er, their - dead bodies, if I don't profit handsomely from it.

Finally, I have several meta-observations about this new report:

• We don't need a "usability maturity model." We need USABILITY. We need the common and longstanding knowledge of software usability in other mission critical sectors to be applied on the development whiteboard and usability testing labs (if any!) at HIT vendors.
  

• Why is it that health IT usability has to be written about, basically as if from the grade school level, in 2011 - some sixty years into the “computer revolution?” Do we still write treatises on why it’s wise to use sterile technique and good lighting in operating rooms? What is the major malfunction in this industry?
  

• Why is this treatise not entitled “Promoting Usability in Health IT Vendor Development Shops: A Vendor Responsibility Model”?

• The frequent use of the terminology "user experience" as applied to healthcare IT in this report struck my eye. It has been adopted in this report, but was uncommon regarding health IT until recently, as in my posts about the health IT mission hostile user experience.
  

The terminology is conspicuously absent in the aforementioned earlier 2009 HIMSS report. It would not take Sherlock Holmes to theorize that I might be an unattributed contributor to the new 2011 HIMSS report. If anyone knows differently, I'd be interested to hear about it.

-- SS