“User-friendly” is a term that comes up increasingly often in the context of Electronic Health Records (EHR) and other healthcare IT systems. Most agree on the importance of user experience (UX), but few understand what this concept truly means. Unfortunately, for clinicians and their administrative staff, the healthcare industry, as a whole, is alarmingly behind when it comes to UX. This blog article will take a critical look at the current state of EHR usability evaluation, as well as the various solutions that are being proposed to improve EHR design:
- How does one determine if a system is truly user-friendly?
- What is the relationship between usability and patient safety?
- What is the role that national healthcare policy plays in improving health IT usability?
- Why are government mandates not enough to increase effective EHR adoption?
- What is the best way to comprehensively evaluate and rate the usability of various EHR products that are currently in the market?
- How can decision makers in provider organizations be better supported to help them achieve their meaningful use targets, objectives, and goals?
- How will value-based care models, distributed care teams, and networked services impact the user experience of EHR systems moving forward?
Read on to find out…
User-Centered Design and Patient Safety
User-centered design (UCD) can dramatically benefit technological systems and applications within any industry, but in healthcare, the need is much more critical. Poor usability can directly and indirectly contribute to massive increases in costs for health plans, providers, patients, and our government. Usability problems can also jeopardize patient safety, and lead to unnecessary treatments, pain, suffering, or even death. User-centered design (once known as “Human Factors Engineering”) first became recognized as a critical path process within the domain of airplane cockpits and military applications, because it is paramount that users are able to operate these systems with little or no human error. It is literally a matter of life or death for the people involved. One could argue that healthcare is quite a similar domain in this regard.
The Current State of EHR Usability and Meaningful Use
At the beginning of 2009, the Health Information Technology for Economic and Clinical Health (HITECH) Act was passed, granting the Health and Human Services (HHS) department “the authority to establish programs to improve healthcare quality, safety, and efficiency through the promotion of health information technology (HIT), including electronic health records and private and secure electronic health information exchange.” At the time, The Office of the National Coordinator for Health Information Technology (ONC) confidently promised that EHRs would offer more complete, accurate information, easily accessed and shared between doctors’ offices, hospitals, health plans, or even patients’ families in a secure manner. They also suggested that EHRs can empower patients so that they are able to become more actively involved in the management of their own health.
However, according to Mosquera, a contributing writer for a HIMSS Media news outlet, most providers and health IT professionals believe there is “uneven usefulness, ease of usability and user satisfaction among EHRs.” During a recent public meeting of the Health IT Policy Committee at the ONC, similar sentiments were expressed. Amongst the committee members, there was widespread concern that vendors have no universal usability standards or benchmarks to drive innovation and measure their products against. Meanwhile, traditional marketing strategies lead vendors to repeatedly favor development of new, poorly vetted functionality over core usability enhancements.
Despite an elevated rhetorical emphasis on patient safety during the past couple decades, current usability initiatives have not translated into more intuitive user interfaces that properly mitigate risk of misdiagnosis or inappropriate, harmful medical procedures. The ONC Health IT Policy Committee vividly discusses some of the major usability issues that persist in many EHRs today:
- Inputting patient record data is inefficient and prone to error. One committee participant claimed that it took up to 500 clicks to complete a single nursing admission form.
- Transfers of Care, and other critical info, are inconsistently tucked away within encounter records and notes.
- Clinical information transferred between various systems and applications is often unusable, because it is not well-structured or easy to find at the times when it is needed most.
Simply cramming more and more information into a patient’s record is extremely overwhelming for providers and their staff. Without true prioritization, intuitive navigation, consistent organization, better collaborative tools, and contextual presentation of relevant data, EHRs will not be able to significantly improve patient care or meet the needs of providers, as the ONC promised over 6 years ago.
In support of the goals and objectives of the HITECH Act, the ONC mandates that EHR products become certified for meaningful use by an Accredited Testing Laboratory (ATL) This certification “ensures that the health IT providers choose to adopt offers the necessary capabilities, functionalities, and security to enable them to achieve MU [Meaningful Use].” As part of this certification, the ONC requires EHR vendors to report on the UCD process they followed for a select set of capabilities, specifically capabilities that have patient safety implications. Unfortunately, these vendors are not currently required to follow a specific set of UCD best practices in order to become certified. They are only expected to disclose their process to the ATL that certifies them.
The government can continue to mandate EHR adoption using a carrot and stick approach, but it is becoming increasingly clear that EHRs will not be effectively utilized until the technology becomes fully centered around user goals, objectives, tasks, and workflows. In fact, the Agency for Healthcare Research and Quality (AHRQ) reports, “One of the key factors driving the adoption and appropriate utilization of electronic health record (EHR) systems is their usability.”
AHRQ’s Contributions to EHR Usability Evaluation
Since 2009, AHRQ has released a series of reports on EHR usability. One consistent theme in these reports is a call for a framework to evaluate existing and developing EHR systems against proven standards in functionality and design. One of these reports suggests developing “a practical and fair process of usability evaluation.” According to another report, there are lots of disagreement amongst vendors about the ideal UCD process or method for ensuring good usability in their health IT software products. Many healthcare IT companies are intimidated or concerned about usability evaluation becoming a required and formal part of the product certification process. However, most express interest in an independent body guiding the development of universal EHR usability standards. The collective body of AHRQ research reveals:
- No consistent adherence to a formal UCD process
- A lack of diversity in usability testing participants, leading to blind spots with undersampled populations.
- Very little focus on accessibility and users with disabilities
- Effectiveness of adaptive/responsive design and user-customizable content is not well understood
Recommendations in these reports include:
- Encouraging vendors to conduct usability tests much earlier in the product development lifecycle
- Establishing formal usability evaluation certification requirements for EHRs
- Generating a common, industry-wide set of realistic, applicable tasks for usability testing validation
- Developing a national, independent EHR usability laboratory to support collaboration on best practices, tools, processes, and implementation of health IT systems.
- Creating a set of guiding design principles specific to the healthcare industry
- Publicly documenting key patterns of information usage amongst clinicians and administrative staff
Despite all of these findings, AHRQ unfortunately stops short of a comprehensive solution to the EHR usability problem. For example, this 2011 report advocates for the construction of a web-based EHR usability toolkit, but it suggests that the assessment component of the toolkit only include usability questionnaires consisting of self-reported, subjective measures related to user satisfaction and explicit perceptions (opinions) of the user interface. Observational, behavior-based methodologies, such as moderated and unmoderated usability testing, are outrightly dismissed, based on their greater resource demand and inherent requirements for trained user research experts.
Even more surprising, the authors recommend that their usability toolkit be primarily utilized by end user provider organizations as a means of critically self-evaluating the user experience aspects of various EHR options when making a purchasing decision for their practice or hospital. I firmly believe that it is not right to pass these intensive evaluative responsibilities onto paying health IT customers. Such usability assessments should be occurring much further upstream, sponsored by the vendors themselves; or independent, policy-based institutions, such as the ONC and National Institute of Standards and Technology (NIST). Provider organizations should then be given transparent access to the summative results of these studies. This is the input that decision-makers truly need in order to make the best technology choices for their clinical offices.
MedStar Health’s EHR User-Centered Design Evaluation Framework
MedStar Health National Center for Human Factors in Healthcare worked directly with the American Medical Association (AMA) to develop a UCD framework and “increase transparency of [EHR] vendor usability processes.” They present this framework as a response to the ONC’s historically anemic emphasis on usability, including the absence of formalized usability evaluation criteria in the meaningful use certification process. MedStar Health came up with a 15-point scale to examine the extent that EHR vendors follow what they determined to be UCD best practices.
While this is a worthwhile effort, it has some considerable limitations and gaps, in terms of comprehensive usability evaluation. MedStar Health openly admits that this framework does not do anything to assess “the actual usability of the EHR as experienced by end users.” Focusing on vendor process improvement is valuable, but direct, independent analysis of current EHR product offerings is equally important. Provider organizations have the right to know which health IT will help them most effectively and safely take care of their patients. Furthermore, the vendors that choose to do the right thing by producing a scientifically-proven better user experience deserve to be showcased and praised for their noble efforts, as this will ultimately lead to an enhanced patient experience; and at a higher level, it can lead to a much stronger healthcare system overall.
Another issue with MedStar Health’s framework is that their defined UCD best practices are heavily skewed toward what I refer to as the “Refine” stage of the UCD process (refer to the diagram of NantHealth’s UCD process below). The Refine stage involves testing and optimizing a UI. However, this validation is not enough on its own. In my professional opinion, the proceeding UCD process stages, “Contextual Field Research” and “Design,” deserve even greater focus than Refinement. Engagement in these first two stages is a key indicator of UX maturity and commitment within any technology-based organization. By participating in these activities, an organization can proactively drive the UCD process to the front of the product development lifecycle, ensuring that user feedback positively impacts core product requirements and system architectural design right from the outset. This means that systems will be built with properly prioritized functionality, less feature bloat, and a thoughtful design that can harmoniously integrate with a user’s dynamic work environment.
Here is a diagram of the UCD process that my UX team uses everyday at NantHealth:
NIST’s Integrative, Mixed Methods Approach to EHR Usability Evaluation
Back in 2011, Mosquera claimed that NIST was actively responding to EHR design issues by developing an official process or consistent set of procedures for evaluating, testing, measuring, and validating usability of EHRs. The goal was to help providers make better choices when they purchase EMR products, ensuring that these providers have access to a transparent, standardized, and objective rating of the user experience quality that various products can offer to their practices or hospitals.
I am very encouraged to discover that NIST is finally beginning to make good on this promise. Just a couple of months ago, they publicly released their most comprehensive EHR usability study to date. In fact, the study serendipitously includes virtually every methodology that I am recommending in this article.
According to the final report:
Researchers applied a multiple research methods approach in order to triangulate user interactions, experience and dispositions of common encounters with EHRs. Multiple forms of data were collected from a variety of user types allowing for a comprehensive view of EHRs. Data include: 1) an online survey; 2) site observations at two geographical locations; 3) interviews with users at these two locations; 4) usability tests with five different EHRs; 5) and expert reviews of the same EHRs. A team of researchers with expertise and extensive experience in research methods executed the effort, aided by three additional researchers, including clinical experts and human factors experts.
This is a huge step in the right direction. The findings in this report are extremely helpful in providing empirically-grounded best practices to “enhance safety-related usability.” The research identifies critical use risk areas and sheds light on real clinical workflows that involve EHR interactions, within inpatient and ambulatory (outpatient) settings. The report also delivers an extremely useful set of EHR-relevant scenarios, use cases, and usability testing tasks, derived from and validated within the field. All of this information can be leveraged by EHR vendors to improve their respective products.
At the same time, the authors of the NIST report deliberately chose to leave out the specific list of EHR products that were tested in the study. They generically state that 5 vendors voluntarily offered their EHRs for usability testing, but they never reveal the EHR product names. Consequently, the relative usability performance of each EHR product is never rated or compared, based on the usability testing, heuristic evaluation, and user survey results. Without this competitive usability analysis and public transparency, provider organizations cannot make practical use of these findings reports, and thus it does not directly help them achieve their meaningful use goals. Unfortunately, this means that NIST still has a long way to go in order to meet its initial, high-level objectives.
My Recommendations for Evaluating EHR Usability
I applaud the effort of the ONC and MedStar Health to review UCD practices across EHR vendors and evaluate their adherence to best practices. However, I believe these best practices should be expanded to cover the full spectrum of the UCD process, as illustrated in the diagram above that my NantHealth UX team produced:
- What UX research and iterative design activities are built into the product development lifecycle?
- How early and often are they conducted?
- What UX artifacts have been constructed and are actively maintained (e.g. UI design pattern library)?
- What are the methodologies used?
In addition, it would be helpful to look at how large the UX team is, where the team is positioned within the organization, the background of the members, their level of expertise, and their years of experience?
Beyond UCD process evaluation, I strongly feel that the actual usability of these critical EHR systems needs to be independently tested, as part of the certification process. This would help provide a more holistic and practical understanding of where the technology in the provider field stands as a whole. Specifically, I would recommend:
- Moderated and unmoderated usability testing, conducted by a neutral, 3rd party UX consultancy. Each EHR system should be tested, based on an identical user task packet, measuring typical metrics, such as Task Success Rate, Error Rate, Time on Task, and User Satisfaction (i.e. perceived intuitiveness or ease-of-use for a given task). The common set of tasks should be derived from contextual field studies to ensure their validity and realism.
- Heuristic evaluations led by trained UX professionals, measuring against literature-based UX best practices
NIST has provided a compelling model for this kind of independent, scientific evaluation in their September 2015 report that I discussed above. However, they must be transparent about the EHR products they test and clearly indicate how each one individually performed in their various tests. Comprehensive, independent usability ratings of these EHRs are critical for this research to be of any value to technology managers on the provider side.
Finally, following from the recommendations of AHRQ, I believe that usability questionnaires can be a useful supplement to task-based observational testing and heuristic evaluations. However, I would extend this methodology to include large-scale customer satisfaction survey data, once again conducted by a neutral, third-party.
The Future of EHR Usability: Value-Based Care, Distributed Care Teams, and Networked Services
As we transition further into a world of value-based care, the EHR user experience is no longer limited to data entry and record retrieval within a practice’s local system. This user experience now extends to a growing constellation of connected services and applications. This trend will only continue as HL7 and FHIR standards mature, evolve, and propagate.
For a provider office, it is important to remember that the EHR still resides at the heart of such a network, especially since it is typically the only software that doctors directly interact with. Consequently, there are many new external (remote) consumers dependent on EHR system data and functionality. The opposite has also become a reality: EHR systems that were once closed, now openly receive information from countless external sources through APIs, EDI message transactions, and various connect adapters. Thus, it is imperative that they start to serve broader, network-based use cases and scenarios. This means that the usability focus must shift towards how nicely these systems play within the collaborative ecosystem of PCPs, specialists, provider staff, payers, patients, outsourced 3rd parties (e.g. billing agencies, authorization processors such as AIM), HIEs, labs, DME vendors, and pharmacies, while they work together as a distributed team to provide quality patient care. Here at NantHealth, we are strategically positioned to play a major role in enabling such capabilities and seamless collaboration.