Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications

Abstract

Rising ownership of smartphones and tablets across social and demographic groups has made mobile applications, or apps, a potentially promising tool for engaging patients in their health care, particularly those with high health care needs. Through a systematic search of iOS (Apple) and Android app stores and an analysis of apps targeting individuals with chronic illnesses, we assessed the degree to which apps are likely to be useful in patient engagement efforts. Usefulness was determined based on the following criteria: description of engagement, relevance to the targeted patient population, consumer ratings and reviews, and most recent app update. Among the 1,046 health care–related, patient-facing applications identified by our search, 43 percent of iOS apps and 27 percent of Android apps appeared likely to be useful. We also developed criteria for evaluating the patient engagement, quality, and safety of mobile apps.

BACKGROUND

Growing evidence suggests that health care is more efficient and effective when patients are actively engaged in their treatment.¹ Engaged, or activated, patients collaborate with their providers, are treated with respect and dignity, receive information related to their care, and are involved in decision-making.² Two separate reviews commissioned by the Institute of Medicine and Robert Wood Johnson Foundation found that chronic disease self-management and promotion of patient engagement are essential to successful care management programs targeting patients with high needs and high costs and are associated with improved quality of life, functional autonomy, and decreased hospital use.³

Mobile health applications, or apps, designed for smartphones can help empower high-need, high-cost patients to self-manage their health.

Nearly two-thirds of Americans now own smartphones, and ownership is rising among older adults (27%) and those with low household incomes (50%).⁴ Moreover, community health centers and clinics perceive mobile health technologies as an ideal tool to engage their patient populations in chronic disease management.⁵

In this issue brief, we describe criteria for evaluating mobile apps for high-need, high-cost populations based on their potential to improve patient engagement and on their quality and safety. We then discuss results of our efforts to test and refine those criteria using a sample of apps available through the Apple iOS and Android app stores.

HOW WE CONDUCTED THE STUDY

We identified mobile applications targeting high-need, high-cost patients through a systematic search of the Apple iOS iTunes and Android Google Play app stores. Our focus was on “patient-facing” apps—those targeted for use by individuals with chronic illnesses. We evaluated the app store descriptions to identify and characterize the different components that may influence level of patient engagement, quality, and safety.

Identifying Potentially Useful, Health-Related, Patient-Facing Applications

We searched for apps using the following terms: alcohol, arthritis, asthma, bipolar, cancer, cirrhosis, cognitive impairment, chronic obstructive pulmonary disease, COPD, coronary artery disease, dementia, depression, diabetes, drug abuse, elderly, heart disease, heart failure, high blood pressure, hypertension, kidney disease, liver disease, lung disease, obesity, pain, smoking, and stroke. We selected the first 50 listings for each of the 26 search terms.

After removing those that were non-health-related, non-patient-facing, non-English, and highly similar (e.g., “lite” or “pro” versions), we evaluated the apps’ potential usefulness based on description, total number of ratings, mean cumulative ratings, consumer reviews, screenshots, and date of most recent update. Apps were considered unlikely to be useful if they had minimal functionality beyond traditional media such as books and videos, were not relevant to the searched illness, were poorly rated or reviewed by consumers, or were not intended for broad use (e.g., inaccessible without specific login credentials).

Developing a Framework for Evaluating Mobile Health Apps

For apps that appeared potentially useful, we developed an initial set of general criteria to evaluate them on the basis of patient engagement, quality, and safety. The criteria were informed by existing app evaluations and frameworks, as well as our review of app store descriptions.⁶ We also solicited open-ended responses from reviewers to each of these domains to determine if criteria needed to be removed, refined, or added. The evaluation framework was piloted by eight reviewers on two to three apps each and adjusted based on the reviewers’ feedback.

We conceptualized patient engagement as the ability for apps to enable collaboration, activation and participation, information-sharing, and decision-making in one’s own health. Patients may differ to the extent to which they participate in their health care, and we wanted to evaluate the degree to which apps may enable patients to increase their engagement. We constructed and used an engagement pyramid (Exhibit 1) to determine how apps are meeting the needs of patients with differing levels of health care engagement. Patients who are least activated in their own care may face health literacy or social barriers to care, which can be addressed through health education, reminders, and recording of health information. These functions do not require smartphone apps, though apps provide a new platform through which to store and deliver information. Moderately activated patients who are informed and able to keep track of their health may benefit from being able to visualize and summarize their health information, receive guidance on next steps, and communicate with family members and health care providers. Finally, the most engaged patients may benefit from peer support delivered through social media or ongoing motivational challenges that can be delivered through “gamification”—that is, using elements of game design, like competition or point scoring, to make an activity more fun.

We considered quality in terms of reliability of information for the targeted population, recommendability, and usability. Two aspects of safety were considered: the ability for an app to handle “dangerous” information entered by a patient (e.g., low blood sugar or suicidal thoughts) and the commitment of the app developer to respect patient information privacy and security.

RESEARCH FINDINGS

Minority of Apps Appear Likely to be Useful in Engaging Patients

A systematic search of the iOS and Android app stores for a broad set of medical conditions revealed 946 iOS apps and 1,173 Android apps. After removing non-health-care, non-patient-facing, non-English, and highly similar apps, 376 iOS apps and 569 Android apps remained (Exhibit 2).

Of the 376 iOS apps:

• 24 (6%) appeared to have limited engagement beyond traditional media
• 66 (18%) were not relevant to the search condition
• 33 (9%) had poor ratings or reviews, 63 (17%) were last updated prior to 2014
• 29 (8%) were otherwise not assessed to be useful.

Of the 569 Android apps:

• 89 (16%) had limited engagement
• 56 (10%) were not relevant to the search condition
• 8 (1%) had poor ratings or reviews, 200 (35%) were last updated prior to 2014
• 64 (11%) were otherwise not assessed to be useful.

In all, 161 (43%) iOS apps and 152 (27%) Android apps were assessed as possibly useful, of which 126 apps existed on both platforms.

Exhibit 2. Search Results for Apps for Engaging Patients with Medical Conditions

Source: Authors’ analysis.

Criteria for Evaluating the Patient Engagement, Quality, and Safety of Mobile Apps

After performing initial reviews of apps, we refined and added evaluation criteria based on group feedback.⁷ Besides noting guideline use and clinical expert involvement in the app development process to determine the reliability of health information, we added an assessment of patient involvement in app development to gauge the relevance of health information. In addition to asking whether apps enabled data-sharing and use of social media, we asked questions about the mechanism that allowed this. We added specific questions to address whether apps shared data with HealthKit and Google Fit, emerging tools that enable data collected by one app to be shared with others. We determined language as an important barrier to care and decided to capture app availability in Spanish. We also noted wide variation in pricing schemes and developed questions to capture “total app cost” by evaluating the purchase price, in-app purchases, and annual subscription costs.

In the next section, we use examples of apps to show how the framework can be used to demonstrate aspects of engagement, quality, and safety.

Case 1. 7 Cups of Tea

7 Cups of Tea is an app that anonymously connects users with trained “active listeners” for emotional support, counseling, and therapy to help address depression, anxiety, and stress. A strong social support network is an essential component of managing these conditions. Individuals who volunteer as “active listeners” must complete a psychologist-developed online training course in compassionate communication.

7 Cups of Tea connects users to one-on-one
and group support for depression, anxiety, and stress

7 Cups of Tea is designed to support highly engaged patients by connecting them to a private social network of active listeners and rewarding patients with points for participating and sharing in one-on-one and group therapy. The development team includes clinical experts in psychology, though no patients with depression, anxiety, or bipolar disorder are specifically noted to have participated in the development process. The two reviewers disagreed on whether the app appropriately handled “dangerous” information. Both the usefulness and safety of the app hinge on its group of volunteer listeners. While it is certainly possible that a listener may not react appropriately to a user’s plea for help, the use of trained volunteers in this app is comparable to volunteer participation in crisis hotlines and thus likely to be a reasonably safe approach.

While the app does not provide functions to support patients with low or moderate engagement levels, both a clinician and nonclinician scored the app highly on recommendability. This app demonstrates that health apps need not provide a wide array of functionalities or target patients across the engagement spectrum to be useful. An app that addresses specific needs of patients at a particular engagement level can be effective and worth recommending.

Case 2. T2 Mood Tracker by the National Center for Telehealth and Technology, U.S. Department of Defense

T2 Mood Tracker is an app designed to help users monitor and track their emotional health, including anxiety, depression, head injury, stress, posttraumatic stress disorder, and general well-being. Originally designed for members of the armed forces where it received first place in the “general wellness” category during the Apps4Army competition, the app is now available for use by the general public.

T2 Mood Tracker allows users to track symptoms of depression, anxiety,
and posttraumatic stress disorder but does not provide feedback

T2 Mood Tracker contains functionalities to support patients with low, moderate, and high levels of engagement but misses key functionalities for each of its potential target audiences. For patients with low engagement levels, the app includes reminders and tracking of depression, anxiety, or posttraumatic stress disorder symptoms but does not provide educational information on why any of the metrics being tracked are relevant. Moderately engaged patients can graph results over time and create a report or spreadsheet of values that can be shared with clinicians though the app, but data-sharing can only be performed through a nonsecure method (such as email) and the app does not provide any guidance on next steps. For highly engaged users, the app provides no venue for seeking social support and no system to motivate continued use of the app.

A psychologist was involved in app development, but it seems patients did not play a role in the development process. This may partially explain a major safety concern identified during testing: the app treats all reported information the same. Whether a user reports feeling “good” or reports feeling “severely depressed” and “unsafe,” the app provides no guidance on next steps. The app does provide an emergency support phone number in a “local resources” section, but not presenting this information when it is contextually relevant—for instance, when a patient reports feeling unsafe—is a missed opportunity for intervention. The app scored poorly on recommendability by both a clinician and a nonclinician.

T2 Mood Tracker’s website contains a privacy policy governing the website but does not mention the app. As such, there is no way to infer what information is collected about users of the app, how it is used, and with whom it is shared.

T2 Mood Tracker’s major limitations are its fragmented functionalities for patients at different engagement levels and its lack of guidance to the user, the latter becoming a safety concern because of the nature of information being collected.

Case 3. Mango Health

Mango Health is a medication management app that provides users with medication reminders, refill alerts, drug interaction warnings, and the ability to track medication side effects. Medication adherence and persistence are particularly problematic among individuals with chronic illnesses, and the number of medications, complexity of administration regimens, and frequency of regimen changes are all known contributors to nonadherence.

Mango Health app rewards users with points and
a chance to win gifts for taking their medications

Mango Health provides functionalities to activate patients across the engagement spectrum. Patients with low engagement can record a medication list and receive reminders to take each medication on time. Moderately engaged patients can view summaries of their medication history and missed doses, and receive guidance on how to identify and manage drug interactions. Highly engaged patients are encouraged to continue tracking their medications through a points system that enters users in a weekly raffle for prizes. The major limitations for Mango Health are its inability to share information with health care providers or to connect users with peer groups.

The Mango Health website reveals that a clinician serves as vice president of medical outcomes, but it does not indicate any formal patient involvement. In our testing, both a clinician and nonclinician found the app to be usable and rated it high on recommendability.

For users who choose to register an account, Mango Health does keep track of medication lists and points across multiple devices. A privacy policy details what information is collected, how it is used, and with whom it is shared. Mango Health does not claim to protect information by the Health Insurance Portability and Accountability Act (HIPAA) standard as it is not storing information on behalf of health care providers and is therefore not bound to HIPAA.

Mango Health is an example of an app that appears to successfully meet the needs of patients with varying levels of engagement. However, while patients may perceive that health information stored in the app is protected under the same standard as a doctor’s office, this is not the case and may be a source of concern in the event of a data breach.

CONCLUSION

In medicine, every treatment needs to be tailored to the patient and prescribed with an understanding of its benefits and risks. This is also true of apps, which appeal to different audiences by offering varied functionalities. App quality and safety do not necessarily align with functionality and must be considered separately. In developing this framework, we discovered several apps that sacrificed quality or safety in the pursuit of added functionality.

Using a framework that considers the engagement, quality, and safety of mobile apps is critical for stakeholders to identify trustworthy apps that serve the needs of high-need, high-cost populations. While apps have tremendous potential to engage high-need, high-cost populations, a minority of patient-facing health applications on both the Apple and Android stores appear likely to be useful to patients. An evaluation of 143 apps targeting high-need, high-cost patients using the framework proposed in this brief has been completed and results are forthcoming.

Notes

¹ J. Greene, J. H. Hibbard, R. Sacks et al., “When Patient Activation Levels Change, Health Outcomes and Costs Change, Too,” Health Affairs, March 2015 34(3):431–37.

² S. M. Brown, R. Rozenblum, H. Aboumatar et al., “Defining Patient and Family Engagement in the Intensive Care Unit,” American Journal of Respiratory and Critical Care Medicine, Feb. 2015 191(3):358–60; and M. A. Grando, R. Rozenblum, and D. W. Bates (eds.), Information Technology for Patient Empowerment in Healthcare (Berlin: Walter de Gruyter Inc., 2015).

³ C. Boult, A. F. Green, L. B. Boult et al., “Successful Models of Comprehensive Care for Older Adults with Chronic Conditions: Evidence for the Institute of Medicine’s ‘Retooling for an Aging America’ Report,” Journal of the American Geriatrics Society, Dec. 2009 57(12):2328–37; and T. S. Bodenheimer and R. Berry-Millett, Care Management of Patients with Complex Health Care Needs (Princeton, N.J.: Robert Wood Johnson Foundation, Dec. 2009).

⁴ Pew Research Center, Pew Research Center Internet Project Omnibus Survey (Washington, D.C.: Pew, 2014); Pew Research Center, Pew Research Center Internet Project Survey (Washington, D.C.: Pew, 2014).

⁵ A. Broderick and F. Haque, Mobile Health and Patient Engagement in the Safety Net: A Survey of Community Health Centers and Clinics (New York: The Commonwealth Fund, May 2015).

⁶ B. Martínez-Pérez, I. de la Torre-Díez, S. Candelas-Plasencia et al., “Development and Evaluation of Tools for Measuring the Quality of Experience (QoE) in mHealth Applications,” Journal of Medical Systems, Oct. 2013 37(5):9976; IMS Institute for Healthcare Informatics, Patient Apps for Improved Healthcare: From Novelty to Mainstream (Danbury, Conn.: IMSIHI, 2013); and S. Stoyanov, L. Hides, D. Kavanagh et al., “Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps,” JMIR mHealth and uHealth, March 2015 3(1):e27.

⁷ We formalized the use of the System Usability Scale (SUS) and Net Promoter Score (NPS) to assess usability and recommendability, respectively. The SUS was created by John Brooke in 1986 and has been used to evaluate hardware, software, mobile devices, websites, and applications. The NPS is a customer loyalty metric developed in 2003 and trademarked by Fred Reichheld, Bain & Company, and Satmetrix. The NPS consists of a single question answered on an 11-point scale: “How likely is it that you would recommend our company/product/service to a friend or colleague?”