Applying a user-centered approach to evaluate the usability of a mobile application for health professionals in home care services (Preprint)
Griselda Manzano Monfort,Guillermo Paluzie,Mercedes Díaz Gegúndez,Carolina Chabrera
DOI: https://doi.org/10.2196/preprints.28263
2021-02-26
Abstract:BACKGROUND Mobile health (mHealth), or the use of mobile devices in medicine and health, is a sub-category of e-health. mHealth interventions are designed to improve healthcare services, and may be divided into different areas, including the Medical Records and Communication (1) (2). Within this area, we find the mobile electronic health records (EHRs), used by healthcare professionals and personal health record (PHR) applications that patients can use to examine and control their own health data, and applications that allow direct patient control over specific diseases. The use of EHR is expected to eliminate data duplication (3) (4), and save time in issuing routine reports, improving clinical practice, facilitating data collection, generating reports, treatment plans, and medical orders, without removing records from circulation (5)(6)(7)(8). These services and applications that utilize mobile functionality are actively being developed in hospitals, organizations, and other groups (9)(10). To assess and improve upon the usability of mHealth applications, a wide range of usability evaluation methods (UEMs) are available to detect problems in user–system interaction. The UEMs allow the identification of those facets of the interaction that need improvement (11) (12) (13) (14). To determine the usability of any new technology, appropriate and rigorously developed measures must be used (15). Although the use of mHealth has increased rapidly in recent decades, there is limited scientific evidence supporting its effectiveness (16) (17) (18), probably due to the lack of reliable information related to proven benefits (19) (20). Study context The mobile study application was originally developed with the aim that the health professionals of the home hospitalization units could have secure access to the patients’ relevant medical information, record follow-up data at home, and automatic upload to the EHR, saving professional’s time, and avoiding transcription errors (21). Data were analyzed one year after implementation, and it was noted that 86% of the professionals used it on a regular basis and considered it an improvement for their daily work. Total theoretically saved hours in the medical information transcription were 256 in a year, which would correspond to 36.5 days (7-hour shifts). The conclusion was that using an application to consult and update a patient’s health record at home avoids transcription errors and saves professionals’ time. Although the mobile application represents an important advance and an improvement in the care provided by professionals, it is currently not being used homogeneously by all health professionals and, consequently, paper documentation is still being used during home visits. This means that patient data and records continue to be duplicated, which is a waste of time and does not completely improve clinical practice or patient safety. OBJECTIVE The objective of this study was to assess the acceptability and usability of a mobile application for health professionals to work with patients’ electronic records at home, and to suggest improvements to the application. METHODS Study Design In this study, different methodologies and techniques were used to evaluate the mobile application described in the literature (22). This study was conducted in three phases: A) Researchers developed an adhoc questionnaire to explore the use of new technologies; B) Tests of usability of the mobile application were performed with the participants, while the interaction of the participant with the mobile application was analyzed through facial gesturing, with a categorical approach, through the six basic facial expressions published by the American Psychological Association (23). C) with the Computer System Usability Questionnaire (CSUQ) (24) user-perceived satisfaction of aspects related to ease of use, ease of learning, simplicity, effectiveness, information and user interface of the mobile application was assessed. The usability tests of this study were performed at the Center of Simulation and Innovation in Health. Recruitment Participants were selected through an open call. Candidates from different areas of healthcare and with different years of care experience were selected. All of them were identified with an ID to ensure confidentiality. Medical professionals, nurses and health professionals from different areas of care, such as hospitals, health centers, geriatric residences, home care, and others, were included. All health professionals who worked or had worked with a mobile healthcare data management application comparable to the one being assessed were excluded from the study. Data collection instrument The instruments used for data collection were as follows: 1. The socio-demographic questionnaire and the use of new technologies, which asked general questions, such as age, years of experience, training, field of work, and personal use of the internet. 2. "Usability questionnaire" inquiring about aspects related directly to effectiveness and efficiency of the application, such as the percentage of tasks successfully completed in the first attempt, the time used to perform a specific task or the time it takes participants to complete the tasks compared to an expert user. The tasks evaluated in the usability tests are shown in Table 1. The think-aloud reactions of the participants and the interaction of participants with the application assessed using the six basic facials expressions valued by the American Psychological Association (23), which are happiness, surprise, fear, disgust, anger, and sadness, were registered. Table 1: Tasks evaluated in the usability tests. Task number Description of task Task 1 Enter the application, identify yourself, search the list of patients and select a specific patient. Task 2 View the patient's personal data, verify the patient’s identity and search for their personal address. Task 3 Review patient's diet and medication allergies. Task 4 Consult and register a clinical notes. Task 5 Consult and record vital signs Task 6 Consult and verify the prescribed medical orders of the admitted patient. The criteria were tested according to the usability measures proposed in the ISO standard 9241–11 (25)(26). 3. Computer System Usability Questionnaire(CSUQ) (24). This is the Spanish adaptation of the Post-Study System Usability Questionnaire (PPSUQ) (27). The CSUQ consists of 16 items rated on a 7-point scale (strongly disagree [1] to strongly agree [7]) (27). The CSUQ consists of a general satisfaction scale and three subscales: system utility (items 1–6), information quality (items 7–12), and interface quality (items 13–15). Higher scores indicate better usability. The socio-demographic questionnaire and the use of new technologies, and the usability questionnaire were developed by the main researchers based on the recommendations described in the bibliography, and subsequently validated by a panel of experts. Data Collection Part A: Socio-demographic data and the use of new technologies. Before evaluating the mobile application, the 32 participants completed a questionnaire online via Google forms about socio-demographic data and the use of new technologies. Part B: Mobile application usability testing. The participants individually performed the usability tests of the mobile application in a room prepared with a filming system. During the tests, the participants completed the tasks that two researchers were indicating from an adjoining room. The evaluation followed a specific order to ensure that every user had an individual perspective of each of the tasks to be performed. During the procedure, the participants’ performance was recorded with cameras at different angles, and the researchers observed the reactions and movements from the adjoining room through a double mirror. At the same time, mobile phone screens were recorded using an external camera, which provided images or screen recordings (Image 1). Participants were asked to voice any feelings, doubts, or limitations they experienced during the exercise (think-aloud) to supplement the information received. Image 1. Participant performing usability tasks The researchers registered all aspects directly related to the effectiveness and efficiency of the participants, and, subsequently, analyzed the interaction of the participants with the mobile application through facial gesturing, with a categorical approach, using the six basic facial expressions published by the American Psychological Association (23). Part C: Questionnaire CSUQ. Finally, all participants completed the CSUQ (24). Ethical aspects The study was approved by the Ethical Committee of the University (CODE: 33/18). Participants signed the informed consent form. Only the principal investigator had access to the identity data, to ensure confidentiality. The results obtained will be maintained for five years. Data analysis Data analysis was based on audio and video recordings collected by cameras. Voice reactions of the participants in the audio recordings were transcribed verbatim. Incident notes, characterized by comments, silences, or repeated actions, and error messages, were collected through the recordings. The contents analysis was performed by two members of the research team. Transcripts and critical incidents were also reviewed to identify the most common usability concerns. In case of discrepancy in contents analysis, a third-party reviewer was consulted. The results of the CSUQ questionnaire were analyzed using the statistical program Jamovi. A descriptive, inferential, and univariate study was conducted. In the univariate analysis, the quantitative variables are expressed as centralization and dispersion parameters (mean, standard deviation, etc.), and as qualitative variables, via frequencies and percentages. RESULTS Participants characteristics A total of thirty-two participants completed the task scenarios and questionnaire. The main characteristics of the participants are summarized in Table 2. Table 2. Characteristics of participants Characteristics of the participants (N=32) N=32 (%) Age mean (years) 38,78 (SD10,35) Sex number Male Female 8 (25%) 24 (75%) Studies Nursing Medicine 29 (90,62%) 3 (9,37%) Post-graduate training Postgraduate studies / Specialization master Official Master Doctoral degree 13 (40,62%) 12 (37,5%) 7 (21,87%) Years of care experience 0 to 4 years 5 to 10 years 11 to 15 years 16 to 20 years More than 21 years 5 (15,62%) 5 (15,62%) 3 (9,37%) 7 (21,87%) 12 (37,5%) Area of care Specialized care (hospital care) Primary health care Social / healthcare 26 (81,25%) 2 (6,25%) 4 (12,5%) Working Area Management Primary care for adults, Primary care for children, Primary home care External consultations Convalescence / long stay / hospital palliative care Conventional hospitalization (internal medicine, surgery, traumatology ...) Hospitalization at home Intra-hospital emergencies, Extra-hospital emergencies 1 (3,13%) 2 (6,25%) 2 (6,25%) 4 (12,5%) 15 (46,87%) 1 (3,13%) 7 (21,87%) How many mobile health applications do you usually use None 1-2 Mobile applications 3-4 Mobile applications 5-6 Mobile applications 6-8 Mobile applications More than 9 Mobile applications 8 (25%) 16 (50%) 6 (18,75%) 1 (3,13%) 0 (0%) 1 (3,13%) Usability test: Completion of Task Scenarios Table 3 shows the completion effectiveness of realistic tasks for the end user, which is measured by the percentage of tasks completed. Table 4 and 5 show the efficiency (whether end users are able to locate the resources using the quickest and most direct route through the application), which is measured by the number of “additional” clicks to do the actions and the time that participants need to complete the tasks, compared to an expert user. Table 3. Effectiveness of participants that were able to complete the task (N=32). Task number Description of task Effectiveness of participants that were able to complete the task, n (%) Task 1 Enter the application, identify yourself, search the list of patients and select a specific patient. 93,75% (30/32) Task 2 View the patient's personal data, verify the patient’s identity, and search for mail address. 96,87% (31/32) Task 3 Review the assigned patient's diet and medication allergies. 96,87% (31/32) Task 4 Consult and register a clinical notes. 100% (32/32) Task 5 Consult and record vital signs 84,37% (27/32) Task 6 Consult and verify the prescribed medical orders of the admitted patient. 96,87% (31/32) All the participants were able to consult and register a clinical notes (Task 4) and more than 96% were able to view the patient's personal data, verify the patient’s identity and search for their personal address (task 2), review the assigned patient's diet and medication allergies (task 3), and consult and verify the prescribed medical orders of the patient (task 6). Participants had more difficulty consulting and recording vital signs (task 5), and only 84.37% (27/32) of them were able to do so. Regarding efficiency, the task that required the highest number of additional clicks was consulting and recording vital signs (task 5) (4,18), followed by entering the application, identifying themselves, searching the list of patients, and selecting a specific patient (Task 1) (1,123). Finally, if we analyze the time it takes an expert to perform each of these tasks compared to the time needed by the participants, we can say that the most difficult tasks are to view the patient's personal data, verify the patient’s identity and search for their postal address (task 2), enter the application, identify themselves, search the list of patients, and select a specific patient (task 1). Table 4: Efficiency of the application: average number of additional clicks participants needed to complete the task (N=32). Task number Description of task Efficiency (average number of additional clicks participants needed to complete the task) Task 1 Enter the application, identify yourself, search the list of patients and select a specific patient 1,123 Task 2 View the patient's personal data, verify the patient’ identity, and search for their personal address. 0,28 Task 3 Review the patient's diet and medication allergies. 0,5 Task 4 Consult and register a clinical notes 0,125 Task 5 Consult and record vital signs 4,18 Task 6 Consult and verify the prescribed medical orders of the admitted patient. 0 Table 5. Efficiency of the application: average participant time (seconds) / average expert time (N=32). Task number Description of task Average participant time (seconds) / average expert time Task 1 Enter the application, identify yourself, search the list of patients and select a specific patient 38,29 / 16,7 = 2,2 Task 2 View the patient's personal data, verify the patient’s identity, and search for their personal address. 17,59 / 6,92 = 2,5 Task 3 Review the patient's diet and medication allergies. 13,45 / 6,93 = 1,9 Task 4 Consult and register a clinical notes 145,57 / 80,88 = 1,7 Task 5 Consult and record vital signs 99,99 / 52 ,21 = 1,9 Task 6 Consult and verify the prescribed medical orders of the admitted patient. 17,95 / 17,17 = 1 Different comments were made during the thinking-aloud procedure. Of the 14 comments recorded, 78.57% (11/14) were related to the task of consulting and recording vital signs (task 5), with comments such as: I can't find them (Participant 4) I do not find the previous vital signs (Participant 5). I am not sure they are here (Participant 6). I do not know where to find the vital signs (Participant 7) I do not know how to do this ... how is this done? (Participant 14) Cannot access the application. I was hoping to find a graph of the vital signs. This is where I need to record the vital signs? I am not sure ... (Participant 19) I was hoping to find a graph of the vital signs ... I do not know if these vital signs I see are the last ones ... where do I have to record them (VS)? (Participant20 ) I cannot find them (previous vital signs ...) (Participant 21) Oh no! I left unintentionally! I do not know where to find the previous vital signs .... they are not here!(Participant 25). I do not see the above vital signs (Participant 28). I am not finding the previous vital signs.... (Participant 33). Figure 1 shows the interaction of the participants with the application. Notice that, in the form of surprise, most reaction were seen on Task 5 (consulting and recording vital signs, 50%), followed by task 4 (consult and register a clinical notes, 31%), and task 6 (consult and verify the prescribed medical orders of the patient, 25%). Figure 1. Interaction of the participant with the mobile application Results from the CSUQ showed that the participants were, overall, satisfied with the usability of the application (see Table 6 for details), as can be seen in the general questions section of the questionnaire. Overall, I was satisfied with the mobile application had a score of 6,18/7 (SD: 0,76), and I would recommend the use of the mobile application to other professionals had a score of 6,21/7 (SD: 0,81). Regarding the system quality, information quality, and interface quality, the best rated category was the quality of the interface with an average score of 6.04 / 7 (SD: 0.22), and the lowest rated was the quality of the information, with an average score of 5.35 / 7 (SD: 0.90). Table 6. Computer System Usability Questionnaire (N=32) CSUQa constructs Scoreb, mean (SD) System quality 5,93 (0,25) Information quality 5,35 (0.90) Interface quality 6,04 (0,22) aCSUQ: Computer System Usability Questionnaire b Score range from 1 =strongly agree to 7=strongly disagree CONCLUSIONS Principal Findings The use of UEMs during the development and testing process of health applications is widely recommended in the literature (29). For this reason, this article has been intended to critically appraise the acceptability and usability of the mobile application for health professionals with different UEMs is available to detect problems in the user–system interaction, and to suggest improvements to the application. Usability tests have shown that the mobile application is intuitive, efficient, and effective., since most of the participants did not have difficulties performing most of the tasks with the application, with few errors, and the time it took them was comparable to that of an expert participant. This is considered an accomplishment since none of the participants had previously used it. The most difficult task for the participants was to consult and record vital signs. In addition, most comments during the thinking aloud process, and the tasks where the participants interacted more with the mobile application through facial gesturing, were also related to consulting and recording vital signs. Overall, end users of this study found the mobile application to be highly usable, as the survey data indicate (CSUQ), with no major bugs and no issues with flow of activities. In addition, most participants said they were satisfied with the mobile application and that they would recommend the use of the mobile application to other professionals. These results suggest that the quality of the information provided with the application should be improved, and that the main task to be improved in terms of accessibility and ease of use that needs improvement is to consult and register a clinical notes of treatment. If we analyze the results obtained in a broader sense, we see that the acceptance and satisfaction of the study participants, who do not use the mobile application on a daily basis, is high, and similar to those obtained by professionals who do use it as a working tool (21). This suggests that, in addition to improving specific aspects of the application, a broader analysis should be made of the reasons why the use of the application is currently not widespread among all professionals, and why paper is still used for queries, and to record clinical data in a complementary way. Some authors conclude that one of the reasons that might explain the low usage rates, resistance, and rejection of the health information technology (IT), and the requests for alternative methods, is that, when adopting mobile applications and technologies, functional features and advanced techniques have been prioritized, and the needs and characteristics of the end users have been neglected (30) (31). Other studies show that the most influential factor in the use of mobile applications is performance expectancy (PE) (9), understood as the degree to which the user expects that the system will help him or her to attain gains in job performance. Other authors conclude that the determining factors are the perceived importance of information security, process orientation, documentation intensity, and e-health related knowledge (32). Therefore, healthcare organizations, apart from designing and developing mobile applications that guarantee the Evidence-based Health Informatics (EBHI) and the utilization of UEMs, should also consider PE as a determining factor in the adoption of new mobile devices, and they should deeply analyze the end-users’ needs to identify useful functions for their workflows (33). Limitations Limitations to the present study include the sample size, although other studies have used a similar or lower samples (34). Moreover, the study design did not allow for “learnability” to be measured, because of the small sample size and the high efficiency and effectiveness of task scenario completion. Conclusion There is clear scientific evidence about the ability of mobile handheld technology to positively impact rapid response, error prevention, information accessibility, and data management in healthcare settings, as well as the beneficial impact of this technology on aspects of healthcare delivery (35). This study has shown that the usability of this mobile application in terms of effectiveness, efficiency, and satisfaction is relevant, but is not the only condition that favors its use in daily practice. That is why, as other authors have recommended, further studies are needed to explore the significant antecedents of this mobile application, i.e., system and information quality and the limitations of mobile devices (9).