Developing and evaluating the usability of a web-based Diabetic Retinopathy Health Education Program for patients with type 2 diabetes mellitus =============================================================================================================================================== * Rif’atunnailah Mat Isa * Mei Chan Chong * Wan Ling Lee * Tajunisah Iqbal * Mardiana Mansor * Airul Azizan Zainudin * Mohd Iman Saiful Suhardi ## Abstract **Objectives:** To describe the development of a webpage based on the Intervention Mapping (IM) protocol and usability testing of the Diabetic Retinopathy Health Education Profram (DRHEP). **Methods:** The mixed methods pilot feasibility study was carried out between April and September 2021, involving 16 patients with type 2 diabetes mellitus and 5 experts. The usability score was rated according to the System Usability Scale (SUS). **Results:** The average SUS score by the experts was 88. The patients gave a higher score of 85 for SUS, with 58 as the lowest. The average SUS score was 72. The findings indicate that the webpage is acceptable, good, and highly usable for users. **Conclusion:** The outcomes of this study signify the relationship between effective health applications and how their design might hamper their effectiveness in changing patients’ behavior. Keywords: * type 2 diabetes mellitus * diabetic retinopathy * usability testing * web-based intervention **D**iabetic retinopathy (DR) often causes disability and becomesa frequent complication of diabetes mellitus (DM), particularly in people of working age.1 Besidesmedicines that result in disease regression, technical advancements in screening and image capturing the fundus features of DR have also cast doubt on the therapeutic significance of the classification of the disease.2 Patients with type 2 diabetes mellitus (T2DM) must make multiple decisions on their progress while attending numerous appointments that require much time and energy. Telemedicine and mobile health can support diabetes self-care by making information and healthcare services easier to understand and more accessible. However, despite the medications that produce disease regression and technical breakthroughs in screening and imaging collection of DR fundus characteristics, therapeutic usesof the Early Treatment of Diabetic Retinopathy Study (ETDRS) classification of DR have raised some doubts.2 Diabetes education videos, patient forums and support groups, and live chats with health coaches make disseminating information and healthcare services more convenient and accessible. Telemedicine and mobile health can also encourage patients to engage in diabetic self-care through a webpage interface. Smartphone notifications can assist users in meeting their health goals, while enhanced connectivity to diabetes care teams allows remote blood glucose monitoring linked with real-time patient progress.3 Thus, technology assistance through a mobile application (app) is a communication tool that monitors disease progression, which benefits patients. Nevertheless, more interactive and user-friendly educational products that provide interactive measures and use layman’s terms should be produced. Assessment of patients’ needs based on a patient-centred approach, especially among non-proliferative diabetic retinopathy (NPDR) patients, could prevent blindness at the early stage of the disease. Therefore, to address this gap, this study aims to develop a Diabetic Retinopathy Health Education Programme (DRHEP) to improve patients’ knowledge, self-care, and visual function quality of life among T2DM patients. Knowledge transfer through technology aid acts as a communication medium. In this study, the Diabetic Retinopathy webpage was established to provide health education among T2DM patients based on diabetes complication risks. The webpage was designed based on clinical research recommendations for T2DM and Diabetic Retinopathy Screening.4,5 Both guidelines provide as main reference in the provision of the webpage. This study describes the development of a webpage based on the Intervention Mapping (IM) protocol and usability testing of the DRHEP webpage. ## Methods A mixed-method pilot feasibility study was employed using a semi-structured interview followed bythe System Usability Scale (SUS) questionnaire containing tenquestions from April to September 2021.6 The study involved 15 experts and 16 patients with T2DM from the ophthalmology clinic, University Malaya Medical Centre, Selangor, Malaysia. The inclusion criteria for the experts were as follows: various backgrounds, including ophthalmology, endocrinology, and nursing. Meanwhile, the inclusion criteria for patients were: diagnosed with T2DM, aged 18 years and above, diagnosed with NPDR until moderate NPDR (NPDR), and have cognitive capability. Blindness, bedridden, dementia, and cognitively impaired patients were excluded from this study. The study was carried out in 4 steps: experts’ interview, webpage development, and validation process of experts and patients. ### Steps 1: Experts’ interview Five experts (group A) were involved in the interview process before the DRHEP webpage testing. The aspects of evaluation include the topic, structure, content, video presentation, and overall effect sizeof the webpage. The panel of experts advised on the alignment with the current Clinical Practice Guideline of Management Type 2 Diabetes Mellitus 6th edition 202064 (Expert 1) and the addition of the signs and symptoms of DR to enhance the understanding of the changes in the patient’s eyes during the progress of the disease (Expert 2). Expert 3 suggested that selecting a proper color scheme and clear font for the subtitle could ease the audience’s experience while watching the educational video. The treatment plan should be updated in line with the diabetic retinopathy guideline. Experts 4 and 5 recommended using layman’s terms in the subtitle to overcome the language barrier for those who did not understand Malay. The overall comments from the experts were acknowledged for amendment and revisionwith information technology (IT) personnel before being implemented. ### Step 2: Webpage development The development of the webpage was based on IM, which involved developing and implementing a basic health education programme.7 The DRHEP webpage is available in English and Malay. It was developed as a web-based application with an adjustable desktop and mobile phone design. The users were given the link to the webpage by the researcher. They were directed to the home page of the webpage, from where they can navigate to other pages. Users can view the choices on the home page, which include health education in PDF form, a PowerPoint presentation, or a video of an ophthalmologist. Patients were given access to the link for a week. ### Usability testing The usability testing in this study applied the SUS questionnaire for experts and patients. The items of the questionnaire and the process are detailed below. ### Usability questionnaire The SUS was invented by John Brooke in 1986 with a 10-item version scored on a 5-point Likert scale, from 1 (strongly disagree) to 5 (strongly agree). The Malay version of SUS (named SKAMA) was translated and validated by 10 experts.The efficacy of the questionnaire was further assessed by testing face validity on 10 mobile phone users, followed by reliability testing involving 54 mobile phone users.8 A system or product that received a score of 68 and above is considered to have good usability. The items are listed below: Item 1: I would use this app often; Item 2: I found the app unnecessarily complex; Item 3: I found the app easy to use; Item 4: I think help from technical support would be needed to be able to use this app; Item 5: I thought the different functions of the app were well integrated; Item 6: I thought there was great inconsistency in this app; Item 7: I would imagine that the majority of people would quickly learn how to use this app; Item 8: I found the app very hard to use; Item 9: I felt confident using this app; Item 10: I needed to learn a series of things before I could use the app. The following is the formula to calculate the total score of SUS: X=Sum of the points for all odd-numbered questions–5. Meanwhile, Y=25 – the sum of the points for all even-numbered questions. Thus, the SUS score = (X+Y) × 2.5. The total scores of SUS between 0 to 100 are as follows:> 80 (excellent), 68-80.3 (good), 60 (okay), 51-68 (poor), and <51 (awful).9 ### Step 3: Experts’ usability testing The usability test for the Diabetic Retinopathy webpage was carried out among 10 experts (group B) to evaluate the implementation for the usersin the study. The panel of experts encompassed 2 with IT speciality in webpage, 5 medical and health science lecturers, and 3 postgraduate students with experience developing webpage from different universities. ### Step 4: Patients’ usability testing A total of 16 T2DM patients participated in the study. The inclusion criteria include patients with T2DM, aged 18 years and above, diagnosed with moderate NPDR, and have cognitive capability. The exclusion criteria for this study were blindness, bedridden, dementia, and cognitive impairment. The Institutional Review Board (IRB number: 202068-8726) approval was issued and obtained from the University Malaya Research Ethic Committee, Malaysia. ### Statistical analysis No formal hypothesis was carried out in this analysis, and descriptive statistics were consistent with the applied qualitative study methods of usability testing. Data were presented using counts, proportions (%), and means. All statistical analyses were conducted using Statistical Package for the Social Sciences for Windows, version 26 (IBM Corp., Armonk, NY, USA). ## Results A total of 16 patients and 10 experts participated in the study. Table 1 reveals the SUS value scores of the DRHEP webpage from the 10 experts (group B). The highest score for SUS given by the experts is 88, and the lowest is 80. The average SUS score is 83. The results indicate that the webpage is acceptable, good, and highly usable. View this table: [Table 1](http://smj.org.sa/content/44/12/1290/T1) Table 1 - The System Usability Scale score by 10 experts on Diabetic Retinopathy Health Education Programme webpage. Table 2 reveals the SUS value scores of the DRHEP webpage from the 16 patients. The highest SUS score given by the patients is 85, and the lowest is 58. The average SUS score is 72. It indicates that the webpage isacceptable, good, and highly usable. View this table: [Table 2](http://smj.org.sa/content/44/12/1290/T2) Table 2 - The System Usability Scale score by 16 participants on Diabetic Retinopathy Health Education Programme webpage. ## Discussion Several interventions are used to disseminate health information on DR and other components like self-care, self-management, peer support, and group-based interventions using traditional or online approaches. Patients with no DR to moderate NPDR were chosen because they would be the influential group for the prevention utilizing DR education in the study. Hence, the practicality and effectiveness of T2DM patient implementation approaches depend on them. The precise elements of instructional content that permit analysis are crucial from a research standpoint. Diabetes management involves lifestyle modification, medications, and patient education to encourage self-care and empowerment.10-12 Rapid changes, especially in the modern era alongside the COVID-19 pandemic, create challenges for keeping education up-to-date and technological usage to spread information.4 In line with the current worldwide pandemic, online health education has become more relevant due to limited physical interaction between patients and healthcare providers. Web-based diabetes research via the Facebook platform to deliver information, cure-focused innovations, and negative sentiment refutation is relevant among users.3 Meanwhile, smartphone application programmes for T2DM patients to monitor physical activity, hemoglobin A1c, blood pressure, waist circumference, waist-to-hip ratio, and interventions that suit the patient’s preference and feasibility issues need to be considered.13 Of 3 studies using mobile apps, only 2 were conducted in person. They acquired verbal education and visual feedback of their retinal images for 3 months, with 2 hours of face-to-face instruction every 2 months, focusing on prevention and self-management strategies, nutrition, physical activity, health-seeking behavior, and counselling.14 Meanwhile, Sight BookTM is a free smartphone app that evaluates patients with DR vision at home using the Amsler grid and visual acuity.15 The Diabetes Carer App, which comprises 3 components, such as the diabetic component, self-management, and patient community, is combined with or without self-monitoring of blood glucose (SMBG) and has an impact on glycaemic control in diabetes patients.16 Baseline retinal imaging as a real-time patient education/engagement tool and telehealth screening are 2 of the 4 technology treatments that make up telehealth. The comments/feedback by the patients regarding goals, obstacles, and change-motivating factors are recorded. An electronic decision-support tool for cardiovascular disease and diabetes based on current recommendations serves as a communication channel between healthcare professionals, T2DM patients, and their caregivers.17 Similar to the findings in this study, other researchers and designers have stressed the need to use an iterative strategy while creating webpages to comprehend end-users’ wants and enhance the feasibility and usability of the apps. Therefore, it is vital to focus on the significance of conducting usability tests on webpages that will be utilized in clinical and patient environments. When creating mobile health apps, user testing is crucial, especially when altering actual patient behavior or influencing patient outcomes. ### Study limitations This study was carried out in a teaching hospital due to logistic issues; thus, the possibility for generalization may be restricted. Reaching out to a wider population/venue might overcome this issue, as it would provide differences in cultures and resources of each health system, particularly in the private sector practice. The interaction with patients and experts was carried out online during the COVID-19 pandemic. Therefore, focus group discussion with the experts is recommended in developing an app. In conclusion, the information on the DRHEP webpage is valuable to the end-users, such as patients with T2DM. The results of this study highlight the association between effective health application sand how their design might hamper their effectiveness in changing patients’ behavior. However, more personalized and interactive elements, such as providing the fundus images of the patient, should be imparted in future app development. A future version of an interactive Diabetic Retinopathy webpage will be a valuable tool for all diabetes patients about blindness-related risks. ## Acknowledgment *The authors gratefully acknowledge ProofreadersUnited for the English language editing.* ## Footnotes * **Disclosure.** Authors have no conflict of interests, and the work was not supported or funded by any drug company. * Received September 13, 2023. * Accepted November 5, 2023. * Copyright: © Saudi Medical Journal This is an Open Access journal and articles published are distributed under the terms of the Creative Commons Attribution-NonCommercial License (CC BY-NC). Readers may copy, distribute, and display the work for non-commercial purposes with the proper citation of the original work. ## References 1. 1.Xiong J, Acemyan CZ, Kortum P. User Experience Magazine. SUSapp: A Free Mobile Application That Makes the System Usability Scale (SUS) Easier to Administer. J Usability Stud 2020; 15: 135-144. 2. 2. Luis Filipe Nakayama, Lucas Zago Ribeiro, Mariana Batista Gonçalves, et al. Diabetic retinopathy classification for supervised machine learning algorithms. Int J Retina Vitreous 2022; 8: 1. 3. 3.Vehof H, Heerdink ER, Sanders J, Das E. Associations Between characteristics of web-based diabetes news and readers’ sentiments: Observational study in the Netherlands. J Med Internet Res 2019; 21: e14554. 4. 4.Ministry of Health. Clinical Practice Guideline. Screening of Diabetic Retinopathy. [Updated 2011; cited 2022 January 15]. [https://www.moh.gov.my/moh/attachments/6601.pdf](https://www.moh.gov.my/moh/attachments/6601.pdf) 5. 5.Ministry of Health. Clinical Practice Guidelines. Management of Type 2 Diabetes Mellitus (6th Edition) Quick Reference Guide for Healthcare Professionals. [Cted 2023 February 2]. Available from: [https://www.moh.gov.my/moh/resources/Penerbitan/CPG/Endocrine/QR\_T2DM\_6th\_Edition\_QR\_Guide\_Digital.pdf](https://www.moh.gov.my/moh/resources/Penerbitan/CPG/Endocrine/QR\_T2DM\_6th\_Edition_QR_Guide_Digital.pdf) 6. 6.Aschbrenner KA. Kruse G, Gallo JJ, Plano Clark VL. Applying mixed methods to pilot feasibility studies to inform intervention trials. Pilot Feasibility Stud 2022; 8: 217. 7. 7.Bartholomew Eldredge LK, Markham CM, Ruiter RAC, Fernández ME, Kok G, Parcel GS. Planning health promotion programs: an Intervention Mapping approach. 4 ed. San Francisco: Jossey-Bass Inc., 2016. 8. 8.José C, Autran R, Melissa Soares Medeiros, Kubrusly M, Marçal E, Arnaldo. Development and Validation of a Mobile Application for the Teaching of Electrocardiogram. Rev Bras Educ Med 2019; 43: 157-165. 9. 9.Lewis JR. The System Usability Scale: past, present, and future. Int J Hum-Comput Int 2013; 7: 577-590. 10. 10.Beck J, Greenwood DA, Blanton L, et al. 2017 National Standards for Diabetes Self-Management Education and Support. Diabetes Educ 2018; 44: 35-50. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1177/0145721718754797&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fsmj%2F44%2F12%2F1290.atom) 11. 11.Fitzgerald JT, Funnell MM, Anderson RM, Nwankwo R, Stansfield RB, Piatt GA. Validation of the Revised Brief Diabetes Knowledge Test (DKT2). The Diabetes Educator 2016; 42: 178-187. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1177/0145721715624968&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=26769757&link_type=MED&atom=%2Fsmj%2F44%2F12%2F1290.atom) 12. 12.Martin CL, Daly A, Laura Shane McWhorter, Shwide-Slavin C, Kushion W. The Scope of practice, standards of practice, and standards of professional performance for diabetes educators. Diabetes Educ 2005; 31: 487-512. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1177/0145721705279719&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=16100327&link_type=MED&atom=%2Fsmj%2F44%2F12%2F1290.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000230826500003&link_type=ISI) 13. 13.Staite E, Bayley A, Ebaa Al-Ozairi, et al. A Wearable technology delivering a web-based diabetes prevention program to people at high risk of type 2 diabetes: randomized controlled trial. JMIR Mhealth Uhealth 2020; 8: e15448. 14. 14.Fenwick E, Xie J, Rees G, Finger RP, Lamoureux EL. Factors associated with knowledge of diabetes in patients with type 2 diabetes using the Diabetes Knowledge Test validated with Rasch analysis. PLoS One 2013; 8: e80593-e80593. 15. 15.Bert F, Giacometti M, Gualano MR, Siliquini R. Smartphones and health promotion: a review of the evidence. J Med Syst 2014; 38: 9995. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1007/s10916-013-9995-7&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=24346929&link_type=MED&atom=%2Fsmj%2F44%2F12%2F1290.atom) 16. 16.Brooke J. SUS: A Retrospective. J Usability Stud 2013; 8: 29-40. 17. 17.Yu Y, Yan Q, Li H, Li H, Wang L, Wang H, et al. Effects of mobile phone application combined with or without self‐monitoring of blood glucose on glycemic control in patients with diabetes: A randomized controlled trial. J Diabetes Investig 2019; 10: 1365-1371. 18. 18.Brazionis L, Jenkins AJ, Keech A, Ryan C, Bursell SE. An evaluation of the telehealth facilitation of diabetes and cardiovascular care in remote Australian Indigenous communities: - protocol for the telehealth eye and associated medical services network [TEAMSnet] project, a pre-post study design. BMC Health Serv Res 2017; 17: 13.