Abstract:Background Preventable patient harm, particularly medication errors, represent significant challenges in healthcare settings. Dispensing the wrong medication is often associated with mix-up of lookalike and soundalike drugs in high workload environments. Replacing manual dispensing with automated unit dose and medication dispensing systems to reduce medication errors is not always feasible in clinical facilities experiencing high patient turn-around or frequent dose changes. Artificial intelligence (AI) based pill recognition tools and smartphone applications could potentially aid healthcare workers in identifying pills in situations where more advanced dispensing systems are not implemented. Objective Most of the published research on pill recognition focuses on theoretical aspects of model development using traditional coding and deep learning methods. The use of code-free deep learning (CFDL) as a practical alternative for accessible model development, and implementation of such models in tools intended to aid decision making in clinical settings, remains largely unexplored. In this study, we sought to address this gap in existing literature by investigating whether CFDL is a viable approach for developing pill recognition models using a custom dataset, followed by a thorough evaluation of the model across various deployment scenarios, and in multicenter clinical settings. Furthermore, we aimed to highlight challenges and propose solutions to achieve optimal performance and real-world applicability of pill recognition models, including when deployed on smartphone applications. Methods A pill recognition model was developed utilizing Microsoft Azure Custom Vision platform and a large custom training dataset of 26,880 images captured from the top 30 most dispensed solid oral dosage forms (SODFs) at the three participating hospitals. A comprehensive internal and external testing strategy was devised, model's performance was investigated through the online API, and offline using exported TensorFlow Lite model running on a Windows PC and on Android, using a tailor-made testing smartphone application. Additionally, model's calibration, degree of reliance on color features and device dependency was thoroughly evaluated. Real-world performance was assessed using images captured by hospital pharmacists at three participating clinical centers. Results The pill recognition model showed high performance in Microsoft Azure Custom Vision platform with 98.7 % precision, 95.1 % recall, and 98.2 % mean average precision (mAP), with thresholds set to 50 %. During internal testing utilizing the online API, the model reached 93.7 % precision, 88.96 % recall, 90.81 % F1-score and 87.35 % mAP. Testing the offline TensorFlow Lite model on Windows PC showed a slight performance reduction, with 91.16 % precision, 83.82 % recall, 86.18 % F1-score and 82.55 % mAP. Performance of the model running offline on the Android application was further reduced to 86.50 % precision, 75.00 % recall, 77.83 % F1-score and 69.24 % mAP. During external clinical testing through the online API an overall precision of 83.10 %, recall of 71.39 %, and F1-score of 75.76 % was achieved. Conclusion Our study demonstrates that using a CFDL approach is a feasible and cost-effective method for developing AI-based pill recognition systems. Despite the limitations encountered, our model performed well, particularly when accessed through the online API. The use of CFDL facilitates interdisciplinary collaboration, resulting in human-centered AI models with enhanced real-world applicability. We suggest that rather than striving to build a universally applicable pill recognition system, models should be tailored to the medications in a regional formulary or needs of a specific clinic, which can in turn lead to improved performance in real-world deployment in these locations. Parallel to focusing on model development, it is crucial to employ a human centered approach by training the end users on how to properly interact with the AI based system to maximize benefits. Future research is needed on refining pill recognition models for broader adaptability. This includes investigating image pre-processing and optimization techniques to enhance offline performance and operation on handheld devices. Moreover, future studies should explore methods to overcome limitations of CFDL development to enhance the robustness of models and reduce overfitting. Collaborative efforts between researchers in this domain and sharing of best practices are vital to improve pill recognition systems, ultimately enhancing patient safety and health -Abstract Truncated-

Real-Time Pill Identification for the Visually Impaired Using Deep Learning

An Efficient and Real-Time Emergency Exit Detection Technology for the Visually Impaired People Based on YOLOv5

Scene Text Detection and Recognition System for Visually Impaired People in Real World

Real-time Pedestrian Crossing Lights Detection Algorithm for the Visually Impaired

REAL TIME OBJECT DETECTION FOR VISUALLY CHALLENGED PEOPLE USING MACHINE LEARNING

Android Based Object Detection System for Visually Impaired

Leveraging code-free deep learning for pill recognition in clinical settings: A multicenter, real-world study of performance across multiple platforms

An Accurate Deep Learning-Based System for Automatic Pill Identification: Model Development and Validation

Assisting Blind People Using Object Detection with Vocal Feedback

Realtime Object Detection and Disease Prediction of Visually Impaired People

CNN-Based Medical Device Design for the Smart Pillbox With Face Recognition

Early detection of visual impairment in young children using a smartphone-based deep learning system

Comparison of RetinaNet, SSD, and YOLO v3 for real-time pill identification

Vision AI: A Deep Learning-Based Object Recognition System for Visually Impaired People Using TensorFlow and OpenCV

AI-Driven Smartphone Solution for Digitizing Rapid Diagnostic Test Kits and Enhancing Accessibility for the Visually Impaired

Object Detection with Voice Guidance to Assist Visually Impaired Using Yolov7

Empowering the Blind: An AI Driven Indoor Assistance for Visually Impaired

Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

Medicine bottle recognition based on machine vision and deep learning in intravenous medicine dispensing robot*

A Novel Hierarchical Deep Learning Framework for Diagnosing Multiple Visual Impairment Diseases in the Clinical Environment

High Accurate and Explainable Multi-Pill Detection Framework with Graph Neural Network-Assisted Multimodal Data Fusion