Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognition
Aggarwal, Sajal,Bhola, Geetanjali,Vishwakarma, Dinesh Kumar
DOI: https://doi.org/10.1007/s11042-024-19582-1
IF: 2.577
2024-06-09
Multimedia Tools and Applications
Abstract:This research work aims to propose an ensemble model of different pre-trained CNN networks combined with LSTM to detect a set of routine human activities practiced by the elderly from the publicly available STAIR Actions Dataset. The prime objective of this work is to visualize the effect of using different networks for implementing activity detection for a set of five activities, compare them based on standard performance parameters and combine the best- performing models into a better-performing ensemble unit. Applying the transfer learning technique helped achieve shorter training times and promising classification results for the dataset. The hybrid CNN-LSTM model uses CNN to extract spatial features from the video frames, followed by classification using sequential learning by the LSTM network. For the set of five routine activities practised by the elderly taken up from the STAIR Actions dataset, we obtained an overall test set accuracy of 82.27%, 82.73%, 73.64%, 72.73%, 64.55%, 71.82%, 83.64%, 88.18%, 86.82% and 86.82% respectively on using VGG16, VGG19, ResNet50, ResNet101, ResNet152, Inception V3, Xception, DenseNet121, DenseNet169 and DenseNet201 models, pre-trained on the ImageNet dataset for feature extraction from the video frames. Apart from test-set accuracy, we evaluated the models using other performance metrics, including AUROC, AUPRC, sensitivity, specificity and precision. Using the best-performing model from each category of CNN networks employed, we developed a weighted voting ensemble that outperformed the best-performing individual model in terms of test set accuracy. The highest test set accuracy achieved by optimising the weights assigned to each model in the proposed ensemble was 93.18%.
computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering