Active Prompt Tuning Enables Gpt-40 To Do Efficient Classification Of Microscopy Images

Abhiram Kandiyana,Peter R. Mouton,Yaroslav Kolinko,Lawrence O. Hall,Dmitry Goldgof

2024-11-05

Abstract:Traditional deep learning-based methods for classifying cellular features in microscopy images require time- and labor-intensive processes for training models. Among the current limitations are major time commitments from domain experts for accurate ground truth preparation; and the need for a large amount of input image data. We previously proposed a solution that overcomes these challenges using OpenAI's GPT-4(V) model on a pilot dataset (Iba-1 immuno-stained tissue sections from 11 mouse brains). Results on the pilot dataset were equivalent in accuracy and with a substantial improvement in throughput efficiency compared to the baseline using a traditional Convolutional Neural Net (CNN)-based approach. The present study builds upon this framework using a second unique and substantially larger dataset of microscopy images. Our current approach uses a newer and faster model, GPT-4o, along with improved prompts. It was evaluated on a microscopy image dataset captured at low (10x) magnification from cresyl-violet-stained sections through the cerebellum of a total of 18 mouse brains (9 Lurcher mice, 9 wild-type controls). We used our approach to classify these images either as a control group or Lurcher mutant. Using 6 mice in the prompt set the results were correct classification for 11 out of the 12 mice (92%) with 96% higher efficiency, reduced image requirements, and lower demands on time and effort of domain experts compared to the baseline method (snapshot ensemble of CNN models). These results confirm that our approach is effective across multiple datasets from different brain regions and magnifications, with minimal overhead.

Image and Video Processing,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reduce the dependence on a large amount of labeled data and the time of domain experts in microscope image classification, and improve the classification efficiency and accuracy. Traditional deep - learning - based methods require a great deal of time to prepare accurate training data and need a large amount of input image data. These methods are not only time - consuming and labor - intensive, but also need to repeat the pre - training, optimization, training and testing processes of the model every time a new data set is processed, which makes the whole process very time - consuming and computationally expensive.

The paper proposes a method using improved prompts (Active Prompt Tuning, APT) and the latest multimodal model GPT - 4o. Through few - shot prompting, the classification task of microscope images can be efficiently completed. This method not only reduces the need for labeled data and the time investment of domain experts, but also significantly improves the efficiency and accuracy of classification. Specifically, the paper uses a low - magnification (10x) microscope image data set from the brains of 18 mice (9 Lurcher mutants and 9 wild - type control groups). Through the APT method, only the data of 6 mice are used as the prompt set, achieving a 92% classification accuracy rate, which is 96% more efficient than the baseline method.