A Method of Dividing Clinical Data Set for Medical Image AI Training

Dezhong Zheng,Yuanyuan Yang,Wentao Li
DOI: https://doi.org/10.1117/12.2549552
2020-01-01
Abstract:In the AI training, the data set is always divided into training set and test set at random, but the clinical image data from hospitals is different from the public data set. The division of public data set is reasonably divided and evenly distributed after many experiments. Accurate understanding of the data distribution directly affects the training model quality. So we proposed a new method of dividing clinical data set based on distance metric learning of the Gaussian mixture model to obtain more reasonable data set divisions. The distance metric learning based on deep neural network, first embeds data into a new metric space, then in the metric space uses in-depth mining based on data characteristics, calculates the distance between samples, finally compares the differences. The method can accurately know the data distribution characteristics to a certain extent. Under the condition of understanding the data distribution characteristics, more reasonable divisions can be obtained. That can greatly affect the accuracy and generalization performance of the models we trained.
What problem does this paper attempt to address?