Identifying data labeling errors using classification models for small datasets

Abstract

 Labeling up data for classification tasks is a complex process, accompanied by unavoidable errors. Manual or automatic labeling of texts for classification includes systematical errors that can be identified using system approaches based on statistics and machine learning models. It is small data sets that are considered, since the consequences of labeling errors are most noticeable in them. However, in case of small datasets, due to the lack of samples, the problem of sparse distributions arises, which prevents the training of models with high complexity. The author uses the effect of overfitting the model to minimize the limitations imposed by insufficient data. Several experiments were conducted as part of the study. An experiment on a large public dataset showed that when classifying short texts, the overfitted model is able to detect data labeling errors. In an experiment with the formation of facets based on user description of goods, the interdependence of the presence of class definition errors and the work of assessors on the labeling of text data based on different rules was determined. Due to its overfitting, the classification model is capable of identifying significant errors that dramatically affect the engineering application of machine learning in highly loaded Internet systems. As a result of the research, the author provides methods and criteria for achieving the state of "productive overfitting" by the model. The best result on the f1-score weighted metric (98%) was shown by the EmbeddingBag-based classification model.