From Qualitative Roles to Measurable Evidence: Classical Text Representations for Action-Type Recognition in Manufacturing Execution Systems

Surya Teja Meesala

Abstract


Manufacturing execution systems generate large volumes of informal knowledge‚ including shift log books‚ work instructions‚ and maintenance logs․ However‚ concrete‚ quantitative evidence of the use of generative artificial intelligence (GenAI) in the shop-floor is rare․ We present an empirical study on a publicly available expert-annotated maintenance short text dataset (MaintIE)․ This dataset comprises 1‚076 records‚ where the task is to predict the maintenance action type (Replace‚ Repair‚ Inspect‚ Service‚ Diagnose‚ Adjust‚ Calibrate) from a free-text work-order description․ We evaluate classical text vector representations combined with linear classifiers (stratified five-fold cross-validation)․ Testing shows that the character n-gram representation (with logistic regression) achieves a macro-F1 score of 0․889 and accuracy of 0․956 while the baseline‚ majority class predictor‚ achieves a macro-F1 score of 0․107․ A noise-injection experiment that simulates typographic errors and common abbreviations shows that character n-grams degrade by 8% at 50% token corruption while words degrade by 38%․ A similarity-based retrieval experiment estimates the coverage-precision trade-off that bounds the case range where lexical retrieval is sufficient․ A comparison of retrieval for the two classes suggests that mid-weight semantic retrieval is on average more precise on the remaining class by about 10 precision points․ This transposes the classical vs․ semantic boundary from a presupposition into a quantifiable measurement․ This work converts qualitative statements about MES knowledge work into reproducible measurements and presents an evidence-based‚ graduated deployment roadmap․


Full Text:

PDF

References


A. Shojaeinasab et al., “Intelligent manufacturing execution systems: A systematic review,” Journal of Manufacturing Systems, vol. 62, pp. 503–522, 2022, doi: 10.1016/j.jmsy.2022.01.004.

A. Kusiak, “Generative artificial intelligence in smart manufacturing,” Journal of Intelligent Manufacturing, vol. 36, no. 1, pp. 1–3, 2025, doi: 10.1007/s10845-024-02480-6.

L. Li, “Reskilling and upskilling the future-ready workforce for Industry 4.0 and beyond,” Information Systems Frontiers, vol. 26, pp. 1697–1712, 2024, doi: 10.1007/s10796-022-10308-y.

F. Akhbardeh, T. Desell, and M. Zampieri, “MaintNet: A collaborative open-source library for predictive maintenance language resources,” in Proc. 28th Int. Conf. Computational Linguistics: System Demonstrations, 2020, pp. 7–11, doi: 10.18653/v1/2020.coling-demos.2.

M. Ghobakhloo et al., “Generative artificial intelligence in manufacturing: opportunities for actualizing Industry 5.0 sustainability goals,” Journal of Manufacturing Technology Management, vol. 35, no. 9, pp. 94–121, 2024, doi: 10.1108/JMTM-12-2023-0530.

S. S. Sengar, A. B. Hasan, S. Kumar, and F. Carroll, “Generative artificial intelligence: a systematic review and applications,” Multimedia Tools and Applications, vol. 84, pp. 23661–23700, 2025, doi: 10.1007/s11042-024-20016-1.

M. Klesel and H. F. Wittmann, “Retrieval-augmented generation (RAG),” Business & Information Systems Engineering, vol. 67, pp. 551–561, 2025, doi: 10.1007/s12599-025-00945-3.

T. K. Bikaun, T. French, M. Stewart, W. Liu, and M. Hodkiewicz, “MaintIE: A fine-grained annotation schema and benchmark for information extraction from maintenance short texts,” in Proc. LREC-COLING, 2024, pp. 10918–10932.

M.-A. Lutz et al., “KPI extraction from maintenance work orders: a comparison of expert labeling, text classification and AI-assisted tagging for computing failure rates of wind turbines,” Energies, vol. 16, no. 24, art. 7937, 2023, doi: 10.3390/en16247937.

J. Walgern et al., “Impact of using text classifiers for standardising maintenance data of wind turbines on reliability calculations,” IET Renewable Power Generation, vol. 18, no. 15, pp. 3463–3479, 2024, doi: 10.1049/rpg2.13151.

M. Malyi, J. Shek, A. McDonald, and A. Biscaya, “A comparative benchmark of large language models for labelling wind turbine maintenance logs,” IET Renewable Power Generation, 2026, doi: 10.1049/rpg2.70212; preprint arXiv:2509.06813, 2025.

M. Dawson, S. Lukens, and M. Sharp, “Advancing technical language processing and large language models in industrial applications,” NIST Special Publication 1500-37, Dec. 2025, doi: 10.6028/NIST.SP.1500-37.

P. Zhao et al., “Retrieval-augmented generation for AI-generated content: a survey,” Data Science and Engineering, vol. 11, pp. 1–29, 2026, doi: 10.1007/s41019-025-00335-5.

J. A. Heredia Álvaro and J. González Barreda, “An advanced retrieval-augmented generation system for manufacturing quality control,” Advanced Engineering Informatics, vol. 64, art. 103007, 2025, doi: 10.1016/j.aei.2024.103007.

D. Shi, J. Li, O. Meyer, and T. Bauernhansl, “Enhancing retrieval-augmented generation for interoperable industrial knowledge representation and inference toward cognitive digital twins,” Computers in Industry, vol. 171, art. 104330, 2025, doi: 10.1016/j.compind.2025.104330.

F. Pedregosa et al., “Scikit-learn: machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность ИТ-КОНГРЕСС ВМК МГУ 2026 СНЭ

ISSN: 2307-8162