Risk identification approach using artificial intelligence and big data analysis

N.N. Goglev, S.A. Migalin, E.V. Kasatkina

Abstract


The use of artificial intelligence technologies and big data analysis in risk management makes it possible to reduce the burden on experts and reduce the influence of the human factor in risk assessment. These technologies are well studied and actively used to determine the probability of known risks and assess the magnitude of the consequences when they occur, but the approach to identifying new types of risks remains poorly developed. The authors have developed an innovative approach to identifying new types of risks based on the use of artificial intelligence methods and big data analysis.

The developed approach involves the identification of new types of risk in three stages: 1) identification of anomalies in the historical data array; 2) division of the identified anomalies into homogeneous clusters; 3) profiling of clusters of anomalies as potentially new types of risks, description of the characteristic features of the identified clusters. To search for anomalous observations, the authors propose to use the technology of ensembling statistical methods and machine learning methods, such as the ellipsoidal data approximation method, the local outlier level method, and the isolation forest method. To form homogeneous clusters of anomalous observations, it is proposed to use one of the cluster analysis methods selected based on the values of internal clustering quality metrics. Correlation and statistical data analysis methods are used to profile anomalous clusters as potentially new types of risks. The proposed approach, in contrast to classical risk identification technologies, makes it possible to increase the efficiency and quality of identification. The developed approach can methodologically be integrated into standard risk management processes and used in various fields of activity for automated identification of new types of risks for the purpose of their subsequent analysis and processing.


Full Text:

PDF (Russian)

References


Committee of Sponsoring Organizations of the Treadway Commision. Enterprise Risk Management. Integrating with Strategy and Performance. Available: https://www.coso.org/documents/2017-coso-erm-integrating-with-strategy-and-performance-executive-summary.pdf.

Artificial Intelligence (AI) Applied to Risk Management. Available: https://www.ferma.eu/publication/artificial-intelligence-ai-applied-to-risk-management.

Aziz S., Dowling M. Machine Learning and AI for Risk Management // SSRN Electronic Journal. DOI: 10.2139/ssrn.3201337.

Svistunova S. A., Muzalev S. V. Usage of Machine Learning in The Process of Risk-Management // Russian Journal of Management – 2021. – No 3. – P. 126–130, 2021. DOI: 10.29039/2409-6024-2021-9-3-126-130.

IEC 31010:2019 «Risk management – Risk assessment techniques»

GOST R 58771-2019. Nacional'nyj standart Rossijskoj Federacii. Menedzhment riska. Tehnologii ocenki riska.

Deberdieva N.P., Voronin A.V. Identifikacija riskov promyshlennyh predprijatij v koncepcii risk-menedzhmenta // Jekonomika, predprinimatel'stvo i pravo. – 2020. – T. 10. – # 5. – S. 1425–1438. DOI: 10.18334/epp.10.5.100952.

Shatalova O. M. O metodologicheskih podhodah k resheniju problemy neopredelennosti v upravlenii tehnologicheskimi innovacijami na predprijatii // Vestnik IzhGTU imeni M. T. Kalashnikova. – 2018 – T. 21. – # 3. – S. 120-126. DOI 10.22213/2413-1172-2018-3-120-126.

Zimek A., Schubert E. Outlier Detection // Encyclopedia of Database Systems. Springer New York – 2017. DOI: 10.1007/978-1-4899-7993-3_80719-1.

Beketnova Yu.M. Comparative Analysis of Machine learning Methods to Identify signs of suspicious Transactions of Credit Institutions and Their Clients. Finance: Theory and Practice. – 2021. No. 25(5). – P. 186–199. DOI: 10.26794/2587-5671-2020-25-5-186-199.

Laimek R., Kaothanthong N., Supnithi T. ATM Fraud Detection Using Outlier Detection. // Intelligent Data Engineering and Automated Learning – IDEAL 2018. Lecture Notes in Computer Science. – Vol. 11314. Springer, Cham. DOI: 10.1007/978-3-030-03493-1_56.

Ray S., Wright A. Detecting anomalies in alert firing within clinical decision support systems using Anomaly / Outlier Detection Techniques // Proc. 7th ACM Int. conf. on bioinformatics, computational biology, and health informatics. New York: Association for Computing Machinery. – 2016. – P. 185–190. DOI: 10.1145/2975167.2975186.

Chesnokov A., Mikhailov V., Dolmatov I. Detection of Structural Deterioration in Hybrid Constructions // 3rd International Conference on Control Systems, Mathematical Modeling, Automation and Energy Efficiency (SUMMA). – 2021. – P. 479–484. DOI: 10.1109/SUMMA53307.2021.9632014.

Hung D.V., Hung H.M., Anh P.H., Thang N.T. Structural damage detection using hybrid deep learning algorithm // Journal of Science and Technology in Civil Engineering (STCE) – HUCE. – 2020. – No. 14(2). – P. 53–64. DOI: 10.31814/stce.nuce2020-14(2)-05.

Alos A., Dahrouj Z. Detecting Contextual Faults in Unmanned Aerial Vehicles Using Dynamic Linear Regression and K-Nearest Neighbour Classifier // Gyroscopy and Navigation. – 2020. – No. 28(1). – P. 66–80. DOI: 10.17285/0869-7035.0024.

Maia P., Meira W. Jr., Barbosa B., Cruz G. Multicriteria Anomaly Detection in Government Purchases. In: Anais do VII Symposium on Knowledge Discovery, Mining and Learning, Fortaleza. – 2019. – P. 97–104. DOI: https://doi.org/10.5753/kdmile.2019.8794.

Kaytaz U., Sivrikaya F., Albayrak S. Competitive Learning for Unsupervised Anomaly Detection in Intelligent Transportation Systems // Conference: IEEE International Conference on Communications, to be published.

Rzaev B.T., Lebedev I.S. Primenenie bjegginga pri poiske anomalij setevogo trafika // Nauchno-tehnicheskij vestnik informacionnyh tehnologij, mehaniki i optiki. – 2021. – T. 21. # 2. – S. 234–240.

Alshawabkeh M., Jang B., Kaeli D. Accelerating the local outlier factor algorithm on a GPU for intrusion detection systems // Conference: Proceedings of 3rd Workshop on General Purpose Processing on Graphics Processing Units, GPGPU. Pittsburgh, Pennsylvania, USA. – 2010. DOI: 10.1145/1735688.1735707.

Campos G.O., Zimek A., Sander J., Campello R., Micenkova B., Schubert E., Assent I., Houle M.E. On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study // Data Mining and Knowledge Discovery. – 2016. – Vol. 30, No. 4. DOI: 10.1007/s10618-015-0444-8.

Pukelsheim F. The Three Sigma Rule // The American Statistician. – 1994. – No. 48(2). – P. 88–91. DOI: 10.2307/2684253

Rousseeuw P.J., Driessen V.K. A fast algorithm for the minimum covariance determinant estimator // Technometrics. – 1999. – Vol. 41(3). – P. 212–223.

D'jakonov A.G., Golovina A.M. Vyjavlenie anomalij v rabote mehanizmov metodami mashinnogo obuchenija // Sbornik nauchnyh trudov XIX Mezhdunarodnoj konferencii DAMDID: Analitika i upravlenie dannymi v oblastjah s intensivnym ispol'zovaniem dannyh. – 2017. – S. 469–476.

Ester M., Kriegel H.-P., Sander J., Xu X. A density-based algorithm for discovering clusters in large spatial databases with noise // Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96) / Evangelos Simoudis, Jiawei Han, Usama M. Fayyad. – AAAI Press. – 1996. – P. 226–231.

Schubert E., Sander J., Ester M., Kriegel H. P., Xu X. DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. // ACM Transactions on Database Systems. – 2017. – Vol. 42(3), No. 19.

Kriegel H.-P., Kröger P., Sander J., Zimek A. Density-based clustering // Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. – 2011. Vol. 1, No. 3. – P. 231–240. DOI: 10.1002/widm.30.

Breunig M.M., Kriegel H.-P., Ng R.T., Sander J.R. OPTICS-OF: Identifying Local Outliers // Principles of Data Mining and Knowledge Discovery. – 1999. Vol. 1704. DOI: 10.1007/978-3-540-48247-5_28.

Knorr E.M., Ng R.T., Tucakov V. Distance-based outliers: algorithms and applications // The VLDB Journal – The International Journal on Very Large Data Bases. – 2000. Vol. 8, No. 3-4. – P. 237–253.

Breunig M. M., Kriegel H.-P., Ng R.T., Sander J. LOF: Identifying Density-based Local Outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data. – 2000. – P. 93–104. DOI: 10.1145/335191.335388.

Lazarevic A., Ozgur A., Ertoz L., Srivastava J., Kumar V. A comparative study of anomaly detection schemes in network intrusion detection // Proc. 3rd SIAM International Conference on Data Mining. – 2003.

Vapnik V.N. The Support Vector method // Artificial Neural Networks – ICANN'97. LNCS. Springer, Berlin, Heidelberg. – 1997. – Vol. 1327. DOI: 10.1007/BFb0020166.

Schölkopf B., Platt J. C., Shawe-Taylor J., Smola A. J., Williamson R. C. Estimating the Support of a High-Dimensional Distribution // Neural Computation. Bernhard. – 2001. – Vol. 13, No. 7. – P. 1443–1471. DOI: 10.1162/089976601750264965.

Schölkopf B, Williamson RC, Smola A, Shawe-Taylor J, Platt J. Support vector method for novelty detection // Advances in neural information processing system. – 1999. – No. 12. – P. 582–588.

Liu F.T., Ting K.M., Zhou Z.-H. Isolation forest // In Data Mining. ICDM'08. Eighth IEEE International Conference on. – 2008. – P. 413-422. DOI: 10.1109/ICDM.2008.17.

Cheng-Yuan Liou, Wei-Chen Cheng, Jiun-Wei Liou, Daw-Ran Liou Autoencoder for words // Neurocomputing. – 2014. – Vol. 139. – P. 84-96.

Zhisheng X., Qing Y., Yali A. Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder // Advances in Neural Information Processing Systems. – 2020.

Aggarwal C.C., Sathe S. Outlier Ensembles: An Introduction. – Springer, 2017. – 276 r.

Benkabou SE., Benabdeslem K., Canitia B. Unsupervised outlier detection for time series by entropy and dynamic time warping // Knowl Inf Syst. – 2018. – Vol. 54. – P. 463-486. DOI: 10.1007/s10115-017-1067-8

Chesnokov M.Ju. Poisk anomalij vo vremennyh rjadah na osnove ansamblej algoritmov DBSCAN // Iskusstvennyj intellekt i prinjatie reshenij. – 2018. – # 1. – S. 99–107.

Arthur D., Vassilvitskii S. K-Means++: The Advantages of Careful Seeding // Conference: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, New Orleans, USA. – 2007. DOI: 10.1145/1283383.1283494.

Shatalova O.M., Kasatkina E.V., Livshic V.N. Klasternyj analiz i klassifikacija promyshlenno orientirovannyh regionov RF po jekonomicheskoj specializacii // Jekonomika i matematicheskie metody. – 2022. – T. 58, # 1. – S. 81–92.

Ketova K.V., Kasatkina E.V., Vavilova D D. Clustering Russian Federation regions according to the level of socio-economic development with the use of machine learning methods // Economic and Social Changes: Facts, Trends, Forecast. – 2021. Vol. 14, No. 6. – P. 70–85. DOI: 10.15838/esc.2021.6.78.4

Shalymov D. S. Algoritmy ustojchivoj klasterizacii na osnove indeksnyh funkcij i funkcij ustojchivosti // Stohasticheskaja optimizacija v informatike. SPb.: Izd-vo S.-Peterburgskogo universiteta. – 2008. # 4. – S. 236-248.

Rousseeuw P.J. Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis // Computational and Applied Mathematics. – 1987. – No. 20, P. 53–65. DOI: 10.1016/0377-0427(87)90125-7.

Davies D.L., Bouldin D.W. A Cluster Separation Measure // IEEE Transactions on Pattern Analysis and Machine Intelligence. – 1979. – P. 224–227. DOI:10.1109/TPAMI.1979.4766909.

Calinski T., Harabasz J. A dendrite method for cluster analysis // Communications in Statistics. – 1974. – Vol. 3. – P. 1-27. DOI: 10.1080/0361092.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность IT Congress 2024

ISSN: 2307-8162