Review of Anomaly Detection Methods During System Call Auditing in OS
Abstract
This paper presents a systematic review of state-of-the-art techniques for anomaly detection in system calls—a critical component of Host-based Intrusion Detection Systems (HIDS). The analyzed approaches include: statistical models, machine learning and deep learning methods, hybrid architectures, and sequence analysis techniques. Particular attention is paid to assessing their applicability in contemporary computing environments, such as Internet of Things (IoT) devices, containers, cloud platforms, and mobile operating systems.
The study results show that state-of-the-art techniques can be categorized based on key characteristics. Deep learning architectures achieve the lowest false positive rate (FPR), but this comes at the cost of significant computational resources. Hybrid approaches demonstrate a more balanced combination of accuracy and efficiency, yet they may remain vulnerable to sophisticated, adaptive attacks. Resource-efficient solutions find application in constrained IoT environments; however, their effectiveness is often diminished by sensitivity to data imbalance. The research confirms that the choice of the optimal method is determined by the target platform. For resource-constrained IoT devices, lightweight models are preferred, while in cloud environments, comprehensive solutions capable of delivering low FPR demonstrate maximum efficiency. Key unresolved challenges remain the high false positive rate and the computational complexity of processing real-time data streams.
Full Text:
PDF (Russian)References
C. Kruegel, D. Mutz, F. Valeur, and G. Vigna, “On the detection of anomalous system call arguments,” // Computer Security – ESORICS 2003, 2003, pp. 326–343
Sokolov, A.M. Modern models of anomaly detection in computer systems // Cybernetics and Computer Science, 1999 // UDC 004.056.53.001.3. - P. 1-18.
Y. Shen, F. Yu, L. Zhang, J. An, and M. Zhu, “An intrusion detection system based on system call,” Proc. The First IEEE and IFIP International Conference in Central Asia on, pp. 1–4, 2005.
C. Warrender, S. Forrest, and B. Pearlmutter, “Detecting intrusions using system calls: alternative data models,” Proceedings of the IEEE Symposium on Security and Privacy, pp. 133–145, 1999.
X. D. Hoang, J. Hu, and P. Bertok, “A multi-layer model for anomaly intrusion detection using program sequences of system calls,” Proc. of the International Conference on Computational Intelligence and Security, pp. 531–536, 2003.
D.-K. Kang, D. Fuller, and V. Honavar, “Learning classifiers for misuse and anomaly detection using a bag of system calls representation,” Proc. Sixth Annual IEEE SMC Information Assurance Workshop, pp. 1–5, 2005.
D. Mutz, F. Valeur, C. Kruegel, and G. Vigna, “Anomalous system call detection,” ACM Transactions on Information and System Security (TISSEC), vol. 9, no. 1, pp. 61–93, 2006.
J. Grandhi, H. Pareek, and P. R. L. Eswari, “Detecting anomalous application behaviors using a system call clustering method over critical resources,” Proc. Communications in Computer and Information Science, pp. 53–64, 2011.
Z. Hu, L. Liu, H. Yu, and X. Yu, “Using graph representation in host-based intrusion detection,” Security and Communication Networks, vol. 2021, Article ID 6291276, 2021.
J. H. Ring, C. M. Van Oort, S. Durst, V. White, J. P. Near, and C. Skalka, “Methods for host-based intrusion detection with deep learning,” Digital Threats: Research and Practice, vol. 2, no. 4, Article 26, pp. 1–29, 2021.
B. Yu and J. Kim, “Using a neural network to detect anomalies given an n-gram profile,” arXiv preprint arXiv:2104.05571v2, 2021. [Online]. Available: https://doi.org/10.48550/arXiv.2104.05571.
J. Carter, S. Mancoridis, M. Nkomo, S. Weber, and K. R. Dandekar, “System call processing using lightweight NLP for IoT behavioral malware detection,” in Lecture Notes in Computer Science, vol. 13247. Cham: Springer, pp. 103–115, 2022.
P. K. Mvula, P. Branco, G.-V. Jourdan, and H. L. Viktor, “Evaluating word embedding feature extraction techniques for host-based intrusion detection systems,” Discover Data, vol. 1, no. 2, pp. 1–27, 2023.
T. Vyšniunas, D. Čeponis, N. Goranin, and A. Čenys, “Risk-based system-call sequence grouping method for malware intrusion detection,” Electronics, vol. 13, no. 1, p. 206, 2024.
V. Van Mieghem, “Detecting malicious behaviour using system calls,” M.S. thesis, Delft University of Technology, Delft, The Netherlands, 2016.
M. Pendleton, “System call anomaly detection in multi-threaded programs,” Ph.D. dissertation, University of Texas at San Antonio, San Antonio, TX, USA, 2017.
F. J. Mora-Gimeno, H. Mora-Mora, B. Volckaert, and A. Atrey, “Intrusion detection system based on integrated system calls graph and neural networks,” IEEE Access, vol. XX, pp. 1–9, 2021.
A. Frossi, F. Maggi, G. L. Rizzo, and S. Zanero, “Selecting and improving system call models for anomaly detection,” in Proc. DIMVA 2009, pp. 206–223, 2009.
G. Canfora, F. Mercaldo, E. Medvet, and C. A. Visaggio, “Detecting android malware using sequences of system calls,” in Proc. 30th Annu. Comput. Security Appl. Conf., pp. 13–20, 2014.
A. S. Abed, T. C. Clancy, and D. S. Levy, “Applying bag of system calls for anomalous behavior detection of applications in Linux containers,” in Proc. IEEE Global Commun. Conf. (GLOBECOM), pp. 1–5, 2015.
Dimjašević, M., Atzeni, S., Ugrina, I., & Rakamarić, Z. “Android malware detection based on system calls UUCS-15-003,” School of Computing, University of Utah.
R. Canzanese, S. Mancoridis, M. Kam, “System call-based detection of malicious processes,” 2015 IEEE 15th International Conference on Quality Software, pp. 147-156, 2015.
Tien, C.-W., Huang, T.-Y., Tien, C.-W., Huang, T.-C., & Kuo, S.-Y. “KubAnomaly: anomaly detection for the docker orchestration platform with neural network approaches,” Engineering Reports, vol. 1, no. 5, p. e12080, 2019.
Peddoju, S. K., H. Upadhyay, J. Soni, N. Prabakar, “Natural language processing based anomalous system call sequences detection with virtual memory introspection,” International Journal of Advanced Computer Science and Applications, vol. 11, no. 5, pp. 445–460, 2020.
M. Shobana and S. Poonkuzhali, “A novel approach to detect IoT malware by system calls using deep learning techniques,” in Proc. 2020 Int. Conf. Innovative Trends in Information Technology (ICITIIT), pp. 436–441, 2020.
Maheswari, K. U., Shobana, G., Bushra, S. N., & Subramanian, N. “Supervised malware learning in cloud through system calls analysis,” in Proc. 2021 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES), pp. 1–8, 2021.
Park, D., Kim, S., Kwon, H., Shin, D., & Shin, D. “Host-based intrusion detection model using Siamese network,” IEEE Access, vol. 9, pp. 1–9, 2021.
G. R. Castanhel, T. Heinrich, F. Ceschin and C. Maziero, “Taking a peek: an evaluation of anomaly detection using system calls for containers,” 2021 IEEE Symposium on Computers and Communications (ISCC), Athens, Greece, pp. 1–6, 2021, doi: 10.1109/ISCC53001.2021.9631251.
P. Brown, A. Brown, M. Gupta, and M. Abdelsalam, “Online malware classification with system-wide system calls in cloud IaaS,” in Proc. 2022 IEEE 23rd International Conference on Information Reuse and Integration for Data Science (IRI), pp. 146–151, 2022.
X. Liao, C. Wang, and W. Chen, “Anomaly detection of system call sequence based on dynamic features and relaxed-SVM,” Security and Communication Networks, vol. 1, no. 2, pp. 1–13, 2022.
I. Tahir and S. Qadir, “Machine learning-based detection of IoT malware using system call data,” in Proc. 2024 4th International Conference on Digital Futures and Transformative Technologies (ICoDT2), Islamabad, Pakistan, pp. 1-8, 2024.
N. Shamim, M. Asim, T. Baker, and A. I. Awad, “Efficient approach for anomaly detection in IoT using system calls,” Sensors, vol. 23, no. 2, Art. no. 652, 2023.
J. Ramamoorthy, K. Gupta, R. C. Kafle, N. K. Shashidhar, and C. Varol, “A novel static analysis approach using system calls for Linux IoT malware detection,” Electronics, vol. 13, no. 15, p. 2906, 2024.
S. Lv, J. Wang, Y. Yang and J. Liu, “Intrusion prediction with system-call sequence-to-sequence model,” in IEEE Access, vol. 6, pp. 71413-71421, 2018.
Y. J. Ham, D. Moon, H.-W. Lee, J. D. Lim, and J. N. Kim, “Android mobile application system call event pattern analysis for determination of malicious attack,” Int. J. Secur. Appl., vol. 8, no. 1, pp. 231–246, 2014.
D. De Bruin, “System call sandboxing: enhancing security through analysis comparing dynamic and static system call analysis for Diff and SSH,” Bachelor's thesis, EEMCS Faculty, Delft University of Technology, Delft, Netherlands, 2024. [Available: https://repository.tudelft.nl/file/File_c7212fb0-5e39-4135-8f7b-0555f30baaca?preview=1]
A. Singh, “System call analysis and visualization,” M.S. thesis, Dept. Comput. Sci., California State Univ., Sacramento, CA, USA, 2018.
Q. Fournier, D. Aloise, S.V. Azhari, and F. Tetreault, “On improving deep learning trace analysis with system call arguments,” in Proc. IEEE/ACM Int. Conf. Mining Software Repositories (MSR), pp. 1–11, 2021.
Slyusarenko I. M. Methods for detection and evaluation of information system anomalies based on system call analysis: PhD dissertation (Candidate of Technical Sciences). Saint Petersburg State Polytechnic University, St. Petersburg, Russia, 2005. 178 p.
Refbacks
- There are currently no refbacks.
Abava Кибербезопасность ИБП для ЦОД СНЭ
ISSN: 2307-8162