Development of a monitoring system for a server application
Abstract
The purpose of this work is to study an approach for creating a monitoring subsystem for a server application of a medical information system for analyzing thyroid ultrasound images. To achieve this goal, the architecture of the subsystem for analyzing the health of the server application was designed, implemented and studied. It is based on a bunch of Prometheus, Grafana, Clickhouse tools and the Apache Spark engine. Prometheus acts as a means of collecting metrics from the nodes of the analyzed system, Grafana - as a means of visualizing data and alerting about failures within the medical system, Clickhouse - for data storage. To obtain information about the operation of the server application of the medical system, author's solutions are used to create exporters of metrics that allow you to both embed them in the application nodes and arrange them in the form of separate containers. This approach allows you to quickly rebuild the monitoring system for changes in the server application. The search for anomalies in the metrics of controlled parameters in real time is performed using Apache Spark and machine learning methods. The proposed solution for creating a monitoring subsystem for a server application has shown its effectiveness in testing the operation of a medical system.
Full Text:
PDF (Russian)References
R. J., "Data Discoverability in Science Gateways
at Scale using Elasticsearch Cluster Architecture," Practice and Experience in Advanced Research Computing, pp. 1-3, 2022.
G. D. E. Sankar P., "Social media monitoring using ELK Stack," 2022 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES), vol. 1, pp. 231-235, 2022.
M. O. K. V. Tokar D., "The IoT Applications Productivity: Data Management Model and ELK Tool Based Monitoring and Research," 2022 IEEE 16th International Conference on Advanced Trends in Radioelectronics, Telecommunications and Computer Engineering (TCSET), pp. 162-167, 2022.
B. K. A. Kalyaeva A. V., "Architecture development of a combined system for monitoring the health and business processes of an online store," Scientific and technical seminar of the Department of MOEM, p. 56., 2021.
W. H. e. al, "An Microservices-Based OpenStack Monitoring System," 2022 11th International Conference on Educational and Information Technology (ICEIT). – IEEE, pp. 232-236, 2022.
T. Leppänen, "Data visualization and monitoring with Grafana and Prometheus," 2021.
T. M. L. Ivanova E. V., "Overview of modern time series processing systems," Bulletin of the South Ural State University. Series: Computational Mathematics and Informatics, vol. 9, no. 4, pp. 79-97, 2020.
R. Booz, "What is ClickHouse, how does it compare to PostgreSQL and TimescaleDB, and how does it perform for time-series data?," Timescale, 21 10 2021. [Online]. Available: https://www.timescale.com/blog/what-is-clickhouse-how-does-it-compare-to-postgresql-and-timescaledb-and-how-does-it-perform-for-time-series-data/. [Accessed 2 5 2023].
B. M. N. B. C., "Commercial and Open Source Cloud Monitoring Tools: A Review.," in Advances in Decision Sciences, Image Processing, Security and Computer Vision, 2020, pp. 480-490.
C. S. B. C. Qian Ma, "A novel model for anomaly detection in network traffic based on kernel support vector machine," in hindawi, 2021.
V. B. D. N. Sarvani A., Anomaly Detection Using K-means Approach and Outliers Detection Technique, Springer Nature Singapore Pte Ltd., 2019.
"PostgreSQL: The World's Most Advanced Open Source Relational Database," The PostgreSQL Global Development Group, [Online]. Available: https://www.postgresql.org/. [Accessed 23 01 2023].
"Django REST framework," Encode OSS Ltd, [Online]. Available: https://www.django-rest-framework.org/. [Accessed 23 01 2023].
"Nginx," F5, Inc, [Online]. Available: https://www.nginx.com/. [Accessed 23 01 2023].
ITSumma, "Monitoring starts with metrics. Part 2: server software," Habr, 2022.
"django-prometheus. Export Django monitoring metrics for Prometheus.io," [Online]. Available: https://github.com/korfuri/django-prometheus. [Accessed 23 01 2023].
"NGINX Prometheus Exporter for NGINX and NGINX Plus," NGINX, Inc., [Online]. Available: https://github.com/nginxinc/nginx-prometheus-exporter. [Accessed 23 01 2023].
"pyNVML. Python bindings to the NVIDIA Management Library," NVIDIA Corporation, [Online]. Available: https://pythonhosted.org/nvidia-ml-py/. [Accessed 23 01 2023].
"Cadvisor. Analyzes resource usage and performance characteristics of running containers.," google, [Online]. Available: https://github.com/google/cadvisor. [Accessed 23 01 2023].
"Serverless. Simple. ClickHouse Cloud.," ClickHouse, Inc., [Online]. Available: https://clickhouse.com/. [Accessed 23 01 2023].
B. A., "How we built monitoring on Prometheus, Clickhouse and ELK," Habr, 2019.
"Grafana. Operational dashboards for your data here, there, or anywhere," Grafana Labs, [Online]. Available: https://grafana.com/. [Accessed 23 01 2023].
U. S., "Monitoring Basics (Review of Prometheus and Grafana)," Habr, 2023.
Malathi K., "Multi Cluster Monitoring for Fault Detection Using Novel Kubernetes with Prometheus over Docker Container," Journal of Pharmaceutical Negative Results, pp. 1548-1555, 2022.
T. J., "Elasticsearch Index and Query Design for Joining Full Text and Time Series Data," Available at SSRN 4325567, 2022.
S. M. V., "Elastic Search Engine Analysis and Its Contribution to Quicker Data Retrieval Solutions for Numerous Issues," International Journal of Engineering Research, vol. 2, no. 1, pp. 05-08, 2022.
Refbacks
- There are currently no refbacks.
Abava Кибербезопасность MoNeTec 2024
ISSN: 2307-8162