Enhancing RAG Configuration Efficiency for Large Language Models Working with Clinical Guidelines: The Case of Allergic Rhinitis

Abstract

The article examines approaches to improving the effectiveness of Large Language Models (LLMs) in solving specialized medical tasks based on clinical guidelines issued by the Ministry of Health of the Russian Federation. Particular attention is paid to comparing different variations of the Retrieval-Augmented Generation (RAG) architecture, which is used to reduce factual errors and improve the relevance of model responses. Clinical guidelines on allergic rhinitis were selected as the domain-specific case study, and Russian GigaChat family models were used as the language models. Within the experimental study, sixteen configurations of model interaction were implemented, differing in LLM capacity, source document format (PDF and pre-adapted Markdown), and knowledge base retrieval strategies (keyword-based, vector-based, and hybrid search). The quality of responses was evaluated by comparing them with reference answers derived from the clinical guidelines using BLEU and METEOR metrics. The results demonstrated that the best performance is achieved when using a more powerful model in combination with a RAG approach based on vector or hybrid retrieval and a knowledge base constructed from structured, machine-readable text. The findings confirm the feasibility of applying RAG in medical information systems and highlight the importance of prior preparation and structuring of regulatory documents to improve the performance of large language models.