Log Probability-Based Evaluation of Code Correctness

Dmitriy Fedrushkov, Ivan Smirnov, Sergey Kovalchuk, Artem Aliev

Abstract


This work investigates probabilistic characteristics of language models as a tool for assessing the correctness of program code. The proposed approach is based on the analysis of token-level log-probabilities assigned by the model to generated solutions and requires neither reference solutions nor code execution. We examine the relationship between log-probability values and functional correctness measured by automated testing, using rank correlation, ROC analysis and logistic regression, and we compare several strategies for aggregating token-level values into a sequence-level score. Because solution length is associated both with the metrics and with the probability of passing tests, every comparison is reported twice: without length control and under length control, implemented as paired comparison within a narrow length band and as partial rank correlation. Experiments on the HumanEval-X benchmark across five programming languages and five open-weight models of up to 16 billion parameters show that probabilistic metrics distinguish correct from incorrect solutions and exhibit a statistically significant association with pass@1. Under length control the gap between self- and cross-evaluation substantially decreases, solution length alone is a weak predictor of correctness, whereas predictive entropy of the token distribution remains competitive with log-probability-based metrics.

Full Text:

PDF (Russian)

References


Kaukaz S., Sarhan Q. A Systematic Survey on Large Language Models for Code Generation // ARO-The Scientific Journal of Koya University. 2025. Vol. 13. P. 83-99. DOI: 10.14500/aro.12159.

Gulwani S., Polozov O., Singh R. Program Synthesis // Found. Trends Program. Lang. 2017. Vol. 4, No. 1–2. P. 1-119. DOI: 10.1561/2500000010.

Papineni K., Roukos S., Ward T., Zhu W.-J. Bleu: a Method for Automatic Evaluation of Machine Translation // Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002. P. 311-318. DOI: 10.3115/1073083.1073135.

Lin C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries // Text Summarization Branches Out. 2004. P. 74-81.

Liu Y., Iter D., Xu Y., Wang S., Xu R., Zhu C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. 2023. arXiv:2303.16634.

Gu J., Jiang X., Shi Z., Tan H., Zhai X., Xu C., Li W., Shen Y., Ma S., Liu H., Wang S., Zhang K., Lin Z., Zhang B., Ni L., Gao W., Wang Y., Guo J. A survey on LLM-as-a-Judge // The Innovation. 2026. P. 101253. DOI: 10.1016/j.xinn.2025.101253.

Pan R., Zhang H., Liu C. CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation. 2025. arXiv:2501.07811.

Ren S., Guo D., Lu S., Zhou L., Liu S., Tang D., Sundaresan N., Zhou M., Blanco A., Ma S. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. 2020. arXiv:2009.10297.

Feng Z., Guo D., Tang D., Duan N., Feng X., Gong M., Shou L., Qin B., Liu T., Jiang D., Zhou M. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. 2020. arXiv:2002.08155.

Fedrushkov D., Kovalchuk S., Aliev A. Natural language processing metrics efficiency for evaluating a generated code: facing the challenge // Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 2026. Vol. 26. P. 135-144. DOI: 10.17586/2226-1494-2026-26-1-135-144.

Zhou S., Alon U., Agarwal S., Neubig G. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code // Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. P. 13921-13937. DOI: 10.18653/v1/2023.emnlp-main.859.

Kovalchuk S., Fedrushkov D., Lomshakov V., Aliev A. Test-based and metric-based evaluation of code generation models for practical question answering // 2023 International Conference on Code Quality (ICCQ). 2023. P. 73-86. DOI: 10.1109/ICCQ57276.2023.10114665.

Shemetova E., Smirnov I., Alekseev A., Shenbin I., Rukhovich A., Nikolenko S., Lomshakov V., Piontkovskaya I. LAMeD: LLM-generated Annotations for Memory Leak Detection // Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering. 2025. P. 1024-1034. DOI: 10.1145/3756681.3756999.

He J., Shi J., Zhuo T. Y., Treude C., Sun J., Xing Z., Du X., Lo D. LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead // ACM Transactions on Software Engineering and Methodology. 2026. DOI: 10.1145/3797276.

Ott M., Auli M., Grangier D., Ranzato M. Analyzing Uncertainty in Neural Machine Translation // Proceedings of the 35th International Conference on Machine Learning. 2018. Vol. 80. P. 3956-3965.

Chen M. et al. Evaluating Large Language Models Trained on Code. 2021. arXiv:2107.03374.

Zheng Q., Xia X., Zou X., Dong Y., Wang S., Xue Y., Shen L., Wang Z.,

Wang A., Li Y., Su T., Yang Z., Tang J. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X // Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023. P. 5673-5684. DOI: 10.1145/3580305.3599790.

Yang A. et al. Qwen3 Technical Report. 2025. arXiv:2505.09388.

Grattafiori A. et al. The Llama 3 Herd of Models. 2024. arXiv:2407.21783.

Zhu Q. et al. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. 2024. arXiv:2406.11931.

Kamath A. et al. Gemma 3 Technical Report. 2025. arXiv:2503.19786.

Abdin M. et al. Phi-4 Technical Report. 2024. arXiv:2412.08905.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность ИТ-КОНГРЕСС ВМК МГУ 2026 СНЭ

ISSN: 2307-8162