Experimental Assessment of the Complexity of Reverse Engineering a TCP/IP Server Using Neural Networks

Abstract

The paper addresses the problem of quantitative evaluation of the complexity of reverse engineering a binary TCP/IP server in the absence of source code and formal protocol specifications. A formalized model of the reverse engineering process is proposed, based on representing the analyzed software as a directed call graph and a set of functional components responsible for network message processing. An integral complexity metric is introduced that aggregates temporal, structural, and semantic parameters of the analysis. An experimental study of the influence of an assistive tool based on neural network function classification on the effort required for binary code analysis is conducted. The results show that preliminary structuring of the search space and prioritization of functions for detailed examination make it possible to reduce the total analysis time by approximately 30 % while maintaining a comparable level of completeness in protocol logic reconstruction. The scientific novelty of the work lies in the formalization of the complexity of the reverse engineering process of a network binary TCP/IP server, the introduction of an integral metric for analysis effort, and the experimental evaluation of the impact of neural network–based assistance on reducing structural and semantic uncertainty during the analysis of binary programs.