Research On Automatic Protocol Reverse Engineering On Binary Programs

Internet communication technology is developing rapidly in recentyears. More and more online applications are used every day, and peopleare paying more and more attention to their security problems. Therefore,the analysis of network protocols is becoming crucial to Internet security.Most applications use encryption technology to protect their sensitive datafrom being leaked or damaged. Meanwhile, many malicious software(malware) authors use encryption to protect the malware from beingdetected or analyzed, and then conduct their malicious behavior. The widespread of malware has been a major thread to Internet security in recentyears. Therefore, it’s of great importance to analyze the network protocolof malware, especially the encrypted network protocol. Security analystscan understand the internal structure and behavior of malware more easilywith the analysis of encryption schemes, and then malware can beprevented to harm the secure internet. Automatic protocol analysis acts animportant role in network security research. It’s extremely useful inprotocol reverse engineering, intrusion detection, network protection, fuzztesting, application signature identification, etc.However, existing techniques of encrypted protocol analysis cannotfulfill current needs. For example, existing techniques require thatdecryption and message processing are separated within the execution of aprogram. This condition can be hardly fulfilled in the complicatedimplementation of today’s malware. The instruction density characteristicsused by existing techniques are usually eliminated by software protectiontechniques such as code obfuscation. Meanwhile, existing techniquescannot extract the parameters of encryption algorithms, such as its type and secret key, which forms difficulties to in-depth analysis. These problemsrequire us to propose new methods to cope with current needs of encryptedprotocol analysis.To solve these issues, we propose a novel approach to encryptednetwork protocol analysis, which can effectively solve current problems.Our contributions are as follows:(1) We can detect and analyze encryption,encoding and checksum algorithms in encrypted protocol processing basedon taint analysis and data dependency analysis, which eliminates therequirements of separation of message decryption and processing inexisting methods;(2) We propose sub-message reconstruction to deal withcommon multi-level structure in malware, which solves the problem ofonly being able to analyze single layer in existing methods;(3) We proposedynamic algorithm parameter extraction methods using data patternanalysis, which solves the weakness of current methods and ease theburden of manual analysis;(4) We propose private algorithm analysismethods and data dependency analysis for private algorithms, which canbe used to analyze common private algorithms effectively in malware; wealso give entropy metrics for the classification and feature extraction ofprivate algorithms;(5) We evaluate our method using malware samples ofhigh threat level in recent years to demonstrate the feasibility, effectivenessand reliability of our method.