Research On Vulnerability Detection On Network Software With Encrypted Communication

Software vulnerability triggerred by unsafely dealing with network communication data is extremely serious. Although fuzz tesing technology can find bugs in network software, existing approaches become invalid in case that the communication is encrypted.Aiming at the above question, this thesis proposes a method towards finding bugs in network software with encrypted communication. Having located the decrypted memory of encrypted data, the method tests the code execution space after the decryption state by treating the decrypted memory as testing object, and after detecting the vulnerability, reconstructing the encrypted sample to confirm the existance of vulnerability in real environment.To realize the above method, based on fine-grained dynamic taint analysis, in accordance with decrypted memory detecion technology, this thesis studies the test data generation technology to test the vulnerability of network software with encrypted communication. The main work can be summarized as follows:1. An optimization method towards fine-grained dynamic taint analysis is proposed, which applies the code optimization idea into dynamic taint analysis. By merging and eliminating unnecessary procedures of taint analysis, this technology improves the efficiency of fine-grained dynamic taint analysis, without changing the original logic of taint propagation, and laying the foundation of the appliaction to the location of decrypted memory.2. Two technologies to locate the decrypted data in memory are proposed. Using the characteristic that the cryptographic function generates decrypted data in phase of loop iteration, a method to locate decrypted memory based on loop I/O is proposed. By dynamically tracing the execution trace of cryptographic function, this method detects the loop structure inside it, extracting and merging loop structures’inputs and outputs, among which filtering the memory of decrypted data. Using the characteristic that cryptographic function highly mixes data, a decrypted data detection method based on high dependency degree is proposed, which uses fine-grained dynamic taint analysis to describe the strong relationship between cipher and plain text, and detects the memory with high dependency as decrypted memory. After comparing the utilization scope of the two methods, an integrated scheme is given by combining the advantages of the two.3. A test case generation method based on "decomposition and reconstruction" is proposed, realizing the goal of testing the vulnerability of network software with encrypted communication. In decomposition phase, by means of detection technology on integrity-checking point and decrypted memory, the data encryption mode and integrity checking mode are detected, as well as the effective testing object. In reconstruction phase, a memory-backtracking algorithm is proposed, which detects the memory none of the duplication of other memories at the other communication side, based on which the encrypted test packet is reconstructed.4. A prototype system (named as EncTracer) for vulnerability detection on network software with encrypted communication is designed and implemented, on which function tests and comparison tests are made. Experimental results demonstrate that EncTracer can accurately locate decrypted memory and integrity-checking point, effectively generate test cases with the method of "decomposition and reconstruction", and find infinite-loop and null-pointer-dereference bugs inside Zeus when dealing with encrypted network data.