Research On Technology Of Software Protection And Malicious Code Detection Based On Code Obfuscation

With the wide application and development of computer network and Internet technology, software security issues are getting more and more attention. Code obfuscation is a beforehand protection mechanism to transform program in order to make it more difficult to analyze and understand while preserving its functionality. It is low-cost technology, and also is one of the representative protection method that preventing program from malicious attack. Along with the application environment becomes increasingly complex and the development of attack artifice, single obfuscation method has been unable to meet the requirements of software protection.Therefore, it is significant to develope more effective and more powerful obfuscation technologies to ensure the safe execution of procedures and data confidentiality. On the other hand, malware writers also make use of obfuscation to construct polymorphic and metamorphic viruses to avoid analysis and detection. It will bring new challenges to the existing malware detection technology. How to determine obfuscated malware effectively, it still need to be further studied. Major achievements of this dissertation are as follows:(1)Matrix data type obfuscation by data refinement. Consider obfuscation as functional refinement to produce equations for proving correctness of operation. Define matrix splitting, and hide information by changing the form of matrix. The obfuscation framework of matrix standard operations is established by functional language combining the property of partitioned matrix, and its complexity does not change. In addition, use matrix to obfuscate scalar and its. Abstract function and transformation function are defined according to the characteristics of matrix determinant. Obfuscation on arithmetic operations are defined and correctness is proved. The method can flexibly define some matrix elements to provide effective obfuscation about scalar.(2)An effective hybrid protection technology is proposed for confidential data obfuscation in large database based on data perturbation and query restriction. Establish the mathematical model of solving optimal subset of queries so that it can be used to answer queries accurately, and approximate solution is obtained by matroid intersection greedy algorithm. Perturbative database is created by designing two random data perturbation method, It can provide perturbation answer which is consistent with accurate answer for the rest queries, and the answer is more exact than standard perturbation method. An additional exact answer to query that is close to original query is provided based on perturbed data. The scheme can maximize the set of query on accurate answer, the confidential data also can not be disclosed. The perturbed data can not affect the results of accurate answer. The approach applies to linear and nonlinear queries and to numerical data and categorical data.(3)Construct model of strengthening control flow flattening by introducing branch function and transfer function based on the existing control flow graph flattening in order to prevent static analysis. Then, use three application models and several attacks to evaluate the strength of the technique. At the same time, the obfuscation algorithm of code block diversification is presented for dynamic reverse engineering and is evaluated.The results show that this obfuscation method not only prevents static reverse engineering effectively, but also limit influence on dynamic analysis. The rate of code reconstructing is greatly reduced.(4)Static analysis of obfuscated malicious code based on semantics. A high-level language Malspec is defined. It is independent of obfuscation to describe malicious behavior. Semantic matching conditions is proposed by Malspec. Construct static analyzer and detection algorithm to detecte executable malicious pattern. The analyzer can analyze executable files effectively and safely. It can resist common obfuscation compared with other virus detectors(5)A more fine-grained detection method is presented for obfuscated code. In addition to control flow graph, the approach also uses functionalities of basic block as the signature of malicious code. The detection framework is constructed. Design signature calculation algorithm to extract the signature of a malicious code fragment based on compiler optimization algorithm, and integrate memory sub-variableoptimization, expression formalization and cross basic block propagation. The expressions of assignment statements are formalized to facilitate comparing the functionalities of two expressions by defining formalization rules. The detection algorithm is used to detecte whether a program is an obfuscated malware instance in term of control flow graph and the functionalities of basic blocks. The experimental results show that the approach can detecte obfuscated malicious code effectively, which can not be detected by previous approach. Avoiding misjudging on the codes with same structures but different functionalities.This dissertation gives new obfuscation approach on abstract data type and confidential data information in large database from data obfuscation and control flow obfuscation. The obfuscation scheme that hides control flow to prevent static and dynamic reverse analysis is achieved. Malicious code static analysis and detection method offer new thought for the problem that results of false positive and false negative are caused by obfuscated malicious code.