Research On Recognition Of Malicious Behavior And Decision Of Maliciousness For Binary Executables

The problem of information system security has been promoted to national strategic level.Nowadays, the malicious code is one of the main threats against information security. On the onehand，driven by economic profit, the number of malicious codes is growing at an alarming rate,which makes the security staffs exhausted to keep up. On the other hand, malicious code writerscontinuously put forward fresh strategies to challenge the traditional software analysis methods,such as polymorphism and metamorphism technologies widely adopted in the malicious codes,which make analyzing and detecting malicious codes more difficult. For this reason, effectiveanalysis and detection against malware will improve the security of information system, and cutthe losses from malicious codes.Decompiling binary executable file to extract the characters and behaviors in the program,which is an effective method to detect and analyze malicious program. This thesis investigatesseveral key techniques used in program maliciousness decision, such as model establishment,program behaviors identification, and design of decision algorithm. The main contents andcontributions of this thesis are as follows.1. An evidential reasoning based program maliciousness analysis model is proposed.Current model of malicious code detection mainly adopt pattern matching method, which onlycompares the known signature to the binary sequences or regular expressions of a program, andthen the false alarms would easily arise. To deal with the above problem, this thesis proposes aprogram maliciousness detection model ERMA, where the maliciousness detection problem isabstracted as multi evidence synthesis problem, which introduces evidential reasoning to thedetection model. Using the identify degree of multi character behaviors as inputs of the ERMAmodel, which can solve the problem that a sole behavior as detection rule to a certain extent. Atthe same time, this model has presented a better performance of false positive.2. A probability distribution function based on neural network is presented. How to switchthe program characters to the probability distribution function in the training set for reasoning,which is a difficulty for utilizing evidence theory to identify malware. The existing methodsacquire the value of the probability distribution function usually by manually setting, which hasmany shortcomings, such as high computational complexity, bad adaptability and strongersubjectivity. For this problem, the neural network has good generalization ability, could learn allthe data with self-inductive learning. Therefore, this thesis adopts the neural network to learnlarge number of knowledge, which can play the role as experts to solve the problem withassigning the value of probability distribution function.3. An evidence combination algorithm based on support degree is presented. ClassicDempster combination rule could generate paradox when synthesing those evidences with theconflict, which makes the results counterintuitive. In the course of maliciousness reasoningalgorithm research, we found that the paradox problem exercises a great influence on theaccuracy and reliability of reasoning result. For the above problem, this thesis proposes theconcept of support degree, which can determine the conflict of evidence by calculating theevidence distance. Therefore, the concept of support degree raises the accuracy of program maliciousness detection.4. A disassemble frame and related algorithm are designed aiming at malware. In order toavoid the static analyzing, hackers rely on various obfuscation techniques to hide its maliciouscharacters. These techniques are very effective against common disassemblers, preventing binaryfile from being disassembled correctly. This thesis explores three kinds of obfuscation techniques,and then develops a strategy to disassemble effective instruction in malware, integrating ControlFlow Graph (CFG) and Data Flow Graph (DFG) information to improve the ability of thedisassembly. This method can fight against common obfuscation techniques, such as junk datainsert, subprogram exception return and direct jump indirectly. The experiment shows that themethod not only improves the accuracy of disassemble, but also greatly deal with malicious files.5. A malicious behavior recognition method based on model checking is presented. Programbehavior is a group of coordinate operations, which can be observed on the computer system.Those behaviors can be seen as the best evidence reflecting the program character when wedetermine the program maliciousness. The program behaviors are not independent but can becomposed to new behaviors, the correlation between them difficultly use specific mathematicalto describe. This thesis proposes a program behaviors recognition method, through analyzingmalware performance in multi layers, such as instructions sequence, system-call map andfunction parameters. Model checker recognizes malicious behavior and denotes detectedbehavior in the CFG.The model and algorithms presented in this thesis have been carried out and applied inNombril system, and the validity has been proved.