Research And Implementation On Key Technology For Processing Branch Instructions In Decompilation

As the front-end of the decompilation system, the instruction decoder translates raw machine instructions into their assemble expression and constructs intraprocedural control flow graph and interprocedural call graph. The thesis researches into the key techniques of processing branch instructions during decoding phase, based on the needs of the static binary translation and the research of decompilation with the 64-bit source machine.Using the static binary translation system ITA that developed by us as an experimental platform, the thesis introduces the decoding algorithm of instruction decoder first, and then analyzes the IA-64 instructions characteristics and points the influence on the recursive decoding algorithm which basing on static control flow of a program caused by branch instructions during the decoding process. Basing on the deep analysis on the mechanism of generating indirect transfer instruction, we present a technique for recovering jump table for n-conditional branches in a machine and compiler independent way, and a processing method for non n-conditional indirect branches as well. Using the interpreter for reference and analyzing the respective characteristics of IA-64 and Alpha architectures, we construct a one-to-one original procedure address to target procedure address mapping table to recover the indirect procedure call in static way. In analyzing the mechanism of tail-call optimization, the thesis introduces three patterns to identify the interprocedural transfer type of direct transfer branch instructions when facing stripped binary code file relying on the context of tail call.Finally, this thesis presents the experimental reulst as well as the analysis on them. The test results prove that the key techniques of processing branch instructions are correct and effective.