Avleak: Profiling commercial anti-virus emulators through black box testing

Binary emulation is an essential part of the antivirus malware detection process. By running binaries in emulated environments, antivirus software is able to identify malware droppers and unpackers, as well as discover novel threats through behavioral heuristic analysis. Antivirus emulators are inherently limited by a number of factors speed, memory, processor power, and copyright law (preventing redistribution of actual Windows software) to name a few. As a result, AV emulators present many artifacts that allow malware to detect that it is being run under emulator analysis and thereby behave differently.;Consumer AV emulators are incredibly vulnerable to detection attacks, but discovering artifacts that can be exploited for detection can be a time-consuming process. Researchers can either spend significant time reverse engineering emulator code, look for artifacts in process memory dumps, or inject "decoy" malware into emulator engines. With decoy malware injection, a program is created that tests some condition of the emulator (ie: will it let a program allocate 500 MB of memory, will it let a program load a given DLL, does it return the right value for a given obscure API call) and either unpacks or does not unpack malware as a result. By checking if malware was detected, researchers are able to leak some information about the internal state of the emulator. Unfortunately, this process can be slow, and often only provides negative results that can be used for detection (ie: API call X is not correctly emulated, DLL X cannot be loaded).;In this thesis we present AVLeak, a novel framework that allows researchers to extract positive data out of emulators (ie: what files are on the file system, what processes are "running" according to the process list, what is the emulated system MAC address) with just a few lines of code and a few minutes of automated testing. Treating AV emulators as a black box, we are able to extract fingerprints without any manual binary reverse engineering. We demonstrate the application of our technique to up-to-date popular commercial AVs including Kaspersky, AVG, VBA32, and the popular BitDefender engine (licensed out to 20+ AV manufacturers). We show how the technique can be used to find a wide range of emulator fingerprints including environmental traits, incorrect OS API behavior, inconsistent network emulation, timing discrepancies, and emulated CPU "red pills".;Our work has applications in both offensive and defensive capacities. In offensive contexts, artifacts discovered through AVLeak may be used by malware authors to create malware which evades detection by antivirus software. AVLeak may also be used by AV manufacturers themselves to "red team" their products, evaluating the security of their emulators with adversarial testing.