Detecting hidden computer processes by deliberate resource exhaustion

Computer attackers often wish to retain control of a successfully compromised computer. Such retention requires evading detection by the computer system's owner and defensive software applications. Retaining control of a compromised computer (and subsequent exploitation of the compromised computer) also requires that the attacker run one or more processes on the computer. To evade detection, the attacker may hide such processes from legitimate system users and defensive software applications. Reliable detection of hidden processes, especially those hidden by novel tools or techniques, remains an unsolved problem. Existing hidden process detection tools require knowledge of the method(s) used to hide the processes or require a known clean version of the computer as a control. In this research, I present an approach for hidden process detection which does not require such prior knowledge or a control system. My approach, called the Method of Induced Observables, is based on the hypothesis that two computer systems, identical except one has an additional (and hidden) process, will produce nondiscriminatory observables under normal conditions, but may produce discriminatory observables when placed under abnormal conditions. I placed systems under the abnormal conditions of memory exhaustion and excessive process creation. I used the resulting induced observables from systems with and without hidden processes to construct classification models, which I then applied to systems employing previously unseen (holdout) process hiding methods. I demonstrated better than 95% hidden process detection accuracy on the holdout test sets.