Assessment of run-time malware detection through critical function hooking and process introspection against real-world attacks

Malware attacks have become a global threat to which no person or organization seems immune. Drive-by attacks and spear-phishing are two of the most prevalent and potentially damaging types of malware attacks, and traditionally both of these rely on exploiting a client application such as a browser or document viewer. This thesis focuses on the detection of a malware attack after a target application has been exploited, and while the malware is still executing in the context of the exploited process. This research presents an implementation of an application monitoring system that hooks critical functions and inspects characteristics of the process state in order to detect malware. The testing of the application monitoring system utilizes a corpus of exploits taken from a variety of real-world malware attacks, including several well-publicized examples. The resulting evaluation demonstrates the utility of function hooking and process state inspection techniques as a platform for detecting and stopping sophisticated malware attacks.