Malware Detection Using Process Activity Monitoring

Date
2023
Authors
Akindele, Sinmi
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

Over the years, researchers have developed different tools to detect malware. However, malware has only increased in complexity. Today, malware actors can evade static analysis through obfuscation and dynamic analysis through sandbox detection. This thesis is focused on using behavioral analysis for malware detection, specifically detecting environment-aware malware samples – the malware that checks the system environment before launching its most destructive actions. In this thesis, we propose a new behavioral malware detection method that extracts frequency counts from Application Programming Interface (API) calls and uses Random Forest (RF) and Support Vector Machine (SVM) machine learning models to train a binary classification function to distinguish between malware and benign processes. Based on extensive literature review, we selected 144 API calls made by environment aware malware. The call counts are then used as feature vectors to test and train the RF and SVM models. Also, we ran malware samples in a virtual environment and extracted the API call counts for each process. By extracting the frequency counts of each API call and using this as inputs for our RF and SVM models, we observed that the RF model achieves competitive performance with an accuracy of 97.65%. However, the accuracy of our SVM model is 84.84%, thus, this indicates that our RF model performs better than our SVM model. Prior research focuses on utilizing API call sequences as feature vectors to train machine/deep learning algorithms. However, we conducted our research using API call counts as feature vectors to investigate whether we could obtain higher detection accuracy. We hope that our dataset and method can be useful as a base architecture for the analysis of environment aware malware

Description
This item is available only to currently enrolled UTSA students, faculty or staff.
Keywords
Application Programming Interface (API), Environment-aware malware, Machine Learning, Malware detection, Malware dwell time
Citation
Department
Computer Science