Online Malware Detection in Cloud Auto-Scaling Systems Using Performance Metrics

Cloud computing is becoming increasingly popular among organizations. The Infrastructure as a Service (IaaS) cloud computing model has become an attractive solution because of the ability of reducing costs and improving resource utilization. Such cloud services are expected to be always available and reliable as per the Service Level Agreements (SLA) between the cloud service providers (CSPs) and their customers. Cloud ecosystems have also become attractive targets to attackers because of the massive amount of data residing on the cloud as well as the massive processing power that can be recruited for malicious intent. Thus, security is a very critical task in cloud ecosystems and the need of continuous security monitoring in the cloud is mandatory for detecting malicious activities.

This dissertation addresses the problem of online malware detection in cloud auto-scaling systems using performance metrics. First, we review the current state-of-the-art malware detection techniques in general with a focus on techniques that target cloud IaaS, specifically virtual machines (VMs). We find that malware detection techniques that target VMs lack taking advantage of cloud unique characteristics. Those techniques can be applied to VMs as well as stand alone servers with nothing specific about cloud.

We then propose a malware detection framework that leverages cloud unique characteristics (i.e. auto-scaling) using black-box features (performance metrics), where data are collected from outside the VMs by the hypervisor in an auto-scaling scenario (e.g. three-tier web architecture with scalability in place). Our approach assumes no prior knowledge of the installed applications on the VMs. In this work, a modified version of sequential K-means clustering algorithm is used to group similar VMs based on workloads (e.g. applications servers, web servers and database servers are three different groups). Then, malware is detected as anomalies when one VM of the same group exceeds a certain threshold.

Despite showing that highly active malware (e.g. ransomware) can be effectively detected by inspecting the performance and resource utilization metrics of VMs as a black-box, this approach is not as effective for detecting malware that maintains a low profile of resource utilization. Accordingly, we propose a white-box approach (where data are obtained from inside the VMs by either the hypervisor or pre-installed agents) for detecting such malware using 2d and 3d Convolutional Neural Networks (CNN). 3d CNN classifiers are introduced to partially mitigate the underestimated mislabeling problem.

The developed white-box approach achieved good results; however, it works only for single VMs. To leverage auto-scalability, we extended the previous approach to handle multiple VMs and introduced a new approach based on paired samples to accommodate for correlations between VMs.

We evaluate the proposed approaches on synthetic data collected from our OpenStack (a popular open-source cloud IaaS software) testbed based on a standard 3-tier web architecture with the ability to scale-up (when multiple copies of the server are spawned) and scale-down (where the number of copies are reduced) on demand.