Hiding Kernel Level Rootkits in Linux Environment

Date

2017

Authors

Honap, Amrita Milind

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

A rootkit is a piece of software that can be installed and hidden on a user's computer without their knowledge. It may be included in a larger software package or installed by an attacker who has been able to take advantage of a vulnerability on the user's computer. Rootkits are not necessarily malicious, but they may hide malicious activities. Kernel Level Rootkits are one such category of malwares which can hide themselves from detection, because of the type of privileges they hold. One of them could be access to the root i.e. working in the ring 0.

There are a few rootkit stealth techniques to perform malicious activities, which are system call hooking, kernel function hooking, kernel patching and Direct Kernel Object Manipulation (DKOM). Deploying these techniques at run time became really easy with the advent of Loadable Kernel Modules (LKMs) for Linux operating systems. The goal of all of these techniques is to carry out malicious execution a stealth mode.

The objective of this thesis is to demonstrate a hiding technique for kernel level rootkits from detection mechanisms which carry out static analysis. We have developed a technique based on the similar behaviors executed by different rootkits. The hiding mechanism makes use of Return Oriented Programming (ROP), which allows user to execute the malicious code in the presence of certain inbuilt security defenses and other detection tools. In this technique, an attacker diverts the control flow without injecting any new code in the program. We chain together short instruction sequences already present in a program's address space, each of which ends in a "return" instruction.

We have implemented a prototype tested it on custom detection tool (at static time) which performs a whole system scan checking for the specified malicious behaviors and returns the results if the rules (behaviors rootkit sample patterns) match with any object file on the system. Experimental results indicate that our prototype was effective in hiding kernel level rootkits.

Description

This item is available only to currently enrolled UTSA students, faculty or staff. To download, navigate to Log In in the top right-hand corner of this screen, then select Log in with my UTSA ID.

Keywords

buffer overflow, hiding, kernel, rootkits, ROP

Citation

Department

Electrical and Computer Engineering