Hiding Kernel Level Rootkits in Linux Environment

Date
2017
Authors
Honap, Amrita Milind
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

A rootkit is a piece of software that can be installed and hidden on a user's computer without their knowledge. It may be included in a larger software package or installed by an attacker who has been able to take advantage of a vulnerability on the user's computer. Rootkits are not necessarily malicious, but they may hide malicious activities. Kernel Level Rootkits are one such category of malwares which can hide themselves from detection, because of the type of privileges they hold. One of them could be access to the root i.e. working in the ring 0.

There are a few rootkit stealth techniques to perform malicious activities, which are system call hooking, kernel function hooking, kernel patching and Direct Kernel Object Manipulation (DKOM). Deploying these techniques at run time became really easy with the advent of Loadable Kernel Modules (LKMs) for Linux operating systems. The goal of all of these techniques is to carry out malicious execution a stealth mode.

The objective of this thesis is to demonstrate a hiding technique for kernel level rootkits from detection mechanisms which carry out static analysis. We have developed a technique based on the similar behaviors executed by different rootkits. The hiding mechanism makes use of Return Oriented Programming (ROP), which allows user to execute the malicious code in the presence of certain inbuilt security defenses and other detection tools. In this technique, an attacker diverts the control flow without injecting any new code in the program. We chain together short instruction sequences already present in a program's address space, each of which ends in a "return" instruction.

We have implemented a prototype tested it on custom detection tool (at static time) which performs a whole system scan checking for the specified malicious behaviors and returns the results if the rules (behaviors rootkit sample patterns) match with any object file on the system. Experimental results indicate that our prototype was effective in hiding kernel level rootkits.

Description
This item is available only to currently enrolled UTSA students, faculty or staff.
Keywords
buffer overflow, hiding, kernel, rootkits, ROP
Citation
Department
Electrical and Computer Engineering