ROOTS 2018 Talk: Kernel-Assisted Debugging of Linux Applications – Tobias Holl, Philipp Klocke, Fabian Franzen
On Linux, most—if not all—debuggers use the ptrace debugging API to control their target processes. However, ptrace proves unsatisfactory for many malware analysis and reverse engineering tasks: So-called split-personality malware often adapts its behavior in the presence of a debugger, yet ptrace makes no attempt to hide from a target process. Furthermore, ptrace enforces a strict one-to-many relation meaning that while each tracer can trace many tracees, each tracee can only be controlled by at most one tracer. Simultaneously, the complex API and signal-based communications provide opportunities for erroneous usage.
Previous works have identified the newer uprobes tracing API as a candidate for building a replacement for ptrace, but ultimately rejected it due to lack of practical use and documentation. Building upon uprobes, we introduce plutonium-dbg, a Linux kernel module providing debugging facilities independent of the limitations of ptrace alongside a GDB-compatible interface. Our approach aims to mitigate some of the design flaws of ptrace that make it both hard to use and easy to detect by malicious software.
We show how plutonium-dbg’s design and implementation remove many of the most frequently named issues with ptrace, and that our method improves on traditional ptrace-based debuggers (GDB and LLDB) when evaluated on software samples that attempt to detect the presence of a debugger.
We asked Tobias, Philipp and Fabian a few more questions about their talk.
Please tell us the top 5 facts about your talk.
- We implemented a debugger using existing linux kernel infrastructure
- Alternative to ptrace API (the usual debugging interface), which has several design flaws
- Use of modern kernel features (uprobes, kprobes, etc.)
- Resists most approaches to detect debuggers
- Compatible to existing Debugger frontends (GDB) and their plugins (pwndbg)
How did you come up with it? Was there something like an initial spark that set your mind on creating this talk?
We encountered a program which ptraced itself, to detect the presence of a debugger. To allow for dynamic analysis, we asked ourselves if we could avoid this well-known mechanism.
Why do you think this is an important topic?
Some Malware also uses this trick to avoid analysis by security researchers and analysis tools in general. Thus, we aim for minimal interference with the target process, which also allows for other kinds of debugging. For example, we can investigate so-called Heisenbugs (bugs that occur in production only, not in debugging).
Is there something you want everybody to know – some good advice for our readers maybe?
Ptrace has major drawbacks, the biggest is that every target can be debugged by only one debugger. Others include the destruction of process order, poor performance in accessing memory and a non-intuitive API.
A prediction for the future – what do you think will be the next innovations or future downfalls when it comes to your field of expertise / the topic of your talk in particular?
Our approach removes obvious detection possibilities in what we believe to be an arms race of debugger detection and debugger stealthiness. Therefore we expect malware authors to develop new evasion techniques, which we will counter, as we think our capabilities are not maxed out yet.
Tobias Holl is a computer science student at TUM with a passion for reverse engineering and IT security. By day, he develops high-performance parallel software in C++, with a focus on computer vision and machine learning.
Philipp Klocke is a hacker, nerd and tech-enthusiast. He occasionally plays CTF and pursues a B.Sc. at the Technical University of Munich.
Since 2018 Fabian Franzen is a PhD student and researcher at the Chair of IT-Security of the Technical University of Munich (TUM). When he is not trying to teach his students the foundations of IT security, he is interested in various research topics. More specifically, these are reverse engineering, binary exploitation, Android security and improving systems security by introducing additional features to the Linux Kernel.