DeepSec 2021 Talk: Firmware Surgery: Cutting, Patching and Instrumenting Firmware for Debugging the Undebuggable – Henrik Ferdinand Nölscher
Embedded systems can be challenging to analyze. Especially on automotive systems, many things that we take for granted on other software such as debugging and tracing do not always work. This is further complicated by watchdogs and peripheral processors, that go haywire when strict timing and communication requirements are violated. On some systems, debugging is even impossible because debugging resources such as pins are either used for something else or they don’t exist at all!
Assuming that code can be dumped, the solution for this can be emulation, however emulating a rich automotive system can be painful and many times, only few aspects of the system can be sufficiently modeled.
What if there was an in-between? How can we debug, fuzz and tamper embedded firmware without access to real-time debugging or emulation?
In this talk, I will show a tool that uses a simple but smart binary instrumentation method and a new, pythonic assembler to automatically patch large firmware binaries, enhancing them with interactive backdoors, as well as function- or basic-block trace capabilities.
Along the way, I share some tricks that can be used to make targets easier to work with (regardless of whether they’re being instrumented) and explore further applications outside of the automotive realm for the tool, which is released specifically for DeepSec.
We asked Henrik a few more questions about his talk.
Please tell us the top 5 facts about your talk.
Fact 1: This talk shows how to tamper, debug, and enhance embedded systems that are highly constrained because of missing/disabled debug functionality or timing requirements. This presentation focuses on bare-metal or RTOS-enabled embedded devices, but most concepts will apply to devices with more complex operating systems, too.
Fact 2: Finding bugs is easier when you can alter the firmware. Being stealthy about it, however, is challenging. In this talk, you will see a demo of static binary instrumentation in action, on an automotive control unit.
Fact 3: You will learn that your target has more usable memory than you think it does
Fact 4: As you probably know, not everything you do on one target (i.e. assembly code) can be transferred to another target. Let’s find the things that do transfer and leverage them!
Fact 5: Many systems employ safety-mechanisms such as watchdogs. Together, we’ll analyze which types of showstoppers we can find in our targets and discuss how to circumvent them.
How did you come up with it? Was there something like an initial spark that set your mind on creating this talk?
I was working on a challenging project and got tired of not being able to debug the system properly. After making some necessary changes to the hardware, I found myself patching firmware or generating payloads with hand-crafted shellcode. Soon, I discovered that today’s tools for turning assembly into bytes either only receive assembly in form of text strings or don’t support the architectures that I work on. Eventually, I got so fed up I ended up writing my own assembler and things got out of control.
Why do you think this is an important topic?
Inspired by the rich tooling available in the non-embedded world, I think we should spend time on finding techniques that a) generalize across many architectures and b) are easily applied.
Unfortunately, we can’t always justify spending several days or weeks for setting up our tooling. Static binary instrumentation can step in when debugging or emulation fails and (as others have also showed), can provide information that is used to build powerful, coverage-guided fuzzers. The result is a reduced time-to-bug and a less tired analyst.
Is there something you want everybody to know – some good advice for our readers, maybe?
As an analyst: Don’t spend too much time focusing only on one type of analysis.
As a designer: Attackers will tamper your system in every way imaginable. What does that mean for your product?
A prediction for the future – what do you think will be the next innovations or future downfalls for your field of expertise / your talk in particular?
On the tooling side, we can expect a growing set of open-source tools that either apply known-good techniques from the non-embedded world or replace older/commercial tools, perhaps with the help of extra hardware. On the target side, we might see more low-level malware and, as microcontrollers get faster and cheaper, we can expect more debugging- and reverse engineering countermeasures being applied. On top of that, there is demand for meeting increasingly strict cybersecurity requirements. In turn, we see a need for tools and techniques that help make black-box penetration tests easier or help implement high-quality vulnerability scanning tools for embedded devices.
Ferdinand has been very passionate about information security ever since he was young. He is specialized in hardware security and reverse engineering techniques and enjoys spending his time analyzing the most challenging security aspects of embedded systems. In the past, he has spoken at Usenix WOOT, Blackhat Arsenal and, along with his great colleagues, he completed many embedded security projects involving secure boot audits, fault injection attacks and binary reverse engineering. In the past, he worked at companies such as Nio and Code White while right now, he’s busy finding bugs and securing embedded systems at Noelscher Consulting GmbH.