ThesisPDF Available

Instrumenting Existing System Components for Dynamic Analysis of Malicious Software

Authors:

Abstract

Analyzing unknown programs and determining their semantic and implementation details is a critical task for computer security. Especially in the context of malicious software it is an indispensable requirement for protecting systems and mitigating the effects of attacks and security breaches. The insights gained with the help of reverse engineering are necessary for developing protection mechanisms and security software, to clean up infected systems, and to forensically reconstruct the details of occurred attacks. Such analysis can be performed statically or dynamically, and both approaches have their advantages and drawbacks. While static analysis allows the examination of every single instruction of an application, it is time consuming and complex. Current malicious software further complicates it by using protection methods such as encryption, obfuscation, or virtual machines. Dynamic methods speed up the analysis process by supplying runtime values of relevant memory locations and register values. As a downside, they only keep track of one single execution path and are unable to deliver complete results with respect to code coverage. Nevertheless, they can be automated and also be utilized for generating behavior reports of unknown software. Only by using such methods it is possible to cope with the ten thousands of new malware samples that emerge every day. Often software emulators are used for dynamic analysis, since they provide full control and isolate the underlying system from the observed malware. Due to the high complexity of modern CISC architectures, emulation is never perfect, and hence can be detected and evaded. Therefore, our approach is to instrument real systems instead and we examine which hardware and software facilities can be used to that end. We start by utilizing the operating system’s memory management, namely by intervening with the MMU and modifying the page fault handler. By further enforcing illegitimate code to always reside in non-executable memory, we are able to intercept all attempts to execute it. Then we make use of the branch tracing facility of contemporary CPUs, which enables us to reconstruct the complete execution path after a program’s termination. We further realize API and system call hooking to obtain insight into the interaction between the observed malware and the operating system. Finally, we utilize hardware-virtualization and develop a novel form of intermodular transition monitoring to further improve our analysis. Our requirements for effective and efficient program analysis are transparency, isolation, soundness and performance. While all of our proposed methods are fast and sound, our hypervisor-based approach in particular fulfills these requirements to a high degree. By combining virtualization with module transition monitoring, we realize a valuable trade-off between executing performance and result quality. Its capability to analyze powerful 64-bit kernel rootkits in particular makes it a powerful and meaningful completion of our work.
Instrumenting
Existing System Components for
Dynamic Analysis of
Malicious Software
Dissertation zur Erlangung des Grades eines Doktor-Ingenieurs
der Fakult¨at f¨ur Elektrotechnik und Informationstechnik
an der Ruhr-Universit¨at Bochum
vorgelegt von
Carsten Willems
aus onchengladbach
Bochum, 10.06.2013
Tag der m¨undlichen Pr¨ufung: 29.04.2013
Gutachter:
Prof. Dr. Thorsten Holz, Ruhr-Universit¨at Bochum
Zweitgutachter:
Prof. Dr. Felix Freiling, Friedrich-Alexander Universit¨at Erlangen-N¨urnberg
Abstract
Analyzing unknown programs and determining their semantic and implementation
details is a critical task for computer security. Especially in the context of malicious
software it is an indispensable requirement for protecting systems and mitigating
the effects of attacks and security breaches. The insights gained with the help of
reverse engineering are necessary for developing protection mechanisms and security
software, to clean up infected systems, and to forensically reconstruct the details of
occurred attacks.
Such analysis can be performed statically or dynamically, and both approaches
have their advantages and drawbacks. While static analysis allows the examination
of every single instruction of an application, it is time consuming and complex.
Current malicious software further complicates it by using protection methods such
as encryption, obfuscation, or virtual machines. Dynamic methods speed up the
analysis process by supplying runtime values of relevant memory locations and
register values. As a downside, they only keep track of one single execution path and
are unable to deliver complete results with respect to code coverage. Nevertheless,
they can be automated and also be utilized for generating behavior reports of
unknown software. Only by using such methods it is possible to cope with the ten
thousands of new malware samples that emerge every day.
Often software emulators are used for dynamic analysis, since they provide full
control and isolate the underlying system from the observed malware. Due to the
high complexity of modern CISC architectures, emulation is never perfect, and hence
can be detected and evaded. Therefore, our approach is to instrument real systems
instead and we examine which hardware and software facilities can be used to that
end. We start by utilizing the operating system’s memory management, namely
by intervening with the MMU and modifying the page fault handler. By further
enforcing illegitimate code to always reside in non-executable memory, we are able to
intercept all attempts to execute it. Then we make use of the branch tracing facility
of contemporary CPUs, which enables us to reconstruct the complete execution path
after a program’s termination. We further realize API and system call hooking to
obtain insight into the interaction between the observed malware and the operating
system. Finally, we utilize hardware-virtualization and develop a novel form of
intermodular transition monitoring to further improve our analysis.
Our requirements for effective and efficient program analysis are transparency,
isolation, soundness and performance. While all of our proposed methods are fast
and sound, our hypervisor-based approach in particular fulfills these requirements to
a high degree. By combining virtualization with module transition monitoring, we
realize a valuable trade-off between executing performance and result quality. Its
capability to analyze powerful 64-bit kernel rootkits in particular makes it a powerful
and meaningful completion of our work.
i
ii
Zusammenfassung
Die Analyse unbekannter Software und Rekonstruktion darin enthaltener Funktio-
nalit¨at ist aus vielerlei Hinsicht eine wichtige Aufgabe. Insbesondere im Kontext
von Schadsoftware liefert sie wertvolle Erkenntnisse, um Schutzmechanismen und
Sicherheitssoftware zu entwickeln, bereits infizierte Systeme zu aubern und erfolgte
Angriffe forensisch aufzuarbeiten. Es kommen dabei sowohl statische als auch dy-
namische Verfahren zum Einsatz und beide haben ihre Vor- und Nachteile. ahrend
statische Methoden eine vollst¨andige Untersuchung amtlicher Programminstruktio-
nen erlauben, sind sie zeitaufwendig und kompliziert. Erschwert werden sie außerdem
durch Schutzmaßnahmen moderner Schadprogramme wie Verschl¨usselung, Codever-
schleierung oder dem Einsatz virtueller Maschinen. Dynamische Ans¨atze onnen die
Analyse beschleunigen, da sie konkrete Laufzeitwerte relevanter Speicherzellen und
CPU-Register zur Verf¨ugung stellen. Es onnen dabei jedoch immer nur einzelne
Programmpfade untersucht werden. Daf¨ur onnen dynamische Methoden jedoch
automatisiert und zur Verhaltensanalyse unbekannter Software eingesetzt werden.
Nur damit ist es letztendlich oglich, mit der aglichen Flut zehntausender neuer
Schadprogramme umgehen zu onnen.
aufig werden bei dynamischen Verfahren Software-Emulatoren eingesetzt, da diese
volle Ausf¨uhrungskontrolle bieten und das eigentliche System von der untersuchten
Software isolieren. Aufgrund der hohen Komplexit¨at moderner CISC -Architekturen
ist Emulation jedoch immer ungenau, was von Schadsoftware aufig erkannt und
ausgenutzt wird. Unser Ansatz ist daher die Verwendung von nativen Systemkom-
ponenten und wir untersuchen ausf¨uhrlich, welche verschiedenen Hardware- und
Software-Mechanismen daf¨ur instrumentalisiert werden onnen. Wir beginnen mit
dem Eingriff in die Speicherverwaltung von Betriebssystemen und interagieren dabei
mit der MMU und dem Page Fault Handler. Dann verwenden wir die Branch Tracing
Funktionalit¨at aktueller Mikroprozessoren, mittels derer auch nach Programmende
noch der komplette Ausf¨uhrungspfad rekonstruiert werden kann. Wir fangen da-
raufhin s¨amtliche Aufrufe von System APIs und System Calls ab, um die Interaktion
zwischen Schadsoftware und Betriebssystem zu beobachten. Schließlich kombinieren
wir Hardware-Virtualisierung mit einer neuartigen Beobachtungsgranularit¨at, bei der
wir Kontrollfluss-Transitionen zwischen einzelnen Programmmodulen betrachten.
Unsere Anforderungen an eine effiziente und effektive Programmanalyse sind Trans-
parenz, Isolierung, Korrektheit und Geschwindigkeit. ahrend alle vorgestellten
Verfahren schnell und korrekt arbeiten, erf¨ullt insbesondere der Hypervisor-basierte
Ansatz auch die anderen Anforderungen in hohem Maße. Durch die Kombination
von Virtualisierung und Modulgrenzen-
¨
Uberwachung bietet er eine hervorragende
Balance zwischen Ergebnisqualit¨at und Ausf¨uhrungsgeschwindigkeit. Vor allem die
ahigkeit damit auch gef¨ahrliche 64-bit Kernelrootkits analysieren zu onnen, macht
dieses Verfahren zu einem logischen und wertvollen Abschluss unserer Arbeit.
iii
iv
Acknowledgments
It is a great pleasure to acknowledge everyone who helped me writing this thesis
successfully. First of all I would like to thank my two advisors Prof. Dr. Thorsten
Holz and Prof. Dr. Felix C. Freiling. They both gave me the opportunity and
flexibility to solely focus on research topics that I was interested in, without any
pressure or dictation at all.
Besides that, they offered me productive environments at the Laboratory for
Dependable Distributed Systems of the University Mannheim and at the Chair for
Systems Security of the Ruhr-University in Bochum. While being there, I enjoyed
working with friendly and talented research staff and enthusiastic graduate students
and I am obliged to many of my colleagues who supported me, especially to Christian
Gorecki, Markus Engelberth, Philipp Trinius, and Jan Goebel.
Furthermore, I am very happy for participating in different challenging projects
together with Konrad Rieck, Ralf Hund, and Benedikt Driessen. It always has been
a pleasure for me to work with people that share the same outstanding enthusiasm
and expertise for security research.
I have been enlightened by many valuable discussions with Frank Boldewin, Jonas
Pfoh, Andreas Dewald, Thomas Schreck, Felix Groebert, Thomas Siebert, Ben Stock,
Felix Schuster, Christian Schneider, Sebastian Vogel, Tilo Mueller, Marco Cova,
Tyler McLellan, Tillmann Werner, and Georg Wicherski. I would like to show my
gratitude to these persons, since they make my daily work fun and give me that
satisfying feeling of belonging to some great community that is pulling in the same
direction.
I owe sincere and earnest appreciation to Chad Loeven, Alex Eckelberry, and Eric
Sites for their trust in me and their success in bringing my software CWSandbox
to the market. This lucky coincidence gave me the autonomy to follow my aims
without any economic constraints and provided me with the chance to get into touch
with so many interesting and helpful people all over the world.
Finally, I would like to express my gratitude to my parents, my siblings and their
partners, and especially to my friend Sandra Gillessen. My deepest thankfulness
goes to them for their love, understanding, and inspiration. Without their blessings
and encouragement, I would not have been able to either start or finish this work.
v
CONTENTS
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Topic of this Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 List of Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Background on Software Analysis and Protection 11
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Approaches to Software Analysis . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Brief Introduction to Program Analysis . . . . . . . . . . . . 13
2.2.2 State of the Art in Reverse Engineering . . . . . . . . . . . . 15
2.3 Software Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.1 Passive Protection Measures . . . . . . . . . . . . . . . . . . 20
2.3.2 Active Protection Measures . . . . . . . . . . . . . . . . . . . 23
2.4 Memory Protection and Exploitation Techniques . . . . . . . . . . . 27
2.4.1 Non-Executable Memory . . . . . . . . . . . . . . . . . . . . 27
2.4.2 Return Oriented Programming . . . . . . . . . . . . . . . . . 28
2.4.3 Address Space Layout Randomization . . . . . . . . . . . . . 30
2.4.4 Heap Spraying . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.5 JIT Spraying . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.6 Other Memory Protections . . . . . . . . . . . . . . . . . . . 33
2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3 Instrumenting Memory Management 35
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2.1 Preventive Measures . . . . . . . . . . . . . . . . . . . . . . . 37
vii
3.2.2 Detection of Illegitimate Code . . . . . . . . . . . . . . . . . 38
3.2.3 Extraction of Illegitimate Code . . . . . . . . . . . . . . . . . 39
3.3 Technical Background . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3.1 Windows Memory Objects . . . . . . . . . . . . . . . . . . . . 40
3.3.2 Windows Paging Data Structures and Algorithms . . . . . . 45
3.4 Model and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.1 Attacker Model . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.2 Illegitimate Code . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 Instrumenting the Page Fault Handler . . . . . . . . . . . . . . . . . 56
3.5.1 Enforcing an Invariant . . . . . . . . . . . . . . . . . . . . . . 57
3.5.2 Trusted Files and Functions . . . . . . . . . . . . . . . . . . . 57
3.5.3 Memory Protection Modifications . . . . . . . . . . . . . . . . 57
3.5.4 Custom Page Fault Handler . . . . . . . . . . . . . . . . . . . 59
3.5.5 Multi Version Dumping . . . . . . . . . . . . . . . . . . . . . 59
3.6 CWXDetector - System Description . . . . . . . . . . . . . . . . . . 60
3.6.1 Trusted Files and Functions . . . . . . . . . . . . . . . . . . . 60
3.6.2 Reconstructing the User Mode Call Stack . . . . . . . . . . . 62
3.6.3 Memory Protection Modifications . . . . . . . . . . . . . . . . 63
3.6.4 Custom Page Fault Handler . . . . . . . . . . . . . . . . . . . 65
3.6.5 Additional System Modifications . . . . . . . . . . . . . . . . 65
3.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.7.1 Analysis of PDF Documents . . . . . . . . . . . . . . . . . . 66
3.7.2 Benign and Malicious PDF Sample Sets . . . . . . . . . . . . 68
3.7.3 Detection Evaluation . . . . . . . . . . . . . . . . . . . . . . . 69
3.7.4 Extraction Evaluation . . . . . . . . . . . . . . . . . . . . . . 77
3.7.5 Additional Experiments . . . . . . . . . . . . . . . . . . . . . 79
3.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4 Instrumenting Processor Features 85
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3 Technical Background . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.1 Commonly used Software Emulators . . . . . . . . . . . . . . 89
4.3.2 Basic Principle: Self-Modifying Code and Atomicity . . . . . 90
4.3.3 Classical Emulator Detection Approach . . . . . . . . . . . . 91
4.4 Delusion Attacks on Software Emulators . . . . . . . . . . . . . . . . 93
4.4.1 Attack 1: The rep movs Instruction . . . . . . . . . . . . . 93
4.4.2 Attack 2: The invd Instruction . . . . . . . . . . . . . . . . 95
4.4.3 Attack 3: The leave Instruction . . . . . . . . . . . . . . . . 97
4.5 Binary Analysis with Branch Tracing . . . . . . . . . . . . . . . . . . 99
viii
4.5.1 General Approach . . . . . . . . . . . . . . . . . . . . . . . . 99
4.5.2 Branch Tracing on Intel . . . . . . . . . . . . . . . . . . . . . 101
4.5.3 Branch Tracing on AMD . . . . . . . . . . . . . . . . . . . . 103
4.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.6.1 Experiment 1: Binning of Malicious PDF Documents . . . . 104
4.6.2 Experiment 2: Enriching Branch Tracing Logs . . . . . . . . 108
4.6.3 Experiment 3: Practical Delusion Attack with a PDF File . . 110
4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5 Instrumenting Code Hooking 113
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.3 Technical Background . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.3.1 Types of Control Flow Transitions . . . . . . . . . . . . . . . 117
5.3.2 Intercepting Control Flow Transitions . . . . . . . . . . . . . 119
5.3.3 Excursion: Windows User Mode Code Hooking . . . . . . . . 122
5.3.4 Code Injection . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.3.5 Excursion: Windows Code Injection . . . . . . . . . . . . . . 127
5.4 CWSandbox - System Description . . . . . . . . . . . . . . . . . . . 130
5.4.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.4.2 Controlling the Monitored Processes . . . . . . . . . . . . . . 132
5.4.3 Monitoring API Calls . . . . . . . . . . . . . . . . . . . . . . 133
5.4.4 Hiding and Protecting the System . . . . . . . . . . . . . . . 135
5.4.5 Generating High Level Analysis Reports . . . . . . . . . . . . 136
5.4.6 Improving the Report Quality . . . . . . . . . . . . . . . . . . 139
5.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.5.1 First Instance of the Malware Sample . . . . . . . . . . . . . 142
5.5.2 Second Instance of the Malware Sample . . . . . . . . . . . . 144
5.5.3 Infected Explorer Process . . . . . . . . . . . . . . . . . . . . 146
5.5.4 Other Infected Processes . . . . . . . . . . . . . . . . . . . . . 150
5.6 Beyond User Mode Hooking . . . . . . . . . . . . . . . . . . . . . . . 151
5.6.1 Drawbacks of User Mode Hooking . . . . . . . . . . . . . . . 151
5.6.2 Excursion: Windows Kernel Mode Code Hooking . . . . . . . 153
5.6.3 Drawbacks of Kernel Mode Hooking . . . . . . . . . . . . . . 156
5.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6 Instrumenting Hardware-Hypervisors 159
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.3 Technical Background . . . . . . . . . . . . . . . . . . . . . . . . . . 163
6.3.1 Virtualization on x86/x64 . . . . . . . . . . . . . . . . . . . . 163
ix
6.3.2 Two-Dimensional Paging . . . . . . . . . . . . . . . . . . . . 164
6.3.3 Memory Regions . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.4 Hypervisor-Assisted System Monitoring . . . . . . . . . . . . . . . . 167
6.4.1 Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
6.4.2 General Approach . . . . . . . . . . . . . . . . . . . . . . . . 170
6.5 CXPInspector - System Description . . . . . . . . . . . . . . . . . . 173
6.5.1 Tracking Memory Regions . . . . . . . . . . . . . . . . . . . . 174
6.5.2 TDP Clone Management . . . . . . . . . . . . . . . . . . . . 174
6.5.3 Virtual Machine Introspection . . . . . . . . . . . . . . . . . 176
6.5.4 Enforcing Transparency . . . . . . . . . . . . . . . . . . . . . 178
6.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
6.6.1 Experiment 1: Analysis of the TDSS/TDL4 Rootkit . . . . . 178
6.6.2 Experiment 2: Spotting Hot Memory Regions . . . . . . . . . 180
6.6.3 Experiment 3: Measuring Performance . . . . . . . . . . . . . 185
6.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7 Summary and Conclusion 189
List of Figures 193
List of Tables 195
List of Listings 197
List of Acronyms 199
x
CHAPTER
ONE
INTRODUCTION
1.1 Motivation
Analyzing unknown software binaries and identifying their inner semantics is an
important task for various reasons. In most cases, no source code is available for the
software utilized at companies, organizations, or in personal environments. Programs
may be bought from external suppliers or downloaded from the Internet. Therefore,
without further analysis one has to trust the developers of the software about their
implementation details [254]. Obviously, the real functionality may differ from the
proposed one, either by intent or unintentionally in case of software faults. Even if
the source code is at hand, the executable generated from it may have a (partially)
different semantic due to modifications the compiler or linker has introduced [16].
These may stem from code transformations such as optimization or may be caused
by maliciously manipulated building tools.
One very important category of software that needs to be analyzed is malicious
software (short: malware). The knowledge gained from such analysis is necessary
to protect computer systems from new infections, to disinfect systems that have
already been exploited and to forensically reconstruct incidents to gain intelligence
about their originators and which data has been stolen. To that end, two important
questions have to be solved: how did the malware get into the system and what did
it do after gaining control over it?
The actual analysis can be performed by different methods, that all differ in
complexity, performance and result quality. If one solely inspects the machine
instructions without actually executing them, it is called a static analysis. This
approach offers the biggest result coverage, since every single operation can be
examined and no secret functionality can be hidden from it. Nevertheless, it also
comes with the highest complexity, since all observations have to be done on symbolic
1
Chapter 1 Introduction
and abstract values only. Furthermore, the code of the examined software may be
obfuscated, compressed or encrypted [175]. In that case, static analysis becomes even
more complex and more time-consuming, since the original and hidden instructions
need to be recovered first. One very effective obfuscation example is the usage
of so-called virtual machine packers [10]. With their help, original code is never
reconstructed, but instead transformed into a complete new virtual machine program
that is only interpreted during runtime. To summarize, static analysis is powerful,
but very complex and extremely time-consuming.
While the static analysis of only one single program is already complex and labori-
ous, an enormous amount of new malicious files appear every day. In 2012 antivirus
vendors claimed to receive up to 100,000 [85,164, 245] new malware samples every
day. These samples are not all unique and new programs, but mostly repackaged and
slightly modified variants of existing ones, specifically crafted to remain undetected
by existing malware signatures. Nevertheless, these variants need to be detected by
security products as well and, hence, it is necessary to extend and adapt the existing
signatures. Besides that, it is very important to determine the new and unknown
threats and examine them in detail. Obviously, no AV company can provide sufficient
resources to manually analyze all of these new viruses, worms, bots, trojans, or
rootkits and more efficient analysis methods are indispensable to keep pace with.
A common way to ease the analysis is to actually execute (parts of) the examined
code to obtain concrete context states, i.e., the values of the CPU registers or certain
data structures in memory. This approach is called dynamic analysis. Although it is
much faster and less complicated than its static counterpart, it comes with some
problems and drawbacks, too. First of all, it is evident that one program run only
reveals the semantic of one particular execution path. Hence, its results are always
incomplete with respect to the semantics of the program and instead only reflect the
semantics of the particular runs taken into account. Furthermore, malware often
tries to actively sabotage dynamic analysis tools or frameworks. This is possible as
they are normally running on the same machine as the malware itself. For instance,
many malware samples try to detect if they are executed within a debugger or inside
an artificial analysis environment and if they detect that they are under observation,
they show a different behavior, e.g., by terminating instantly or just performing
some bogus operations. Sometimes it is even possible for the malware to evade
the analysis and affect the underlying system without being noticed. Nevertheless,
dynamic analysis is a very powerful methodology and capable of delivering high
quality results in very short time and also can be fully automated.
2
1.2 Topic of this Work
1.2 Topic of this Work
In this thesis we examine different dynamic analysis techniques that all are based
on using existing system components. Deliberately we refrain from using software
emulators or abstract interpretation techniques. While these latter approaches are
capable of generating valuable data, they mostly lack of performance and suffer from
imperfect simulation. We have developed and evaluated efficient implementations
of existing methods and novel approaches that, as far as we know, had not been
utilized so far. The target platform of our research is the x86/x64 architecture
and the Microsoft Windows operating system. However, we utilize hardware and
software features that are available on all commodity systems and, hence, can be
easily transferred. Although we solely focus on the analysis of malicious software,
the techniques and approaches can be used for all software in general. The reason
for focusing on malware is the real-world relevance and impact of this topic. Due to
the vast amount [244] of daily new malware samples and their heavy utilization of
various obfuscation and protection techniques, efficient and effective analysis tools
and frameworks are highly required.
The starting point of our research was the most intuitive form of dynamic analysis:
monitoring each single instruction while it gets executed on the CPU. It became
clear very quickly, that this approach has too many drawbacks and limitations. First
of all, intercepting the execution of a program or complete system for every single
instruction poses a huge computation overhead and results in very bad performance.
Furthermore, this approach produces an incredibly huge amount of data that has to
be processed and interpreted appropriately. Especially the reconstruction of higher
level semantics from a complete sequence of executed machine instructions is a
tremendously complicated task. Finally, single stepping is traditionally performed
with the help of a debugger or with a process/system emulator and, therefore, can
be easily detected by the observed code. Starting from these drawbacks, we come up
with an informal list of requirements that should be met by an efficient and effective
dynamic malware analysis framework:
The interception frequency should be as low as possible in order to limit the
performance impact of the monitoring.
The amount of monitored data should be as small as possible to allow its
efficient processing and interpretation.
The kind of monitored data should allow the reconstruction of high level
semantics of the observed code.
The used monitoring facility should be hard/impossible to detect or evade.
3
Chapter 1 Introduction
With these requirements in mind, we take a look at the existing different hard-
ware and software facilities that are usually involved in program execution on a
contemporary operating system and examine which of them can be instrumented
to achieve the desired goals. Figure 1.1 shows a schematic overview of the involved
parts, correlates them to our different research topics and states in which chapters
they are discussed. In the following, we refer to all handled topics and describe the
related contributions of our research.
Hypervisor
Userland
OS LIBRARIES
OS
SERVICE
OS
SERVICE
OS
SERVICE
APPLICATION
Kernelland
OS kernel
driver
driver
driver
driver driver
driver
driver
Hardware
CPU
MMU devicedevicedevice
VMM
vmexit
vmenter
Chapter 6:
Hypervisor
Instrumentation
Chapter 4:
Processor
Instrumentation
Chapter 3:
Memory Management
Instrumentation
Chapter 5.4:
API Call
Monitoring
Chapter 5.6:
System Call
Monitoring
Figure 1.1: Topics of this Thesis
1.3 Contributions
The contributions of this thesis are described in the following.
Chapter 2: Background on Software Analysis and Protection.
In the introduc-
tory background chapter of this thesis, we describe the state of the art
1
in Reverse
Engineering in a detailed manner. We give a comprehensive overview of existing
static and dynamic analysis approaches and point out the corresponding publi-
cations that are relevant. Furthermore, we give a broad survey on the analysis
1
with respect to the beginning of 2012
4
1.3 Contributions
countermeasures that software uses for mitigation. We also describe the applicable
methods to overcome these mitigation attempts in the ongoing arms race between
developers of benign and malicious software on the one hand and analysts on the
other. Besides that, we present contemporary hardening measures that are used in
operating systems and applications to prevent malicious code execution. As this
is a very broad field, the given overview is far from complete and only covers the
methods that are relevant to the following chapters.
Chapter 3: Instrumenting the Memory Management.
We start with a novel
generic and fully automatic approach to detect and extract illegitimate code by
instrumenting the memory management features of an operating system. Illegitimate
code is also known under the term shellcode and usually embedded within other data,
e.g., in a malicious network packet or a specially crafted PDF or Word document. By
exploiting a vulnerability in the parsing application, the attacker diverts the control
flow to the embedded malicious instructions, which in turn install or download the
real malware. To detect and extract these malicious instructions automatically, we
modify essential parts of the OS memory manager. Roughly speaking, the idea is to
mark certain memory pages as non-executable, whenever illegitimate code may be
contained in them. This ensures that upon execution of code in these regions the
page fault handler of the operating system is invoked and we can react appropriately
on the execution attempt.
For actually implementing this approach, we need a very detailed understanding
of the operating system’s memory management. Since we are focusing on Windows
and this is not an open-source system, we first have to reverse engineer parts
of the implementation of the related facilities. The findings of this examination
are explained in detail, covering the involved data structures and the algorithms
operating on them. With their help we were able to develop and present CWX-
Detector, the Windows implementation of our method. We successfully evaluate
this tool using malicious PDF documents as example and show that we can improve
application specific state-of-the-art analysis tools. In addition, our generic approach
can also be applied to other kinds of malicious documents and network packets as
we demonstrate in three case studies.
Chapter 4: Instrumenting Processor Features.
With the help of CWXDetector
it is possible to automatically detect the execution of illegitimate code and, hence, to
divide a set of unknown files into two subsets: one that contains files with embedded
malicious code and another without. Nevertheless, without further investigation it
is not possible to learn anything about the malicious semantic contained in the files.
In our second presented method, we are especially interested in the root cause of
the vulnerability that is exploited to divert the control flow. To that end, we utilize
5
Chapter 1 Introduction
the Branch Tracing facility of modern x86/x64 processors that records runtime
information about each performed branch instruction. With that logged data we
are able to reconstruct the full execution path. To demonstrate the effectiveness
and applicability of our approach, we show how our method is sufficient to analyze
malicious PDF documents. In an empirical evaluation, we demonstrate that the
branch tracing results can be used to automatically cluster similar vulnerabilities
which are exploited within the analyzed documents: a set of 4,869 PDF documents
can be clustered into eight different root causes based on the analysis results of our
tool.
We further show that the usage of hardware features for malware analysis is
able to outperform software methods under certain circumstances. To that end,
we introduce several novel mechanisms by which an attacker can delude a software
emulator. In contrast to existing detection approaches that perform a dedicated
test on the environment and combine the test with an explicit conditional branch,
our detection mechanisms introduce code sequences that have an implicitly different
behavior on a native machine when compared to an emulator. This inhibits the
correct analysis of binaries that use such instruction sequences. However, we show
that hardware features cannot be fooled by such sequences and branch tracing data
indeed reflects the real semantics of the examined code.
Chapter 5: Instrumenting Code Hooking.
The approaches shown so far can
be effectively used to detect the execution of illegitimate code and backtrack the
execution path that lead from executed shellcode to the exploited vulnerability.
However, they are inappropriate to realize an efficient examination of complex
malware programs, since they do not provide any insights into the executed operations.
For instance, they are able to log the memory addresses of executed code, but are
incapable of delivering further information such as function parameters or return
values. Besides that, the methods so far are inflexible with respect to which triggers
can be used to intercept the control flow. On the contrary, this can be realized with
the help of a behavior analysis, which monitors the execution of an unknown binary
by intercepting solely its interactions with the operating system. Such interactions
are normally performed by either calling functions of the system’s Application
Programming Interface (
API
) or directly issuing system calls into its kernel. By
intercepting these system calls and redirecting execution to so-called hook functions,
each call and its parameters can be examined easily.
We first explain the technical background of code hooking by showing what kind of
control flow transitions can be manipulated with the help of which techniques. We
then describe CWSandbox, an efficient and powerful malware analysis framework
that utilizes function inline hooking of relevant system API calls. After presenting
a comprehensive behavior analysis of a current malware sample, we discuss the
6
1.4 List of Publications
general drawbacks of user mode hooking. Finally, we show techniques and methods
to overcome these drawbacks by moving the control flow interception completely
into the kernel of the operating system.
Chapter 6: Instrumenting Hardware-Hypervisors.
Our final method combines
the utilization of modern CPU features with the effectiveness of monitoring system
function calls. To this end we utilize a hypervisor to perform the actual monitoring
and develop a powerful method to reduce drastically the amount of information
that needs to be logged. The main advantage of the proposed method is that all
modifications are done on the outside of the analyzed system, i.e., it does not have
to be modified at all. It thus offers a maximum of transparency and isolation.
Essentially our approach is based on the concept of Currently Executable Pages,
i.e., we closely control which memory regions are currently active and use this
information as a basis for our analysis. We demonstrate how the Two-Dimensional
Paging mechanism available on current x86/x64 architectures can be used to analyze
a full system (or only certain processes/memory regions). We further present CXP-
Inspector, a prototype implementation of the proposed methodology. To the best
of our knowledge, this tool is the first malware analysis environment capable of
analyzing both user and kernel mode malware for machines powered by a 64-bit
processor. Our evaluation shows how it can be efficiently used to analyze the
execution of the TDSS/TDL4 rootkit. Beyond applications in the area of malware
analysis, we further show how this method can be used to perform system profiling,
e.g., measuring how much time is spent in which modules/memory pages of a running
application or system.
1.4 List of Publications
This thesis is based on previous academic publications but also contains novel and
unpublished material. The background chapter on reverse code engineering and
program protection is based on joint work together with Felix C. Freiling and was
originally published in the Information Technology journal [281]. Chapter 3 on
utilizing the OS memory management for detecting illegitimate code execution
results from a publication [282] at the ACSAC conference in 2012, in cooperation
with Felix C. Freiling and Thorsten Holz, and from a technical report [280] created
in 2011. In Chapter 4 we present another publication [284] from the same conference.
In a collaborated work with Ralf Hund, Andreas Fobian, Dennis Felsch, Thorsten
Holz and Amit Vasudevan we introduce a new form of malicious code sequences that
can be used to delude software emulators. To encounter this problem we instrument
the branch tracing feature of modern x86/x64 processors for analyzing such code.
The material on code hooking in Chapter 5 is unpublished yet, but to some extent
7
Chapter 1 Introduction
inspired by early work together with Thorsten Holz and Felix C. Freiling that resulted
in an IEEE Security & Privacy journal article [283] and one chapter from the book
Botnets. The killer web app [219], both from 2007. Finally, our hypervisor-based
approach, described in Chapter 6, was developed with Ralf Hund and Thorsten Holz
and is not published yet but still in submission.
Besides the mentioned publications, our research was influenced by many other
projects from the last six years. While being out of the focus of this thesis, they still
had a vast impact on the way and direction of our work. Therefore, in the following
we briefly present these other projects that have been published in several journals
and conferences.
Together with Jan Goebel and Thorsten Holz, we examined autonomous spreading
malware in a university environment [102]. With the help of CWSandbox we analyzed
2
,
454 malware samples and presented comprehensive results regarding their network
communication. Another application of this analysis tool was developed with Markus
Engelberth and Thorsten Holz [84]. In this case, the generated behavior reports were
used to create security policies for automatically discriminating between malicious
and benign Office documents.
More sophisticated applications of CWSandbox were done later to automatically
cluster unknown malware into groups of samples that show the same behavior and,
hence, belong to the same malware family. This long lasting research resulted in
several publications. A first version [208] was created in cooperation with Konrad
Rieck, Thorsten Holz, Patrick Duessel and Pavel Laskov. While already providing
good results, the work was later refined and for that purpose the Malware Instruction
Set (MIST) specification was created [256] together with Philipp Trinius, Thorsten
Holz and Konrad Rieck. With the help of this instruction set, we were able to
enhance our clustering approach crucially in a joint work [209] with Konrad Rieck,
Philipp Trinius, and Thorsten Holz.
Not directly related to malware and behavior monitoring, but nonetheless con-
nected to dynamic analysis in the security context, our next work focused on the
automatic identification of cryptographic routines in unknown software. In combina-
tion with Felix Groebert and Thorsten Holz [105] we applied existing methods and
developed new ones to locate such cryptographic primitives by means of instruction
tracing. To some extent we were capable to identify even the used implementation
variant of the applied algorithm. This basis was later used to identify cryptographic
routines in the firmware of satellite phones. In collaboration with Benedikt Driessen,
Ralf Hund, Christof Paar and Thorsten Holz we were able to reverse engineer and
partially break the crypto-algorithms used in the GMR-1 and GMR-2 standards [79].
As a continuation of our hardware- and processor-related research [284], we
utilized timing side channels to actually break the kernel ASLR protection of current
Windows operating systems [115]. Together with Ralf Hund and Thorsten Holz we
utilized the Translation Lookaside Buffers and other caches from modern x86/x64
8
1.5 Outline
hardware to infer the mapping of critical kernel structures, such as the kernel image
and loaded kernel drivers.
1.5 Outline
The remainder of this thesis is structured as follows. In Chapter 2 we start with
a detailed overview on reverse code engineering in general, were amongst others
we contrast existing static from dynamic analysis methods. We further present
techniques and approaches that current software uses to resist and distort its analysis
and how these can be encountered.
After that, Chapter 3 describes our approach to utilize the memory management
of an operating system to automatically detect and extract illegitimate code during
its execution. The idea is to divide a system’s memory into trusted and untrusted
locations and enforce that untrusted code is automatically placed in non-executable
memory. By further modifying the page fault handler of the OS, we are able to
intercept all attempts to run illegitimate code. Besides developing our general
approach, we also present CWXDetector, an actual Windows implementation of
it. We use it to perform a large scale analysis of malicious PDF documents for
evaluating its effectiveness and accuracy.
Chapter 4 covers the utilization of processor features for efficient and reliable
system monitoring. We start by introducing so-called delusion attacks, which are
short instruction sequences that have an inherent different behavior when being
executed either on a native system or within a system emulator. Using hardware
facilities such as Branch Tracing allow the recording of all taken branch operations
and cannot be tricked like an emulator. We extent CWXDetector by this feature and,
thus, become capable of automatically clustering the malicious documents from the
previous chapter into groups from which all elements exploit the same vulnerability
within the used PDF viewer application.
In Chapter 5 we present the fundamentals of code hooking, one of the main
parts of our research. We first cover the theoretical aspects, such as the different
types of code transitions and look at the different possibilities to intercept them.
Subsequently, we present CWSandbox, one malware analysis framework that has
been become a commercial product meanwhile and is licensed by many companies
and organizations worldwide. We also discuss the drawbacks and limitations of
user mode hooking and present possible improvements that move all interception
mechanisms completely into the kernel.
Chapter 6 then presents our final approach that actually combines the utilization
of modern CPU features with code hooking for efficient malware analysis. To
that end we modify an existing hypervisor software and develop a novel powerful
monitoring strategy. By focusing solely on control flow transitions between different
9
Chapter 1 Introduction
memory modules, we gain a previously unseen trade-off between performance and
data granularity. After presenting our implementation CXPInspector, we evaluate it
by several experiments. Amongst others, we use it to analyze the behavior of the
infamous 64-bit kernel bootkit TDSS/TDL4.
Finally, we conclude this thesis in Chapter 7 by summarizing the presented material
and pointing out necessary and interesting topics for related future research.
10
CHAPTER
TWO
BACKGROUND ON SOFTWARE ANALYSIS AND
PROTECTION
2.1 Introduction
Before we can go into detail and present the different dynamic analysis approaches
we have developed and evaluated, we need to handle essential background knowledge
that is a necessary prerequisite for the following chapters. We start by giving a very
brief introduction into program analysis in general and then present the current state
of the art
1
in Reverse Engineering (
RE
) techniques. To further categorize them,
we follow their common division into static and dynamic approaches. Although
all described instrumentation methods of this thesis solely fall into the group of
dynamic analysis, the theoretical foundation of static analysis cannot be neglected
when working with binary code and trying to identify the semantics of unknown
applications. After introducing into
RE
, we present the protection techniques
that software applies to resist its analysis. These are grouped into passive and
active measurements, from which the former aim at resisting the static analysis and
the latter respectively the dynamic approaches. For each presented anti-analysis
measurement, we also describe the mitigation techniques used to overcome it. Finally,
we take a look at current protection techniques that contemporary operating systems,
applications and building tools use to prevent the execution of malicious code. Since
there is a constant arms race between attackers and benign developers, a vast number
of such hardening techniques and their counterparts exist. Hence, we only discuss
those methods that are relevant to the approaches and methods covered in this
thesis.
1
with respect to the beginning of 2012
11
Chapter 2 Background on Software Analysis and Protection
This chapter is based on joint work together with Felix C. Freiling and was originally
published in the Information Technology journal [281]. It is organized as follows:
Section 2.2 starts by putting
RE
into the broader context of program analysis and
then presents the current state of the art in static and dynamic program analysis.
After that, Section 2.3 describes the active and passive countermeasures that software
utilizes to resist its analysis and points out relevant academic publications and tools.
Finally, Section 2.4 covers contemporary protection techniques that are used by
operating systems and applications to prevent malicious code execution. Besides the
actual prevention approaches, also countermeasures are presented that are commonly
used by malware to overcome them.
2.2 Approaches to Software Analysis
The term engineering generally refers to a constructive process, in which hardware
or software systems are built and brought into operation for commercial or private
purposes. Many such systems exist today for which we do not have access to the
details of their construction process, for example because of lost documentation or
legal requirements (intellectual property protection). This has promoted the advent
of a corresponding deconstructive process commonly known as Reverse Engineering
or simply reversing. In general, reverse engineering refers to the process of “analyzing
a subject system to identify the system’s components and their interrelationships
and create representations of the system in another form or at a higher level of
abstraction” [52]. While the term originally was introduced with respect to hardware
devices and machines, nowadays it usually relates to reverse code engineering or
protocol/file format reversing [44, 65, 70, 133, 152, 255, 275, 286]. Protocol reverse
engineering tries to recover the structure of a secret network protocol. Similarly, file
format reverse engineering tries to reconstruct non-public file specifications from raw
data files.
In the past, RE has often been associated with shady and illegitimate uses in the
context of software piracy. This bad image is probably due to public advertising
and legal campaigns of large companies trying to protect their intellectual property.
Today, there are an increasing number of totally legitimate applications of RE, such
as closed source auditing, vulnerability research and the analysis of malware. Famous
examples for successful applications of RE include the recovery and recreation of
the original IBM BIOS, which empowered the enormous success of the IBM PC
compatible computers, the reversing of the Microsoft SMB/CIFS protocol which
lead to the development of Samba [255], the disclosure of the Sony Rootkit by
Russinovich [214], and the recent reversing of the Stuxnet worm [89].
These successes have given rise to an active academic research community that,
however, is still conceptually dispersed and therefore hard to enter. In this chapter,
12
2.2 Approaches to Software Analysis
we attempt to lower the entrance barrier to RE research by surveying and structuring
this exciting research area. We focus on techniques to recover the control and data
flow of programs for which no source code is available and which, in the case of
malware, often are further protected to resist the analysis. More specifically, we
concentrate our survey around RE of binary native code for the x86/x64 architecture
and neglect byte-code like languages such as Java or .NET. Nevertheless, most of
the techniques discussed in the following can be transferred to other architectures
and operating system as well. We also disregard the legal aspect of RE, a research
area of its own [25, 29,109, 218].
2.2.1 Brief Introduction to Program Analysis
RE is strongly related to program analysis [184], which tries to statically predict
approximations about the run-time behavior of a program or its reachable states. One
of its requirements is computability and, hence, often under- or over-approximations
are used. It is mainly used to support compiler optimization, perform automated
program verification and assist in security research. The main fields are control flow
analysis, data flow analysis, and type recovery.
When observing the control flow, it is examined which code locations can be
reached from or lead to which other parts of the program. In data flow analysis one
is interested in the dependency and relationship between different data structures.
Often the semantics of the used operations on the data are neglected and only data
propagation is observed. On machine level the CPU only operates on integer and
float values. Therefore, all complex data structures have to be mapped onto those
atomic data types during compilation. Recovering the original data structures is
essential for understanding the analyzed program and, hence, performed during
data type recovery. To support this, the data types of memory accesses have to
be identified, e.g., by observing the access patterns [15, 231], inferring from the
prototypes of called functions [153], or by applying unsupervised machine learning
methods on memory images of running processes [68].
One essential model used in program analysis is the Control Flow Graph (
CFG
) as
shown in Figure 2.1. Its nodes consist of basic blocks, which constitute sequentially
executed instructions and have exactly one single entry and one exit node. The
connecting edges represent the control flow between them and, thus, two nodes are
connected if there is a branch from one to the other. Figure 2.2 displays another
important graph type that is often used. In a Call Graph (
CG
) each node represents
one function of an application and each directed edge stands for a call-relationship
between them, pointing from the caller to the callee. Constructing such graphs is
a non-trivial task for executables that are only available in binary form. In a first
step the program has to be disassembled, i.e., the particular machine instructions
and their encoding have to be resolved. This is exceedingly complicated on Complex
13
Chapter 2 Background on Software Analysis and Protection
Figure 2.1: Control Flow Graph Example
Instruction Set Computer (
CISC
) architectures, since they often have a variable
length instruction set and the instruction boundaries need to be determined first. In
a second step the function boundaries have to be identified, which is impeded by
overlapping and cross-jumping functions. A related effect arises through optimization
techniques, such as selectively placing frequently used function chunks near to each
other for performance reasons. Additional problems are caused by indirect branches,
since their effective destination address are unknown. Data flow techniques can
determine their possible destinations, e.g., Value Set Analysis (VSA) [15] can be
applied to statically compute the set of possible values that a certain memory location
or register may contain during runtime.
Another related structure is the Program Dependence Graph (
PDG
), which comes
in different forms, e.g., Code Dependence Graph (
CDG
) or Data Dependence Graph
(
DDG
).
CDG
s and
DDG
s are used for mapping numerous program analysis problems
to the realm of graph theory. One powerful related method is slicing [28,139, 277],
in which program complexity is reduced by eliminating all instructions that do not
affect the reaching of a particular program state. Modern slicing techniques are
based on
PDG
s and constitute a reachability problem. In backward slicing one
wants to know what variables or operations influence the value of a variable at one
given statement. In forward slicing it is verified which variables or statements are
affected by a variable value at some given point. Traditionally slicing was restricted
to one-procedure programs, but newer techniques allow inter-procedural slicing as
well [122, 139, 204]. Differential slicing [127] takes two different runs of the same
binary into account and compares them towards their different input values. By that,
the input and control flow differences can be identified that distinguish the runs.
14
2.2 Approaches to Software Analysis
Figure 2.2: Call Graph Example
Finally, dynamic slicing [143] reduces the complexity by taking the concrete values
from one particular run into account. Hence, several conditional statements can be
ignored and removed from the graph since the tested variable values are known.
2.2.2 State of the Art in Reverse Engineering
While fundamental limits of program analysis arise from the undecidability of the
halting problem [258] and Rice’s theorem [206], RE suffers from additional problems
beyond that. First of all, when reversing programs, the source code normally is not
available. Even if it is at hand, it cannot be trusted due to compiler optimizations [16]
and, therefore, the assembly output has to be taken into account. Typical difficulties
that complicate RE of binaries are
compilation imposes information loss,
lack of symbol names and comments,
compiler optimizations complicate resulting code,
hardware knowledge is necessary to understand code,
data structures have become chunks of bytes, and
programs try to protect from analysis.
In general there are two different approaches: static and dynamic analysis. Static
analysis tries to model all possible program behaviors and, hence, operates on a
high level of abstraction. For that purpose the program under observation is not
15
Chapter 2 Background on Software Analysis and Protection
executed, but all examinations are performed in an abstract way, mostly by using
CFG
s,
CG
s or
PDG
s. By that, the analysis is sound and covers all possible program
states, while on the other hand, it is very complex and time-consuming and may
encounter serious problems in the case of obfuscated and protected binaries.
In contrast to static analysis, dynamic analysis has its origin in program testing
and profiling and takes actual data values from one or more program executions
into account. Since concrete values are at hand, no abstraction and approximation
has to be used. Nevertheless, no complete view of an application can be generated,
since only a subset of all possible execution paths is observed. To that end, dynamic
analysis is incomplete. One special form of dynamic analysis is behavior analysis,
in which the executable is seen as a black box and only its runtime effects on the
environment are observed, e.g., what files have been created or what processes have
been started. Although this kind of analysis only gives a very high level view of a
binary, it often is helpful and sufficient.
Static Analysis
Prior to any static analysis, the executable binary file has to be disassembled, e.g.,
the raw byte stream has to be transformed into valid assembler instructions and
(unformatted) data regions. Two different disassembling algorithms are used mainly
nowadays: linear sweep and recursive traversal. Linear sweep algorithms start
with the first byte of the code section and consecutively analyze each successive
instruction. The method is simple and fast, but it has some serious drawbacks
which arise due to variable instruction size and data or garbage that is embedded
into the code stream. It easily happens that this algorithm erroneously interprets
data as code and, accordingly, propagates disassembling errors throughout all the
following regular instructions. Recursive traversal addresses this problem by not
using a strictly sequential approach, but by starting to disassemble at the known
code entry-points and recursively following each branch-instruction. By that, only
valid code is observed and unaligned instructions or embedded data do not disturb
the process. The main disadvantage of this method is the assumption that each
jump target can be identified by static analysis, which is