The Art of Unpacking

The Art of Unpacking
by Mark Vincent Yason
=====================================================
=====================================================

Abstract: Unpacking is an art—it is a mental challenge and is one of the most exciting mind
games in the reverse engineering field. In some cases, the reverser needs to know the
internals of the operating system in order to identify or solve very difficult anti-reversing tricks
employed by packers/protectors, patience and cleverness are also major factors in a
successful unpack. This challenge involves researchers creating the packers and on the other
side, the researchers that are determined to bypass these protections.

The main purpose of this paper is to present anti-reversing techniques employed by
executable packers/protectors and also discusses techniques and publicly available tools that
can be used to bypass or disable this protections. This information will allow researchers,
especially, malcode analysts to identify these techniques when utilized by packed malicious
code, and then be able decide the next move when these anti-reversing techniques impede
successful analysis. As a secondary purpose, the information presented can also be used by
researchers that are planning to add some level of protection in their software by slowing
down reversers from analyzing their protected code, but of course, nothing will stop a skilled,
informed, and determined reverser.

Table of Contents……………………………………………………………………………………………………. 2
1. INTRODUCTION……………………………………………………………………………………………………… 3

2. TECHNIQUES: DEBUGGER DETECTION……………………………………………………………………………………… 4
2.1. PEB.BeingDebugged Flag: IsDebuggerPresent() …………………………………………………………………… 4
2.2. PEB.NtGlobalFlag, Heap Flags ……………………………………………………………………………..5
2.3. DebugPort: CheckRemoteDebuggerPresent() / NtQueryInformationProcess()………………………………………….6
2.4. Debugger Interrupts ………………………………………………………………………………………… 7
2.5. Timing Checks ………………………………………………………………………………………………..8
2.6. SeDebugPrivilege ……………………………………………………………………………………………. 9
2.7. Parent Process ……………………………………………………………………………………………… 10
2.8. DebugObject: NtQueryObject() …………………………………………………………………………. 11
2.9. Debugger Window …………………………………………………………………………………………. 12
2.10. Debugger Process ……………………………………………………………………………………… 12
2.11. Device Drivers ………………………………………………………………………………………….. 12
2.12. OllyDbg: Guard Pages…………………………………………………………………………………. 13

3. TECHNIQUES: BREAKPOINT AND PATCHING DETECTION…………………………………………………. 14
3.1. Software Breakpoint Detection………………………………………………………………………….. 14
3.2. Hardware Breakpoint Detection…………………………………………………………………………. 15
3.3. Patching Detection via Code Checksum Calculation…………………………………………………. 16

4. TECHNIQUES: ANTI-ANALYSIS…………………………………………………………………………………. 17
4.1. Encryption and Compression…………………………………………………………………………….. 17
4.2. Garbage Code and Code Permutation………………………………………………………………….. 18
4.3. Anti-Disassembly ………………………………………………………………………………………….. 20

5. TECHNIQUES : DEBUGGER ATTACKS …………………………………………………………………………. 22
5.1. Misdirection and Stopping Execution via Exceptions ………………………………………………… 22
5.2. Blocking Input ……………………………………………………………………………………………… 23
5.3. ThreadHideFromDebugger ……………………………………………………………………………….. 24
5.4. Disabling Breakpoints …………………………………………………………………………………….. 25
5.5. Unhandled Exception Filter ………………………………………………………………………………. 26
5.6. OllyDbg: OutputDebugString() Format String Bug ………………………………………………….. 26

6. TECHNIQUES : ADVANCED AND OTHER TECHNIQUES …………………………………………………….. 27
6.1. Process Injection…………………………………………………………………………………………… 27
6.2. Debugger Blocker………………………………………………………………………………………….. 28
6.3. TLS Callbacks ………………………………………………………………………………………………. 29
6.4. Stolen Bytes …………………………………………………………………………………………………30
6.5. API Redirection …………………………………………………………………………………………….. 31
6.6. Multi-Threaded Packers…………………………………………………………………………………… 32
6.7. Virtual Machines……………………………………………………………………………………………. 32

7. TOOLS ………………………………………………………………………………………………………………. 34
7.1. OllyDbg………………………………………………………………………………………………………. 34
7.2. Ollyscript…………………………………………………………………………………………………….. 34
7.3. Olly Advanced………………………………………………………………………………………………. 34
7.4. OllyDump…………………………………………………………………………………………………….34
7.5. ImpRec ………………………………………………………………………………………………………. 34

8. REFERENCES……………………………………………………………………………………………………….. 35

Link:http://sdrv.ms/QDCDJX