Software obfuscation is a longstanding and open research challenge in computer security. While theoretical results indicate that provably secure obfuscation in general is impossible to achieve, many application areas (e.g. malware, commercial software, etc.) show that software obfuscation is indeed employed in practice. Still, it remains largely unexplored to what extent today's software obfuscation state-of-the-art can keep up with the progress in code analysis and where we stand in the arms race between attackers and defenders. The first part of this thesis thus analyzes how effective software obfuscation is in the presence of ever more sophisticated deobfuscation techniques and off-the-shelf code analysis tools. To this end, we develop a novel classification scheme for the resilience of different types of obfuscations in specific attack scenarios. The answer heavily depends on the goals of the attacker and his available resources. Even simple obfuscation techniques can be quite effective against analysis techniques employing pattern matching or static analysis, which explains the unbroken popularity of obfuscation among malware writers. Dynamic analysis methods, in particular if assisted by a human analyst, are much harder to cope with; this makes software obfuscation for the purpose of intellectual property protection highly challenging. The subsequent part of this thesis therefore concentrates on code obfuscation for the protection of intellectual property in software. Software diversification is an effective method for preventing that automated attacks developed against one instance of a program work against other instances (class break). However, distribution of diversified software is challenging as each copy has to be different. The second research problem we consider in this thesis is the development of a concept for software diversification which does not require the individual copies of the program to be different on the binary code layer and thus provides a solution to the distribution problem. We introduce a novel code obfuscation scheme that applies the concept of software diversification to the control flow graph of the software. Our approach makes dynamic reverse-engineering considerably harder as the information an attacker can retrieve from the analysis of a single run of the program with a certain input is useless for understanding the program's behavior on other inputs. While the resilience of code obfuscation remains unclear and ultimately depends only on available resources and patience of the attacker, hardware-based solutions (trusted computing) provide a wide range of protection mechanisms such as remote attestation and secure storage for secrets. However, until now almost no systematic research has been done on the interplay between hardware- and software-based protection mechanisms. The third research problem we tackle in this thesis is how code obfuscation can be assisted by lightweight hardware. We propose minimal modifications to Intel¿s AES-NI instruction set in order to make it suitable for application in software protection scenarios and use these modifications for parametrization of our control flow obfuscation scheme. The combined approach provides strong hardware-software binding and restricts the attack context to pure dynamic analysis ¿ two major limiting factors of reverse-engineering. In the final part of this thesis, we focus on the problem of malware obfuscation. Recently, the concept of semantic-aware malware detection has been proposed in the literature. Instead of relying on a syntactic analysis (i.e., comparison of a program to pre-generated signatures of malware samples), semantic-aware malware detection tries to model the effects a malware sample has on the machine and thus does not depend on a specific syntactic implementation. For this purpose a model of the underlying machine is used. The fourth research problem we deal with in this thesis is the implementation of hidden functionality based on properties that are difficult to cover with a model of the hardware. We present COVERT COMPUTATION, a concept for the implementation of functionality in side effects of the microprocessor. We further give a comprehensive analysis of side effects in the x86 architecture and demonstrate the suitability of COVERT COMPUTATION for malware obfuscation.
