Abstract:
Most research discussing malware detection completely dismisses signatures as being a thing of the past, accusing signatures of suffering from a weak ability to detect zero-day malware. This indeed could be the case if we are still referring to the classic definition of signatures, which renders them specific to only a single malicious executable binary. But what if these signatures grouped more malicious executables under a single signature? They would then make a valuable defense towards the fight against malware. To create such signatures, we need to develop new methods and techniques to constantly advance the state of the art as malware gets more and more elusive under old methods and approaches. The methods I will discuss not only give a good chance of creating effective signatures for malware, but also provide something just as important giving the malware analyst an automated approach to understanding key characteristics of the analyzed malware.
This dissertation has many contributions. The main contribution is a fully automated malware analysis system that can create families of malware, each able to be classified into its appropriate family, including zero-day malware. Another contribution is a new pruning algorithm that tests cluster strength and ensures the tightness of a malware family. The dissertation also incorporates a novel application of blockmodeling to the problem of malware analysis, which takes the form of a visual component in the system. It also creates a novel malware family signature based on n-gram frequencies composed of instructions and API function calls. Two experiments were carried out testing the accuracy and scalability of the system. The experimental results show that this system is highly accurate and scalable.