Black-box Android Code Coverage: Case studies in App Testing and Analysis with ACVTool Speaker: Olga Gadyatskaya, University of Luxemburg

Android is the most popular mobile platform today enjoying billions of active devices and millions of third-party applications. It is also a target of many adversaries, with a new malicious sample being created every 7 seconds. As third-party apps are distributed without source code, it is crucial that app market owners and security companies are able to test these packaged apps automatically for security issues or even bugs. 

Code coverage is a metric used by dynamic analysis and testing tools to evaluate how well an app has been exercised or even to guide the code exploration process. In this talk, I will present ACVTool that measures code coverage in black-box third-party app testing. I will discuss how ACVTool works and present findings of two case studies performed with ACVTool on Android apps. 

 The first study features Sapienz, a state-of-art automated testing tool for finding faults in Android apps. Testing Google Play apps with Sapienz, we found that different code coverage granularities uncover different bugs. This finding opens up new avenues for optimizing the testing process by combining different coverage metrics.

 The second case study explores sensitive API coverage in automated Android testing. Sensitive APIs are those that represent critical Android platform capabilities, like sending SMS or accessing Camera. We have evaluated sensitive API coverage achieved by the popular automated testing tools Sapienz and Monkey on datasets including malicious and benign applications. Our findings show that the evaluated tools are not suitable to cover most of sensitive APIs, and more advanced testing strategies are need. 

 I will conclude the talk by summarizing the research challenges that need to be addressed for reliable bug detection and malware detection in dynamic analysis of third-party Android apps.