Profile Coverage: Using Android Compilation Profiles to Evaluate Dynamic Testing

Abstract

The rising complexity of Android apps makes comprehensive dynamic testing infeasible, especially for third-party apps. Knowing which methods are exercised by real users typically requires costly user studies or access to usage telemetry. We show that Android’s compilation profiles, specifically Cloud Profiles collected by the Google Play Store, offer a readily available, underutilized source of such information. These operational profiles aggregate which methods are commonly executed across users and guide ahead-of-time compilation during app installation. We provide the first in-depth characterization of Baseline Profiles and Cloud Profiles and show that over 99.89% of the top 1,000 apps include usage-derived cloud profiles.Based on this insight, we introduce profile coverage, a novel metric that measures how well dynamic testing exercises the methods real users interact with. This metric builds on the idea of operational coverage and enables a more holistic evaluation of automated test input generators. To enable profile coverage measurements, we develop a lightweight tracer, ProfTrace, based on Linux kernel uprobes that requires no app or system modifications. We demonstrate its utility by comparing three tools and a no-interaction baseline on 50 popular apps, showing that profile coverage reveals differences that traditional code coverage misses. For instance, in Candy Crush, automated testing achieves only 2.22% method coverage, but 21.39% profile coverage—indicating better alignment with user behavior than traditional code coverage would suggest.