Kolchin, M., Wetz, P., Kiesling, E., & Tjoa, A. M. (2016). YABench: A Comprehensive Framework for RDF Stream Processor Correctness and Performance Assessment. In A. Bozzon, P. Cudre-Mauroux, & C. Pautasso (Eds.), Web Engineering (pp. 280–298). Springer. https://doi.org/10.1007/978-3-319-38791-8_16
RDF stream processing (RSP) has become a vibrant area of research in the semantic web community. Recent advances have resulted in the development of several RSP engines that leverage semantics to facilitate reasoning over flows of incoming data. These engines vary
greatly in terms of implemented query syntax, their evaluation and
operational semantics, and in various performance dimensions. Existing
benchmarks tackle particular aspects such as functional coverage, result correctness, or performance. None of them, however, assess RSP engine behavior comprehensively with respect to all these dimensions. In this paper, we introduce YABench, a novel benchmarking framework for RSP engines. YABench extends the concept of correctness checking and provides a flexible and comprehensive tool set to analyze and evaluate RSP engine behavior. It is highly configurable and provides quantifiable and reproducible results on correctness and performance characteristics. To validate our approach, we replicate results of the existing CSRBench benchmark with YABench. We then assess two well-established RSP engines, CQELS and C-SPARQL, through more comprehensive experiments.
In particular, we measure precision, recall, performance, and
scalability characteristics while varying throughput and query complexity.
Finally, we discuss implications on the development of future stream
processing engines and benchmarks.
