Sertkan, M., Althammer, S., Hofstätter, S., Knees, P., & Neidhardt, J. (2023). Exploring Effect-Size-Based Meta-Analysis for Multi-Dataset Evaluation. In Proceedings of the 3rd Workshop Perspectives on the Evaluation of Recommender Systems 2023 co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023). PERSPECTIVES 2023 - Perspectives on the Evaluation of Recommender Systems Workshop co-located with the 17th ACM Conference on Recommender Systems, Singapore, Singapore. CEUR-WS.org. https://doi.org/10.34726/5352
Proceedings of the 3rd Workshop Perspectives on the Evaluation of Recommender Systems 2023 co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023)
-
Volume:
3476
-
Date (published):
19-Sep-2023
-
Event name:
PERSPECTIVES 2023 - Perspectives on the Evaluation of Recommender Systems Workshop co-located with the 17th ACM Conference on Recommender Systems
In this paper, we address the essential yet complex task of evaluating Recommender Systems (RecSys) across multiple datasets. This is critical for gauging their overall performance and applicability in various contexts. Owing to the unique characteristics of each dataset and the variability in algorithm performance, we propose the adoption of effect-size-based meta-analysis, a proven tool in comparative research. This approach enables us to compare a “treatment model” and a “control model” across multiple datasets, offering a comprehensive evaluation of their performance. Through two case studies, we highlight the flexibility and effectiveness of this method in multi-dataset evaluations, irrespective of the metric utilized. The power of forest plots in providing an intuitive and concise summarization of our analysis is also demonstrated, which significantly aids in the communication of research findings. Our work provides valuable insights into leveraging these methodologies to draw more reliable and validated conclusions on the generalizability and robustness of RecSys models.
en
Project title:
Christian Doppler Labor für Weiterentwicklung des State-of-the-Art von Recommender-Systemen in mehreren Domänen: CDL Neidhardt (Christian Doppler Forschungsgesells)
CC BY 4.0
