Hybrid Human-Machine Evaluation of Knowledge Graphs

Abstract

Abstract. Knowledge Graphs (KGs) enable real-world knowledge to be structured in a machine-readable format and act as a skeleton to many advanced intelligent applications (e.g. conversational agents). The correctness and quality of KGs are of high importance, as incorrectly represented information or controversial concepts modeled from a single viewpoint can lead to invalid application outputs and biased systems. Several KG quality issues can be detected automatically, such as syntax errors or hierarchy cycles, however, others require human involvement, e.g., identifying incorrectly modeled statements, or discovering concepts not compliant with how humans think. Human Computation and Crowdsourcing (HC&C) techniques have been used as a promising approach to outsource human-centric tasks to human contributors at a lower cost. Nevertheless, the scalable evaluation of large KGs relying on HC&C techniques alone remains a challenge and the need for a hybrid human-machine approach arises. In this paper, a hybrid intelligence system for the use case of Knowledge Verification is proposed, which utilizes techniques from the Semantic Web, HC&C and Multi-Agent Systems communities for ensuring a transparent, well-coordinated and auditable KG evaluation process.