Scalable Learning of Challenging Normative Behaviours with Deep RL

Abstract

The acceptance of AI agents in daily life hinges on their alignment with social, moral, and legal norms, and in recent years, attempts to build norm-sensitive AI agents --- including reinforcement learning (RL) agents --- have gained traction. However, while there are many promising approaches, they tend to be geared toward environments with modest state spaces, and adaptations to the deep RL context are lacking. In this paper we present a deep learning adaptation of normative restraining bolts (NRBs). Our contributions are twofold; first we combine NRBs with deep Q-networks (DQNs) and proximal policy optimization (PPO) to learn optimal behaviour compliant with challenging norms in a complex environment. We demonstrate our agent's ability to learn difficult normative behaviours in the game Pac-Man, which has been used in the literature to benchmark normative RL techniques. Secondly, while past work has assumed a lexicographic ordering over conflicting norms when finding appropriate weights for norms, we discuss the shortcomings of this approach, and provide a more scalable alternative which allows for the selection of policies deemed ideal by more complex metrics.