Wireless systems are becoming increasingly complex, with a growing number of parameters to tune, a rising variety and heterogeneity of devices and equipment, and continuously evolving, diverse quality-of-service requirements. While centralized optimization may be theoretically optimal, it is often impractical in real-world deployments. This creates a need for methods that support distributed optimization and coordination among the goals of individual agents (e.g., users, operators, applications), while maintaining or improving network efficiency with manageable computational effort. In this talk, we explore the principles behind using deep reinforcement learning (DRL) as a promising approach for optimizing distributed multi-agent wireless systems. We illustrate its application to cell-free MIMO power allocation and the assignment of reconfigurable intelligent surfaces (RISs) in multi-operator scenarios, highlighting both the potential benefits and the challenges introduced by non-stationary multi-agent environments.