Particles on the nano- to micrometer scale can be levitated and cooled down towards their motional ground state using optical forces. A major challenge in the field of levitation consists in extending existing methods to many particles. This would allow for the entanglement of mesoscopic objects or could provide a platform to test many-body quantum effects at the mesoscale. Indeed, a significant roadblock so far has been the requirement to monitor the particles' many degrees of freedom simultaneously and to engineer complex light fields to respond to their motion in real time. We solve both of these problems by introducing and computationally verifying a novel multi-particle cooling approach using a generalization of the Wigner-Smith time-delay operator [Phys. Rev. Lett. 130, 083203 (2023)]. For electromagnetic fields we connect the eigenvalues of this operator with the energy shift the corresponding fields induce in the particles. Through this, we can identify spatially modulated wave-fronts that optimally counteract the motion of multiple particles in parallel. Remarkably, our approach decouples the degrees of freedom of the fields from those of the particles, naturally leading to good scaling properties. We can thus propose an experimental implementation, where continuously shaped wave-fronts cool an ensemble of levitated objects.