We introduce the quantum Wigner-Smith (QWS) operator, a Hermitian operator describing the interaction between the spatial as well as the quantum degrees of freedom of light and a local classical parameter of a linear, but otherwise arbitrarily complex scattering medium through which the light propagates. The QWS operator builds a bridge between quantum micromanipulation, vacuum forces and quantum metrology on the one side, and the formalism of classical scattering matrices, which are experimentally measurable in a noninvasive manner, on the other side. The QWS operator can be used to describe generalized forces (momentum transfer, angular momentum transfer, pressure) that quantum light exerts on classical target objects. From the classical far-field scattering matrix and its dependence on the corresponding local parameter, the effect of quantum light in the near-field (in the vicinity of the target object) can be inferred. Our formalism makes it possible to identify quantum states of light that have an optimal effect (largest or smallest possible force, least possible quantum noise in the force) on the target object. If the light field is in the vacuum state, the formalism naturally provides the vacuum contributions to the forces, also known as Casimir forces. Another application of the QWS operator lies in quantum metrology. The variance of the QWS operator is proportional to the quantum Fisher information (QFI), which in turn provides a measure on how precisely a parameter of the scattering system (e.g. the position of a scatterer or its orientation) can be measured. The optimization of the QFI determines --- even in complex, open scattering systems --- how the spatial structure and the quantum degrees of freedom of the light must be designed in order to achieve the physically best possible measurement precision.