Forces between colloidal particles cannot generally be measured directly while they contribute to particle movements and ultimately drive self-assembly. We develop methods to infer the forcefield in colloidal systems from experimentally accessible trajectories. According to the Overdamped Langevin Equation (OLE) [1-2], colloidal motion is the sum of two terms: a deterministic motion caused by the total force between the colloids that depends on their (respective) coordinates and a stochastic motion due to the solvent thermal agitation, known as diffusion. Consequently, direct measurement of colloidal motion gives only access to a noisy estimator of the total force and nothing can be said about the different components in the forcefield (i.e external field, pairwise interaction, …). Furthermore, hydrodynamic couplings make the diffusion state dependent and the motion of one colloid becomes correlated with the motion of all the surrounding colloids [3]. Following the core idea developed in [4], we introduce a basis with physical inductive biases to capture the different phenomena behind hydrodynamic couplings. Then, using the inferred hydrodynamic couplings, we decorrelate colloidal motions and apply a similarly designed basis to disentangle the different forcefield components. Within this approach, we validate our method on both passive and active 3D systems simulated using Brownian dynamics with hydrodynamics [5]. We show that we are able to get very good quantitative agreements between the exact and inferred hydrodynamic couplings as well as the self-activity and pairwise interactions. [1] I. C. Jenkins, Soft Matter, 11 (2015), 6948-6956 [2] G. Volpe, Rep. Prog. Phys., 79 (2016), 053901 [3] E. Wajnryb, Journal of Fluid Mechanics 731 (2013), 3 [4] A. Frishman, Phys. Rev. X, 10 (2020), 021009 [5] A. Callegari, F. Toschi, M. Sega, Flowing Matter. Soft and Biological Matter. Springer, Cham. (2019)