An optimization approach to generalized linear models

Abstract

Statistical model building is a difficult and time-consuming process. Modelers need to balance various competing goals, such as predictive power, sparsity and fairness. Currently, practitioners follow model building rules and only check if a model meets the desired properties afterwards. These model selection techniques often rely on heuristics. As a result, final models often do not meet the desired criteria, let alone fulfill them satisfactorily. In this work, we investigate an optimization approach to Generalized Linear Models (GLMs), resulting in Holistic Generalized Linear Models (HGLMs). This class of models allows for the inclusion of additional constraints such as sparsity, limited multicollinearity or linear constraints, directly into the maximum likelihood estimation problem for the corresponding GLMs. To model and solve the resulting optimization problems, we use (mixed-integer) conic optimization or (mixed-integer) linear optimization while linearly approximating the objective function. Furthermore, we provide the software package holiglm for conveniently fitting HGLMs. Adding additional constraints to underlying optimization problem ensures that the desired properties of the statistical model, such as sparsity or non-negative coefficients, are always satisfied. The usual method for fitting GLMs is Maximum Likelihood Estimation via Iteratively Reweighted Least Squares (IRLS). However, when encountering linear constraints, which can occur for certain family/link combinations even without additional constraints, IRLS tends to converge slowly or does not converge at all. Conic optimization, as a generalization of linear optimization, offers a unified and reliable way for fitting HGLMs. In cases where the log-likelihood cannot be represented by the available cones, we resort to linear approximations. Since conic and linear optimization are exact optimization methods, the objective function will have a global optimum at the provided solution. In conclusion, the proposed class of HGLMs makes the process of model building less manual and more holistic since it is driven by optimization. Moreover, the holistic constraints can be used to incorporate additional knowledge or enable automated model selection by including information criteria into the objective function. Through our linear approximation scheme for differentiable log-likelihoods, we can fit HGLMs for a wide range of family/link combinations.