Composite NUV Priors and Applications

Normal with unknown variance (NUV) priors are a central idea of sparse Bayesian learning and allow variational representations of non-Gaussian priors. More specifically, such variational representations can be seen as parameterized Gaussians, wherein the parameters are generally unknown. The advantage is apparent: for fixed parameters, NUV priors are Gaussian, and hence computationally compatible with Gaussian models. Moreover, working with (linear-)Gaussian models is particularly attractive since the Gaussian distribution is closed under affine transformations, marginalization, and conditioning. Interestingly, the variational representation proves to be rather universal than restrictive: many common sparsity-promoting priors (among them, in particular, the Laplace prior) can be represented in this manner. In estimation problems, parameters or variables of the underlying model are often subject to constraints (e.g., discrete-level constraints). Such constraints cannot adequately be represented by linear-Gaussian models and generally require special treatment. To handle such constraints within a linear-Gaussian setting, we extend the idea of NUV priors beyond its original use for sparsity. In particular, we study compositions of existing NUV priors, referred to as composite NUV priors, and show that many commonly used model constraints can be represented in this way. In Part I, we derive composite NUV representations of discretizing constraints, which enforce a model variable to take on values in a finite set (e.g., binary: {0, 1}, or M-ary: {0, 1, . . . ,M-1}). Furthermore, we derive composite NUV representations of linear inequality constraints, which enforce a model variable to be lower-bounded, upper-bounded, or both. In addition, we derive a composite NUV representation of an exclusion constraint, which enforces a model variable to stay outside of an exclusion region. In Part II, we review the standard linear state space representation to model physical systems. Linear state space models (LSSMs) are defined only by a few parameters, bring flexible modeling capabilities, and pave the way for efficient algorithms thanks to their linearity and recursive structure. Kalman-type algorithms are commonly used to perform inference in Gaussian LSSMs. We will use a Gaussian message passing scheme based on factor graphs which offers several improvements and can be seen as a generalization of the standard Kalman filter/smoother. In particular, we will apply the modified Bryson-Frazier (MBF) smoother (augmented with input estimation), which is numerically stable and avoids matrix inversions. The expressive power of composite NUV priors and their computational compatibility with Gaussian models allow us to reformulate a variety of (constrained) optimization problems as statistical estimation problems in a linear-Gaussian model with unknown parameters. We propose an efficient iterative algorithm based on Gaussian message passing with closed-form update rules for the unknown parameters. An asset of the algorithm is the linear computational complexity in time (per iteration). Consequently, the method is able to efficiently handle long time horizons, which is generally the bottleneck of other algorithms. Finally, in Part III and IV, we demonstrate the applicability of the proposed method using pertinent problems from signal processing and constrained control. We consider problems ranging from digital-to-analog conversion, discrete-phase beamforming, trajectory planning, to obstacle avoidance, power converter control, and more. The results are promising and suggest that the proposed method is a versatile toolbox to handle various challenging practical applications.