Estimation of Jointly Constrained Mean-Covariance of Multivariate Normal Distribution
Abstract
Estimation of the mean vector and covariance matrix is of central importance in the analysis of
multivariate data. In the framework of generalized linear models, usually the variances are certain functions of the means with the normal distribution being an exception. We study some implications of functional relationships between covariance and the mean by focusing on the maximum likelihood and Bayesian estimation of the mean-covariance under the joint constraint Σμ = μ for a multivariate normal distribution. It can be viewed as a multivariate counterpart of the classical estimation problem in the N (θ,θ²) distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. I have tackled these issues in two ways and verified the solutions using extensive simulation studies.
In the first case, we derive the non-linear likelihood equations for the constrained maximum
likelihood estimator of (μ,Σ) and solve them using iterative methods. Generally, the MLE of covariance matrices computed using iterative methods do not satisfy the constraints. We propose a novel algorithm to modify such (infeasible) estimators or any other (reasonable) estimator. The key step is to re-align the mean vector along the eigenvectors of the covariance matrix using the idea of regression. In using the Lagrangian function for constrained MLE (Aitchison and Silvey, 1958), the Lagrange multiplier entangles with the parameters of interest and presents another computational challenge. We handle this by either iterative or explicit calculation of the Lagrange multiplier. The existence and nature of location of the constrained MLE are explored within a data-dependent convex set using recent results from random matrix theory.
In the second case, a novel structured covariance is proposed through reparameterization of the spectral decomposition of Σ involving its eigenvalues and μ which lets us study some implications of the functional relationships between covariance and the mean by focusing on the maximum likelihood and Bayesian estimation of the mean-covariance under the joint constraint Σμ = μ for a multivariate normal distribution. This is designed to address the challenging issue of positive-definiteness and to reduce the number of covariance parameters from quadratic to linear function of the dimension. We propose a fast (noniterative) method for approximating the maximum likelihood estimator by maximizing a lower bound for the profile likelihood function, which is concave. We use normal and inverse gamma priors on the mean and eigenvalues, and approximate the maximum aposteriori estimators by both MH within Gibbs sampling and a faster iterative method.
Subject
Maximum Likelihood EstimationIterative Methods
Lagrange Multiplier
Positive-Definite Matrices
Covariance matrix
Joint Mean-Covariance Estimation
Structured Covariance
Shrinkage Inverse Wishart
Normal-Inverse Gamma
Metropolis-Hastings within Gibbs Sampling
Citation
Kundu, Anupam (2022). Estimation of Jointly Constrained Mean-Covariance of Multivariate Normal Distribution. Doctoral dissertation, Texas A&M University. Available electronically from https : / /hdl .handle .net /1969 .1 /197849.