Quadratic form (statistics)

If $$\epsilon$$ is a vector of $$n$$ random variables, and $$\Lambda$$ is an $$n$$-dimensional square matrix, then the scalar quantity $$\epsilon'\Lambda\epsilon$$ is known as a quadratic form in $$\epsilon$$.

Expectation
It can be shown that


 * $$\operatorname{E}\left[\epsilon'\Lambda\epsilon\right]=\operatorname{tr}\left[\Lambda \Sigma\right] + \mu'\Lambda\mu$$

where $$\mu$$ and $$\Sigma$$ are the expected value and variance-covariance matrix of $$\epsilon$$, respectively. This result only depends on the existence of $$\mu$$ and $$\Sigma$$; in particular, normality of $$\epsilon$$ is not required.

Variance
In general, the variance of a quadratic form depends greatly on the distribution of $$\epsilon$$. However, if $$\epsilon$$ does follow a multivariate normal distribution, the variance of the quadratic form becomes particularly tractable. Assume for the moment that $$\Lambda$$ is a symmetric matrix. Then,


 * $$\operatorname{var}\left[\epsilon'\Lambda\epsilon\right]=2\operatorname{tr}\left[\Lambda \Sigma\Lambda \Sigma\right] + 4\mu'\Lambda\Sigma\Lambda\mu$$

In fact, this can be generalized to find the covariance between two quadratic forms on the same $$\epsilon$$ (once again, $$\Lambda_1$$ and $$\Lambda_2$$ must both be symmetric):


 * $$\operatorname{cov}\left[\epsilon'\Lambda_1\epsilon,\epsilon'\Lambda_2\epsilon\right]=2\operatorname{tr}\left[\Lambda _1\Sigma\Lambda_2 \Sigma\right] + 4\mu'\Lambda_1\Sigma\Lambda_2\mu$$

Computing the variance in the non-symmetric case
Some texts incorrectly state the above variance or covariance results without enforcing $$\Lambda$$ to be symmetric. The case for general $$\Lambda$$ can be derived by noting that


 * $$\epsilon'\Lambda'\epsilon=\epsilon'\Lambda\epsilon$$

so


 * $$\epsilon'\Lambda\epsilon=\epsilon'\left(\Lambda+\Lambda'\right)\epsilon/2$$

But this is a quadratic form in the symmetric matrix $$\tilde{\Lambda}=\left(\Lambda+\Lambda'\right)/2$$, so the mean and variance expressions are the same, provided $$\Lambda$$ is replaced by $$\tilde{\Lambda}$$ therein.

Examples of quadratic forms
In the setting where one has a set of observations $$y$$ and an operator matrix $$H$$, then the residual sum of squares can be written as a quadratic form in $$y$$:


 * $$\textrm{RSS}=y'\left(I-H\right)'\left(I-H\right)y$$

For procedures where the matrix $$H$$ is symmetric and idempotent, and the errors are Gaussian with covariance matrix $$\sigma^2I$$, $$\textrm{RSS}/\sigma^2$$ has a chi-square distribution with $$k$$ degrees of freedom and noncentrality parameter $$\lambda$$, where


 * $$k=\operatorname{tr}\left[\left(I-H\right)'\left(I-H\right)\right]$$
 * $$\lambda=\mu'\left(I-H\right)'\left(I-H\right)\mu/2$$

may be found by matching the first two central moments of a noncentral chi-square random variable to the expressions given in the first two sections. If $$Hy$$ estimates $$\mu$$ with no bias, then the noncentrality $$\lambda$$ is zero and $$\textrm{RSS}/\sigma^2$$ follows a central chi-square distribution.