Solving Low-rank Factor Analysis Problem using NExOS.jl¶

Shuvomoy Das Gupta

In this tutorial, we will show how to solve low-rank factor analysis problem using NExOS.

Factor analysis is a very popular method in statistics that reduces linear dimensionality of a particular model. The best way to understand factor analysis is to consider a generative model.

Basic factor analysis model¶

A basic factor analysis model is of the form

$$\begin{eqnarray} x & = & Lf+\epsilon, \end{eqnarray}$$

where $x\in\mathbf{R}^{n}$ is the observed random vector, $f\in\mathbf{R}^{r}$ (with $r\leq n$) is a random vector of common factor variables or scores, $L\in\mathbf{R}^{n\times r}$ is a matrix of factor loadings, and $\epsilon\in\mathbf{R}^{n}$ is a vector of uncorrelated random variables.

The observed random vector $x$ may contain series of achievement tests, psychological evaluation, intellectual performance etc. Without loss of generality, we assume that $x$ is mean-centered i.e., $\mathbf{E}(x)=0$, the vectors $f$ and $\epsilon$ are uncorrelated, and the covariance matrix of $f$ is the identity matrix.

We will denote $\mathbf{cov}(\epsilon)=D=\mathbf{diag}(d_{1},d_{2},\ldots,d_{n}).$ Then the covariance matrix of $x$, denoted by $\Sigma$ can be written as:

$$\Sigma=X+D,$$ where $X=LL^{T}$ is the covariance matrix corresponding to the common factors. The statistical method of factor analysis involves looking at $N$ samples generated by the model (1), i.e., given $x_{1},\ldots,x_{N}$ generated by (1) we want to estimate the matrices $X$ and $D$.

Optimization problem in consideration¶

The optimization problem to determine the matrices $X,D$ can be written as:

$$\begin{equation} \begin{array}{ll} \textrm{minimize} & \left\Vert \Sigma-X-D\right\Vert _{F}^{2}\\ \textrm{subject to} & D=\mathbf{diag}(d)\\ & d\geq0\\ & X\succeq0\\ & \mathbf{rank}(X)\leq r\\ & \Sigma-D\succeq0\\ & \|X\|_{2}\leq M, \end{array} \end{equation}$$

where $X\in\mathbf{S}^{n}$ and the diagonal matrix $D\in\mathbf{S}^{n}$ with nonnegative diagonal entries are the decision variables, and $\Sigma\in\mathbf{S}_{+}^{n}$ (a positive semidefinite matrix), $r\in\mathbf{Z}_{+},$ and $M\in\mathbf{R}_{++}$ are the problem data. The term $r$ is called the number of factors for a factor analysis model.

A proper solution for the optimization problem above requires that both $X$ and $D$ are positive semidefinite. Furthermore, when $\Sigma-D$, which is the covariance matrix for the common parts of the variables, is not positive semidefinite, that would as embarrassing as having a negative unique variance in $D$, as noted by ten Berge here, page 326. To prevent the aforementioned undesriable situation we enforce the constraint $\Sigma-D\succeq0$.

Next, we are going to show how to solve factor analysis problem using NExOS, which has a built in function for this purpose.

First, we load the packages.

Next, we load the data. You can change it to any other dataset as desired.

Now, we construct this problem using NExOS.

Now, we construct the settings.

Let us take a look at the quality of the found solution.