---
title: "Descriptions of the Tests for the Number of Factors"
author: "Brian O'Connor"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
    number_sections: false
vignette: >
  %\VignetteIndexEntry{Descriptions of the Tests for the Number of Factors}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---


```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = FALSE,
  comment = "",
  fig.width = 7,
  fig.align = "center",
  background='#FFFFFF'
)
```

<!-- ```{css echo=FALSE} -->

<!-- .custom-inline { -->

<!--   color: red; -->

<!--   font-weight: 700 -->

<!-- } -->

<!-- ``` -->

```{css, echo=FALSE}
h1 {
  font-size: 30px;
  color: black !important;
}
h2 {
  font-size: 20px;
  color: blue !important;
}

#TOC {
  position: fixed;
  left: 10px;
  top: 10px;
  width: 200px;
  height: 100%;
  overflow:auto;
}

#TOC::before {
  content: "";
}

#TOC li {
  list-style-type: none;
}

body {
  max-width: 800px;
  margin: auto;
  margin-left:230px;
  line-height: 20px;
}
```

# Tests for the Number of Factors:

## CD

The CD (comparison data) method uses the CD function (and its defaults) in the 
EFAtools package.

From Auerswald & Moshagen (2019):

"Ruscio and Roche (2012) suggested an approach that finds the number of factors 
by determining the solution that reproduces the pattern of eigenvalues best 
(comparison data, CD). CD takes previous factors into account by generating 
comparison data of a known factorial structure in an iterative procedure. 
Initially, CD compares whether the simulated comparison data with one underlying 
factor (j = 1) reproduce the pattern of empirical eigenvalues significantly 
worse compared with a two-factor solution (j + 1). If this is the case, CD 
increases j until further improvements are nonsignificant or a preset maximum 
of factors is reached."

"No single extraction criterion performed best for every factor model. In 
unidimensional and orthogonal models, traditional PA, EKC, and Hull 
consistently displayed high hit rates even in small samples. Models with 
correlated factors were more challenging, where CD and SMT outperformed 
other methods, especially for shorter scales. Whereas the presence of 
cross-loadings generally increased accuracy, non-normality had virtually 
no effect on most criteria. We suggest researchers use a combination of 
SMT and either Hull, the EKC, or traditional PA, because the number of 
factors was almost always correctly retrieved if those methods converged. 
When the results of this combination rule are inconclusive, traditional 
PA, CD, and the EKC performed comparatively well. However, disagreement 
also suggests that factors will be harder to detect, increasing sample 
size requirements to N >= 500."

## EMPKC

The empirical Kaiser criterion method (Braeken & van Assen, 2017).
	
The code for this option was adapted from the code provided by Auerswald & Moshagen (2019).
	
From Braeken & van Assen (2017):

"We developed a new factor retention method, the Empirical Kaiser Criterion, which 
is directly linked to statistical theory on eigenvalues and to researchers' goals to 
obtain reliable scales. EKC is easily visualized, and easy to compute and apply (no 
specialized software or simulations are needed). EKC can be seen as a sample-variant 
of the original Kaiser criterion (which is only effective at the population level), 
yet with a built-in empirical correction factor that is a function of the 
variables-to-sample-size ratio and the prior observed eigenvalues in the series. 
The links with statistical theory and practically relevant scales allowed us to derive 
conditions under which EKC accurately retrieves the number of acceptable scales, 
that is, sufficiently reliable scales and strong enough items.

"Our simulations verified our derivations, and showed that (a) EKC performs about as 
well as parallel analysis for data arising from the null, 1-factor, or orthogonal 
factors model; and (b) clearly outperforms parallel analysis for the specific case of 
oblique factors, particularly whenever interfactor correlation is moderate to high and 
the number of variables per factor is small, which is characteristic of many applications 
these days. Moreover, additional simulations suggest that our method for predicting 
conditions of accurate factor retention also work for the more computer- intensive 
methods ... The ease-of-use and effectiveness of EKC make this method a prime candidate 
for replacing parallel analysis, and the original Kaiser criterion that, although it 
empirically does not perform too well, is still the number one method taught in 
introductory multivariate statistics courses and the default in many commercial 
software packages. Furthermore, the link to statistical theory opens up possibilities 
for generic power curves and sample size planning for exploratory factor analysis studies.

"Generally, the EKC accurately retrieved the number of factors in conditions whenever 
it was predicted to work well, and its performance was worse when it was not predicted 
to work well. More precisely, hit rate or power exceeded .8 in accordance with 
predictions under the null model, 1-factor model, the orthogonal factor model, and 
the oblique factor model with more than three variables per scale. Only in the case 
of minimal scales, that is, with three items per scale, did EKC sometimes not accurately 
retrieve the number of factors as predicted; dropping the restriction that eigenvalues 
should exceed 1 then mended EKC's performance. A general guideline for application that 
can be derived from our results (and would not need a study-specific power study), is 
that EKC will accurately retrieve the number of factors in samples of at least 100 
persons, when there is no factor, one practically relevant scale, or up to five 
practically relevant uncorrelated scales with a reliability of at least .8." (pp. 463-464)

From Auerswald & Moshagen (2019):

"The Empirical Kaiser Criterion (EKC; Braeken & van Assen, 2017) is an approach 
that incorporates random sample variations of the eigenvalues in Kaiser's criterion. 
On a population level, the criterion is equivalent to Kaiser's criterion and extractions 
all factors with associated eigenvalues of the correlation matrix greater than one. 
However, on a sample level, the criterion takes the distribution of eigenvalues for 
normally distributed data into account." (p. 474)

## HULL

The HULL method option uses the HULL function (and its defaults) in the EFAtools package.

From Auerswald & Moshagen (2019):

"The Hull method (Lorenzo-Seva et al., 2011) is an approach based on the Hull 
heuristic used in other areas of model selection (e.g., Ceulemans & Kiers, 2006). 
Similar to nongraphical variants of Cattell's scree plot, the Hull method attempts 
to find an elbow as justification for the number of common factors. However, instead 
of using the eigenvalues relative to the number of factors, the Hull method relies 
on goodness-of-fit indices relative to the model degrees of freedom of the proposed 
model."

## MAP

Velicer's minimum average partial test

This method for determining the number 
of components focuses on the common variance in a correlation matrix. It involves 
a complete principal components analysis followed by the examination of a series 
of matrices of partial correlations. Specifically, on the first step, the first 
principal component is partialled out of the correlations between the variables 
of interest, and the average squared coefficient in the off-diagonals of the 
resulting partial correlation matrix is computed. On the second step, the first 
two principal components are partialled out of the original correlation matrix 
and the average squared partial correlation is again computed. These computations 
are conducted for k (the number of variables) minus one steps. The average squared 
partial correlations from these steps are then lined up, and the number of components 
is determined by the step number in the analyses that resulted in the lowest average 
squared partial correlation. The average squared coefficient in the original 
correlation matrix is also computed, and if this coefficient happens to be lower 
than the lowest average squared partial correlation, then no components should be 
extractioned from the correlation matrix. Statistically, components are retained as 
long as the variance in the correlation matrix represents systematic variance. 
Components are no longer retained when there is proportionately more unsystematic 
variance than systematic variance (see O'Connor, 2000, p. 397).

## NEVALSGT1

The number of eigenvalues greater than one method is often referred to as the "Kaiser", 
"Kaiser-Guttman", or "Guttman-Kaiser" rule for determining
the number of components or factors in a correlation matrix.

The rationale for this traditional procedure is that a component with an 
eigenvalue of 1 accounts for as much
variance as a single variable. Extracting components with eigenvalues of 1 
or less than 1 would defeat the usual purpose of component and factor analyses. 
Furthermore, the reliability of a component will always be nonnegative when its 
eigenvalue is greater than 1.

There are a number of problems with this rule of thumb. Monte Carlo investigations have 
found that its accuracy rate is not acceptably high (Zwick & Velicer, 1986)). The rule was  
originally intended to be an upper bound for the number of components to be retained, but  
it is most often used as the criterion to determine the exact number of components or factors. 
Guttman's original proof applies only to the population correlation matrix and the 
sampling error that occurs in specific samples results in the rule often overestimating 
the number of components. The rule is also considered overly mechanical, e.g., a component 
with an eigenvalue of 1.01 achieves factor status whereas a component with an 
eigenvalue of .999 does not.
	         
## RAWPAR

Parallel analysis of eigenvalues, with real data as input.

The parallel analysis procedure for deciding on the number of 
components or factors involves extracting eigenvalues from random data 
sets that parallel the actual data set with regard to the number of cases 
and variables. For example, if the original data set consists of 305 
observations for each of 8 variables, then a series of random data matrices 
of this size (305 by 8) would be generated, and eigenvalues would be computed 
for the correlation matrices for the original, real data and for each of the 
random data sets. The eigenvalues derived from the actual data are then 
compared to the eigenvalues derived from the random data. In Horn's original 
description of this procedure, the mean eigenvalues from the random data served 
as the comparison baseline, whereas the more common current practice is to use 
the eigenvalues that correspond to the desired percentile (typically the 95th) 
of the distribution of random data eigenvalues. Factors or components are 
retained as long as the ith eigenvalue from the actual data is greater than 
the ith eigenvalue from the random data.
	
The RAWPAR function permits users to specify PCA or PAF or image as the  factor 
extraction method. Principal components eigenvalues are often used to determine the number
of common factors. This is the default in most statistical software packages and it is the
primary practice in the literature. It is also the method used by many factor analysis experts, 
including Cattell, who often examined principal components eigenvalues in his scree plots to 
determine the number of common factors. Principal components eigenvalues are based on all of 
the variance in correlation matrices, including both the variance that is shared among 
variables and the variances that are unique to the variables. In contrast, principal axis 
eigenvalues are based solely on the shared variance among the variables. The procedures are 
qualitatively different. Some therefore claim that the eigenvalues from one extraction method 
should not be used to determine the number of factors for another extraction method. 
The PAF option in the extraction argument for the RAWPAR and PARALLEL functions was included solely for 
research purposes. It is best to use PCA as the extraction method for regular data analyses. 

## SALIENT

The salient loadings criterion for determining the number of factors was
recommended by Gorsuch. Factors are retained when 
each factor has at least a specified minimum number variables (e.g., 3) that have loadings 
that are greater than or equal to a specified minimum loading value (e.g., .40).
Factor are considered trivial when they do not contain a sufficient number 
of salient loadings (Gorsuch, 1997, 2015; Boyd, 2011).
	
The procedure begins by extracting and rotating (if requested) an excessive
number of factors. If the initial factor loadings do not meet the specified criteria,
then the factor analysis is conducted again with one less factor and the
loadings are again examined to determine whether the factor loadings meet the 
specified criteria. The procedure stops when a loading matrix meets the criteria,
in which case the number of columns in the loading matrix is the number
of factors according to the salient loadings criteria.
	
The initial, excessive number of factors for the procedure is determined using the
min_eigval argument (for minimum eigenvalue). The default is .70, which can be 
adjusted (raised) when analyses produce an error caused by there being too few variables.
	
Although there is no consensus on what constitutes a 'salient' loading, an
absolute value of .40 is common in the literature.
	
There are different versions of the salient loadings criterion method, which has not
been extensively tested to date. The procedure is nevertheless considered promising
by Gorsuch and others.
	
Some versions involve the use of multiple salient loading values, each with its own 
minimum number of variables. This can be done in the SALIENT function by providing a
vector of values for the salvalue argument and a corresponding vector of values for the
numsals argument. The maximum number of possible values is three, and there 
should be a logical order in the values, i.e., increasing values for salvalue and 
decreasing values for numsals.
	
It is also possible to place a restriction of the maximum value of the cross-loadings
for the salient variables, e.g., requiring that a salient loading is not
accompanied by cross-loadings on other variables that are greater than .15.
Use the max_cross argument for this purpose, although it may be difficult to claim
that cross-loadings should be small when the factors are correlated.

## SESCREE

The Standard Error Scree test (Zoski & Jurs, 1996) is a linear regression 
operationalization of the 
scree test for determining the number of components. The results are purportedly 
identical to those from the visual scree test. The test is based on the standard 
error of estimate values that are computed for the set of eigenvalues in a scree 
plot. The number of components to retain is the point where the standard error 
exceeds 1/m, where m is the numbers of variables.

## SMT

The sequential chi-square model test for the number of common factors uses the likelihood 
ratio test statistic values from maximum likelihood factor analysis estimations.

From Auerswald & Moshagen (2019):

"The fit of common factor models is often assessed with the likelihood ratio test 
statistic (Lawley, 1940) using maximum likelihood estimation (ML), which tests 
whether the model-implied covariance matrix is equal to the population covariance 
matrix. The associated test statistic asymptotically follows a Chi-Square distribution 
if the observed variables follow a multivariate normal distribution and other 
assumptions are met (e.g., Bollen, 1989). This test can be sequentially applied to 
factor models with increasing numbers of factors, starting with a zero-factor model. 
If the Chi-Square test statistic is statistically significant (with e.g., p < .05), 
a model with one additional factor, in this case a unidimensional factor model, is 
estimated and tested. The procedure continues until a nonsignificant result is 
obtained, at which point the number of common factors is identified.

"Simulation studies investigating the performance of sequential Chi-Square model 
tests (SMT) as an extraction criterion have shown conflicting results. Whereas 
some studies have shown that SMT has a tendency to overextraction (e.g., Linn, 1968; 
Ruscio & Roche, 2012; Schonemann & Wang, 1972), others have indicated that the SMT 
has a tendency to underextraction (e.g., Green et al., 2015; Hakstian et al., 1982; 
Humphreys & Montanelli, 1975; Zwick & Velicer, 1986). Hayashi, Bentler, and 
Yuan (2007) demonstrated that overextraction tendencies are due to violations of 
regularity assumptions if the number of factors for the test exceeds the true 
number of factors. For example, if a test of three factors is applied to samples 
from a population with two underlying factors, the likelihood ratio test statistic 
will no longer follow a Chi-Square distribution. Note that the tests are applied 
sequentially, so a three-factor test is only employed if the two-factor test was 
incorrectly significant. Therefore, this violation of regularity assumptions does 
not decrease the accuracy of SMT, but leads to (further) overextractions if a 
previous test was erroneously significant. Additionally, this overextraction 
tendency might be counteracted by the lack of power in simulation studies with 
smaller sample sizes. The performance of SMT has not yet been assessed for 
non-normally distributed data or in comparison to most of the other modern techniques 
presented thus far in a larger simulation design." (p. 475) 

<br>

### &nbsp; {-}

# References / Sources

Auerswald, M., & Moshagen, M. (2019). How to determine the number of factors to retain in exploratory factor analysis: A comparison of extraction methods under realistic conditions. Psychological Methods, 24(4), 468-491.

Boyd, K. C. (2011). Factor analysis. In M. Stausberg & S. Engler (Eds.), The Routledge Handbook of Research Methods in the Study of Religion (pp. 204-216). New York: Routledge. 

Braeken, J., & van Assen, M. A. (2017). An empirical Kaiser criterion. Psychological Methods, 22, 450 - 466.	

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.

Gorsuch, R. L. (1997). Exploratory factor analysis: Its role in item analysis. Journal of Personality Assessment, 68, 532-560.

Gorsuch, R. L. (2015). Factor analysis. Routledge/Taylor & Francis Group.

Guttman, L. (1954). Some necessary conditions for common factor analysis. Psychometrika, 19, 149-161.

Hayton, J. C., Allen, D. G., Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191-205.	

Kaiser, H. F. (1960). The application of electronic computer to factor analysis. Educational and Psychological Measurement, 20, 141-151.

Lorenzo-Seva, U., Timmerman, M. E., & Kiers, H. A. (2011). The Hull method for selecting the number of common factors. Multivariate Behavioral Research, 46(2), 340-364.

O'Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer's MAP test. Behavior Research Methods, Instrumentation, and Computers, 32, 396-402.

Ruscio, J., & Roche, B. (2012). Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychological Assessment, 24, 282292. doi: 10.1037/a0025697

Velicer, W. F. (1976). Determining the number of components from the matrix of partial  correlations. Psychometrika, 41, 321-327.

Velicer, W. F., Eaton, C. A., and Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes, eds., Problems and solutions in human assessment (p.p. 41-71). Boston: Kluwer.	

Zoski, K., & Jurs, S. (1996). An objective counterpart to the visual scree test for factor analysis: the standard error scree test. Educational and Psychological Measurement, 56(3), 443-451.

Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining  the number of components to retain. Psychological Bulletin, 99, 432-442.