Cross-Validation in Psychometrics: Advances to the State of the Art

Abstract

When researchers do not have two independent samples with which to cross-validate their instrument, the common advice is to randomly split the sample and conduct the exploratory factor analysis (EFA) on the first half and the confirmatory factor analysis (CFA) on the second half. However, this approach does not improve the generalizability of the instrument and also allows for the randomness of group selection to affect the results. An alternative approach, though less commonly used, is to use the same full sample for both the EFA and CFA. This study uses empirical and simulated data to explore the differences in recovery of factor patterns, model fit, and power between EFA/CFA cross-validation using random split and full sample approaches.

Empirical and simulated results showed that different randomly split subsamples are not equivalent in their performance recovering factor patterns as results varied due to random chance. Additionally, simulation results showed increased precision and power in the full samples compared to split samples. From a pragmatic perspective, the increase in n required to maintain adequate power for both an EFA and a CFA when the sample has been split in half involves a large amount of additional participant recruitment. If researchers are committed to additional recruitment, perhaps efforts would be better served in recruiting an independent sample for a cross-validation that can withstand threats against generalizability. If additional recruitment is not feasible, using a full sample rather than a split sample provides more stable estimates, particularly with smaller sample sizes.