Exploring the Construct-Related Validity of a Situational Judgment Test of Agreeableness and Conscientiousness: The Role of Social Competence

Abstract

Organizations frequently use personality assessments to inform employment decisions. In response to concerns about the susceptibility of single-statement personality measures to socially desirable responding (SDR; faking), researchers have explored alternative assessment methods. This study examined the construct-related validity of one such assessment approach—a situational judgment test (SJT) of agreeableness and conscientiousness (SJT-AC). SJTs, although theorized to make SDR or faking a nonissue by virtue of their predetermined scoring keys, suffer from issues surrounding construct measurement. Historically, because researchers have taken a work-sample focus, the specific construct(s) measured by SJTs have often been unclear. This study further explored the construct-related validity of construct-focused SJTs as measures of distinct, unidimensional personality constructs by examining the SJT-AC. In doing so, this study sought to permit comparisons between SJTs and single-statement methods designed to measure the same noncognitive constructs. Unlike single-statement methods, the SJT-AC measured two personality traits using scenarios embedded in social contexts. Consequently, this study examined whether a proposed third construct reflecting “how to act in social situations”—operationalized as social competence—influenced the construct-related validity of the SJT-AC. A sample of 495 undergraduate students completed the SJT-AC and single-statement measures of agreeableness, conscientiousness, and social competence. The results showed that a two-factor model fit the SJT-AC scores better than a one-factor model. A three-factor model including social competence failed to provide better fit. Moreover, convergent and discriminant validity of the SJT-AC scores were not better after controlling for social competence. Future research should examine other constructs thought to affect the construct-related validity of noncognitive SJTs with interpersonal scenarios.