Situational Judgment Test Responding: Best and Worst or Rate Each Response

Abstract

This study explores the differential validity of SJT responding formats (i.e., selecting a response to an SJT item). It was hypothesized that the SJT on which respondents identified the best and worst options would be related to cognitive ability because this type of SJT has a high cognitive load and thus acts like a knowledge test. It was also hypothesized that the SJT on which respondents rated the effectiveness of each option on a Likert scale would be related to personality because it taps into test taker?s behavioral tendencies. Results show that the best-and-worst SJT was not related to measures of education (a proxy for cognitive ability) or measures of personality. The SJT on which respondents rated the effectiveness of each option on a Likert scale was related to measures of personality but not education. Finally, because the Likert SJT has a greater number of responses, it was rescored as following the best-and-worst SJT keying. This converted SJT was related to neither measures of education nor measures of personality. The choose best-and-worst SJT significantly predicted performance, while the Likert SJT and a Likert SJT converted to a best-and-worst SJT did not predict performance.