The comparative accuracy of performance distribution assessments versus conventional summary judgment assessments

Abstract

The purpose of this study was to test the accuracy of performance appraisal ratings based on either frequency estimations or summary judgments. The study required students at a large southwestern university to observe and rate the performance of two videotaped assessment center participants. One group (N = 88) reported frequency information, while the other group (N = 115) provided traditional summary judgments of performance. Accuracy indices (Kane, 2000), comparing the participants' ratings to experts' ratings, were used to investigate the comparative accuracy of frequency ratings versus traditional summary judgments of performance. This study also investigated the relationship between performance variability and interrater agreement in traditional summary judgments of performance. The results provided mixed support for the hypothesis that the reporting of frequency information would result in greater accuracy in performance appraisals. Frequency-based performance appraisals were more accurate on some performance dimensions, and traditional summary judgments were more accurate on others. No relationship between performance variability and interrater agreement in traditional summary judgment appraisal was found. Results of the present study suggest that in situations where it is important to capture the variability of an individual's performance, frequency estimation may be a more informative rating method. However, in terms of comparative accuracy, additional research is needed before a recommendation can be made for the widespread use of frequency estimation in organizational performance appraisals.