Estimating third -party examiners' scoring stability on selected applications to the Texas Award for Performance Excellence

Abstract

This study was an attempt to add to existing research by estimating the ability of third-party examiners to assess whether or not an organization successfully implemented strategies based on the criteria of the Texas Award for Performance Excellence (TAPE). The TAPE is given each year by the Quality Texas Foundation and recognizes organizations that demonstrate superior performance as it is defined by customer satisfaction and continuous improvement. The TAPE is a state-level award for quality that uses the same criteria as the Malcolm Baldrige National Quality Award for Performance Excellence. This research was an analysis of the TAPE process at the level of examiners, also known as the Board of Examiners. The Board is made up of approximately 150 experienced professionals from several types of business sectors and is responsible for evaluating organizational self-assessments. In this quantitative study, data were converted from the Quality Texas Foundation into a database. Because the set of the TAPE applicants included in the study consisted of the entire population of TAPE applicants selected from 2001 to 2004, descriptive statistics were appropriate for producing informative data that could be analyzed for variation and stability in the scoring process. Exploration of patterns in descriptive statistics and multivariate analysis of variance were the primary tools used in this particular study along with Cronbach’s Alpha as an indicator of reliability. Since scoring for the TAPE is based on an individual examiner’s best subjective assessment, it was impossible to have one objective score against which all the other scores could be measured. The team consensus score was therefore used as the true score for measurement. Establishing reliability of examiners’ scores was a problem due to the fact that organizations and teams did not repeat. Results from the study led to the conclusion that there was insufficient evidence to make a determination on what influences examiners’ scoring consistency. More data will need to be collected in such a way so as to make it possible to identify that impact consistency of examiner scores.