The development and psychometric analysis of the conceptual level teacher behavior observation tool

Abstract

The research literature is replete with information about the teacher shortage. The connection between teacher shortages and teacher classroom effectiveness with student achievement substantiates the need for interventions. Research has identified the potential of developmental mentoring and supervision programs for increasing teacher effectiveness, teacher retention, and student achievement. The purpose of this study was to develop and to analyze the psychometric properties of the Conceptual Level Teacher Behavior Observation Tool (CLTBOT). The purpose of this study was important because the development of the CLTBOT filled a void in the literature for an observation tool that would evaluate teacher behaviors in the conceptual domain. The potential use for these data is tied to mentoring or supervisory practices designed specifically for the teacher’s current need for structure, as well as for showing evidence of growth resulting from program activities. This study was organized into three steps. Step one focused on the development of the CLTBOT. Step two, of this study, explored the validity of the first draft of the CLTBOT in a pilot study. The pilot study indicated a moderate association between an adapted version of Hunt’s Paragraph Completion Method (PCM), the established measure for conceptual development, and the CLTBOT, the focal instrument of this study. The pilot proved an essential step in the process of developing and analyzing the CLTBOT as revisions were made following the results. Step three was the research study designed to answer the research questions. Research question one required an item by item analysis of the CLTBOT. Cohen’s kappa coefficients of between .699 and .867 demonstrated that the two raters’ scores were consistent. Research question two was answered with evaluations of the CLTBOT by two experts who awarded high ratings for the items based on relevance and clarity. A Cramer’s V coefficient of .56 revealed a strong relationship between the CLTBOT and the PCM, establishing evidence for concurrent validity and answering research question three. The results provided preliminary validity and reliability evidence for the use of the Conceptual Level Teacher Behavior Observation Tool (CLTBOT).