Alternative assessment of at-risk students in mathematics : the effects of context and setting

Abstract

Performance-based mathematics assessment scenarios were developed using real-life situations which were intended to be both meaningful and relevant to the students. Scenarios were based on four contexts: familiar activity, social issue, hands-on and technology, and they were designed to be closely aligned with the Texas Assessment of Academic Skills (TAAS) test's mathematical content. Three assessment settings were investigated which were individual, aided, and small group. A two facet, mixed model Generalizability study was conducted with students and raters as the objects of measurement. Two repeated measures ANOVAs were conducted with context and setting as main factors, and with TAAS content and setting as main factors. The G-study indicated that the raters were dependable when rating the TAAS knowledge domains of problem solving and operations with G-coefficients of 1.00 and 0.65 respectively. Raters were less reliable when rating concepts which produced a G-coefficient of 0.37. The G-coefficients for students were 0.88 for problem solving, 0.92 for operations, and 0.86 for concepts which suggested that similar results could be obtained with another sample of at-risk students. The ANOVA for the main effects of context and setting indicated that the contexts did not differ among each other with an F-value of 1.64 (p $<$ 0.1871). The setting was significant with an F-value of 3.30 (p $<$ 0.0422). The two-way interaction of context by setting was not significant with an F-value of 1.62 (p $<$ 0.1530). The ANOVA results for content and setting as main factors indicated that setting and content were significant with F-values of 5.12 (p $<$ 0.0095) and 3.23 (p $<$ 0.0136) respectively. At-risk students produced the largest mean scores in the group setting. Overall, the familiar activity and social issue produced the greatest mean scores for each content area with some variations. It was concluded that situated contexts which have relevancy to at-risk students' past knowledge and experiences tend to produce greater achievement in certain TAAS content areas than others associated with a particular context.