National Center for Research on Evaluation, Standards and Student Testing

#387 – Specifications for the Design of Problem-Solving Assessments in Science
Brenda Sugrue

Summary

In Specifications for the Design of Problem-Solving Assessments in Science, CRESST researcher Brenda Sugrue draws on the CRESST performance assessment model to develop a new set of test specifications for science. Sugrue recommends that designers follow a straightforward approach for developing alternative science assessments. "Carry out an analysis of the subject matter content to be assessed," says Sugrue, "identifying key concepts, principles, and procedures that are embodied in the content." She adds that much of this analysis already exists in state frameworks or in the national science standards. Either multiple choice, open-ended, or hands-on science tasks can then be created or adapted to measure individual constructs, such as concepts and principles, and the links between concepts and principles. In addition to measuring content-related constructs, Sugrue's model advocates measuring metacognitive constructs and motivational constructs in the context of the content. This permits more specific identification of the sources of students' poor performance. Students may perform poorly because of deficiencies in content knowledge, and/or deficiencies in constructs such as planning and monitoring, and/or maladaptive perceptions of self and task. The more specific the diagnosis of the source of poor performance, the more specific can be instructional interventions to improve performance. Sugrue's model includes specifications for task design, task development, and task scoring, all linked to specific components of problem-solving ability. An upcoming CRESST report will discuss the results of a study designed to evaluate the effectiveness of the model for attributing variance in performance to particular components of problem solving and particular formats for measuring them.

#473 – Principals' Views of Mathematics Standards, Frameworks, and Assessment in a Context of Reform
Maryl Gearhart

Summary

The purpose of this study was to gather information on principals' views regarding standards, frameworks, and assessment in mathematics. Based on surveys completed by 96 principals from 35 public school districts in Greater Los Angeles - each principal a past participant in events sponsored by the UCLA Principals' Center - our findings reflect the views of principals interested in improving educational practice.

With regard to standards and frameworks, the findings indicate that the principals' schools were not currently building mathematics programs closely on existing standards and frameworks; however, these principals were prepared to support the future implementation of state and/or district mathematics standards in their schools, and they requested resources and assistance with implementation. The principals disagreed on the need for standards at the school level. With regard to testing, the principals were concerned that parents and students may not understand the results of norm-referenced tests and that norm-referenced tests are not aligned with their instructional programs in mathematics. The principals were likely to favor performance-based measures for program evaluation and reporting and for guiding instruction, and they requested resources and assistance for building teacher capacity with new assessments. However, a large minority of the principals favored the use of both forms of mathematics testing, and some principals favored norm-referenced testing. Thus, although these principals represented administrators engaged in school improvement, they differed in their views regarding accountability testing.

The findings suggest that resolution among the views of administrators lies in the design of mathematics standards that embrace a breadth of knowledge and skill, together with the design of a coherent, standards-based assessment system that integrates multiple measures.

#395 – How Does My Teacher Know What I Know?
Kathryn Davinroy, Carribeth Bliem, and Vicky Mayfield

Summary

Regardless of the type of assessment used in the classroom, students continue to have the same traditional understandings of assessment suggests a new CRESST study. "...students believe that assessment activities are often aimed at measuring their handwriting, punctuation, [and] expression when reading out loud," say researchers Kathryn Davinroy, Carribeth Bliem, and Vicky Mayfield, in a new CRESST report. The third-grade students involved in the study had been exposed to performance assessments in reading and mathematics for over a year, yet their concepts of assessment did not shift significantly. The authors found that this limited framework applied to multiple topics. When asked what does a math test look like, for example, students still referenced the timed math-fact test. "It has a hundred problems on it," said one student, "and you have to get as many problems as you can down in five minutes." Since students are typically the last to be exposed to changes in assessment, the data tend to confirm that students attitudes towards assessment may also be the last to change. "Our findings," conclude the authors, "about student perceptions, regarding reading, mathematics, and assessment support contentions that reform takes time if perceptions and understandings are going to change significantly."

#471 – Teachers' Shifting Assessment Practices in the Context of Educational Reform in Mathematics
Geoffrey B.Saxe, Megan L. Franke, Maryl Gearhart, Sharon Howard, and Michele Crockett

Summary

This paper presents a study of primary and secondary mathematics teachers' changing assessment practices in the context of policy, stakeholder, and personal presses for change. Using survey and interviews, we collected teachers' reports of their uses of three forms of assessment, one linked to traditional practice (exercises), and two linked to reforms in mathematics education (open ended problems and rubrics). Findings revealed several trajectories of change in the interplay between assessment forms and the functions that they serve. Teachers may implement new assessment form in ways that serve 'old' functions; teachers may re-purpose 'old' assessment forms in ways that reveal students' mathematical thinking. Our developmental framework provides a way to understand the dynamics of teacher development in relation to ongoing educational reforms.

#692 – Automated Assessment of Domain Knowledge With Online Knowledge Mapping
Gregory K. W. K. Chung, Eva L. Baker, David G. Brill, Ravi Sinha, Farzad Saadat, and William L. Bewley

Summary

A critical first step in developing training systems is gathering quality information about a trainee's competency in a skill or knowledge domain. Such information includes an estimate of what the trainee knows prior to training, how much has been learned from training, how well the trainee may perform in future task situations, and whether to recommend remediation to bolster the trainee's knowledge. This paper describes the design, development, testing, and application of a Web-based tool designed to assess a trainee's understanding of a content domain in a distributed learning environment. The tool, called the CRESST Human Performance Knowledge Mapping Tool (HPKMT), enables trainees to express their understanding of a content area by creating graphical, network representations of concepts and links that define the relationships of concepts.

#705 – Using Artifacts to Describe Instruction: Lessons Learned from Studying Reform-Oriented Instruction in Middle School Mathematics and Science
Brian Stecher, Hilda Borko, Karin L. Kuffner, Felipe Martinez, Suzanne C. Arnold, Dionne Barnes, Laura Creighton and Mary Lou Gilbert

Summary

It is important to be able to describe instructional practices accurately in order to support research on “what works” in education and professional development as a basis for efforts to improve practice. This report describes a project to develop procedures for characterizing classroom practices in mathematics and science on the basis of collected classroom artifacts. A data collection tool called the “Scoop Notebook” was used to gather classroom artifacts (e.g., lesson plans, instructional materials, student work) and teacher reflections. Scoring guides were developed for rating the Notebooks (and observed classroom behaviors) along ten dimensions of reform-oriented practice in mathematics and science. Field studies were conducted in middle school science and mathematics classrooms to collect information about the reliability, validity, and feasibility of the Scoop Notebook as a measure of classroom practice. The studies yielded positive results, indicating that the Scoop Notebooks and associated scoring guides have promise for providing accurate representations of selected aspects of classroom practice. The report summarizes these results and discusses lessons learned about artifact collection and scoring procedures.

#782 – Year 3 ASK/FOSS Efficacy Study
Ellen Osmundson, Yunyun Dai, and Joan Herman

Summary

In this interim report, CRESST researchers examine the effect on teaching and student learning between several different science curricula. Using randomly assigned treatment and control groups of 3rd and 4th grade teachers, this interim report provides important lessons for the upcoming 4th year study.

#466 – I've Seen This Before? The Effects of Self-Monitoring and Multiple Context Instruction on Knowledge Representation and Transfer Among Middle School Students
Davina C. D. Klein

Summary

Both multiple context learning and self-reflection training are posited to affect students' knowledge representations by fostering decontextualization, abstraction, and schema formation. Schemata, in turn, theoretically facilitate transfer. One hundred eighty-six low-SES middle school students of mixed ethnicities were taught to use concept mapping as a means of understanding material in either one subject area or two subject areas. In addition, half of the students in each group were trained in metacognitive self-monitoring techniques. The transfer task was a problem in a third subject area.

Students were asked to complete the transfer task and then to complete three questionnaires, one eliciting alternative solutions to the transfer task, one assessing their schemata, and one addressing their metacognitive activity. In addition, a small, randomly selected subsample of students from each treatment group did not take the transfer task, instead completing only the questionnaires.

It was hypothesized that students who both engaged in self-monitoring and were exposed to two subject areas would form better schemata, engage in greater metacognitive activity, and perform better on the transfer measure than other students. Although the main predictions were not confirmed, some support was found for the beneficial effects of monitoring on schema formation. In addition, it was found that, given a relatively brief treatment period, at-risk students were able to learn the cognitive strategy of concept mapping, to engage in metacognitive activities such as self-monitoring, to construct good concept mapping schemata, and to transfer to a large degree. Results are discussed and suggestions are made for future work in this area.

#681 – Measuring Reading Comprehension and Mathematics Instruction in Urban Middle Schools: A Pilot Study of the Instructional Quality Assessment
Lindsay Clare Matsumura, Sharon Cadman Slater, Brian Junker, Maureen Peterson, Melissa Boston, Michael Steele, and Lauren Resnick

Summary

The quality of reading comprehension and mathematics instruction was explored in five urban middle schools using the Instructional Quality Assessment (IQA) toolkit (N = 34 teachers). The IQA is comprised of protocols for rating observed instruction and the quality of teachers’ assignments with student work. The purpose of this research was to investigate the reliability and potential validity of the ratings of these data sources. Commensurate with other research on the quality of middle schools, our results indicated that the quality of instruction varied a great deal within schools and was of a ‘basic’ quality overall. Results indicated a moderate to high level of reliability. Four assignments with student work yielded a stable estimate of quality in both content areas, and when teachers complied with the requirements of the research as few as two observations yielded a stable estimate of teaching quality in both content areas as well. The quality of teachers’ observations and assignments were significantly associated in mathematics, but not in reading comprehension. Because of the small sample size it was not possible to apply multi-level models. The relation between the IQA and student achievement on the SAT-10 was explored using linear regression techniques. Results indicated that after controlling for students’ prior achievement, socio-economic status (SES), ethnicity, language, and IEP status, the IQA assignment measure in reading comprehension predicted student achievement on the Total Reading, Reading Comprehension, and Vocabulary subscores of the SAT-10. The observation measure in reading comprehension predicted student outcomes on the Reading Comprehension subscore of the SAT-10 only. In mathematics, the quality of teachers’ assignments predicted students’ achievement on the Procedures subscore of the SAT-10. The quality of observed instruction in mathematics predicted students’ achievement on the Procedures and Total Math subscores. Without accounting for clustering within classrooms and schools as multilevel models do, our linear regression analyses may lead to results that appear stronger than they actually are. Nevertheless our analyses indicate 2 the direction of trend in these relationships and raise important questions regarding which data sources may be best (classroom assignments or observations) for measuring specific aspects of instruction and student outcomes. Additional research with larger samples of teachers is needed to make definitive conclusions about the validity of the IQA ratings and under what conditions one might choose to either observe in classrooms or collect assignments with student work.

#526 – Learning to Write in Urban Elementary and Middle Schools: An Investigation of Teachers' Written Feedback on Student Compositions
Lindsay Clare Matsumura, Rosa Valdés, G. Genevieve Patthey-Chavez

Summary

In writing instruction, feedback from teachers provides a critical opportunity for students to revise their work and improve as writers. Contexts in which students routinely receive feedback on their work include peer reviews and teacher-student conferences. For many teachers, however, written comments on student papers remain a significant method of response. Despite the importance of teacher responses to student work in facilitating the learning process, little research has examined the relationship between teacher feedback on early drafts of student work and the quality of students' subsequent drafts. Even less research has examined the nature of teachers' written feedback to students in K-12 settings. This study investigates the nature of written instructor responses to student writings and the relationship of these written responses to the quality of subsequent student work in urban elementary and middle schools. Most of the 22 instructors who provided the study's corpus of student writings (N = 114) provided their students with some written feedback, and most of their students incorporated that feedback into their subsequent drafts. Instructors tended to focus most on standardizing their students' written output, with measurable success. Student papers received little feedback about content or organization, and these qualities generally did not change over successive drafts.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators