Reports
Please note that CRESST reports were called "CSE Reports" or "CSE Technical Reports" prior to CRESST report 723.
#364 – Dilemmas and Issues for Teachers Developing Performance Assessments in Mathematics
Roberta J. Flexer and Eileen A. Gerstner
Roberta J. Flexer and Eileen A. Gerstner
CSE Report 364, 1993 
Summary
Summary
This report examines some of the dilemmas and issues that arose during the first two terms of work with teachers participating in a mathematics performance assessment development program. Additionally, the authors report on changes in teachers' instruction and assessment as a result of the project. During the study, many dilemmas and issues arose that were unique to each of the three schools studied, but the most challenging problems across all schools were the teachers' focus on what was important to teach, and therefore assess; and how children could learn what was taught--all within the constraints of limited teacher time. As expected, preliminary results of the project were mixed, but hopeful. Researchers believe that future development and implementation of performance assessments in these classrooms hinge on teachers' beliefs in these assessments as useful and practical tools.
#406 – Teachers' and Students' Roles in Large-Scale Portfolio Assessment: Providing Evidence of Competency With the Purposes and Processes of Writing
Maryl Gearhart and Shelby Wolf
Maryl Gearhart and Shelby Wolf
CSE Report 406, 1995
Summary
Summary
From 1992-1994, the California Department of Education and the Center for Performance Assessment of Educational Testing Service were engaged in the development of a new standards-based portfolio component for the California Learning Assessment System (CLAS). Based on interviews with four teachers from different school settings, the researchers sought answers to the following questions: How did teachers participating in trials of the program understand the CLAS Portfolio Assessment Program and how did they use the dimensions of learning to guide their language arts curriculum and assessment practices? How did their students understand the dimensions of learning, and how did they use the dimensions to guide their portfolio choices? What implications do the findings have for large-scale portfolio assessment?
The CRESST researchers found that teachers' curriculum varied, providing students with quite different opportunities to learn about the dimensions of learning measured by the portfolios; teachers also varied in their approach to documentation of students' writing, providing students with different opportunities to demonstrate their competencies with portfolio choices. Findings suggest a need to balance the vision of student choice as a desirable goal for students with what is needed to ensure that portfolio raters are provided appropriate evidence of student performance.
The CRESST researchers found that teachers' curriculum varied, providing students with quite different opportunities to learn about the dimensions of learning measured by the portfolios; teachers also varied in their approach to documentation of students' writing, providing students with different opportunities to demonstrate their competencies with portfolio choices. Findings suggest a need to balance the vision of student choice as a desirable goal for students with what is needed to ensure that portfolio raters are provided appropriate evidence of student performance.
#512 – Professional Development: A Key to Kentucky's Reform Effort
Hilda Borko, Rebekah Elliott, and Kay Uchiyama
Hilda Borko, Rebekah Elliott, and Kay Uchiyama
CSE Report 512, 1999
Summary
Summary
Educational reform leaders generally agree that professional development opportunities for teachers are crucial to the success of any effort to make meaningful, sustainable changes in educational practice. As Fullan (1991) explained, "Continuous development of all teachers is the cornerstone for meaning, improvement, and reform. Professional development and school development are inextricably linked" (p. 315). Kentucky Department of Education (KDE) personnel charged with the responsibility to operationalize the Kentucky Educational Reform Act (KERA) understood this link. They developed an extensive professional development (PD) program to help Kentucky educators achieve the ambitious KERA goals. In this paper we describe the Department's multi-faceted approach to professional development and provide evidence for its impact on schools' achievement of KERA goals. We draw upon data from the exemplary case study component of a larger research project, The Effects of Standards-Based Assessments on Schools and Classrooms.
Perhaps the biggest challenge that KDE faced in providing PD services was geography. Many of Kentucky's school districts are located in remote rural areas, accessible only by mountain roads which are particularly treacherous to travel during the winter months. To reach these districts, KDE relied on a system of nine regional service centers, which provided a wide variety of services to districts, schools, and individual teachers. However, as Ed Reidy, then-Deputy Commissioner of Education, explained, "We have a real commitment that what kids learn should not be a function of geography --You could draw a circle around [the regional service] centers. Most of our audited schools were outside those circles and most were poor." To supplement the work of the centers, KDE developed a variety of materials and activities specifically designed to meet emerging needs of teachers as they worked to achieve KERA goals. This paper focuses on the two major categories of services--school-based professional development and professional development for mathematics and writing portfolios.
All four case study schools exhibited a strong commitment to professional development and a belief in the importance of ongoing support for teacher learning. They used state PD resources to enhance their instructional programs in areas explicitly connected to KERA, such as curriculum alignment and development of materials and activities keyed to the core content standards. Further, teachers at each school served in leadership roles in the KDE Division of Portfolio Initiatives professional development activities. These teachers saw their leadership roles as benefiting their schools, their students, and their colleagues, as well as supporting their own professional growth. Thus, using state resources and opportunities, these four exemplary schools created extensive professional development programs to suit the specific needs of their teachers and students. Through their successful efforts, they provide an existence proof that Kentucky's approach to professional development can provide the resources needed to support statewide, standards-based educational reform. The paper concludes with recommendations for approaches to professional development that seem to hold promise for facilitating statewide standards-based educational reform efforts.
Perhaps the biggest challenge that KDE faced in providing PD services was geography. Many of Kentucky's school districts are located in remote rural areas, accessible only by mountain roads which are particularly treacherous to travel during the winter months. To reach these districts, KDE relied on a system of nine regional service centers, which provided a wide variety of services to districts, schools, and individual teachers. However, as Ed Reidy, then-Deputy Commissioner of Education, explained, "We have a real commitment that what kids learn should not be a function of geography --You could draw a circle around [the regional service] centers. Most of our audited schools were outside those circles and most were poor." To supplement the work of the centers, KDE developed a variety of materials and activities specifically designed to meet emerging needs of teachers as they worked to achieve KERA goals. This paper focuses on the two major categories of services--school-based professional development and professional development for mathematics and writing portfolios.
All four case study schools exhibited a strong commitment to professional development and a belief in the importance of ongoing support for teacher learning. They used state PD resources to enhance their instructional programs in areas explicitly connected to KERA, such as curriculum alignment and development of materials and activities keyed to the core content standards. Further, teachers at each school served in leadership roles in the KDE Division of Portfolio Initiatives professional development activities. These teachers saw their leadership roles as benefiting their schools, their students, and their colleagues, as well as supporting their own professional growth. Thus, using state resources and opportunities, these four exemplary schools created extensive professional development programs to suit the specific needs of their teachers and students. Through their successful efforts, they provide an existence proof that Kentucky's approach to professional development can provide the resources needed to support statewide, standards-based educational reform. The paper concludes with recommendations for approaches to professional development that seem to hold promise for facilitating statewide standards-based educational reform efforts.
#672 – Assessing Academic Rigor in Mathematics Instruction: The Development of the Instructional Quality Assessment Toolkit
Melissa Boston and Mikyung Kim Wolf
Melissa Boston and Mikyung Kim Wolf
CSE Report 672, 2006
Summary
Summary
The development of an assessment tool to measure the quality of instruction is necessary to provide an informative accountability system in education. Such a tool should be capable of characterizing the quality of teaching and learning that occurs in actual classrooms, schools, or districts. The purpose of this paper is to describe the development of the Academic Rigor in Mathematics (AR-Math) rubrics of the Instructional Quality Assessment Toolkit and to share the findings from a small pilot study conducted in the Spring of 2003. The study described in this paper examined the instructional quality of mathematics programs in elementary classrooms in two urban school districts. The study assessed the reliability of the AR-Math rubrics, the ability of the AR-Math rubrics to distinguish important differences between districts, the relationships between rubric dimensions, and the generalizability of the assignment collection. Overall, exact reliability ranged from poor to fair, though 1-point reliability was excellent. Even with the small sample size, the rubrics were capable of detecting difference in students’ opportunities to learn mathematics in each district. The paper concludes by suggesting how the AR-Math rubrics might serve as professional development tools for mathematics teachers.
#791 – Evaluation of the Enhanced Assessment Grants (EAGs) SimScientists Program: SITE VISIT FINDINGS
Joan Herman, Yunyun Dai, Aye Mon Htut, Marcela Martinez, and Nichole Rivera
Joan Herman, Yunyun Dai, Aye Mon Htut, Marcela Martinez, and Nichole Rivera
CRESST Report 791, May 2011
Summary
Summary
This evaluation report addresses the implementation, utility, and feasibility of simulation-based assessments for middle school science classrooms, with particular attention to the use of accommodations available in the program. The SimScientists program includes embedded, formative assessments; reflection activities designed to deepen student understanding of key ideas and processes; and benchmark assessments to gauge student learning at the end of the unit. While teachers and students alike were very positive about their experiences with SimScientists, the evaluation team offered several recommendations for improvementin particular, recommendations for refining the program¹s embedded and benchmark assessments and for increasing the feasibility of the reflection activities.
#495 – Tensions Between Competing Pedagogical and Accountability Commitments for Exemplary Teachers of Mathematics in Kentucky
Hilda Borko and Rebekah Elliott
Hilda Borko and Rebekah Elliott
CSE Report 495, 1998
Summary
Summary
This paper presents a focused case study of Ann and Kay, a team of exemplary elementary teachers, as they worked to modify their mathematics instruction to be consistent with the goals of the Kentucky Education Reform Act (KERA) and Kentucky Instructional Results Information System (KIRIS, its innovative high-stakes assessment system). At the time of our work with Ann and Kay, the mathematics component of KIRIS included three types of measures: open response items, multiple choice items, and mathematics portfolios (in a research and development phase), which together assessed students' understanding of concepts and procedures, as well as their ability to use this understanding to solve problems in other disciplines and real life.
Ann and Kay's efforts to guide students' creation of mathematics portfolios and prepare them for the open response item format focused on increased attention to problem solving, mathematical communication, and connections to real world situations. They often found themselves faced with tensions and struggles as they attempted to put policy into practice without compromising their pedagogical goals and beliefs. .
In this case study, we discuss how they worked with these tensions to create a successful reform-based mathematics program in their 4-5 classroom.
Ann and Kay's efforts to guide students' creation of mathematics portfolios and prepare them for the open response item format focused on increased attention to problem solving, mathematical communication, and connections to real world situations. They often found themselves faced with tensions and struggles as they attempted to put policy into practice without compromising their pedagogical goals and beliefs. .
In this case study, we discuss how they worked with these tensions to create a successful reform-based mathematics program in their 4-5 classroom.
#471 – Teachers' Shifting Assessment Practices in the Context of Educational Reform in Mathematics
Geoffrey B.Saxe, Megan L. Franke, Maryl Gearhart, Sharon Howard, and Michele Crockett
Geoffrey B.Saxe, Megan L. Franke, Maryl Gearhart, Sharon Howard, and Michele Crockett
CSE Report 471, 1997
Summary
Summary
This paper presents a study of primary and secondary mathematics teachers' changing assessment practices in the context of policy, stakeholder, and personal presses for change. Using survey and interviews, we collected teachers' reports of their uses of three forms of assessment, one linked to traditional practice (exercises), and two linked to reforms in mathematics education (open ended problems and rubrics). Findings revealed several trajectories of change in the interplay between assessment forms and the functions that they serve. Teachers may implement new assessment form in ways that serve 'old' functions; teachers may re-purpose 'old' assessment forms in ways that reveal students' mathematical thinking. Our developmental framework provides a way to understand the dynamics of teacher development in relation to ongoing educational reforms.
#671 – Overview of the Instructional Quality Assessment
Brian Junker, Yanna Weisberg, Lindsay Clare Matsumura, Amy Crosson, Mikyung Kim Wolf, Allison Levison, and Lauren Resnick
Brian Junker, Yanna Weisberg, Lindsay Clare Matsumura, Amy Crosson, Mikyung Kim Wolf, Allison Levison, and Lauren Resnick
CSE Report 671, 2006
Summary
Summary
Educators, policy-makers, and researchers need to be able to assess the efficacy of specific interventions in schools and school Districts. While student achievement is unquestionably the bottom line, it is essential to open up the educational process so that each major factor influencing student achievement can be examined; indeed as a proverb often quoted in industrial quality control goes, “That which cannot be measured, cannot be improved”. Instructional practice is certainly a central factor: if student achievement is not improving, is it because instructional practice is not changing, or because changes in instructional practice are not affecting achievement? A tool is needed to provide snapshots of instructional practice itself, before and after implementing new professional development or other interventions, and at other regular intervals to help monitor and focus efforts to improve instructional practice. In this paper we review our research program building and piloting the Instructional Quality Assessment (IQA), a formal toolkit for rating instructional quality based primarily on classroom observation and student assignments. In the first part of the paper we review the need for, and some other efforts to provide, direct assessments of instructional practice. In the second part of this paper we briefly summarize the development of the IQA in reading comprehension and in mathematics at the elementary school level. In the third part of the paper we report on a large pilot study of the IQA, conducted in Spring 2003 in two moderately large urban school Districts. We conclude with some ideas about future work and future directions for the IQA.
#466 – I've Seen This Before? The Effects of Self-Monitoring and Multiple Context Instruction on Knowledge Representation and Transfer Among Middle School Students
Davina C. D. Klein
Davina C. D. Klein
CSE Report 466, 1998
Summary
Summary
Both multiple context learning and self-reflection training are posited to affect students' knowledge representations by fostering decontextualization, abstraction, and schema formation. Schemata, in turn, theoretically facilitate transfer. One hundred eighty-six low-SES middle school students of mixed ethnicities were taught to use concept mapping as a means of understanding material in either one subject area or two subject areas. In addition, half of the students in each group were trained in metacognitive self-monitoring techniques. The transfer task was a problem in a third subject area.
Students were asked to complete the transfer task and then to complete three questionnaires, one eliciting alternative solutions to the transfer task, one assessing their schemata, and one addressing their metacognitive activity. In addition, a small, randomly selected subsample of students from each treatment group did not take the transfer task, instead completing only the questionnaires.
It was hypothesized that students who both engaged in self-monitoring and were exposed to two subject areas would form better schemata, engage in greater metacognitive activity, and perform better on the transfer measure than other students. Although the main predictions were not confirmed, some support was found for the beneficial effects of monitoring on schema formation. In addition, it was found that, given a relatively brief treatment period, at-risk students were able to learn the cognitive strategy of concept mapping, to engage in metacognitive activities such as self-monitoring, to construct good concept mapping schemata, and to transfer to a large degree. Results are discussed and suggestions are made for future work in this area.
Students were asked to complete the transfer task and then to complete three questionnaires, one eliciting alternative solutions to the transfer task, one assessing their schemata, and one addressing their metacognitive activity. In addition, a small, randomly selected subsample of students from each treatment group did not take the transfer task, instead completing only the questionnaires.
It was hypothesized that students who both engaged in self-monitoring and were exposed to two subject areas would form better schemata, engage in greater metacognitive activity, and perform better on the transfer measure than other students. Although the main predictions were not confirmed, some support was found for the beneficial effects of monitoring on schema formation. In addition, it was found that, given a relatively brief treatment period, at-risk students were able to learn the cognitive strategy of concept mapping, to engage in metacognitive activities such as self-monitoring, to construct good concept mapping schemata, and to transfer to a large degree. Results are discussed and suggestions are made for future work in this area.
#706 – Moving to the Next Generation System Design: Integrating Cognition, Assessment, and Learning
Eva L. Baker
Eva L. Baker
CSE Report 706, 2007
Summary
Summary
This paper will describe the relationships between research on learning and its application in assessment models and operational systems. These have been topics of research at the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) for more than 20 years and form a significant part of the intellectual foundation of our present research Center supported by the Institute of Education Sciences. This description serves as the context for the presentation of CRESST efforts in building the POWERSOURCE© assessment system as described in subsequent papers delivered at Session N2 of the 2006 annual meeting of the National Council on Measurement in Education.
