The purpose of this study was to gather information on principals' views regarding standards, frameworks, and assessment in mathematics. Based on surveys completed by 96 principals from 35 public school districts in Greater Los Angeles - each principal a past participant in events sponsored by the UCLA Principals' Center - our findings reflect the views of principals interested in improving educational practice.

With regard to standards and frameworks, the findings indicate that the principals' schools were not currently building mathematics programs closely on existing standards and frameworks; however, these principals were prepared to support the future implementation of state and/or district mathematics standards in their schools, and they requested resources and assistance with implementation. The principals disagreed on the need for standards at the school level. With regard to testing, the principals were concerned that parents and students may not understand the results of norm-referenced tests and that norm-referenced tests are not aligned with their instructional programs in mathematics. The principals were likely to favor performance-based measures for program evaluation and reporting and for guiding instruction, and they requested resources and assistance for building teacher capacity with new assessments. However, a large minority of the principals favored the use of both forms of mathematics testing, and some principals favored norm-referenced testing. Thus, although these principals represented administrators engaged in school improvement, they differed in their views regarding accountability testing.

The findings suggest that resolution among the views of administrators lies in the design of mathematics standards that embrace a breadth of knowledge and skill, together with the design of a coherent, standards-based assessment system that integrates multiple measures.

Collaborative problem solving and collaborative skills are considered necessary skills for success in today's world. Collaborative problem solving is defined as problem solving activities that involve interactions among a group of individuals. Large-scale and small-scale assessment programs increasingly use collaborative group tasks in which students work together to solve problems or to accomplish projects. This study attempts to research the role of feedback on a computer-based collaborative problem solving task by extending Hsieh and O’Neil’s (2002) computer-based collaborative knowledge mapping study. In their study, groups of students searched a web environment of information to improve a knowledge map. Various types of feedback were investigated. They found that searching has a negative relationship with group outcome (knowledge map scores). By teaching searching and by providing different types of feedback, this study explores the effects of students’ teamwork and problem solving processes on students’ knowledge mapping performance. Moreover, the effects of two types of feedback (adapted knowledge of response feedback and task-specific adapted knowledge of response feedback) were also investigated. One hundred and twenty college students (60 groups) participated in the main study. The students were randomly assigned either to be a group leader whose responsibility was to construct the map or to be a group searcher whose responsibility was to help the leader construct the map by seeking information and accessing feedback from the Web environment. Results showed that task-specific adapted knowledge of response feedback was significantly more beneficial to group outcome than adapted knowledge of response feedback. In addition, as predicted, for the problem solving process, information seeking including request of feedback, browsing, searching for information and searching using Boolean operators were all significantly related to group outcome for both groups. 2 Lastly, this study confirmed that computer-based performance assessment in collaborative problem solving was effective. The collaboration between team members and individual students’ problem solving processes and strategies were effectively recorded by the computers. In addition, the use of computers to assess and report group interaction and students’ thinking processes was proven to be more inexpensive and less time consuming than other alternatives.

We have developed a prototype authoring system for knowledge mapping that provides the capability to (a) specify map tasks (i.e., specify terms and links), (b) launch a knowledge map related to a task, (c) specify criterion or expert maps (for scoring purposes), (d) define groups of users and associate tasks to those groups, and (e) publish tasks for global access. The prototype interface is a simple HTML interface, with any destination function accessible within four mouse clicks. A usability study was conducted with one teacher and 62 fourth- and fifth-grade students. Students were considered beta testers, and the teacher, who had used our existing knowledge mapper, was considered an expert. The teacher reported using the authoring system to create pretest and posttest tasks for students and to access and evaluate student maps. Students used the authoring system to create knowledge maps of their group research projects. The teacher reported that the authoring system was easy to use and thought student authoring promoted ownership of work. Students using the authoring system created more sophisticated knowledge maps, especially in the links specified by students. This anecdotal report is bolstered by analyses of the links used across all student-authored maps. Thirty-eight percent of the links were causal (e.g., causes) or functional (e.g., protects), and 24% were part-whole (e.g., part of). Limited screen real estate, confusing map access, complicated login functionality, and system crashes limited usability. Student performance on a usability task showed that, in general, students were able to independently carry out major authoring and mapping tasks; however, a positive relationship was found between the amount of time a student was at the keyboard and the amount of help that student needed during the usability task, suggesting that practice using the authoring system improves fluency with the authoring system.

In the latter part of the 1990s, education in California was caught in a whirlwind of change. Schools scrambled to find enough teachers and enough classroom space to fulfill state-mandated class-size reduction requirements. The voters eliminated bilingual education, leaving schools with no specific classroom tool for teaching English-language learners. Schools were required to administer a new standardized test each spring to Grades 2 through 11 that was not aligned to classroom work and yet carried great weight for both students and educators.

Amidst this upheaval, a major new school-reform initiative was trying to make headway in Los Angeles County. From 1994 through 2000, the Los Angeles Annenberg Metropolitan Project, or LAAMP, was one of 18 major school improvement initiatives across the country to be funded by the $1.1 billion Annenberg Challenge. Its centerpiece was a new educational structure known as the School Family, which brought together teachers, administrators, and parents from high schools and their feeder middle schools and elementary schools, plus others with an interest in education. LAAMP organizers hoped the School Families would create a stable learning environment for students by encouraging coordination among schools and between grade levels.

Today, the Annenberg Challenge has drawn to a close. A final report released in June called the national effort a partial success. The report credited the program with strengthening urban, rural, and arts education and with raising the quality of teaching. The report also found that school-reform programs must learn to deal with rapid leadership turnover, changes in direction, and other setbacks. And it found that the grant money, while generous, frequently was spread too thin over too many schools.

The national findings parallel conclusions drawn about the 6-year Los Angeles program, which received $53 million from the Annenberg Challenge in December 1994. LAAMP commissioned a group of education researchers from UCLA and USC to evaluate the local project. Known as the Los Angeles Consortium for Evaluation, or LACE, the researchers found that LAAMP accomplished some of what it set out to do. But for a variety of reasons, it did not attain its ultimate goal of improving student performance.

The tumultuous period of California history that suctioned off time, energy, and financial resources from schools and the people working in them bears much of the blame. Researchers found other explanations. Among them were:
• School Family teams of teachers, administrators, and parents needed more time than was anticipated to develop the group process skills necessary for success and spent much of their time trying to learn how to collaborate instead of instituting change.
• The teams needed time to learn about and understand the concepts of results- or standards-based school reform and to develop the skills required to analyze available data and use them in the planning process.
• There were insufficient resources to ensure adequate support for teachers attempting to implement programs devised by the School Families. There also was no mechanism for extending the reforms to teachers not directly involved in the reform project.

LACE also acknowledged that the research methodologies it used to evaluate LAAMP, although the best that were available, might not have presented a full and accurate picture of the effects the program had on its participating schools. In addition, researchers suggested the need for more sensitive gauges of student accomplishment that measure the actual curriculum taught. The primary measurement used—California’s Stanford 9 test—may not have been the best tool for detecting the effects of specific changes in teaching and learning.

Overall, the researchers found that the LAAMP reform can claim many achievements that have benefited K-12 education in Los Angeles County, including:
• Creation of the School Family concept, which in many cases was responsible for productive changes that could not have been realized by a single school working alone.
• Strengthening of schools’ acceptance of accountability, their focus on performance, and their capacity for self-evaluation especially in regard to accessing and using student-achievement data.
• Creation of valuable teacher professional development activities and access to new instructional programs, which were especially helpful for the many new and uncredentialed teachers who were hired to fulfill class-size reduction requirements.
• Encouragement of parental involvement in the schools and in children’s learning at home, which had demonstrable effects on student performance.
• Demonstration of the potential of stable learning communities for curing many of the ills facing urban schools.

Looking at test scores, LACE researchers saw improvement at LAAMP schools over the 3-year period from 1997-1998 to 2000-2001. However, there was no statistically significant difference between LAAMP schools and non-LAAMP schools with regard to student performance on the state’s Stanford 9 standardized test.

Researchers also found no indication that LAAMP has had a wide impact on classroom practices. In other words, its core school-reform principles have not yet permeated participating schools. However, researchers saw signs that LAAMP initiatives were starting to move into the classroom in the later years of the program after so much time and energy were spent initially on developing the School Family structure.

When Walter Annenberg issued his challenge in December 1993 by giving what at the time was the largest gift ever dedicated to improving public education, he called it a “crusade for the betterment of our country.” Nine years later, that crusade has made a difference. The public schools “in most major cities are still not doing the job they must,” the June report said, but they are “better today than they were a decade ago and teachers are better equipped to help children overcome obstacles and achieve higher standards.”

This study examines measures of formative assessment practice using data from a study of the implementation and effects of adding curriculum embedded measures to a hands-on science program for upper elementary school students. The authors present a unifying conception for measuring critical elements of formative assessment practice, illustrate common measures for doing so, and investigate the relationships among and between scores on these measures. Findings raise important issues with regard to both the challenge of obtaining valid measures of teachers’ assessment practice and the uneven quality nature of current teacher practice.


To cite from this report, please use the following as your APA reference:

Herman, J., L., Osmundson, E., & Silver, D. (2010). Capturing quality in formative assessment practice: Measurement challenges. (CRESST Report 770). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

The National Assessment of Educational Progress (NAEP) test has been accused of underestimating student achievement because this "low-stakes" assessment has no consequences for students, their teachers, or their schools. In contrast, "high-stakes" tests--those assessments that have serious consequences for students, teachers, and schools--are assumed to motivate greater student performance because of the positive or negative consequences (such as college entrance) associated with student performance on the test. The purpose of this study was to investigate whether differences in test administration conditions and presumed levels of motivation created by the different testing environments affect student performance on the NAEP test. The testing conditions studied were the "low-stakes" environment of the current NAEP administration and a "higher-stakes" environment typified by many state assessment programs. The results of the study lead to the conclusion that estimates of achievement from NAEP would not be substantially higher if the stakes were increased to the level associated with a "higher-stakes" test.

The characteristics of expert knowledge-interconnectedness, understanding, and ability to transfer-are inextricably linked, a point that is critically important for educators and constitutes a major theme of this paper. In this paper we explore how an analysis of the architecture of expert knowledge can inform the development of assessments to help teachers move students toward greater expertise in mathematics, and we present examples of such assessments. We also review student responses and preliminary results from pilot tests of assessments administered in sixth-grade classes in a large urban school district. Our preliminary analyses suggest that an assessment strategy based on the structure of mathematical knowledge can reveal deficiencies in student understanding of and ability to apply fundamental concepts of pre-algebra, and has the potential to help teachers remediate those deficiencies.

Findings from this study of fifth grade students provided further evidence that critical differences exist between students who think and reason well with their knowledge and those who do not. In the study, students received six mystery boxes and were asked to identify the contents by making the components into circuits. The research team found that students who displayed consistently high levels of learning and understanding were able to describe a comprehensive plan for an experiment. Further, these same students demonstrated an efficient approach to problem-solving which included the use of scientific principles. In contrast, lower-performing students invoked a trial-and-error strategy of "hook something up and see what happens" to guide their experiments.

Only 20% of the students performed at high levels, suggesting that even low ability students could complete a problem without understanding the processes or principles involved. The researchers concluded that "Strategies for how to represent problems must be taught as well as strategies for how to solve problems." They suggest that teachers use performance assessments, such as this science experiment, to integrate instruction, assessment, and high levels of student learning.

A number of educational researchers are currently developing alternatives to survey and case study methods for measuring instructional practice. These alternative strategies involve gathering and analyzing artifact data related to teachers’ use of instructional materials and strategies, classroom learning activities, and students’ work, and other important features of practice. “The Impact of Accountability Systems on Classroom Practice” is one such effort. The goals of this 5-year project, funded through the National Center for Research on Evaluation, Standards, and Student Testing (CRESST), are to develop artifact collection and scoring procedures designed to measure classroom practice in mathematics and science; validate these procedures through classroom observations, discourse analysis, and teacher interviews; and then use the procedures, in conjunction with other CRESST projects, to conduct comparative studies of the impact of different approaches to school reform on school and classroom practices. The first phase of the project was a set of pilot studies, conducted in a small number of middle school science and mathematics classrooms, to provide initial information about the reliability, validity, and feasibility of artifact collections as measures of classroom practice. This report presents results of these pilot studies.

The purpose of this report is to provide the theoretical rationale for the approach to academic language that was adopted to meet the research goals of the second phase of this project as well as to report on the results from the pilot training program that was developed to create the conditions under which varying levels of direct instruction in academic language occurs. The challenge was to find an approach for the instruction of academic language that would serve a dual purpose. The first purpose was aimed at building teachers' understanding of the key components of academic language to improve their instructional decision-making. The second goal was to provide teachers with tools for providing ELLs with direct instruction on academic language and thereby support their English language development. After careful review of the literature, we found that the functional linguistic approach to language development best met these goals. We developed training modules on writing instruction based on the functional linguistic approach, as it has the strongest potential in providing explicit instruction to support ELL student writing development. Overall, teachers responded positively to the functional linguistic approach and were optimistic about its potential for improving ELL writing development. Responses to the pre-and post institute survey revealed that teachers felt better prepared in evaluating student writing from a functional linguistic perspective as well as in developing instructional plans that targeted specific learning needs.