National Center for Research on Evaluation, Standards and Student Testing

#384 – A Conceptual Framework for Analyzing the Costs of Alternative Assessment
Lawrence Picus

Summary

Despite the fact that many states are investing millions of dollars into the development of alternative assessments, little is known about the actual costs of such assessments. In A Conceptual Framework for Analyzing the Costs of Alternative Assessments, CRESST partner Lawrence O. Picus analyzes many of the issues related to identifying the costs of new assessments including the relationship between costs and goals. "If, as is often the case in education," says Picus, "there are multiple goals established for an alternative assessment program, then estimation of the costs of that program must include all of the resources necessary to accomplish all of those goals." To identify alternative assessment costs, Picus suggests the use of a three dimensional model comprised of levels of expenditures, kinds of expenditures, and expenditure components. Levels of expenditures are the source of expense such as national, state, district, school, classroom, or private market levels. Kinds of expenditures include personnel, materials, supplies, and travel. Components include assessment development, production, training, scoring, reporting and program evaluation. "The largest single expenditure item in any assessment program," concludes Picus, "seems likely to be personnel." Opportunity costs must also be considered, adds Picus. Resources committed to creating an alternative assessment program are resources used to support a former testing program or resources that could be spent on other programs, such as bilingual education. The framework developed in A Conceptual Framework for Analyzing the Costs of Alternative Assessment addresses both opportunity costs and assessment costs matched to goals.

#678 – Consequences and Validity of Performance Assessment for English Language Learners: Integrating Academic Language and ELL Instructional Needs into Opportunity to Learn Measures
Zenaida Aguirre-Munoz, Christy Kim Boscardin, Barbara Jones, Jae-Eun Park, Marjorie Chinen, Hye Sook Shin, Janet Lee, Anastasia Aimee Amabisca, and Aprile Benner

Summary

To further the understanding of ELL (under) achievement and broaden the current scope of OTL models, the primary focus of this study was to investigate process and content opportunities that are particularly relevant to improving ELL achievement with particular attention to the relationship between opportunities to acquire academic language and ELL achievement. The work reported here operationalized academic language within a systemic functional linguistics theory (also called “functional grammar”). This theory of language use provided teachers and students with a framework for analyzing language in a manner that both built on existing language knowledge (or schemas) and provided them with an accessible structure for examining more complicated language (and content) concept. Based on our literature review, previous findings, and new ELL-sensitive OTL model, we investigated the following research questions in this study: (a.) To what extent and in what ways are students being exposed to key OTL variables in classrooms? (b). What is the impact of academic language and other OTL indicators on ELLs’ and non-ELLs’ performance on LAPA? After an introduction and literature review, the remainder of this report addresses sources of data, description of instruments and procedures, and types of analyses selected for the study; findings related to the research questions; and concludes with a discussion based on the findings. One of the most important findings from this study points to the need for explicit instruction on academic language. The positive impact of functional grammar implementation on student outcomes also suggests that in order for ELLs to fully benefit from assessment-driven reform, teachers need the capacity to make the linguistic expectations clear to students by focusing on the linguistic elements that are characteristic of academic registers.

#385 – The Evolution of a Portfolio Program: The Impact and Quality of the Vermont Program in Its Second Year (1992-1993)
Daniel Koretz, Brian Stecher, Stephen Klein, and Daniel McCaffrey

Summary

Part of an ongoing evaluation of the Vermont portfolio assessment program by RAND/CRESST researchers, this reports presents recent analyses of the reliability of Vermont portfolio scores, and the results of school principal interviews and teacher questionnaires. The message, especially from Vermont teachers, say the researchers, remains mixed. Math teachers, for example, have modified their curricula and teaching practices to emphasize problem solving and mathematical communication skills, but many feel they are doing so at the expense of other areas of the curriculum. About one-half of the teachers report that student learning has improved, but an equal number feel that there has been no change. Additionally, teachers reported great variation in the implementation of portfolios into their classroom, including the amount of assistance provided to students. "One in four teachers," found the authors, "does not assist his or her own students in revisions, and a similar proportion does not permit students to help each other. Seventy percent of fourth-grade teachers and thirty-nine percent of eighth-grade teachers forbid parental or other outside assistance." Consequently, students who receive more support from teachers, parents and other students, may have a significant advantage over students who receive little or no outside help. Reliability problems continue. "The degree of agreement," wrote the authors, "among Vermont's portfolio raters was much lower than among raters in studies with other types of constructed response measures." The authors suggest that one cause of the low reliability was the diversity of tasks within each portfolio. Because teachers and students are free to select their own pieces, performance on the tasks is much more difficult to assess than if the work were standardized. Despite these problem areas, support for the portfolio program remains high. Teachers, for example, expressed strong support for expanding portfolios to all grade levels. Seventy percent of principals said that their schools had extended portfolio usage beyond the original Vermont state mandate.

#795 – Progress Report Year 4: National Center for Research on Evaluation, Standards, and Student Testing (CRESST) The Development and Impact of POWERSOURCE©
Eva L. Baker

Summary

The POWERSOURCE© intervention is intended to be a powerful formative assessment strategy that can be integrated with any on-going mathematics curriculum. The key goal is to to improve teachers' knowledge and practice and, in turn, student learning in middle school mathematics. In this 4th year study, the authors provide updated results from the 2006-07 experimental (randomized) field test of POWERSOURCE© and present findings from the 2007-08 school year on student and teacher outcomes. The authors found that a short targeted POWERSOURCE© intervention on key mathematical principles had a positive student performance impact on a transfer measure of related content. POWERSOURCE© had more impact on the higher-performing students than the lower-performing students. They also found a stronger effect on more difficult mathematics items.

#473 – Principals' Views of Mathematics Standards, Frameworks, and Assessment in a Context of Reform
Maryl Gearhart

Summary

The purpose of this study was to gather information on principals' views regarding standards, frameworks, and assessment in mathematics. Based on surveys completed by 96 principals from 35 public school districts in Greater Los Angeles - each principal a past participant in events sponsored by the UCLA Principals' Center - our findings reflect the views of principals interested in improving educational practice.

With regard to standards and frameworks, the findings indicate that the principals' schools were not currently building mathematics programs closely on existing standards and frameworks; however, these principals were prepared to support the future implementation of state and/or district mathematics standards in their schools, and they requested resources and assistance with implementation. The principals disagreed on the need for standards at the school level. With regard to testing, the principals were concerned that parents and students may not understand the results of norm-referenced tests and that norm-referenced tests are not aligned with their instructional programs in mathematics. The principals were likely to favor performance-based measures for program evaluation and reporting and for guiding instruction, and they requested resources and assistance for building teacher capacity with new assessments. However, a large minority of the principals favored the use of both forms of mathematics testing, and some principals favored norm-referenced testing. Thus, although these principals represented administrators engaged in school improvement, they differed in their views regarding accountability testing.

The findings suggest that resolution among the views of administrators lies in the design of mathematics standards that embrace a breadth of knowledge and skill, together with the design of a coherent, standards-based assessment system that integrates multiple measures.

#675 – Preschool Participation and the Cognitive and Social Development of Language-Minority Students
Russell W. Rumberger, Loan Tran

Summary

This study examined participation in preschool and its relationship with the cognitive and social development of language-minority students. Although there is a large body of research that demonstrates the cognitive and social benefits of attending preschool (Barnett, 1995; Gorey, 2001; National Research Council, Committee on Early Childhood Pedagogy, 2000; Vandell, 2004), very little of this research has included language-minority students, or at least those who do not speak English. Either non-English speaking families are not included in the design of the study, such as with the widely cited National Institute for Child Health and Development (NICHD) Early Child Care Study, or the studies are based on cognitive and social assessments that are only conducted in English (e.g., Magnuson, Meyers, Ruhm, & Waldfogel, 2004). Consequently, little is known about participation in and outcomes of preschool for the growing population of language-minority students.

#400 – Portfolio-Driven Reform: Vermont Teachers' Understanding of Mathematical Problem Solving and Related Changes in Classroom Practice
Brian M. Stecher and Karen J. Mitchell

Summary

The Vermont portfolio assessment program, conclude the authors of the report, has had substantial positive effects on fourth-grade teachers' perceptions and practices in mathematics.

"Through the Vermont state training materials and network meetings," say CRESST/RAND researchers Brian Stecher and Karen Mitchell, "teachers have incorporated problem solving into their curriculum and have gained a greater insight into teaching problem-solving skills."

However, the researchers add that teachers do not yet share a common understanding of mathematical problem solving and have not reached agreement on the most essential skills to be taught. The emphasis on the scoring rubrics, for example, has helped teachers focus on some of the important and observable aspects of students' problem solving, but may have caused teachers to neglect other important problem-solving skills not addressed in the scoring rubrics. Additionally, significant variability exists in teaching methods-some teachers "preteach" portfolio tasks by assigning similar, simpler problems prior to student work on portfolio pieces so that assessment problems are not overly novel or difficult for students. Differential help may threaten the validity of portfolio scores for comparison of students, classrooms or schools.

The Vermont Department of Education, conclude the authors, should orient existing professional teacher development programs towards increasing teachers' basic understanding of mathematical problem solving and related instructional practices.

Effects of Introducing Classroom Performance Assessments on Student Learning is one of the first empirical examinations of the link between student achievement and performance assessment. Achievement results were compared for both treatment and control schools, where the schools were matched on demographics and socioeconomic factors. Assessments from the Maryland State Department were selected as independent measures of student performance in both sets of classrooms because they are still relatively standardized test-like compared to many performance assessments, but markedly different from traditional standardized tests.

"Our concluding advice," write the researchers, "is that reformers take seriously the current rhetoric about `delivery standards' and the need for sustained professional development to implement a thinking curriculum. The changes that did occur...confirm our beliefs that many more students can develop conceptual understandings presently exhibited by only the most able students-if only they are exposed to relevant problems and given the opportunity-to-learn."

#799 – Evaluation of Green Dot's Locke Transformation Project: Findings from the 2007-08, 2008-09, and 2009-10 School Years
Joan L. Herman, Jordan Rickles, Mark Hansen, Larry Thomas, Alice Gualpa and Jia Wang

Summary

In the fall of 2007, Alain Leroy Locke High School, historically one of California's lowest performing secondary schools, began its transition into a set of smaller, Green Dot Charter High Schools. This report summarizes early CRESST evaluation findings, examining how Green Dot Locke (GDL) students were performing in terms of school persistence, attendance, course-taking and completion, and achievement on standardized tests. CRESST found positive early trends. Relative to comparison high schools, GDL students' overall total enrollment in core courses increased over the years. Furthermore, GDL pass rates increased for particular courses and California High School Exit Exam (CAHSEE) scores continued to rise. The results from matched samples suggest that on various student outcome measures, 9th graders who entered GDL often did better than (and at least as well as) they would have if they attended a comparable LAUSD high school. Given the progressive pattern of increasing results for each new group of entering students, broader effects may well materialize as current students progress through high school and on to graduation.

#815 – Evaluation of Green Dot’s Locke Transformation Project: Findings for Cohort 1 and 2 Students
Joan L. Herman, Jia Wang, Jordan Rickles, Vivian Hsu, Scott Monroe, Seth Leon, and Rolf Straubhaar

Summary

With funding from the Bill and Melinda Gates Foundation, CRESST conducted a multi-year evaluation of a major school reform project at Alain Leroy Locke High School, historically one of California’s lowest performing secondary schools. Beginning in 2007, Locke High School transitioned into a set of smaller, Green Dot Charter High Schools, subsequently referred to as Green Dot Locke (GDL) in this report. Based on 9th grade students who entered GDL in 2007 and 2008 respectively, CRESST used a range of student outcomes to monitor progress of the GDL transformation. The CRESST evaluation, employing a strong quasi-experimental design with propensity score matching, found statistically significant, positive effects for the GDL transformation including improved achievement, school persistence, and completion of college preparatory courses.

#484 – Instructional Validity, Opportunity to Learn and Equity: New Standards Examinations for the California Mathematics Renaissance
Bokhee Yoon and Lauren Resnick

Summary

In this report, CRESST researchers examined the relationship between professional development opportunities for teachers, the kinds of instruction offered to students, and student performance on the New Standards Mathematics Reference Examination. By comparing teachers (and their students) who had participated in the California Mathematics Renaissance professional development program with teachers and students elsewhere the researchers were able to evaluate both the effectiveness of the Renaissance program and the instructional validity of the Reference Examination.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators