National Center for Research on Evaluation, Standards and Student Testing

#637 – Children Left Behind In AYP and Non-AYP Schools: Using Student Progress and the Distribution Of Student Gains to Validate AYP
Kilchan Choi, Michael Seltzer, Joan Herman, and Kyo Yamashiro

Summary

The No Child Left Behind Act (NCLB, 2002) establishes ambitious goals for increasing student learning and attaining equity in the distribution of student performance. Schools must assure that all students, including all significant subgroups, show adequate yearly progress toward the goal of 100% proficiency by the year 2014. In this paper, we wish to illustrate an alternative way of measuring AYP that both emphasizes individual student growth over time, and focuses on the distribution of student growth between performance subgroups. We do so through analyses of a longitudinal dataset from an urban school district in Washington. We also examine what these patterns tell us about schools that meet their AYP targets and those that do not. This alternative way of measuring AYP helps bring to light potentially important aspects of school performance that might be masked if we limit our focus to classifying schools based only on current AYP criteria. In particular, we are able to identify some schools meeting the Washington state criteria for AYP, for example, that have above average students making substantial progress but below average students making little to no progress. In contrast, other schools making AYP have below average students making adequate progress but above average students making little gains. These contrasts raise questions about the meaning of "adequate" progress and to whom the notion of progress is referring. We believe that closely examining the distribution of student progress may provide an important supplementary or alternative measure of AYP.

#633 – Issues in Assessing English Language Learners’ Opportunity to Learn Mathematics
Joan L. Herman and Jamal Abedi

Summary

The Annual Yearly Progress (AYP) requirements of No Child Left Behind (NCLB) underscore both the mandate and the challenge of assuring that English Language Learners (ELL) achieve the same high standards of performance that are expected of their native English speaking peers. The intent indeed is laudable: states, districts, schools, and teachers must be accountable for the learning of their ELLs as are the students themselves. ELLs can no longer be invisible in the educational system, their learning needs must be met, and they too must make steady progress the goal of all students being judged proficient based on statewide testing by the year 2014. Already, however, NCLB results suggest a different reality: ELL subgroups are being left behind and schools and districts serving significant proportions of ELLs are less likely to meet their AYP goals and more likely to be subject to corrective action. Fairness demands that ELLs have equitable opportunity to learn (OTL) that upon which they are assessed, especially if those assessments carry significant future consequences. Moreover, if NCLB goals are to be met and achievement gaps reduced, schools must move beyond the performance only orientation of AYP to understand why results are as they are and how to improve them. OTL data can help to provide guidance in these areas and to acknowledge the reality that ELLs’ learning is unlikely to improve unless and until students have more effective opportunities to attain expected performance standards. We view this study as an interesting beginning. It was conceived as a pilot, the results of which add fuel to the concern for and underscore some of the complexities of adequately measuring OTL, and we look forward to the full study involving a larger and more representative sample of teachers and classrooms and a more robust outcome measure.

#732 – Issues in Assessing English Language Learners: English Language Proficiency Measures and Accommodation Uses--Practice Review
Mikyung Kim Wolf, Jenny Kao, Noelle Griffin, Joan L. Herman, Patina L. Bachman, Sandy M. Chang, Tim Farnsworth

Summary

The No Child Left Behind (NCLB) has had a great impact on states’ policies in assessing English language learner (ELL) students. The legislation requires states to develop or adopt sound assessments in order to validly measure the ELL students’ English language proficiency, as well as content knowledge and skills. While states have moved rapidly to meet these requirements, they face challenges to validate their current assessment and accountability systems for ELL students, partly due to the lack of resources. Considering the significant role of an assessment in guiding decisions about organizations and individuals, it is of paramount importance to establish a valid assessment system. In light of this, we reviewed the current literature and policy regarding ELL assessment in order to inform practitioners of the key issues to consider in their validation process. Drawn from our review of literature and practice, we developed a set of guidelines and recommendations for practitioners to use as a resource to improve their ELL assessment systems. We have compiled a series of three reports. The present report is the second component of the series, providing a comprehensive picture of states’ current policies related to ELL assessment. The areas reviewed include the procedures of ELL identification and redesignation, the characteristics of English language proficiency assessments, including validity information, and the use of accommodations in the assessment of content knowledge.

#568 – Collaborative, School-Based Professional Development Settings for Teachers: Implementation and Links to Improving the Quality of Classroom Practice and Student Learning
Lindsay Clare Matsumura and Joan Rector Steinberg

Summary

This study examined how cognitive and motivational factors jointly contributed to science achievement, engagement, and choice of science-related majors and careers in a sample of 491 high school students. Students completed cognitive and motivational measures in three different sessions: (a) a survey of motivational processes, including competence beliefs, task values, and behavioral engagement in the science classroom; (b) assessments of fluid, crystallized, and spatial abilities; and (c) a science achievement test. Results of regression analyses showed that the inclusion of motivational variables enhanced the predictive validity for science achievement. General ability was the strongest predictor of achievement outcomes, whereas motivational variables were the strongest predictors of engagement and choice. General ability had a direct effect on achievement and an indirect effect through the mediation of competence beliefs. Competence beliefs and task values had direct effects on achievement and indirect effects through the mediation of engagement. The study highlights the differential predictive validity of cognitive and motivational factors for different types of outcome and corroborates the mediational pathways linking self-system processes, action, and outcomes.

#349 – Cognitive Theory as the Basis for Design of Innovative Assessment: Design Characteristics of Science Assessments
Robert Glaser, Kalyani Raghavan, and Gail Baxter

Summary

Part of a long-range goal to investigate the validity of reasoning and problem-solving assessment tasks in science, this report describes progress in analyzing several science performance assessment projects. The authors discuss developments from Connecticut's Common Core of Learning Assessment Project, the California Assessment Program, and the University of California, Santa Barbara/California Institute of Technology research project "Alternative Technologies for Assessing Science Understanding." The analysis framework articulates general aspects of problem-solving performance, including structured, integrated knowledge; effective problem representation; proceduralized knowledge; automaticity; and self-regulatory skills.

#439 – American Students' Perspectives on Alternative Assessment: Do They Know It's Different?
Joan Herman, Davina C.D.Klein, and Sara T.Wakai

Summary

This study used the 1993 California Learning Assessment System (CLAS) Middle Grades Mathematics Performance Assessment as a platform to examine alternative assessment in actual practice in the U.S. Reported here is information gathered using the CLAS regarding student attitudes and approaches toward this new type of assessment. At issue is whether students find alternative assessments to be more motivating and interesting than traditional types of tests, and whether they appreciate the difference between traditional and alternative tasks. Data were collected in 13 schools across the state of California, involving more than 800 students. Instrumentation used in data collection included student surveys and in-depth student retrospective interviews. Findings suggest that students do indeed understand the differences in approaches necessitated by novel open-ended tasks versus more familiar multiple-choice tasks. In addition, student attitudes toward these two types of tasks are discussed in detail.

#750 – Some Aspects of the Technical Quality of Formative Assessments in Middle School Mathematics
Julia Phelan, Taehoon Kang, David N. Niemi, Terry Vendlinski, Kilchan Choi

Summary

While research suggests that formative assessment can be a powerful tool to support teaching and learning, efforts to jump on the formative assessment bandwagon have been more widespread than those to assure the technical quality of the assessments. This report covers initial analyses of data bearing on the quality of formative assessments in middle school mathematics. Specifically, these data address the question of whether relatively short assessments can provide reliable and useful information on middle school students' understanding of conceptual domains in pre-algebra. Items and test forms were developed and tested in four domains (rational number equivalence, properties of arithmetic, principles for solving equations, and applications of these concepts to other domains), all of which are critical to eventual mastery of algebra. We tested the items with sixth-grade students in classrooms in four districts. We then pared down the items to create eight assessment forms that were further tested alongside instructional support materials and professional development. Results of this study suggest that relatively brief formative assessments focused on key conceptual domains can provide reliable and useful information on students’ levels of understanding and possible misunderstandings in the domain.

#352 – Collaborative Group Versus Individual Assessment in Mathematics: Group Processes and Outcomes
Noreen Webb

Summary

Several states, such as Connecticut and California, are attempting to incorporate group assessment into their large-scale testing programs. One intention of such efforts is to use scores from group assessments as indicators of individual performance. However, a key technical question for such assessments is "to what extent do scores on a group assessment actually represent individual performance or knowledge." This study by UCLA professor and CRESST researcher Noreen Webb sheds some light on this substantial technical question. Webb gave two seventh-grade classes an initial mathematics test as a group assessment, where exchange of information and assistance was common. Several weeks later, she administered a nearly identical individual test to the same students where assistance was not permitted. The results showed that some students' performance dropped significantly from the group assessment to the individual test. These students apparently depended on the resources of the group in order to get correct answers and when the same resources were not available during the individual test, many of the students were not able to solve the problem. Webb concluded: "Scores from a group assessment may not be valid indicators of some students' individual competence. Furthermore, achievement scores from group assessment contexts provide little information about group functioning." Webb's study suggests that states or school districts who intend to assign individual scores based on group assessments may want to seriously rethink their intentions.

#793 – The Development and Impact of POWERSOURCE©: Year 3
Julia Phelan, Terry Vendlinski, Kilchan Choi, Joan Herman, and Eva L. Baker

Summary

The POWERSOURCE© intervention is intended as a powerful formative assessment-based strategy that can be integrated with any ongoing mathematics curriculum to improve middle school teachers' knowledge and practice and, in turn, student learning. In this Year 3 (of 5) study, researchers found significant differences between POWERSOURCE© and control students' performance on project-developed measures for all districts and domains.

#738 – Providing Validity Evidence to Improve the Assessment of English Language Learners
Mikyung Kim Wolf, Joan L. Herman, Jinok Kim, Jamal Abedi, Seth Leon, Noelle Griffin, Patina L. Bachman, Sandy M. Chang, Tim Farnsworth, Hyekyung Jung, Julie Nollner, Hye Won Shin

Summary

This research project addresses the validity of assessments used to measure the performance of English language learners (ELLs), such as those mandated by the No Child Left Behind Act of 2001 (NCLB, 2002). The goals of the research are to help educators understand and improve ELL performance by investigating the validity of their current assessments, and to provide states with much needed guidance to improve the validity of their English language proficiency (ELP) and academic achievement assessments for ELL students. The research has three phases. In the first phase, the researchers analyze existing data and documents to understand the nature and validity of states’ current practices and their priority needs. This first phase is exploratory in that the researchers identify key validity issues by examining the existing data and formulate research areas where further investigation is needed for the second phase. In the second phase of the research, the researchers will deepen their analysis of the areas identified from Phase I findings. In the third phase of the research, the researchers will develop specific guidelines on which states may base their ELL assessment policy and practice. The present report focuses on the researchers' Phase I research activities and results. The report also discusses preliminary implications and recommendations for improving ELL assessment systems.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators