National Center for Research on Evaluation, Standards and Student Testing

#824 – Evaluation of Green Dot’s Locke Transformation Project: From the Perspective of Teachers and Administrators
Joan Herman, Jia Wang, Christine Ong, Rolf Straubhaar, Jon Schweig, and Vivian Hsu

Summary

In the fall of 2007, Alain Leroy Locke High School, historically one of California’s lowest performing secondary schools, underwent a transformation. Suffering from a history of extremely low academic performance, student unrest and even violence, the nonprofit charter organization Green Dot Public Schools was charged to transition Locke into a set of smaller charter academies, in partnership with the Los Angeles Unified School District (LAUSD).With a grant from the Bill and Melinda Gates Foundation, the National Center for Research on Evaluation, Standards and Student Testing (CRESST), was charged with monitoring the progress and effects of Green Dot Public Schools’ Locke transformation from 2007 to the present. Previous annual reports have presented findings related to the academic performance of Green Dot Locke (GDL) students. The primary focus of this current report, is to use both quantitative (including teachers’ value-added data based on state test scores) and qualitative data (interviews with 13 teachers and four administrators across GDL academies) to explore potential teacher factors influencing students’ academic progress since the transformation, particularly focusing on teacher recruitment/selection, retention, and support.

#815 – Evaluation of Green Dot’s Locke Transformation Project: Findings for Cohort 1 and 2 Students
Joan L. Herman, Jia Wang, Jordan Rickles, Vivian Hsu, Scott Monroe, Seth Leon, and Rolf Straubhaar

Summary

With funding from the Bill and Melinda Gates Foundation, CRESST conducted a multi-year evaluation of a major school reform project at Alain Leroy Locke High School, historically one of California’s lowest performing secondary schools. Beginning in 2007, Locke High School transitioned into a set of smaller, Green Dot Charter High Schools, subsequently referred to as Green Dot Locke (GDL) in this report. Based on 9th grade students who entered GDL in 2007 and 2008 respectively, CRESST used a range of student outcomes to monitor progress of the GDL transformation. The CRESST evaluation, employing a strong quasi-experimental design with propensity score matching, found statistically significant, positive effects for the GDL transformation including improved achievement, school persistence, and completion of college preparatory courses.

#799 – Evaluation of Green Dot's Locke Transformation Project: Findings from the 2007-08, 2008-09, and 2009-10 School Years
Joan L. Herman, Jordan Rickles, Mark Hansen, Larry Thomas, Alice Gualpa and Jia Wang

Summary

In the fall of 2007, Alain Leroy Locke High School, historically one of California's lowest performing secondary schools, began its transition into a set of smaller, Green Dot Charter High Schools. This report summarizes early CRESST evaluation findings, examining how Green Dot Locke (GDL) students were performing in terms of school persistence, attendance, course-taking and completion, and achievement on standardized tests. CRESST found positive early trends. Relative to comparison high schools, GDL students' overall total enrollment in core courses increased over the years. Furthermore, GDL pass rates increased for particular courses and California High School Exit Exam (CAHSEE) scores continued to rise. The results from matched samples suggest that on various student outcome measures, 9th graders who entered GDL often did better than (and at least as well as) they would have if they attended a comparable LAUSD high school. Given the progressive pattern of increasing results for each new group of entering students, broader effects may well materialize as current students progress through high school and on to graduation.

#723 – Recommendations for Building a Valid Benchmark Assessment System: Interim Report to the Jackson Public Schools
David Niemi, Julia Vallone, Jia Wang, Noelle Griffin

Summary

Many districts and schools across the U. S. have begun to develop and administer assessments to complement state testing systems and provide additional information to monitor curriculum, instruction and schools. In advance of this trend, the Jackson Public Schools (JPS) district has had a district benchmark testing system in place for many years. To complement and enhance the capabilities of district and school staff, the Stupski Foundation and CRESST (National Center for Research on Evaluation, Standards, and Student Testing at UCLA) worked out an agreement for CRESST to provide expert review and recommendations to improve the technical quality of the district’s benchmark tests. This report (which represents the first of two deliverables on this project) focuses on assessment development and is consistent with the district goal of increasing the predictive ability of the assessments for students’ state test performance, as well as secondary goals.

#724 – Recommendations for Building a Valid Benchmark Assessment System: Second Report to the Jackson Public Schools
David Niemi, Jia Wang, Haiwen Wang, Julia Vallone, Noelle Griffin

Summary

There are usually many testing activities going on in a school, with different tests serving different purposes, thus organization and planning are key in creating an efficient system in assessing the most important educational objectives. In the ideal case, an assessment system will be able to inform on student learning, instruction and curricula, and district and school administration, as well as providing information to identify and solve educational problems. This report represents the second of two deliverables provided to the Jackson Public Schools (JPS) and provides recommendations, based on ongoing discussions with the district and review of information from the benchmark assessments, on topics related to building a valid benchmark assessment system. We consider how item data can be used to improve benchmark tests over time and also cover what needs to be done to insure that results of a test are correctly interpreted, reported and used.

#719 – Impact of Different Performance Assessment Cut Scores on Student Promotion
Jia Wang, David Niemi, and Haiwen Wang

Summary

The goals of the current effort were to take these previously drafted tasks and create prototypes by trying out the tasks for the first time with 224 students from native English and English language learner (ELL) backgrounds. Students across the 4th-6th grades, as well as native-English students, are included in the studies because native speakers and adjacent grades provide critical information about the targeted language abilities of mainstream students at the 5th grade level. Phase 1 (n= 96) involved various tryouts of 101 draft tasks to estimate duration of administration, clarity of directions, whole-class administration procedures, and an opportunity to administer verbal protocols to provide further information about task accessibility and characteristics. Phase 2, the pilot stage, involved administration of 40 retained tasks (35 of which were modified as a result of Phase 1) to students in whole-class settings (n=128). Analyses included item difficulty and item discrimination. The rationale for retaining or rejecting tasks is presented along with psychometric/linguistic profiles documenting the evolution of example effective and ineffective prototype tasks. The final chapter of the report reflects on the lessons learned from the test development process we adopted and makes suggestions for further advances in this area.

#729 – Predictive Validity of an English Language Arts Performance Assessment
Jia Wang, David Niemi, Haiwen Wang

Summary

The main goal of this report is to present evidence on the predictive validity of an English language arts (ELA) performance assessment (PA) administered in Grades 2–9 in a large urban school district. To account for the hierarchical structure of the data (students are nested within schools), we employed hierarchical linear modeling (HLM) to distinguish individual and aggregated explanatory variables. Based on a sub-sample of 5,427 students, we found that students’ 2001 ELA PA scores were predictive of their probability of passing the California High School Exit Exam (CAHSEE). We also found a significant correlation between student performances on the ELA performance assessment and other standardized tests. We believe that the ELA PA may be a dependable and useful indicator to identify at-risk students.

#718 – Examining the Generalizability of Direct Writing Assessment Tasks
Eva Chen, David Niemi, Jia Wang, Haiwen Wang, Jim Mirocha

Summary

This study investigated the level of generalizability across a few high quality assessment tasks and the validity of measuring student writing ability using a limited number of essay tasks. More specifically, the research team explored how well writing prompts could measure student general writing ability and if student performance from one writing task could be generalized to other similar writing tasks. A total of four writing prompts were used in the study, with three tasks being literature-based and one task based on a short story. A total of 397 students participated in the study and each student was randomly assigned to complete two of the four tasks. The research team found that three to five essays were required to evaluate and make a reliable judgment of student writing performance.

#676 – Evaluation of Seeds of Science/Roots of Reading Project: Shoreline Science and Terrarium Investigations
Jia Wang, Joan Herman

Summary

This project was initiated in order to evaluate two literacy and science integrated instruction units, Shoreline Science and Terrarium Investigations, designed by the Lawrence Hall of Science Seeds of Science/Roots of Reading Project (Seeds/Roots). We examined how the integrated units affect student interest, motivation, and learning, as well as evaluating the units’ quality, usability, and utility using both quantitative and qualitative approaches. Through analyses of student performance and teacher interviews, we found Shoreline Science and Terrarium Investigations beneficial to both students and teachers. Teachers were highly motivated to use the materials, and Shoreline Science and Terrarium Investigations students learned significantly more than the control group students in all science and literacy measures on which the differences were expected.

#501 – Final Report for Validation of Problem-Solving Measures
Howard E. Herl, Harold O'Neil, Jr., Gregory K. W. K. Chung, Cecilia Bianchi, Shu-ling Wang, Richard Mayer, Charlotte Yeh Lee, Angela Choi, Thomas Suen, and Aileen Tu

Summary

In this report we summarize several studies regarding the measurement of problem solving and investigation of the impact of providing content information (Yes/No) and scoring criteria (Yes/No) for two knowledge mapping tasks proposed for use in an international assessment context (the International Life Skills Survey). Reliability and validity issues related to (a) the use of knowledge maps for measuring content understanding, (b) domain-specific problem-solving prompts for measuring participants' problem-solving strategies, and (c) a self-regulation questionnaire for measuring levels of participants' metacognition and motivation are presented within the larger context of measuring problem solving. Analyses showed that providing scoring instructions to participants did not significantly affect their performance on knowledge mapping tasks, including measures of content understanding, number of terms of used, and number of links constructed. For one of the two tasks, no gender effects were found for knowledge mapping measures. However, for the other task, there were significant effects found for two knowledge mapping measures, where males constructed significantly greater numbers of links and achieved significantly higher content understanding scores. Finally, providing content information resulted in higher performance scores. In general, there was sufficient reliability and validity to use these new measures at the cross-national level but not at the individual level.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators