National Center for Research on Evaluation, Standards and Student Testing

#744 – Examining Differential Item Functioning in Reading Assessments for Students with Disabilities
Jamal Abedi, Seth Leon, Jenny C. Kao

Summary

This study examines performance differences between students with disabilities and students without disabilities students using differential item functioning (DIF) analyses in a high-stakes reading assessment. Results indicated that for Grade 9, many items exhibited DIF. Items that exhibited DIF were more likely to be located in the second half of the assessment subscales. After accounting for reading ability using a proxy score from items on the first half of the subscales, students with disabilities consistently underperformed on items located in the second half relative to the items located in the first half, as compared with students without disabilities. These results were seen in Grade 9 for data from two different states. These results were not seen for Grade 3. This study has several limitations. There was no access to information regarding the testing accommodations that students with disabilities might have received, and no access to the type of disabilities. Results of this study can shed light on potential factors affecting the accessibility of reading assessments for students with disabilities, in an ultimate effort to provide assessment tools that are conceptually and psychometrically sound for all students. A companion report is available examining differential distractor functioning for students with disabilities.

#738 – Providing Validity Evidence to Improve the Assessment of English Language Learners
Mikyung Kim Wolf, Joan L. Herman, Jinok Kim, Jamal Abedi, Seth Leon, Noelle Griffin, Patina L. Bachman, Sandy M. Chang, Tim Farnsworth, Hyekyung Jung, Julie Nollner, Hye Won Shin

Summary

This research project addresses the validity of assessments used to measure the performance of English language learners (ELLs), such as those mandated by the No Child Left Behind Act of 2001 (NCLB, 2002). The goals of the research are to help educators understand and improve ELL performance by investigating the validity of their current assessments, and to provide states with much needed guidance to improve the validity of their English language proficiency (ELP) and academic achievement assessments for ELL students. The research has three phases. In the first phase, the researchers analyze existing data and documents to understand the nature and validity of states’ current practices and their priority needs. This first phase is exploratory in that the researchers identify key validity issues by examining the existing data and formulate research areas where further investigation is needed for the second phase. In the second phase of the research, the researchers will deepen their analysis of the areas identified from Phase I findings. In the third phase of the research, the researchers will develop specific guidelines on which states may base their ELL assessment policy and practice. The present report focuses on the researchers' Phase I research activities and results. The report also discusses preliminary implications and recommendations for improving ELL assessment systems.

#723 – Recommendations for Building a Valid Benchmark Assessment System: Interim Report to the Jackson Public Schools
David Niemi, Julia Vallone, Jia Wang, Noelle Griffin

Summary

Many districts and schools across the U. S. have begun to develop and administer assessments to complement state testing systems and provide additional information to monitor curriculum, instruction and schools. In advance of this trend, the Jackson Public Schools (JPS) district has had a district benchmark testing system in place for many years. To complement and enhance the capabilities of district and school staff, the Stupski Foundation and CRESST (National Center for Research on Evaluation, Standards, and Student Testing at UCLA) worked out an agreement for CRESST to provide expert review and recommendations to improve the technical quality of the district’s benchmark tests. This report (which represents the first of two deliverables on this project) focuses on assessment development and is consistent with the district goal of increasing the predictive ability of the assessments for students’ state test performance, as well as secondary goals.

#821 – Evaluation of Calipers II: Using Simulations to Assess Complex Learning Site Visit Findings
Deborah La Torre Matrundola, Sandy Chang, and Joan Herman

Summary

The purpose of these case studies was to examine the ways technology and professional development supported the use of the SimScientists assessment systems. Qualitative research methodology was used to provide narrative descriptions of six classes implementing simulation-based assessments for either the topic of Ecosystems or Atoms and Molecules. Results revealed both strengths and weaknesses concerning technology support for the assessments, as well as technology and professional development support of the teachers. Furthermore, recommendations are provided concerning potential improvements to the assessments, reflection activities, and professional development.

#698 – Celebrating 20 Years of Research on Educational Assessment: Proceedings of the 2005 CRESST Conference
Anne Lewis

Summary

The 2005 CRESST conference marked the 20th year of work on critically important accountability topics by the UCLA institution, "a tremendous accomplishment for a research center," according to Aimee Dorr, dean of the Graduate School of Education and Information Studies. In her welcoming remarks, Dean Dorr described why CRESST has achieved such longevity. The center is "independent, very lively, grounded in practice, and very forward looking, with many top accountability experts from around the nation," said Dorr, "interested in new technologies and helping to shape the future of education." She also noted that although it was "good fortune" for the center's senior partner to be located at UCLA, "it is a partnership throughout the country, and one that enriches us here as the partners do on the national scene."

The anniversary for CRESST was an opportunity for the conference program to focus on the achievements in the use of assessment to improve student learning. The two-day gathering described many of the lessons learned from a century of testing. The discussions also featured the newest CRESST initiative, known as POWERSOURCE, a $10 million grant from the Institute of Education Sciences at the U.S. Department of Education to develop formative mathematics assessments in the middle grades to improve student performance and learning.

#726 – The Role of Teacher Discourse in Effective Groupwork
Noreen M. Webb, Megan L. Franke, Marsha Ing, Angela Chan, Tondra De, Deanna Freund, Dan Battey

Summary

Prior research on collaborative learning identifies student behaviors that significantly predict student achievement, such as giving explanations of one’s thinking. Less often studied is how teachers’ instructional practices influence collaboration among students. This report investigates the extent to which teachers engage in practices that support students’ explanations of their thinking, and how these teacher practices influence the nature of explanations that students give when asked by the teacher to collaborate with each other. In this study, we videotaped and audiotaped teacher and student participation, and measured student achievement, in second- and third-grade mathematics classrooms working on algebraic concepts of equality and relational thinking. The teachers observed here, all of whom received specific instruction in eliciting the details of student thinking, varied significantly in the extent to which they asked students to elaborate on their suggestions. This variation corresponded strongly to variation across classrooms in the nature and extent of student explanations during collaborative conversations, and to differences in student achievement.

#602 – Teachers’ Assignments and Student Work: Opening a Window on Classroom Practice
Lindsay Clare Matsumura, Jenny Pascal

Summary

In this report, four years of CRESST's research is described developing indicators of classroom practice that have the potential to be used in large-scale settings and that draw attention to important aspects of standards-based learning and instruction. CRESST's method was based on the collection of teachers' assignments with student work. The assignments then were rated and results were summarized to create indicators of classroom practice. Results to date indicated an acceptable level of inter-rater reliability across study years. It likely would be necessary to collect as many as three or four assignments from teachers to obtain a stable estimate of quality. Additionally, this method was reliable when teachers created their own assignments, but not when teachers submitted assignments created by outside sources. The quality of classroom assignments was associated with the quality of observed instruction, as well as the quality of students' written work. Students who were exposed to teachers who created more cognitively challenging assignments and who had clearer grading criteria also made greater gains on the Stanford Test of Achievement, 9th Edition (Stanford 9). The quality of teachers' assignments submitted at each of the study years, however, tended to be of basic quality only. Teachers' reactions to the data collection and implications for the use of this method in collaborative professional development sessions also are discussed.

#704 – Developing Expertise With Classroom Assessment in K-12 Science: Learning to Interpret Student Work
Interim Findings From a 2-Year Study
Maryl Gearhart, Sam Nagashima, Jennifer Pfotenhauer, Shaunna Clark, Cheryl Schwab, Terry Vendlinski, Ellen Osmundson, Joan Herman, Diana J. Bernbaum

Summary

This article reports findings on growth in three science teachers’ expertise with interpretation of student work over 1 year of participation in a program. The program was designed to strengthen classroom assessment in the middle grades. Using a framework for classroom assessment expertise, we analyzed patterns of teacher learning, and the roles of the professional program and the quality of the assessments provided with teachers’ instructional materials.

#621 – Towards the Characterization of Academic Language in Upper Elementary Science Classrooms
Alison L. Bailey, Frances A. Butler, Charmien LaFramenta, & Christine Ong

Summary

This report details an exploratory study that employs qualitative methods to characterize the academic language used by teachers and students in 4th and 5th grade mainstream science classrooms. Teacher oral language, and to some degree student talk, was observed during content instruction. This type of data allows for a broad descriptive representation of language use and vocabulary choice rather than finegrained structural analyses. Print materials and student written products were also examined in this manner. Primary research questions include (1) how does teacher language function in the classroom? (2) how do teachers support academic vocabulary growth? The inductive approach taken led to the development of a matrix of teacherproduced academic language within three instructional contexts; concept instruction, vocabulary instruction, and process/application instruction. Describing the language embedded in oral and written classroom discourse has implications for both educational research and practice. The focus on mainstream classroom talk will serve as a baseline for future research of academic language and English language learners. Specifically, this study has created a conceptual framework with empirical illustrations that should prove valuable for (1) conducting further study of effective classroom instruction that includes academic language, and (2) the development of language assessments that measure student progress towards academic language proficiency.

#797 – An Exploratory Study of the Relationship Between Collaboration and Mathematics and Game Outcomes
Rebecca E. Buschang, Gregory K. W. K Chung, and Jinok Kim

Summary

This is an exploratory study of the relationship between collaboration and mathematics and game performance. 243 middle school students played a video game either with one partner or individually for 40 minutes. Results suggest that participants with low prior math knowledge tended to perform better by working collaboratively, while participants with high prior math knowledge tended to perform better by working individually. These results are similar to prior findings from classroom research which indicates that collaboration is more effective for low-performing students. The results have implications for designing game environments for low-performing students.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators