Reports 692-683

#692 – Automated Assessment of Domain Knowledge With Online Knowledge Mapping
Gregory K. W. K. Chung, Eva L. Baker, David G. Brill, Ravi Sinha, Farzad Saadat, and William L. Bewley

Summary

A critical first step in developing training systems is gathering quality information about a trainee's competency in a skill or knowledge domain. Such information includes an estimate of what the trainee knows prior to training, how much has been learned from training, how well the trainee may perform in future task situations, and whether to recommend remediation to bolster the trainee's knowledge. This paper describes the design, development, testing, and application of a Web-based tool designed to assess a trainee's understanding of a content domain in a distributed learning environment. The tool, called the CRESST Human Performance Knowledge Mapping Tool (HPKMT), enables trainees to express their understanding of a content area by creating graphical, network representations of concepts and links that define the relationships of concepts.

#691 – Beyond Summative Evaluation: The Instructional Quality Assessment as a Professional Development Tool
Amy C. Crosson, Melissa Boston, Allison Levison, Lindsay Clare Matsumura, Lauren B. Resnick, Mikyung Kim Wolf, and Brian W. Junker

Summary

In order to improve students' opportunities to learn, educators need tools that can assist them to reflect on and analyze their own and others' teaching practice. Many available observation tools and protocols for studying student work are inadequate because they do not directly engage educators in core issues about rigorous content and pedagogy. In this conceptual paper, we argue that the Instructional Quality Assessment (IQA)-a formal toolkit for rating instructional quality that is based primarily on classroom observations and student assignments-has strong potential to support professional development within schools at multiple levels. We argue that the IQA could be useful to teachers for analyzing their own and their colleagues' practice; additionally, the IQA could aid the efforts of principals in their work as instructional leaders, identifying effective practitioners to help lead professional development within a school and targeting professional development needs that would require external support. Although the IQA was designed for summative, external evaluation, we argue that the steps taken to improve the reliability of the instrument- particularly the efforts to make the rubric descriptors for gradations of instructional quality as transparent as possible-also serve to make the tool a resource for professional growth among educators.

#690 – Using Classroom Artifacts to Measure Instructional Practice in Middle School Science: A Two-State Field Test
Hilda Borko, Brian M. Stecher, Felipe Martinez, Karin L. Kuffner, Dionne Barnes, Suzanne C. Arnold, Joi Spencer, Laura Creighton, Mary Lou Gilbert

Summary

This report presents findings from two investigations of the use of classroom artifacts to measure the presence of reform-oriented teaching practices in middle-school science classes. It complements previous research on the use of artifacts to describe reform-oriented teaching practices in mathematics. In both studies, ratings based on collections of artifacts assembled by teachers following directions in the "Scoop Notebook" are compared to judgments based on other sources of information, including direct classroom observations and transcripts of discourse recorded during classroom observations. For this purpose, we developed descriptions of 11 dimensions of reform-oriented science instruction, and procedures for rating each on a dimension-specific five-point scale.

#689 – Closing the Gap: Modeling Within-School Variance Heterogeneity in School Effect Studies
Kilchan Choi and Junyeop Kim

Summary

Effective schools should be superior in both enhancing students' achievement levels and reducing the gap between high- and low-achieving students in the school. However, the focus has been placed mainly on schools' achievement levels in most school effect studies. In this article, we attend to the school-specific achievement dispersion as well as achievement level in determining effective schools. The achievement dispersion in a particular school can be captured by within-school variance in achievement (ó2). Assuming heterogeneous within-school variance across schools in hierarchical modeling, we identified school factors related to high achievement level and a small gap between high- and low-achieving students. Schools with a high achievement level tended to be more homogeneous in achievement dispersion, but even among schools with the same achievement level, schools varied in their achievement dispersion, depending on classroom practices.

#688 – Effects of Misbehaving Common Items on Aggregate Scores and an Application of the Mantel-Haenszel Statistic in Test Equating
Michalis P. Michaelides

Summary

Consistent behavior is a desirable characteristic that common items are expected to have when administered to different groups. Findings from the literature have established that items do not always behave in consistent ways; item indices and IRT item parameter estimates of the same items differ when obtained from different administrations. Content effects, such as discrepancies in instructional emphasis, and context effects, such as changes in the presentation, format, and positioning of the item, may result in differential item difficulty for different groups. When common items are differentially difficult for two groups, using them to generate an equating transformation is questionable. The delta-plot method is a simple, graphical procedure that identifies such items by examining their classical test theory difficulty values. After inspection, such items are likely to drop to a non-common-item status.

#687 – Educational Accountability Systems
Robert L. Linn

Summary

Test-based educational accountability systems have considerable appeal to politicians, policymakers, and the general public. Such systems have been widely used by states for more than a decade and with the enactment of the No Child Left Behind Act of 2001 all states must now implement an accountability system that uses results from assessments in mathematics and English/language arts and is administered each year in grades 3 through 8 plus one high school grade. A wide variety of test-based accountability systems of states and as required by NCLB are described and their strengths and weaknesses are evaluated. It is argued that the sanctions for schools that are part of the accountability system require causal inferences about school effectiveness. It is concluded, however, that basing causal inferences about school quality on the results that can be obtained from the existing school accountability systems is not scientifically defensible. It would be better to view accountability results as a source of descriptive information about schools and the basis of hypotheses that can be evaluated by gathering addition information about instructional staff and practice.

#686 – An Investigation of Language-Minority Children: Demographic Characteristics, Initial Performance, and Growth in Achievement
Douglas Ready and Gerald Tindal

Summary

Research on language-minority (LM) children has generally focused on language as the primary mediator of student achievement without considering other student demographic characteristics. This is unfortunate, as studies that approach languageminority children as a homogeneous group will misestimate relationships between language status and academic achievement. Moreover, extant research is often hampered by its lack of focus on language-minority students’ cognitive growth over time. Using data from the Early Childhood Longitudinal Study, Kindergarten Cohort of 1998-1999 (ECLS-K) and growth curve analyses within a three-level hierarchical framework, this report examines the academic skills of LM children as they enter kindergarten and progress through first grade. This report defines language-minority children as those having a primary home language other than English. A further distinction is made between language-minority children who are English proficient (LM/P) and those who are not proficient (LM/NP), although examinations of LM/NP children using the regular ECLS-K cognitive assessments are limited to mathematics and Hispanic (Spanish-speaking) LM/NP children. Descriptive findings stress the diverse socio-demographic and academic backgrounds of language-minority children. Many LM children have highly-educated and affluent parents, while others come from families with few social or economic resources. Some LM children enter school with well-developed literacy and numeracy skills, while others exhibit few initial academic competencies. Despite this considerable variability, one theme that permeates this report is the socioeconomic and academic disadvantage among Hispanic LM children. Unsurprisingly, Hispanic LM/P children enter kindergarten with fewer English skills than non-LM children. However, Hispanic LM/P children’s initial literacy skills also lag behind those of other LM/P children. Much of this initial disadvantage is explained by the relative socioeconomic disadvantage of Hispanic LM/P children. Not coincidentally, non-Hispanic LM/P students and their non-LM peers share similar 2 socioeconomic and academic backgrounds. In terms of literacy learning, during kindergarten, Asian LM/P children erase the small literacy gap that separates them from non-LM children. The Hispanic LM/P literacy disadvantage, however, remains relatively constant during kindergarten and first grade and actually increases during the intervening summer months. As with literacy, Hispanic LM/P children enter kindergarten with fewer mathematics skills than their non-LM peers. An even larger mathematics skills gap separates Hispanic LM/NP and non-LM children. For both groups, a substantial proportion of these initial achievement differences can be explained by their socioeconomic disadvantage compared to non-LM children. Although Hispanic LM/P children gain mathematics skills at rates comparable to Hispanic non-LM children, Hispanic LM/NP children fall even further behind during kindergarten (but learn at similar rates during the summer and first grade). Non-Hispanic LM/P children enter kindergarten and end first grade with mathematics skills equal to non-LM children.

#685 – Alignment of Mathematics State-level Standards and Assessments: The Role of Reviewer Agreement
Noreen Webb, Joan Herman, and Norman Webb

Summary

In this report we explore the role of reviewer agreement in judgments about alignment between tests and standards. Specifically, we consider approaches to describing alignment that incorporate reviewer agreement information in different ways. The essential questions were whether and how taking into account reviewer agreement changes the picture of alignment between tests and standards. This study showed a wide range of reviewer agreement during the process of aligning standards and assessments, with substantial reviewer disagreement about important elements such as correspondence between objectives and items on the assessments. Taking this reviewer disagreement into account changed conclusions about alignment, not only showing weaker alignment than previously demonstrated, but also changing the profiles of alignment about, for example, relative coverage of specific standards. The results of this study point to the need for greater clarity in objectives and standards, more extensive reviewer training during the alignment process, and possibly also inspection of items to uncover characteristics that may lead to uncertainty among reviewers.

#684 – Role of a Task-Specific Adapted Feedback on a Computer-Based Collaborative Problem-Solving Task
San-hui (Sabrina) Chuang and Harold F. O'Neil

Summary

Collaborative problem solving and collaborative skills are considered necessary skills for success in today's world. Collaborative problem solving is defined as problem solving activities that involve interactions among a group of individuals. Large-scale and small-scale assessment programs increasingly use collaborative group tasks in which students work together to solve problems or to accomplish projects. This study attempts to research the role of feedback on a computer-based collaborative problem solving task by extending Hsieh and O’Neil’s (2002) computer-based collaborative knowledge mapping study. In their study, groups of students searched a web environment of information to improve a knowledge map. Various types of feedback were investigated. They found that searching has a negative relationship with group outcome (knowledge map scores). By teaching searching and by providing different types of feedback, this study explores the effects of students’ teamwork and problem solving processes on students’ knowledge mapping performance. Moreover, the effects of two types of feedback (adapted knowledge of response feedback and task-specific adapted knowledge of response feedback) were also investigated. One hundred and twenty college students (60 groups) participated in the main study. The students were randomly assigned either to be a group leader whose responsibility was to construct the map or to be a group searcher whose responsibility was to help the leader construct the map by seeking information and accessing feedback from the Web environment. Results showed that task-specific adapted knowledge of response feedback was significantly more beneficial to group outcome than adapted knowledge of response feedback. In addition, as predicted, for the problem solving process, information seeking including request of feedback, browsing, searching for information and searching using Boolean operators were all significantly related to group outcome for both groups. 2 Lastly, this study confirmed that computer-based performance assessment in collaborative problem solving was effective. The collaboration between team members and individual students’ problem solving processes and strategies were effectively recorded by the computers. In addition, the use of computers to assess and report group interaction and students’ thinking processes was proven to be more inexpensive and less time consuming than other alternatives.

#683 – Language-minority Students' Cognitive School Readiness and Success in Elementary School
Lindsay Taggart Rutherford

Summary

A significant amount of research treats students who speak a language other than English at home, or language-minority students, as a single demographic group and compares them to students who speak only English at home. If important disparities in early school experiences among language-minority students have been overlooked, then policies aimed at helping them as they begin formal schooling may fall short, as they will not attend to the needs of specific subpopulations. This paper uses data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) to address this gap in the literature by exploring language-minority students’ experiences with grade retention and special education placement and specifically examining variation among language-minority students based on race, immigrant status and socioeconomic status. Findings indicate that language-minority students are no more likely to be retained than their English-only counterparts, while they are less likely than their English-only counterparts to be placed in special education. Furthermore, there was no variation among language-minority students by race or immigrant status. These findings and their implications for language-minority students are explored in the conclusion.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators