Reports 682-673

#682 – Performance Assessment Models and Tools for Complex Tasks
Gregory K. W. K. Chung, Girlie C. Delacruz, and William L. Bewley

Summary

Assessment models support the design of quality performance assessments. Assessment tools are being developed to enable easy and effective application of the models. Based on representations of assessment design knowledge and domain knowledge in ontologies, the tools provide guidance to assessment designers, and through constraint processing check the completeness and accuracy of designs. With the addition of Bayesian networks, the tools can also enable individualized instruction by identifying knowledge gaps and prescribing instruction to fill the gaps. This paper describes the technical approach to developing the tools and discusses applications of ontologies and Bayesian networks for assessment authoring and individualized instruction.

#681 – Measuring Reading Comprehension and Mathematics Instruction in Urban Middle Schools: A Pilot Study of the Instructional Quality Assessment
Lindsay Clare Matsumura, Sharon Cadman Slater, Brian Junker, Maureen Peterson, Melissa Boston, Michael Steele, and Lauren Resnick

Summary

The quality of reading comprehension and mathematics instruction was explored in five urban middle schools using the Instructional Quality Assessment (IQA) toolkit (N = 34 teachers). The IQA is comprised of protocols for rating observed instruction and the quality of teachers’ assignments with student work. The purpose of this research was to investigate the reliability and potential validity of the ratings of these data sources. Commensurate with other research on the quality of middle schools, our results indicated that the quality of instruction varied a great deal within schools and was of a ‘basic’ quality overall. Results indicated a moderate to high level of reliability. Four assignments with student work yielded a stable estimate of quality in both content areas, and when teachers complied with the requirements of the research as few as two observations yielded a stable estimate of teaching quality in both content areas as well. The quality of teachers’ observations and assignments were significantly associated in mathematics, but not in reading comprehension. Because of the small sample size it was not possible to apply multi-level models. The relation between the IQA and student achievement on the SAT-10 was explored using linear regression techniques. Results indicated that after controlling for students’ prior achievement, socio-economic status (SES), ethnicity, language, and IEP status, the IQA assignment measure in reading comprehension predicted student achievement on the Total Reading, Reading Comprehension, and Vocabulary subscores of the SAT-10. The observation measure in reading comprehension predicted student outcomes on the Reading Comprehension subscore of the SAT-10 only. In mathematics, the quality of teachers’ assignments predicted students’ achievement on the Procedures subscore of the SAT-10. The quality of observed instruction in mathematics predicted students’ achievement on the Procedures and Total Math subscores. Without accounting for clustering within classrooms and schools as multilevel models do, our linear regression analyses may lead to results that appear stronger than they actually are. Nevertheless our analyses indicate 2 the direction of trend in these relationships and raise important questions regarding which data sources may be best (classroom assignments or observations) for measuring specific aspects of instruction and student outcomes. Additional research with larger samples of teachers is needed to make definitive conclusions about the validity of the IQA ratings and under what conditions one might choose to either observe in classrooms or collect assignments with student work.

#680 – Measurement Models as Narrative Structures
Robert J. Mislevy, Chun-Wei Huang

Summary

Advances in cognitive research increase the need for assessment that can address the processes and the strategies by which persons solve problems. Several psychometric models have been introduced to handle claims cast in information-processing terms, explicitly modeling performance in terms of theory-based predictions of performance. Cognitively based item response theory IRT models incorporate features of items that influence persons' responses, and relate these features to response probabilities. A number of these models are discussed in terms of the interplay between the formal characteristics of the probability models and the substantive narratives they support. Examples include the Linear Logistic Test Model, mixtures of IRT models, and multivariate structured IRT models.

#679 – Language Background and Early Academic Achievement: Disentangling Language-Minority Status, Social Background, and Academic Engagement
Marcel Paret

Summary

Due to the increasing influx of immigrants from Asia and Latin America since the passage of the 1965 Immigration Reform Act, the American population has been steadily shifting. The National Research Council estimated that during the 1990s, entering immigrants accounted for 37% of the total growth in the U.S. population (Smith & Edmonston, 1997). An important element of this population shift is the use of language, as many immigrants arrive from countries where English is not the primary language. In 1990 a substantial portion (43%) of recent immigrants either spoke English “not well,” or “not at all” (Smith & Edmonston, 1997: Table 8.3). Further, even if immigrants enter the country with English language skills or acquire them after arrival, there is still the possibility that they will not use English when interacting with their children. This prevalence of non-English languages suggests that examining the educational experiences of language-minority students—those for whom English is not the first language—is extremely important to larger issues of educational and social stratification.

#678 – Consequences and Validity of Performance Assessment for English Language Learners: Integrating Academic Language and ELL Instructional Needs into Opportunity to Learn Measures
Zenaida Aguirre-Munoz, Christy Kim Boscardin, Barbara Jones, Jae-Eun Park, Marjorie Chinen, Hye Sook Shin, Janet Lee, Anastasia Aimee Amabisca, and Aprile Benner

Summary

To further the understanding of ELL (under) achievement and broaden the current scope of OTL models, the primary focus of this study was to investigate process and content opportunities that are particularly relevant to improving ELL achievement with particular attention to the relationship between opportunities to acquire academic language and ELL achievement. The work reported here operationalized academic language within a systemic functional linguistics theory (also called “functional grammar”). This theory of language use provided teachers and students with a framework for analyzing language in a manner that both built on existing language knowledge (or schemas) and provided them with an accessible structure for examining more complicated language (and content) concept. Based on our literature review, previous findings, and new ELL-sensitive OTL model, we investigated the following research questions in this study: (a.) To what extent and in what ways are students being exposed to key OTL variables in classrooms? (b). What is the impact of academic language and other OTL indicators on ELLs’ and non-ELLs’ performance on LAPA? After an introduction and literature review, the remainder of this report addresses sources of data, description of instruments and procedures, and types of analyses selected for the study; findings related to the research questions; and concludes with a discussion based on the findings. One of the most important findings from this study points to the need for explicit instruction on academic language. The positive impact of functional grammar implementation on student outcomes also suggests that in order for ELLs to fully benefit from assessment-driven reform, teachers need the capacity to make the linguistic expectations clear to students by focusing on the linguistic elements that are characteristic of academic registers.

#677 – A Multi-Method and Multi-Source Approach for Studying Fidelity of Implementation
Maria Araceli Ruiz-Primo

Summary

Even the best program in education will fail to have the intended impact if its essential elements are not implemented properly. Degree of implementation is, then, critical to draw valid conclusions on program outcomes (e.g., Scheirer & Rezmovic, 1983). Especially important is the information on the fidelity with which a program is implemented. Fidelity of Implementation (FOI) has been defined as the determination of how close the program is implemented according to its original design or as intended (e.g., Dobson & Shaw, 1988; Dusenbury, Brannigan, Falco, & Hanse, 2003; Witt & Elliot, 1985).1 Unfortunately, empirical evidence on the effect of FOI on program success is limited. Many evaluation studies do not collect data on FOI and even fewer examine its impact on program outcomes (Dane & Schenider, 1998; Dusenbury et al., 2003; Lillehoj, Griffin, Spoth, 2004). Furthermore, studies on FOI differ considerably on their approaches (Dane & Schenider, 1998; Dusenbury et al., 2003; Huntley, 2004; Lillehoj, Griffin, Spoth, 2004); there is no set of methods and procedures that is universally known and used as standard procedure in the study of FOI. Whereas the characteristics of each program determine what has to be measured during implementation, there are some commonalities across types of programs and, therefore, some general strategies that can be developed. This paper addresses FOI at three levels: general, conceptual, and applied. The first section provides a short review of literature on the main issues of FOI. The second section proposes a conceptual approach for studying FOI in the context of inquiry-based science curricula. The third section describes a series of studies, currently in progress, in which this conceptual approach is being used.

#676 – Evaluation of Seeds of Science/Roots of Reading Project: Shoreline Science and Terrarium Investigations
Jia Wang, Joan Herman

Summary

This project was initiated in order to evaluate two literacy and science integrated instruction units, Shoreline Science and Terrarium Investigations, designed by the Lawrence Hall of Science Seeds of Science/Roots of Reading Project (Seeds/Roots). We examined how the integrated units affect student interest, motivation, and learning, as well as evaluating the units’ quality, usability, and utility using both quantitative and qualitative approaches. Through analyses of student performance and teacher interviews, we found Shoreline Science and Terrarium Investigations beneficial to both students and teachers. Teachers were highly motivated to use the materials, and Shoreline Science and Terrarium Investigations students learned significantly more than the control group students in all science and literacy measures on which the differences were expected.

#675 – Preschool Participation and the Cognitive and Social Development of Language-Minority Students
Russell W. Rumberger, Loan Tran

Summary

This study examined participation in preschool and its relationship with the cognitive and social development of language-minority students. Although there is a large body of research that demonstrates the cognitive and social benefits of attending preschool (Barnett, 1995; Gorey, 2001; National Research Council, Committee on Early Childhood Pedagogy, 2000; Vandell, 2004), very little of this research has included language-minority students, or at least those who do not speak English. Either non-English speaking families are not included in the design of the study, such as with the widely cited National Institute for Child Health and Development (NICHD) Early Child Care Study, or the studies are based on cognitive and social assessments that are only conducted in English (e.g., Magnuson, Meyers, Ruhm, & Waldfogel, 2004). Consequently, little is known about participation in and outcomes of preschool for the growing population of language-minority students.

#674 – Child Care Arrangements of Language-Minority Children: Care Provider’s Language Use
Hiromi Ishizawa

Summary

In the United States, child care arrangements serve increasingly linguistically diverse populations of children. However, little is known about patterns of childcare arrangements for language-minority children and the linguistic environment of child care arrangements. Using the Early Childhood Program Participation Survey of the 2001 National Household Education Surveys Program (ECPP-NHES, 2001), this report asks three research questions: (1) what are the patterns of child care arrangements for language-minority children?, (2) do child care arrangements differ by language use of the child?, and (3) what factors are associated with the child having a care provider who speaks a non-English language? The results show that after controlling for the characteristics of the child, children are less likely to be in nonparental care (i.e., center-based programs, and relative care or nonrelative care) when the child speaks a non-English language mostly. However, this association disappears when characteristics of parents and household are controlled. The results also suggest that language use of the child as well as parents are important factors associated with the care provider’s language use. These findings may suggest that an additional context, the linguistic environment of child care arrangements, contribute to a child’s non-English language use.

#673 – Exploring Models of School Performance: From Theory to Practice
Kilchan Choi, Pete Goldschmidt, Kyo Yamashir

Summary

Our purpose in this report is to present and discuss competing accountability approaches, or models, designed to systematically indicate how a school’s students are performing academically. Within the framework of the current federally mandated accountability legislation, increased interest in models measuring school performance has caused educational policymakers to consider several key issues. These issues include whether results from different accountability models yield different inferences about a school’s performance; what assumptions underlie each of the models; how different models are implemented; and ultimately which model is best suited for a particular context.

Like this Product? Get CRESST News Every Month!

Reports

Download Our Reports

Partners & Collaborators