Like this Product? Get CRESST News Every Month!

Don't miss out on the latest CRESST news including our FREE reports and products. Subscribe to the monthly CRESST E-newsletter right now!

We only use this address for the monthly CRESST E-Newsletter and will ask for confirmation before adding you to our list.



No thanks | Don't ask again

Reports

Please note that CRESST reports were called "CSE Reports" or "CSE Technical Reports" prior to CRESST report 723.

#853 – CRESST Shiphandling Automated Assessment Engine: Underway Replenishment (UNREP)
Alan D. Koenig, John J. Lee, and Markus R. Iseli

Summary

The U.S. Navy is looking to automate the assessment of shiphandling skills to allow for less supervised practice which in turn should reduce instructor load. As part of a broader initiative at the Surface Warfare Officers School (SWOS), CRESST has been working to develop this capability using an automated assessment engine (AAE), which infers student shiphandling proficiency based on meaningful, observable actions taken in the shiphandling simulator, the Conning Officers Virtual Environment (COVE). Ultimately, four separate instances of the AAE will be deployed at SWOS, one for each of the core shiphandling skill areas (called evolutions): Mooring to a Pier, Underway Replenishment (UNREP), Getting Underway from a Pier, and Harbor Transit. This report describes the rubrics and inference model (a Bayesian network) used in the AAE for the UNREP evolution. 


#852 – CRESST Shiphandling Automated Assessment Engine: Mooring at a Pier
Alan D. Koenig, John J. Lee, and Markus R. Iseli

Summary

To meet the challenges of training shiphandling skills more effectively, the U.S. Navy seeks to automate the assessment of shiphandling skills to allow for less supervised practice and, therefore, reduced instructor load. As part of a broader initiative at the Surface Warfare Officers School (SWOS), CRESST has been working to develop this capability using an automated assessment engine (AAE), which infers student shiphandling proficiency based on meaningful, observable actions. This report describes the rubrics used in the AAE, as well as the inference model used therein. Plans for a future validation study are also outlined.


#851 – Assessment for Deeper Learning in CCSS: Progress Report on the Status of Smarter Balanced and PARCC Assessment Consortia
Joan L. Herman, Rebecca E. Buschang, Deborah La Torre Matrundola, and Jia Wang

Summary

This report examines progress made by the Smarter Balanced Assessment Consortium (Smarter Balanced) and the Partnership for Assessment of Readiness for College and Careers (PARCC) through the spring of 2013. This is done by examining efforts concerning the development of their respective assessments, achievement level descriptors, accessibility and accommodations guidelines, and technology guidelines. This report also projects expectations for deeper learning in the consortia’s summative assessments and compares these results to related studies. Results indicate that both consortia made substantial progress in their assessment development over the course of the year. In addition, analyses show a range of depth of knowledge (DOK) expectations across those aligned to the Common Core State Standards (CCSS) as well as respected national and international tests. Based on these analyses, this study recommends benchmarks for deeper learning (DOK3 and DOK4) of 33% of total score points in mathematics, 33% in English language arts (ELA) at the elementary level, and 50% in ELA at the secondary level.


#850 – An Exploratory Study Examining the Feasibility of Using Bayesian Networks to Predict Circuit Analysis Understanding
Gregory K. W. K. Chung, Gary B. Dionne, and William J. Kaiser

Summary

Renewed interest in individualizing instruction, particularly with the use of technology, has resulted in a search for methods that can accurately diagnose student knowledge gaps and prescribe appropriate remediation. In this study we gathered validity evidence for the use of a Bayesian network to model students’ understanding of circuit analysis concepts. Thirty-four undergraduate students completed tasks designed to measure conceptual knowledge, procedural knowledge, and problem-solving skills. Results suggested that the Bayesian network was generally working as intended. When high- and low-performing groups were formed on the basis of posterior probabilities, significant group differences were found favoring the high-performing group with respect to circuit definitions and circuit analysis problems, for both actual and self-assessments, and higher major GPA. The Bayesian network also predicted participants’ performance on problem-solving items on average 75% of the time. The findings of this study are promising for developing scalable and feasible online automated reasoning techniques to diagnose student knowledge gaps.


#849 – On the Road to Assessing Deeper Learning: What Direction Do Test Blueprints Provide?
Joan L. Herman, Deborah La Torre Matrundola, and Jia Wang

Summary

This study examines the extent to which deeper learning is expected to be present in the new college and career ready (CCR) standards. This is done by examining the distribution of items and tasks at high levels of cognitive demand (DOK3 and DOK4) in the summative test blueprints developed by the Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (Smarter Balanced). The study found that while only 10–20% of the consortia’s assessment items and tasks appear to require higher levels of cognitive demand, approximately 30–45% of the total possible raw scores are allocated to deeper learning. Furthermore, the analyses indicated that while the end-of-year (EOY) exams are focused on relatively lower level items, components of the performance tasks primarily concentrate on deeper learning and higher levels of thinking. If the consortia maintain the levels of cognitive demand specified in their blueprints, there is no doubt that this will result in an increase in intellectual demand from prior state tests.


#848 – The Implementation and Effects of the Literacy Design Collaborative (LDC): Early Findings in Eighth-Grade History/Social Studies and Science Courses
Joan L. Herman, Scott Epstein, Seth Leon, Yunyun Dai, Deborah La Torre Matrundola, Sarah Reber, and Kilchan Choi

Summary

The Bill and Melinda Gates Foundation invested in the Literacy Design Collaborative (LDC) as one strategy to support teachers’ and students’ transition to the Common Core State Standards (CCSS) in English language arts. This report provides an early look at the implementation of LDC in eighth-grade history/social studies and science classes in two states, and the effectiveness of the intervention in these settings. The study found that across states and subjects, teachers understood LDC and implemented it with fidelity. Teachers also generally reported positive attitudes about the effectiveness of LDC and its usefulness in introducing literacy instruction into content area classrooms. Quasi-experimental analyses using Coarsened Exact Matching (CEM) techniques and hierarchical linear modeling (HLM) found a small statistically significant positive effect on reading scores in the one state where suitable data were available, but no effects on writing scores. However, students generally performed at low levels on assessments designed to align with the intervention, suggesting the challenge of meeting CCSS expectations. Exploratory analyses suggest that LDC may have been most effective for higher achieving students. However understandable, the findings thus suggest that, in the absence of additional scaffolding and supports for low-achieving students, LDC may be gap enhancing.


#847 РMeasuring the Causal Effect of the National Math + Science Initiative’s College Readiness Program
Richard S. Brown and Kilchan Choi

Summary

This study employs a potential outcomes modeling approach to estimate the causal effect of the National Math + Science Initiative’s College Readiness Program on Advanced Placement test taking and qualifying score earning for three recent cohorts of schools. Results indicate substantial and significant increases in both AP test taking and qualifying score earning for all students. In addition, significant effects for AP test taking and qualifying score earning over baseline were found for female students and minority students when analyzed separately. This study provides evidence of the effectiveness of a College Readiness Program that is having a significant and important impact on preparing more students to succeed in math and science careers and improve the future of math and science education in this country.


#846 – The Implementation and Effects of the Literacy Design Collaborative (LDC): Early Findings in Sixth-Grade Advanced Reading Courses
Joan L. Herman, Scott Epstein, Seth Leon, Yunyun Dai, Deborah La Torre Matrundola, Sarah Reber, and Kilchan Choi

Summary

The Bill and Melinda Gates Foundation invested in the Literacy Design Collaborative (LDC)
as one strategy to support teachers’ and students’ transition to the Common Core State
Standards (CCSS) in English language arts. This report provides an early look at the
implementation of LDC in sixth-grade Advanced Reading classes in a large Florida district,
and the effectiveness of the intervention in this setting. The study found that teachers
understood LDC and implemented it with fidelity and that curriculum modules were well
crafted. Teachers also generally reported positive attitudes about the effectiveness of LDC
and its usefulness as a tool for teaching CCSS skills. Although implementation results were
highly positive, quasi-experimental analyses employing matched control group and
regression discontinuity designs found no evidence of an impact of LDC on student
performance on state reading or district writing assessments. Furthermore, students generally
performed at basic levels on assessments designed to align with the intervention, suggesting
the challenge of meeting CCSS expectations. Exploratory analyses suggest that LDC may
have been most effective for higher achieving students. However understandable, the findings
thus suggest that, in the absence of additional scaffolding and supports for low-achieving
students, LDC may be gap enhancing.


#845 – The Implementation and Effects of the Mathematics Design Collaborative (MDC): Early Findings From Kentucky Ninth-Grade Algebra 1 Courses
Joan L. Herman, Deborah La Torre Matrundola, Scott Epstein, Seth Leon, Yunyun Dai, Sarah Reber, and Kilchan Choi

Summary

With support from the Bill and Melinda Gates Foundation, researchers and experts in mathematics education developed the Mathematics Design Collaborative (MDC) as a strategy to support the transition to Common Core State Standards in math. MDC provides short formative assessment lessons known as Classroom Challenges for use in middle and high school math classrooms. UCLA CRESST’s study of ninth-grade Algebra 1 classrooms in Kentucky implementing MDC showed strong support from teachers for the intervention and a statistically significant positive impact on student scores on the PLAN Algebra assessment, as compared to similar students statewide in Kentucky.


#844 – Semi-Parametric Item Response Functions in the Context of Guessing
Carl F. Falk, Li Cai

Summary

We present a logistic function of a monotonic polynomial with a lower asymptote, allowing additional flexibility beyond the three-parameter logistic model. We develop a maximum marginal likelihood based approach to estimate the item parameters. The new item response model is demonstrated on math assessment data from a state, and a computationally efficient strategy for choosing the order of the polynomial is demonstrated and tested.