Please note that CRESST reports were called "CSE Reports" or "CSE Technical Reports" prior to CRESST report 723.
Carl F. Falk, Li Cai
We present a logistic function of a monotonic polynomial with a lower asymptote, allowing additional flexibility beyond the three-parameter logistic model. We develop a maximum marginal likelihood based approach to estimate the item parameters. The new item response model is demonstrated on math assessment data from a state, and a computationally efficient strategy for choosing the order of the polynomial is demonstrated and tested.
Jia Wang, Jonathan D. Schweig, Joan L. Herman
Magnet schools are one of the largest sectors of choice schools in the United States. In this study, we explored whether there is heterogeneity in magnet school effects on student achievement by examining the effectiveness of 24 recently funded magnet schools in 5 school districts across 4 states. We used a two-step analysis: First, separate magnet school effects were estimated using a propensity score matched regression approach to address selection bias. Second, the magnet effects were synthesized across schools using a multi-level random-effects meta-analytic framework. Results indicated that there is significant variation in magnet school effects on student outcomes, with some magnet schools showing positive effects, and others showing negative effects. This variation can be explained by program implementation and magnet support.
Scott Monroe, Li Cai, and Kilchan Choi
This research concerns a new proposal for calculating student growth percentiles (SGP, Betebenner, 2009). In Betebenner (2009), quantile regression (QR) is used to estimate the SGPs. However, measurement error in the score estimates, which always exists in practice, leads to bias in the QR-based estimates (Shang, 2012). One way to address this issue is to estimate the SGPs using a modeling framework that can directly account for the measurement error. Multidimensional IRT (MIRT) is one such framework, and the one utilized here. To maximize the generality of the approach, the SNP-MIRT model (Monroe, 2014), which estimates the shape of the latent variable density, is used to obtain model parameter estimates. These estimates are then used with the calibrated projection linking methodology (Thissen, Varni, et al., 2011, Thissen, Liu, Magnus, & Quinn, 2014, Cai, in press-a, Cai, in-press-b) to produce SGP estimates. The methods are compared using simulated and empirical data.
Gregory K.W.K Chung, Kilchan Choi, Eva L. Baker, and Li Cai
A large-scale randomized controlled trial tested the effects of researcher-developed learning games on a transfer measure of fractions knowledge. The measure contained items similar to standardized assessments. Thirty treatment and 29 control classrooms (~1500 students, 9 districts, 26 schools) participated in the study. Students in treatment classrooms played fractions games and students in the control classrooms played solving equations games. Multilevel multidimensional item response theory modeling of the outcome measure produced scaled scores that were more sensitive to the instructional treatment than standard measurement approaches. Hierarchical linear modeling of the scaled scores showed that the treatment condition performed significantly higher on the outcome measure than the control condition. The effect (d = 0.58) was medium to large (Cohen, 1992).
Mark Hansen, Li Cai, Scott Monroe, and Zhen Li
Although diagnostic classification models have become increasingly popular in educational and psychological measurement, one noted limitation has been a lack of established methods for evaluating their goodness-of-fit. This is a significant problem, since inferences made on the basis of these models (including classification of examinees) may be invalid when the models are badly misspecified. This study examines the potential utility of two indices for testing the fit of these models: Maydeu-Olivares and Joe's (2006) M2 and Chen and Thissen's (1997) local dependence (LD) chi-square. These two indices have been previously applied to standard item response theory models. Here we evaluate their performance when applied to diagnostic classification models. Both were found to have good calibration under a wide range of data generating conditions and were sensitive to some--but not all--types of misspecification examined. The study suggests that M2 and the LD index may be quite useful for detecting and characterizing diagnostic classification model misfit.
Li Cai and Scott Monroe
We propose a new limited-information goodness of fit test statistic C2 for ordinal IRT models. The construction of the new statistic lies formally between the M2 statistic of Maydeu-Olivares and Joe (2006), which utilizes first and second order marginal probabilities, and the M2* statistic of Cai and Hansen (2013), which collapses the marginal probabilities into means and product moments. Unlike M2*, C2 may be computed even when the number of items is small and the number of categories is large. It is as well calibrated as the alternatives
and can be more powerful than M2. When all items are dichotomous, C2 becomes equivalent to M2*, which is also equivalent to M2. We analyze empirical data from a patient-reported outcomes measurement development project to illustrate the potential differences in substantive conclusions that one may draw from the use of different statistics for model fit assessment.
Educational video games provide an opportunity for students to interact with and explore complex representations of academic content and allow for the examination of problem-solving strategies and mistakes that can be difficult to capture in more traditional environments. However, data from such games are notoriously difficult to analyze. This study used a three-step process to examine mistakes students make while playing an educational video game about the identification of fractions. First, cluster analysis was used to identify common misconceptions in the game. Second, a survey was given to determine if the identified in-game misconceptions represented real-world misconceptions. Third, a second educational video game was analyzed to determine whether the same misconceptions would be identified in both games. Results indicate that the in-game misconceptions identified in this study represented real-world misconceptions and demonstrate that similar misconceptions can be found in different representations.
Digital games offer an appealing environment for assessing student proficiencies, including skills and misconceptions in a diagnostic setting. This paper proposes a dynamic Bayesian network modeling approach for observations of student performance from an educational video game. A Bayesian approach to model construction, calibration, and use in facilitating inferences about students on the fly is described.
Deirdre Kerr, Hamid Mousavi, and Markus R. Iseli
The Common Core assessments emphasize short essay constructed response items over multiple choice items because they are more precise measures of understanding. However, such items are too costly and time consuming to be used in national assessments unless a way is found to score them automatically. Current automatic essay scoring techniques are inappropriate for scoring the content of an essay because they rely on either grammatical measures of quality or machine learning techniques, neither of which identifies statements of meaning (propositions) in the text. In this report, we explain our process of (1) extracting meaning from student essays in the form of propositions using our text mining framework called SemScape, (2) using the propositions to score the essays, and (3) testing our system’s performance on two separate sets of essays. Results demonstrate the potential of this purely semantic process and indicate that the system can accurately extract propositions from student short essays, approaching or exceeding standard benchmarks for scoring performance.
Deirdre Kerr and Gregory K.W.K. Chung
Student mathematical errors are rarely random and often occur because students are applying procedures that they believe to be accurate. Traditional approaches often view such errors as indicators of students’ failure to understand the construct in question, but some theorists view errors as opportunities for students to expand their mental model and create a deeper understanding of the construct. This study examines errors in an educational video game that are indicative of two specific misunderstandings students have about fractions (unitizing and partitioning) in order to determine whether the occurrence of those errors makes students more likely to learn from the game or more likely to be confused by the game. Analysis indicates that students who made unitizing errors were more likely to be confused by the game while students who made partitioning errors were more likely to learn from the game.