In this primer, two CRESST researchers describe the use of data logging to
support measurement of student learning through games. Their research-based
recommendations include: specify the behavior to log ahead of time; record
data that reflects behavior rather than inferences about the behavior; log
in-game behaviors that map directly to targeted knowledge, skills, and
attitudes; and encode sufficient information so that the data elements are
unambiguous at the desired grain size.

Educational videogames can provide instructional feedback, but research to date indicates that students often ignore such feedback and instead, use less effective help-seeking strategies. Using a pre-algebra videogame and a randomized design, this study compared learning outcomes for students who received an incentive to use feedback compared to students not receiving the incentive. Results indicated that students given the incentive to use feedback had significantly higher math scores than those not receiving the incentive. Effects were stronger for students with low academic motivation.

In this study we compared the effects of two math game designs on math and game performance, using discrete-time survival analysis (DTSA) to model players’ risk of not advancing to the next level in the game. 137 students were randomly assigned to two game conditions. The game covered the concept of a unit and the addition of like-sized fractional pieces. The math content in the baseline version of the game focused on procedures and did not elaborate on the math topics. The experimental version of the game provided more conceptual instruction by emphasizing the underlying concepts in fractional addition. Students played the game for 30 minutes. DTSA was used to examine student game performance, and whether and how game performance relates to students’ prior math knowledge and game experience. Students who played the experimental version of the game were less likely to fail in the game relative to students who played the baseline version of the game (odds ratio = 0.64). Students with higher prior knowledge of fractions also were less likely to fail in the game (odds ratio = 0.41), and students with more game experience were less likely to fail (odds ration = 0.58). The use of DTSA provided an analytical method to understand game performance and game process data. DTSA enabled examination of the game play progression of students with various characteristics over sequences of game levels.

A central challenge in efficacy studies centers on the issue of “fidelity of implementation,” that is, the extent to which participants use the curriculum specified by curriculum developers. In this study, we describe and discuss a “fidelity of implementation” model using multiple methods and instruments to compare two versions of a science curriculum and embedded assessment system. We present results from our validation study and discuss the challenges in determining the extent to which teachers use a curriculum as designed. We focus as well on the practical curriculum implementation issues amidst the needs and perspectives from different stakeholders.

This report examines enrollment history, achievement gaps, and persistence in school for ELL students and reclassified ELL students as compared to non-ELL students. The study uses statewide individual-level data sets merged from students’ entry to exit in a state public school system for graduate cohorts of 2006, 2007, and 2008. Analytic methods include multilevel logistic regression in which students are nested within districts to study correlates of dropouts. The results reconfirmed other literature showing large achievement and socio economic gaps between ELL and non-ELL students. Results also show that after accounting for academic achievement, behavioral issues, background, and district contexts, the longer a student is designated as an ELL, the more likely he or she is to drop out of school.

Drawing from a large efficacy study in upper elementary science, this report
had three purposes: First to examine the quality of teachers'
content-pedagogical knowledge in upper elementary science; second, to
analyze the relationship between teacher knowledge and their assessment
practice; and third, to study the relationship between teacher knowledge,
assessment practice, and student learning. Based on data from 39 teachers,
CRESST researchers found that students whose teachers frequently analyzed
and provided feedback on student work had higher achievement than students
whose teachers spent less time on such activities. The findings support
other research indicating the power of well-implemented formative assessment
to improve learning.

Understanding the meaning of rational numbers and how to perform mathematical operations seems to be a perennial problem in the United States for both adults and children. CRESST researchers hypothesized that giving students more time to practice using rational numbers in an environment that enticed them to apply their understanding might prove educationally beneficial. They developed a video game, based on two key ideas about addition and rational numbers, to investigate their hypothesis. in this initial study, they found that students using an appropriately design game increased their ability to add rational numbers even when playing the game for a relatively short period of time. The authors discuss implications for the larger efficacy study to follow.

In recent years, a large number of school districts have adopted interim or benchmark assessments to help inform instruction during the school year. In this study, researchers drew from interviews across seven school districts and analyzed: 1) teachers understandings of their school districts’ purposes for interim assessments; 2) professional development to support teachers effective use of assessment information; and 3) reported teacher use of the data to adjust instruction. The researchers found that although many teachers expressed an interest in using interim assessment results as intended, that they often received minimal professional development and frequently had a different understanding regarding the intended use of the assessments than did district leaders. The researchers concluded that, in general, the interim assessment data did not provide teachers with insights about what to do next to help students other than to reteach.

In order to provide more frequent information about student progress during the year, many school districts have been implementing "interim" or "benchmark" assessment programs. To date, little research has examined the implementation of interim assessments or their effects on teaching and learning. This new CRESST report investigates purposes in adopting interim or benchmark assessments, ensuing implementation efforts, and actual assessment uses. The researchers found a number of substantial barriers to success including test questions that were predominantly multiple-choice, lack of professional development for teachers, and minimal coherence in shared understandings of assessment purposes and uses across district, school, and classroom levels. Based on the results, the researchers provide recommendations for a successful interim or benchmark assessment system.

This paper introduces a new text mining framework using a tree-based Linguistic Query Language, called LQL. The framework generates more than one parse tree for each sentence using a probabilistic parser, and annotates each node of these parse trees with main-parts information. Using main-parts-annotated parse trees for a given textual dataset, the system can efficiently answer individual queries as well as mine the text for a given set of queries. The framework also has the ability to support grammatical ambiguity through probabilistic rules and linguistic exceptions in order to increase the quality of the extracted information.