Do low-stakes international assessments of student performance accurately describe student ability? According to this recent NBER Working paper, a country’s results may depend on their students’ intrinsic motivation to do well.
The authors used an experimental approach that offered treatment students a small financial incentive to do well on a test. Students in the treatment group were told just before the test that an envelope with the equivalent of $25 (US) was theirs, but that $1 would be subtracted for each incorrect answer. The researchers conducted the experiment in three high schools in China and two in the US. They found no impact of the financial incentive in China, but the effect size in the US was 0.20-0.23 standard deviations.
The authors conclude that the massive difference in treatment effect between countries indicates that “success on low states assessments does not solely reflect differences in ability between students across countries” and that “low-stakes tests do not measure and compare ability in isolation.”
With this in mind, researchers, policymakers, and the public should be a little more cautious when interpreting the results of low-stakes assessments like PISA. Student motivation clearly plays a significant role in the results of a country’s performance - a topic that deserves further investigation. For example, NCES will release new NAEP results in early 2018. Does student motivation vary significantly for NAEP across state lines? If so, how much of a role (if any) is it contributing to state performance?
Low-stakes assessments can be informative, but they may be measuring more than student ability. This study is a reminder to policymakers and advocates to exercise caution when comparing performance on these tests across jurisdictions.