Even though information plays a central role in markets with consumer choice, we know relatively little about how individual behavior can distort information and affect markets. In the context of education, a critical piece of information is school quality, which is often measured via standardized tests. However, critics of these tests points towards Holmstrom and Milgrom (1991) classic work, and argue that high-stakes testing might generate undesirable behavioral responses that introduce distortions in the metric itself. Despite increasing evidence of undesirable behavioral responses, quantification of these potential distortions and their market-level effects is surprisingly scarce.

There is substantial academic evidence showing behavioral responses to accountability systems. For example, Figlio and Winicki (2005) show that schools increased the caloric intake of student lunches on testing days and affected test scores. Another classic example is the selective assignment of students to special education programs to manipulate test scores in Jacob (2005). Recent work has, however, started to collect evidence on the impact of these distortions on relevant educational and labor market outcomes of students that were subject to these behavioral responses. For instance, we know that grade inflation around passing grades in the New York school district and in Sweden have long lasting effects on students. On the one hand Dee at al (2017) find that students who approve a class due to manipulation have slightly higher high school graduation rates, but often take less advanced coursework and are less likely to enroll in college. On the other hand, Diamond and Persson (2017) find that students who approve a class due to manipulation are more likely to enroll in college, complete more years of education, are less likely to have children and have higher earnings than students who were at the margin of being passed due to manipulation but were not. This research highlights that behavioral responses to accountability system may have sizable and long-lasting consequences on students, with a direction that depends on the context.

One could expect, however, that behavioral responses to accountability systems have consequences beyond individuals and propagate to the educational system as a whole. In a recent study, we explore this possibility. We study one of the most developed accountability systems in the world –Chile’s market-oriented educational system– and show that behavioral responses exist and indeed affect important market outcomes. In the Chilean educational system, the government relies on a standardized test called SIMCE to generate a school-specific quality metric. These measures are used for quality assessment, performance evaluation, and as a disclosure system in school choice. These features make Chile a convenient setting to quantify the consequences of behavioral responses to standardized tests.

Our analysis goes from measuring distortions to understanding their consequences for the market as a whole. We begin by showing that low-performing students are more likely to be absent on test days relative to other students. Using national administrative data on Chilean school children, we compare daily attendance of test takers (fourth graders) and non-takers (third graders) within schools on test and non-test days. High performing test-takers increase their attendance on test days relative to low performing test-takers. This result suggests that a behavioral response to standardized testing is at work. Importantly, the degree of student non-representativeness varies considerably across schools, suggesting that the extent to which average test scores are distorted by this behavior is different across schools. In a second stage, we measure the effects of absenteeism on test days. We use a statistical procedure called multiple imputations to predict the test scores of absent students and thus the associated distortions in school quality signals. We show that there are substantial differences in distortions across schools.

The key question we ask is: what are the impacts of these distortions on the educational system? We focus on two relevant margins, school choice and the allocation of public programs. To address the first dimension, we estimate a school choice model. In a counterfactual simulation, we find that if we remove informational distortions we would induce three percent of students to switch schools. Our results suggest that households that would change their choices would be willing to pay as much as 117 U.S. dollars annually for undistorted information. Importantly, high-income households would be willing to pay more than low-income households for this information, given the former are on average more quality-sensitive and less price-sensitive than the latter. This finding suggests that policies complementing informational policies in educational markets with policies that increase quality-sensitivity might help preventing these distributional effects. An additional dimension where distortions have impacts is the allocation of public programs that are assigned based on school performance. To estimate the effect in this margin, we proceed to simulate what the allocation of these programs would be in absence of informational distortions. We focus on a teacher incentives program that compensates teachers for both the level and improvement of school performance. We show that as much as 13 percent of bonuses are incorrectly allocated because of distortions, implying substantial misallocation of public funds.

Overall, we attempt to highlight the strong link between behavioral responses to accountability systems in education and relevant outcomes for the system. While we implement our analysis in the Chilean educational market, the implications of it go beyond this country and schooling. Multiple markets in which quality is imperfectly observed have quality disclosure systems, many of which may create incentives for undesirable behavioral responses. Moreover, whenever quality signals generated by the disclosure system feed into consumer and government choices, implications similar to those discussed in our study might arise. This highlights the relevance of designing disclosure systems that are unlikely to be affected by participants in the market.

References:

Cuesta, J. I., González, F., Larroulet, C. (2017). Distorted quality signals in school markets. Manuscript.

Dee, T. S., Dobbie, W., Jacob, B. A., and Rockoff, J. (2017). The causes and consequences of test score manipulation: Evidence from the New York regents examinations. Manuscript.

Diamond, R. and Persson, P. (2016). The long-term consequences of teacher discretion in grading of high-stakes tests. Manuscript.

Figlio, D., Winicki, J. (2005). Food for thought: the effects of school accountability plans on school nutrition. Journal of Public Economics, 89(2-3):381-394.

Holmstrom, B, Milgrom, P. (1991). Multitask principal-agent analyses: incentive contracts, asset ownership, and job design. Journal of Law, Economics & Organization, 7:24-52.

Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics, 89:761 – 796.