Ideals, Meet Reality
In short, because states didn’t take on this data-collection and evidence-building work, yet some large-market states require it, publishers are performing these tasks. It’s fair game to scrutinize their efforts—and they could certainly invest more in careful research both to improve their products and to demonstrate effectiveness. But many obstacles impede that vision and underscore the need for more independent, centralized curriculum evaluations, ideally led by states or foundations.
Unrealistic timelines present one set of obstacles. Relatively few states require that curriculum materials meet ESSA evidence standards for statewide approval, but the number is not zero. And, of course, districts are making purchases all the time, often corresponding to their states’ curriculum adoption cycles, so publishers need to produce evidence quickly. But how can they evaluate their materials if they can’t sell them to schools and districts in order to conduct an evaluation? Often curriculum adoption decisions come shortly after standards revisions, but companies need time to adapt their materials to new standards. Add in the time needed to do a serious evaluation, and the process can take several years. And that’s if researchers are only looking at short-term outcomes like test score gains over a single year—outcomes that may be misleading about a curriculum’s long-term effects.
Logistics, especially the challenge of recruiting participants, also present challenges. To conduct an impact evaluation that meets current Tier 1 standards, publishers would need to get some set of schools or districts to agree to random assignment of curriculum materials, with researchers deciding by chance which schools or districts would use a given curriculum and which would not. Realistically, this is hard: Districts aren’t exactly champing at the bit to have their core instructional materials randomly assigned. Probably the best case is that publishers could convince an existing district customer to allow some subset of their schools to undergo random assignment. In that case, though, the evaluation would compare the company’s new curriculum with its older one. But because companies want to sell both products, that study design is guaranteed to produce results that won’t benefit the company. It’s hardly good marketing practice to present a conclusion like “Our new curriculum is 0.25 standard deviations more effective than the one we’ve been selling for a decade.”
Questions of validity. Studies relying on random assignment or analyzing early adoption of curriculum materials may well be sacrificing the generalizability of their results at the altar of causal surety. That is, the kinds of schools and districts likely to allow random assignment or to be early adopters may differ systematically from those that are not, so the results produced may not represent what would occur if the materials were more widely adopted. And random assignment studies may well entail ideal implementation conditions that don’t mirror most school environments. For instance, the publisher may provide high-quality professional learning directly to educators in participating districts, in contrast to the more typical conditions in more naturalistic, quasi-experimental studies of districts that choose to adopt a particular set of materials.
Measuring student outcomes. Evaluation of curriculum materials requires some measure of student outcomes, usually the state test. But state tests often lag behind the development of new state learning standards, or, as has been true in the past, they may fail to accurately assess the full breadth of content prescribed by the standards. Beyond this, state tests infamously struggle with their sensitivity to instruction, and different tests correlate differently with measures of instructional quality. Why wouldn’t we expect the same variability to show up in curriculum effectiveness studies? What’s more, many children are performing far below grade level, but state tests are required to assess grade-level standards, potentially obscuring meaningful student growth. We certainly can’t expect that districts participating in curriculum-evaluation studies will also allow for additional assessment using a neutral external test. If they were to do so, however, and if the results based on different tests were to disagree, how would we determine which set of results is right?
