New Insights and Opportunities
Web scraping is relatively inexpensive, disaggregates job postings by subject area, and can capture data twice a week. Because job postings are available at this frequency, they provide a clear, nuanced, and timely signal of school staffing needs.
The importance of timeliness cannot be overstated, especially when considering traditional data reporting and state legislative calendars. In Washington State, legislators conclude their work by late April, or sooner, each year. In spring 2023, for example, any state funding or policy decisions aimed at addressing teacher staffing challenges in the 2023–24 school year would have been based on data from 2021 (a May 2021 report on teacher shortages and the state education department’s 2021 report card) and early 2022 (the national teacher shortage survey from 2021–22). Data from job postings would have provided a detailed snapshot from the fall of 2022—more current by almost a full year.
The timebound nature of scraped job postings data also sheds light on school and district hiring processes as they relate to teacher quality. Research has shown that late hires, which happen in late summer or early fall, can be detrimental to student learning. Observing when districts post open jobs can signal whether they are engaged in early hiring, late hiring, or something in between, where the difference can mean new teachers have months to prepare or are scrambling to get their footing after the school year has already begun. Such information could inform policy and targeted supports for more effective management practices.
Job postings data can also serve as a ready measure of policy impact. State personnel data tell us when new people are hired, but those data are generally not available until the following school year. This lag is limiting when policymakers are facing immediate issues, such as the imminent expiration of federal Covid-era relief funding. For example, in a separate analysis, we looked at scraped job postings data in Washington State to determine the impact of federal ESSER (Elementary and Secondary School Emergency Relief Fund) dollars on school staffing. That analysis found that 12,000 school staff and roughly 5,100 teachers were hired statewide with ESSER funds—suggesting that district budgets will likely face cuts when the funds are gone.
Scraped job postings data do not capture everything education leaders and policymakers need to know. Job postings data don’t show the share of teachers working out-of-field or with emergency credentials, or illuminate trends in teacher retention, diversity, or quality. However, the data can serve as an important accompaniment to traditional data collection.
Many states are currently investing millions of dollars to support new pathways into the teaching profession; scraped data could steer investment toward subject area and school types most in need of new teachers. Incentives such as loan forgiveness, bonuses, housing stipends, and salary boosts could also be directed toward districts that struggle to hire sufficient staff. Such policies have helped address shortages in the past; for example, in Hawaii, a $10,000 bonus program for new special education teachers reduced vacancies by 32 percent, research by one of us (Roddy Theobald) has found. States could also use scraped data to coordinate with undergraduate institutions to recruit students in shortage areas like STEM to pursue a teaching credential. Additionally, typical counts of unfilled classroom jobs tend to overemphasize the shortage of elementary teachers, which are the largest share of the workforce, and understate the immediate needs for special education and STEM teachers, who are employed in smaller numbers overall.
Common sense and job postings data tell us that teacher shortages are multifaceted and complex. There are multiple, distinct challenges to bring credentialed educators into the classroom based on subject area and school type, and hiring enough qualified candidates will require a variety of inputs and incentives. Web scraped job postings data highlight inequities in teacher demand and supply that go beyond what is captured in administrative data, and they appear to be a valid signal of district and school hiring needs. This low-cost, timely method of data collection can provide vital information to policymakers and educators who hope to address systemic inequities in public schools.
