Although students are fundamentally impacted by policies made at the school and district level, education researchers have traditionally faced severe data limitations in studying local policy variation. Administrative datasets capture only a fraction of the policy decisions districts make, and it can be expensive to collect more nuanced implementation data from teachers and leaders. Natural language processing and web-scraping techniques can help address these challenges by assisting researchers in locating and processing policy implementation documents located online. School district policies and practices are commonly documented in student and staff manuals, school improvement plans, and meeting minutes that are posted for the public. This paper introduces and end-to-end framework for collecting these sorts of policy documents and extracting structured policy data: the researcher gathers all potentially-relevant documents from district websites, narrows the text corpus to spans of interest using a text classifier, and then extracts specific policy data using additional NLP techniques. Using this framework, a researcher can describe variation in policy implementation at the local level, aggregated across state- or nation-wide populations even as policies evolve over time.

Given the widespread use of nonexperimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the same target population. The goal is to assess whether the stringent assumptions required for NE methods are likely to be met in practice. This essay provides an overview of recent efforts to empirically evaluate NE method performance in field settings. We discuss a brief history of the design, highlighting methodological innovations along the way. We also describe papers that are included in this two-volume special issue on WSC approaches and suggest future areas for consideration in the design, implementation, and analysis of WSCs.