Kylie L. Anglin

PhD Student in Education Policy

EdPolicyWorks at the University of Virginia

Kylie L. Anglin is a Ph.D. student in Education Policy at the University of Virginia. Her research focuses on developing and using data science approaches to examine variations in policy and intervention implementation, as well as the impact of intervention heterogeneity on student outcomes. In an article published in the Journal of Research on Educational Effectiveness, she proposed a framework for using web scraping and natural language processing techniques to collect data on district responses to state policies from online policy documents. In that article, Kylie illustrates this framework in the context of statewide education deregulation under the Texas District of Innovation statute. In her dissertation, she builds on her descriptive work surrounding the District of Innovation statute to evaluate the impact of the policy, and heterogeneous district responses, on student outcomes.

Kylie is also developing methods for using natural language processing techniques to assess treatment fidelity and replicability in intervention evaluations that take place in educational settings. She has published on methods related to causal inference, replication, and open science in Evaluation Review, Oxford Bibliography in Education, and Zeitschrift für Psychologie, and she is a regular presenter at APPAM, AEFP, and SREE.

Prior to coming to University of Virginia, Kylie earned a B.A. in Political Science from Southwestern University, a Post-Baccalaureate in Mathematics from Northwestern University, and a Masters in Public Policy from the University of Virginia. Kylie has worked as a 7th grade English teacher and as an evaluator for an after-school program.

Interests

Education Regulation
Causal Inference
Effect Heterogeneity and Implementation
Natural Language Processing

Education

PhD in Education Policy Evaluation, 2021 (expected)

University of Virginia
Masters in Public Policy, 2018

University of Virginia
Post-Baccalearate in Mathematics, 2015

Northwestern University
BA in Political Science, 2013

Southwestern University

Publications and Working Papers

December 2019 Kylie L. Anglin (2019) Gather-Narrow-Extract: A Framework for Studying Local Policy Variation Using Web-Scraping and Natural Language Processing, Journal of Research on Educational Effectiveness, 12:4, 685-706, DOI: 10.¹⁰⁸⁰⁄_19345747.2019.1654576

Gather-Narrow-Extract: A Framework for Studying Local Policy Variation Using Web-Scraping and Natural Language Processing

Education researchers have traditionally faced severe data limitations in studying local policy variation; administrative data sets capture only a fraction of districts’ policy decisions, and it can be expensive to collect more nuanced implementation data from teachers and leaders. Natural language processing and web-scraping techniques can help address these challenges by assisting researchers in locating and processing policy documents located online. School district policies and practices are commonly documented in student and staff manuals, school improvement plans, and meeting minutes that are posted for the public. This article introduces an end-to-end framework for collecting these sorts of policy documents and extracting structured policy data: The researcher gathers all potentially relevant documents from district websites, narrows the text corpus to spans of interest using a text classifier, and then extracts specific policy data using additional natural language processing techniques. Through this framework, a researcher can describe variation in policy implementation at the local level, aggregated across state- or nationwide populations even as policies evolve over time.

January 2019 Anglin, K. L., Krishnamachari, A. & Wong, V. C. (2020). Methodological Approaches for Impact Evaluation in Educational Settings. In Oxford Bibliographies in Education. New York: Oxford University Press.

Methodological Approaches for Impact Evaluation in Educational Settings

Since the start of the War on Poverty in the 1960s, social scientists have developed and refined experimental and quasi-experimental methods for evaluating and understanding the ways in which public policies, programs, and interventions affect people’s lives. The overarching mission of many social scientists is to understand “what works” in education and social policy. These are causal questions about whether an intervention, practice, program, or policy affects some outcome of interest. Although causal questions are not the only relevant questions in program evaluation, they are assumed by many in the fields of public health, economics, social policy, and now education to be the scientific foundation for evidence-based decision making. Fortunately, over the last half-century, two methodological advances have improved the rigor of social science approaches for making causal inferences. The first was acknowledging the primacy of research designs over statistical adjustment procedures. Donald Campbell and colleagues showed how research designs could be used to address many plausible threats to validity. The second methodological advancement was the use of potential outcomes to specify exact causal quantities of interest. This allowed researchers to think systematically about research design assumptions and to develop diagnostic measures for assessing when these assumptions are met. This article reviews important statistical methods for estimating the impact of interventions on outcomes in education settings, particularly programs that are implemented in field, rather than laboratory, settings.

October 2018 Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870

What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?

Given the widespread use of nonexperimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the same target population. The goal is to assess whether the stringent assumptions required for NE methods are likely to be met in practice. This essay provides an overview of recent efforts to empirically evaluate NE method performance in field settings. We discuss a brief history of the design, highlighting methodological innovations along the way. We also describe papers that are included in this two-volume special issue on WSC approaches and suggest future areas for consideration in the design, implementation, and analysis of WSCs.