This blog post is the first in a series highlighting the ways in which interoperable data can enable more timely research insights for policymakers and practitioners. Future blog posts will explore in more depth several research projects using this data, including early findings, lessons learned, and recommendations for the field.
Molly Stewart
Libby Pier
In U.S. education research, we face a persistent challenge: for the billions of dollars invested during the last 50 years, why are student outcomes in this country not improving? As education researchers trained in traditional doctoral programs and currently working at the crux of education research, policy, and technology, we believe the current processes and systems involved in doing research are nearing the limit of their usefulness.
This blog aims at not only examining these challenges but also identifies ways in which we and our partners in state and local education agencies, institutions of higher education, and other non- and for-profit technology groups are working on innovative solutions to these challenges.
This first installment provides a brief history of the trends of the previous fifty years of dominant quantitative education research. Stay tuned for upcoming blog installments, where we’ll dive deeper into:
a) how we and our partners currently use interoperable data to answer immediate, locally relevant research questions, and
b) how we are beginning to pilot researcher-agency partnerships that can provide intermediary support for researchers and agencies to achieve faster, more granular, and more relevant research that can positively impact K-12 education in the United States.
While very early education research was primarily observational and subjective (Lagemann, 1997), methods have become increasingly rigorous as seen in randomized controlled trials (RCTs), known as the “gold standard.” RCTs began growing in popularity starting around 2002, due in large part to the policies established in the No Child Left Behind Act, funding from the federal Institute of Education Sciences (IES), and the establishment of the What Works Clearinghouse (Hedges & Schauer, 2018). However, there have always been feasibility and ethical concerns with using RCTs to study educational supports for children, (e.g., Parra & Edwards, 2024), as well as calls for pairing RCTs with other methods in order to more fully understand contextual implications (Deaton & Cartwright, 2018), of which there are countless in the realm of education.
RCTs and other methods have also proven to be difficult to replicate across contexts, which has led to a reevaluation in the field regarding the limitations of experimental and quasi-experimental designs for helping us understand what works for students at scale.
Part of the challenge for educational research, however, is not just the methods at our disposal–it is deeply entwined with data availability issues. Data silos, by which data in different school district departments and levels of government are disconnected, increase the difficulty of data collection by researchers, requiring data to be collected in disparate formats, sometimes with different types of identifiers, and requests for data collection from oversubscribed agency staff makes compliance with such requests difficult to prioritize over the immediate needs of district operations and workflows.
It may take years to get permission to collect or use data from an educational setting or agency. Combining and cleaning various data sets to answer one set of research questions often takes the bulk of a research team’s effort and cost. By the time the steps in the research process are complete, years may have passed, and the policy or practice under study may no longer be of priority interest to policies and/or educators. And, most importantly, delayed research on that policy or practice means that that work may never be able to positively impact or support schools and students.
A common example is pandemic-era research being published after most schools have returned to in-person learning. Research-based recommendations on how to address pandemic-related disruptions would have been far more valuable during the height of school lockdowns. Without this kind of research, educators likely made rational instructional decisions, but without evidence to guide them, sometimes resulted in potential exacerbation of the original challenges of the pandemic.
Our own training and experience in traditional research methods have shown us that change is necessary. We need a stronger, more integrated data infrastructure. In today’s landscape, research relies on data obtained through one of two primary pathways:
- Researchers collect data themselves, which is slow, costly, and not scalable—but customized to the research question; or
- Researchers rely on data collected or generated for other purposes, which is faster, cheaper, and more scalable, but generic and often only a proxy for the data needed to fully answer the research question
When we can leverage more accurate and complete data at scale—and create tighter feedback loops with practitioners and policymakers—education research will increase its impact dramatically. These advancements will allow us to generate more timely, relevant insights that address the most pressing educational issues and support better outcomes for students and families across the United States.
In our experience, a promising path for educational research modernization is leveraging interoperable data infrastructure and related technologies. This blog series will highlight cutting-edge work by researchers who are using these technologies in partnership with state and local education agencies. Our vision for modern research infrastructure in education is a technology-driven, data-rich ecosystem where research is continuously embedded into schools’ operational systems. Instead of relying on static data collections, this infrastructure integrates dynamic data streams to ensure that research directly informs and improves teaching, learning, and policy with quick and relevant feedback loops. The increased feedback loop from researchers to educators also provides information to researchers about the variables in their studies—do we need to look at additional factors that might affect a given outcome? Researchers can quickly evaluate the extent of implementation or take-up by policymakers or practitioners who might use their results to improve teaching or decision-making (e.g., Demszky et al., 2023).
The future of research using standardized, near-real-time data flows is still not fully understood, and researchers and education practitioners are likely to go through several iterations before determining the full range of possibilities. However, we at Education Analytics are embarking on several research partnerships that we will showcase in this blog series. Stay tuned, and if you are a researcher interested in learning more about these possibilities, please reach out to us to start a conversation!
References
Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science and Medicine, 210, 2-21.
Demszky, D., et al. (2021). Can automated feedback improve teachers’ uptake of student ideas? Evidence from a randomized controlled trial in a large-scale online course. Educational Evaluation and Policy Analysis, 46(3).
Hedges, L. V., & Schauer, J. (2018). Randomised trials in education in the USA. Educational Research, 60(3), 265-275.
Lagemann, E. C. (1997). Contested terrain: A history of education research in the United States, 1890-1990. Educational Researcher, 26(9), 5-17.
Parra, J. D., & Edwards, D. B., Jr. (2024). Challenging the gold standard consensus: Randomised controlled trials (RCTs) and their pitfalls in evidence-based education. Critical Studies in Education, 65(5).