Building Shared Data Infrastructure for Public Education: EDU, Databricks, and the Power of Community Code

When Education Analytics built Enable Data Union (EDU), it was built on a principle that shared infrastructure for public education should be open, standards-based, well-governed, and never beholden to any single technology provider. EDU was built to lower or eliminate roadblocks and give education agencies access to a modern, interoperable data warehouse — regardless of their technology environment. One of the core benefits of building interoperable, open tools is that no agency gets left behind when the infrastructure improves. When one state or agency contributes, everyone benefits. Public agencies don’t always have the flexibility to change or add a new cloud provider. Additionally, contracts are often statewide, procurement environments are complex, and technology decisions made years earlier can shape what’s possible today.

However, development has to start somewhere, and EDU's path to multi-platform support started with a single option: AWS for cloud hosting, Snowflake as a data platform. As EDU expanded and states and districts began seriously exploring implementation, it became clear that it was time to expand support to additional cloud platforms. When the Tennessee Department of Education (TDOE) approached us about implementing EDU, they had a clear platform constraint: their environment ran on Microsoft and Databricks, and switching to AWS and Snowflake was not an option. This gave us the opportunity to show that EDU’s commitment to open, vendor-neutral infrastructure wasn’t just a principle – it was built into the code and an offering that all states and agencies could access. After a close collaboration with TDOE, including direct code contributions by their team, EDU is compatible with Databricks.

A Platform Question Is Really an Access Question 

Previously, states and districts in Microsoft-only environments, or locked into existing procurement agreements, couldn’t adopt EDU regardless of how much they wanted to. Platform constraints were keeping agencies on the sidelines of a community that was already building and improving shared infrastructure without them.

This is what EDU’s multi-platform support solves. The result is an EDU that works for more agencies than before — and a community that gets stronger every time another state joins it. EDU exemplifies EA's mission by choosing collaboration, transparency, and access over individualism, black boxes, and lock-in. Databricks compatibility is a practical move that falls in line with our commitment to community and collaboration by making EDU more accessible. 

Built Once, Ready for All 

For states and agencies looking to implement EDU on Databricks, the path has already been cleared. EA worked with the Tennessee Department of Education’s lead data engineer, who learned how EDU was architected and wrote the Databricks-specific implementation needed to make it work. Rather than keeping that work in a private fork, it was shared back to the core codebase — so every agency that comes after Tennessee gets the benefit of what Tennessee built. Due to the technical architecture of EDU, this one-time effort in Tennessee will lead to the ongoing compatibility of EDU with Databricks.

Now, one state’s contribution becomes a starting point for many. 

How Architecture Made This Possible 

The technical decisions we made early in the development of EDU are what made Tennessee’s contribution, and that of future states and school districts, achievable in the first place. 

EDU is built on two foundational open-source tools: dbt for data transformation and Apache Airflow for orchestration. We chose these tools deliberately because they’re powerful and platform-agnostic by design. 

On the dbt side, we use a feature called adapter dispatch. Instead of writing Snowflake-specific SQL directly into our models, we write a function call and then define what that function does on each platform separately. Here’s a simple example: 

The generic call that lives in EDU’s models: 

{{ json_array_size(column) }} 

What runs on Snowflake: 

array_size(column) 

What Tennessee wrote for Databricks: 

size(try_cast(column as array<string>)) 

The model itself doesn’t know or care which platform it’s running on – it just calls a function. That means when Tennessee wrote the Databricks version of that function, our core models didn’t need to change. The state plugged in its implementation, it worked, and the rest of EDU came with it. 

We did the same thing on the Airflow side, defining clean interfaces for both database connections and object storage. Snowflake and Databricks each have their own implementation behind a shared interface and adding an additional platform in the future means writing a new implementation, not rewriting the core logic. If you’re a data engineer working on a platform we haven’t yet supported — Google Cloud Storage, BigQuery, or something else entirely — this is an open invitation. We’ve documented how to add a new storage or database interface, and we welcome contributions from the community.  

The result is a codebase that’s configurable and able to meet the needs of the present and of the future.  

Implementation from Months to Days 

Once the Databricks work was merged, the door opened for more agencies to get started quickly. For agencies with a populated Ed-Fi ODS, you can wire into EDU and have a working data warehouse — in either Snowflake or Databricks — in days, not months or years. Tennessee did it in hours. 

Wondering what that looks like once EDU is up and running? Our Getting Started guide explains exactly what happens when EDU is populated — from pre-built dimensional models to analytics-ready data a team can query from day one. 

Building a custom data warehouse from scratch or purchasing a boutique product tailored to every specification can seem attractive at first. But what often gets overlooked is the dependency it creates—on a single vendor, specialized internal expertise, and the time and resources required to build, maintain, and evolve the solution. A community-maintained, open framework means not waiting on a single vendor to respond to needs.  

What Platform Neutrality Means 

EDU’s compatibility with Databricks is evidence that the architecture we chose does what we said it would do. 

The same patterns that made Databricks support possible — adapter dispatch in dbt, interface-based design in Airflow — mean that a future contributor could add support for another platform, like Postgres, Google BigQuery, and whatever comes next.  

EDU is built on the Ed-Fi Data Standard because we believe standard data models should be shared across states to build on one another's success. We chose dbt and Airflow because open-source tools democratize development and keep agencies from being locked into proprietary systems. And we released EDU under the PolyForm Noncommercial license as a genuine commitment that this software will remain available to education agencies, free from commercial profit.  

One Principle, Many Layers 

Databricks support is one expression of something that runs deeper in how EDU is built. Our commitment to data standards is grounded in Ed-Fi, but extends to current and future integration with standards such as CASE, OneRoster, the Comprehensive Learner Record Standard, and other interoperability frameworks, because we believe that standard code and data models shared across states produce better outcomes than every agency solving the same problem in isolation. Platform flexibility and data standards flexibility are the same principle applied at different layers of the stack. 

The goal has always been the same: reduce the cost of entry, eliminate redundant work, and let agencies focus on what matters — understanding and improving outcomes for students. 

For agencies ready to move forward but without the internal capacity to implement EDU on their own — or who want implementation expertise, managed infrastructure, and access to EA’s full product suite including AI and downstream analytics tools — Stadium is the managed path built on the same open code. 

What Comes Next 

We’re actively exploring how EDU can extend beyond K-12 — into workforce outcomes, post-secondary data, and work-based learning — building on the same principle that standard code shared across states produces better results than every agency starting from scratch. 

We’re also building community-driven, governed standards for how education agencies store and structure the context that AI tools need – so that the AI work happening in education today isn’t locked into any single model or vendor tomorrow. Additionally, we are engaging with the Open Semantic Interchange, a new standard for semantics in AI and BI, especially for this purpose. 

If your state or district has been watching EDU from the sidelines, let this be a signal that there is a community ready to support your work.