Referred to as “one of the most transformative Texas education bills in recent history,” the Texas House Bill 3 that was passed in 2019 established the Teacher Incentive Allotment (TIA) and created a sea change for teacher compensation in the state. TIA funds are used for teacher pay and retention efforts across the state based on district-developed systems for identifying their most effective teachers—especially those working to close opportunity gaps for students with the most need.
Ninety percent of TIA funds must be used for teacher compensation, and funds are distributed according to a formula based on teachers’ “designation” levels, the level of socioeconomic need at a school, and whether the school is rural. Teachers can be designated as Recognized, Exemplary, or Master.
In order to determine the designation of each teacher, districts must develop multiple-measures systems that are reviewed and approved by the Texas Education Agency (TEA) in partnership with Texas Tech University. These systems must include the following components:
Although many Texas districts already have existing observational rating systems in place, developing and using student growth measures is a new endeavor for most. Districts do have flexibility in the way in which they measure student growth; they may choose from the following approaches (each described in more detail in this TEA presentation):
- Pre- and post-tests
- Value-added measures
- Student learning objectives (SLOs)
- Portfolios
Regardless of the approach used, districts’ growth measures all need to be “congruent” with teacher observation measures. This means that growth and observation measures should generally be somewhat correlated, or related, to each other.
It is important to ensure that both observational measures and growth measures are providing a valid and reliable view of teacher practice and student performance. Both as part of TEA requirements for TIA approval and as best practice, districts should monitor congruence on a regular basis so that measures and their underlying processes can be adjusted and refined as needed.
Below is an example of what congruence might look like between student growth measures (on the x-axis) and observation measures (on the y-axis), where the dots are schools within a district. Here, as the average share of students meeting growth targets increases, average teacher observation scores in the school increase. Because they are related, but do not measure exactly the same things, correlations between these two measures in congruent systems is typically between 0.1 and 0.25 (where 0 would mean they are completely unrelated and 1 would mean they are perfectly correlated).
Creating growth measures that are congruent is no small feat. EA has helped districts and states build growth measures for nearly a decade, and it is using that vast experience to offer support to Texas districts as they create their TIA designation systems. As part of this work, we’ve sat down with representatives from several Texas districts and Education Service Centers (which provide support to school districts in their region) to learn more about the particular challenges Texas districts face when building student growth measures as part of their designation systems.
The Four Most Common Challenges when Building Student Growth Measures for Designation Systems
1. Identifying appropriate assessments and approach to calculating growth metrics
Deciding on a student growth measure involves a complex process with multiple steps. First, districts must administer a set of assessments that have properties that lend themselves to growth measure calculation. TEA provides a list of commonly used assessments that meet TIA statutory requirements, but districts may also develop their own assessments in-house, provided they are valid and reliable.
Alongside assessment administration, districts must also choose which of the four types of growth measure calculations they will use and then develop the metrics. Regardless of the growth measure, metric development is difficult and complex for a district to do on its own.
Beyond Texas, an approach districts have taken to tackle this challenge is to partner together in a consortium to run a common growth model using common assessments in order to generate annual growth measures. This approach achieves economies of scale with regards to cost, and it takes the operational burden off of a single district to design its own growth measure independently. EA is currently working on a similar project with districts in New York State using their MAP tests from NWEA. More information on this project can be found here.
2. Knowing how to set meaningful growth targets
Should districts choose to calculate growth measures in-house, they will need to establish student growth targets, or the score (or performance level) that students are expected to achieve. This is an essential step in the calculation of pre-post test measures and SLOs (two of the possible growth measures in Texas designation systems), because a teacher’s growth measure is created by comparing students’ actual performance to their expected performance. Thus, student growth targets are the building blocks of growth measures.
Often, teachers and school leaders are the ones tasked with establishing student growth targets. Depending on the assessment used, educators may find it difficult to establish student growth targets that are data-informed and ambitious yet feasible. In order to reduce burden and ensure both consistency and rigor across classrooms, solutions such as a web-based tool to review student assessment results could be helpful for teachers and school leaders engaged in this work.
For example, an SLO tool could provide information on potential student growth targets based on students’ prior academic history and that of other similar students, by allowing educators to identify threshold scores of various percentiles; for example, this could mean the 50th percentile score among students with similar achievement levels for a moderate target, or the 75th percentile score for a more ambitious target. Such a tool could also evaluate if students met or didn’t meet targets at the end of the year.
The figures below show an example from one such tool that EA created as part of the Rally Analytics Platform. These snapshots are from a teacher-facing dashboard that assists educators in understanding a) where students were predicted to score in the absence of COVID, b) where they actually scored during and after COVID, and c) how each of the teacher’s classrooms are doing as a whole.
3. Data sources and data systems
Once a district has designed and calculated its growth measures, the next step on the road to approval for its TIA system is to determine the congruence between teacher observation and student growth measures. This requires the joining of two datasets that may live in different source systems. Depending on their systems, not all districts are able to easily pull and join observation and growth data. Since time is such a limited commodity among district staff, particularly in the smallest districts, having an efficient and user-friendly way to pull the measures together is an important challenge to overcome for some districts.
One way to mitigate this kind of problem is by using standards-based data collection approaches. An example of this type of approach involves building and implementing a data system that enables interoperability between different data sources, such as through the technology made available by the Ed-Fi Alliance. Although organizations such as the Ed-Fi Alliance can provide information and technical assistance for implementing these systems, this type of approach is a long-term solution and extremely involved for a single district to embark on. In fact, some state education agencies, such as the South Carolina Department of Education, have begun providing statewide support for the implementation of such systems. For those districts without statewide support or where transitioning independently to a fully interoperable data system is not practical, a more feasible short-term approach could be to streamline data collection specific to just teacher observation and student growth data, in order to make them more standards aligned and more easily joined together.
4. Expertise for using measures
In their TEA application, districts are responsible for operationalizing congruence checks—meaning, they must articulate how they will check for congruence in practice. However, not all districts have the expertise to conduct checks on congruence or to easily make meaning of congruence findings; for example, many small districts lack dedicated staff with a strong data background. As a result, these districts may not have staff on hand who know how to conduct congruence analyses, or they may lack the tools to take action on next steps when the results show that the measures are misaligned.
Districts grappling with this challenge could benefit from effective technical assistance that provides tangible resources, like a timeline for congruence checks and inquiry cycles, or a map of the type of data that are best to utilize for congruence checks. TEA has established a list of approved technical assistance providers who can provide support on multiple aspects related to developing and implementing TIA designation systems, including monitoring and addressing congruence.
On The Horizon
To date, four “cohorts” of districts have submitted applications to TEA for approval of their TIA designation systems. Districts interested in joining the fifth cohort, “Cohort E,” must submit their applications to TEA by April 2022. As this and future cohorts dig into the work of developing their systems, it is very likely they will encounter one or more of the challenges described above. But they don’t need to do so alone. As more districts in Texas join the ranks of others across the country who have embarked on developing multiple-measures systems to recognize effective teachers, a body of lessons learned, resources, and partner organizations grows wider to support those whose efforts have just begun.