Laurene L. Christensen, Vitaliy V. Shyyan, Kelly Koenig, and Crystal Fields
December 2016
All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Christensen, L. L., Shyyan, V. V., Koenig, K., & Fields, C. (2016). Variables used by states in the calculation of Annual Measureable Achievement Objectives (AMAOs) for Title III reporting. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
This report is dedicated to Crystal Fields, a tireless advocate for English learners.
Executive Summary
Overview
Method
Results
Summary and Conclusions
Appendix A: Survey Instrument
All states receiving federal funds for Title III of the 2002 Elementary and Secondary Education Act (ESEA) were required to calculate annual measurable achievement objectives (AMAOs). This report highlights variables used by states in the calculations of AMAOs for Title III accountability purposes. The variables were reported by 29 states that responded to a survey on this topic administered by the National Center on Educational Outcomes (NCEO) in collaboration with the English Language Learner State Collaborative on Assessments and Student Standards (ELL SCASS). Survey findings based on the 2014-15 academic year indicate that states take a wide range of approaches in how they calculate each of the AMAOs. Of particular note is the diversity of approaches states take in including or excluding students who may be considered nonparticipants for one reason or another.
Under Title III of the Elementary and Secondary Education Act (ESEA) of 2002, states were required to report annual measurable achievement objectives (AMAOs) related to the language acquisition and academic achievement of students with limited English proficiency (LEP), or English learners (ELs). Specifically, there were three AMAOs that states were required to report:
With the implementation of the flexibility waivers, AMAO 3 changed somewhat to account for the various approaches to accountability taken by states.
The purpose of this report is to present information on the approaches used by states to calculate each of the AMAOs. Although states report their AMAO calculations to the federal government, there is no mechanism of comparison across states. This report provides a snapshot related to the inputs states used for calculating AMAOs for the 2014-15 school year.
In collaboration with the English Language Learner State Collaborative on Assessments and Student Standards (ELL SCASS), the National Center on Educational Outcomes (NCEO) sent out a survey to all 50 states in order to gather information on how states calculate and report each of the AMAOs. The survey was developed by NCEO and revised based on extensive feedback from ELL SCASS members. Information was gathered via SurveyMonkey in the fall of 2015. Appendix A includes the survey instrument. The results of the survey are compiled here.
The results of the AMAO survey are presented here by each question posed in the survey. It is important to note that not all states answered every question, so the n-size varies from question to question.
Question 1 asked for background information related to the state completing the survey. A total of 29 states responded to the survey. States included large and small states, including those with both high and low populations of ELs. The sample represents a cross-section of the United States. Most responses came from Title III Coordinators/Directors (N=11). Other respondents included Data Analysts/Administrators (N=2), or Division Administrators (N=2).
States were asked what English language proficiency assessment (ELPA) is used for AMAO calculations. A total of 20 states reported using the WIDA ACCESS for ELLs for their AMAO Calculations. Four states reported using ELPA21. One state reported using LAS Links Form C. Two states reported using their own assessments to measure English language proficiency (ELP) of their students.
States were asked whether they use different exit criteria for services from the measure(s) used for AMAO 2. If a state answered yes to this item, the state was asked to elaborate its answer. A total of 27 states responded to this question. As shown in Figure 1, most states do not use different exit criteria for services from the measures used for AMAO 2.
Figure 1. Exit Criteria for Services Other Than AMAO 2
Some states reported that there were additional criteria for exiting EL services, such as a higher overall composite score, or proficiency in specific domains such as reading and writing. Three states reported that exit criteria were a local decision.
States were asked whether AMAO calculations relate to each other. If a state answered yes to this item, the state was encouraged to provide more information.
Figure 2. Relationship of AMAO Calculations
As seen in Figure 2, most states reported that one AMAO calculation does not factor into the calculation of another AMAO. Only two of the 26 states responding to this question answered "yes." One of these states noted that "if students meet exit criteria for AMAO 2, they meet growth for AMAO 1 regardless of whether they otherwise meet the growth criteria." The other state that responded affirmatively to this question stated, "students exiting as proficient in a given year are included as making a gain in AMAO 1."
This question asked for the data used to calculate AMAO 3. The response options included the following: (1) percent proficient for reading and math, for all students in the EL subgroup; (2) growth targets set for district or state level; and (3) specific graduation rate targets for all students or specific to ELs. A total of 26 states responded to this question.
Figure 3. Data Used for AMAO 3 Calculations
There is wide variability among states in the data used to calculate AMAO 3 (see Figure 3). A total of 23 states used the percent proficient in reading and math for AMAO 3. Growth targets set for the district or state level were used in 14 states. In 19 states, specific graduation rate targets for all students or specific to ELs were used. Seven states commented that participation rates were also used as part of the calculation for AMAO 3.
States were also asked whether alternate assessments were included in the calculations for AMAO 3 (see Figure 4). Just over half of the responding states (N=16) reported that alternate assessments were used in AMAO 3 calculations. Four of the states that reported that they did not include alternate assessments in their calculations said that they would start including these data in the upcoming year.
Figure 4. Alternate Assessment in AMAO 3 Calculations
States were asked whether they have a policy on how students could be excluded from AMAO 1 (see Figure 5). A total of 26 states responded to this question. Slightly more than half of the states (N=16) reported that they had no polices on how students could be excluded from AMAO 1. Of the 10 states that reported having such a policy, the most common policy reason noted was that two data points were required for calculating growth, so students with only one data point were excluded from calculations.
Figure 5. States with Policies Excluding Students from AMAO 1 Calculations
States were asked whether they had a policy on excluding students from being counted in AMAO 2 (see Figure 6). A total of 26 states responded to this question. Only two states reported having a policy that allowed for students to be excluded from this AMAO calculation. One state reported excluding a student who had a documented medical reason. The other state that allowed for exclusions reported that a student must have an overall score in order to be included in calculations for AMAO 2.
Figure 6. State Policies Excluding Students from AMAO 2 Calculations
This question asked states whether they had policies that allowed for excluding students from calculations for AMAO 3 (see Figure 7). A total of 25 states responded to this question. Slightly more than half of the states answering this question (N=14) reported that they did have a policy allowing for a student to be excluded from AMAO 3 calculations. Eleven states reported not having such a policy. Ten states reported that students who were in their first year of U.S. schooling were eligible to be exempt from AMAO 3 calculations. Two states reported that there was a minimum N-size for AMAO 3 calculations. Two states reported that students could be exempted from AMAO 3 calculations if they had not been enrolled in school for a certain number of days or by a certain point in the school year.
Figure 7. State Policies Excluding Students from AMAO 3 Calculations
States were also asked whether they used confidence intervals in the calculations of AMAOs. A total of 23 states responded to this question. About two-thirds (N=16) of the states that responded to this question reported not using confidence intervals in AMAO calculations (see Figure 8). Seven states reported using confidence intervals in their calculations. Two states reported using a 95% confidence interval for each calculation. One state said that a 99% confidence interval was used if a district or consortium initially had a "not met" status. Another state noted that confidence intervals were used when a Title III sub-grantee had fewer than 30 students taking the English language proficiency assessment.
Figure 8. Use of Confidence Intervals in AMAO Calculations
States were asked whether they used a minimum N-size for AMAO 1 calculations and reporting. Of the 24 states responding to this question, 10 states reported using a minimum N-size for AMAO 1 (see Figure 9). Among the states that used a minimum N-size, there was a wide range of N-sizes given, including 7 (one state), 10 (four states), 20 (one state), 30 (two states), and 34 (one state). One state did not report the minimum N-size used.
Figure 9. Use of a Minimum N for AMAO 1
This question asked states whether they use a minimum N-size for AMAO 2 calculations. A total of 24 states responded to this question. Of those responding, nine states reported using a minimum N-size; 15 states did not use a minimum N-size for AMAO 2 calculations (see Figure 10). States that used a minimum N-size generally reported the same minimum N-size that they did for AMAO 1 calculations. These N-sizes ranged from 7 to 34.
Figure 10. Use of a Minimum N for AMAO 2
States were also asked whether they used a minimum N-size for reporting AMAO 3 calculations. A total of 24 states responded to this question. Of the states that responded, most (N=19) stated that a minimum N-size was used for AMAO 3. Only five states said that it was not used (see Figure 11). Not all states that answered yes to this question provided the minimum N-size used. The minimum N-sizes reported ranged from 5 to 30, with 30 being the most commonly reported number, used in four of the states.
Figure 11. Use of a Minimum N for AMAO 3
States were asked whether districts were required to report a minimum N-size on AMAO 1 in order to receive Title III funding. A total of 26 states responded to this question. Of those responding, 10 reported no minimum N-size was used, and 16 reported having a minimum N-size. Among states reporting this requirement, answers varied widely. Some states listed a specific N-size. These included N-sizes of 10 (reported by 2 states), 20 (1 state), 100 (1 state), and 112 (1 state). Other states reported that the district or consortium must have enough ELs to generate $10,000 worth of funding (reported by 8 states).
Figure 12. Use of a Minimum N for AMAO 1 in Order to Obtain Title III Funds
States were also asked whether there was a minimum N-size for AMAO 2 that districts must meet in order to obtain Title III funds. A total of 24 states responded to this question. Of the states that did, eight states reported requiring a minimum N-size; a total of 16 states reported not having this requirement (see Figure 13). Similar to Question 14, some states reported a specific minimum N-size. In this case, the minimum N-size ranged from 10 (2 states), to 112 (1 state). Similar to Question 14, other states reported that districts must meet a funding threshold of $10,000 in order to receive funding.
Figure 13. Use of a Minimum N for AMAO 2 in Order to Obtain Title III Funds
States were also asked whether there was a minimum N-size required by districts or district consortia on AMAO 3 in order to obtain Title III funding. A total of 24 states responded to this question. Of those that did, 8 states reported a minimum N-size on AMAO 3 in order to receive Title III funding; a total of 16 states did not have this requirement (see Figure 14). Similar to Question 14 and 15, some states reported a specific minimum N-size, including 20 (2 states), 100 (1 state), and 112 (1 state). In other cases, the funding threshold of $10,000 was mentioned as a requirement.
Figure 14. Use of a Minimum N for AMAO 3 in Order to Obtain Title III Funds
States were asked whether cohorts were used in the calculation of AMAO 1. If a state answered yes, it was prompted to share the characteristics that were used. A total of 26 states answered this question. The majority of states (N=22) answered no to this question (see Figure 15). A total of four states reported using cohorts in AMAO 1 calculations. One state noted that number of years in that state's schools was considered. Another state included years spent in program. One additional state reported that sub-grantees must include at least 65% of the test takers with prior scores in AMAO 1 calculations.
Figure 15. Use of Cohorts in AMAO 1 Calculations
States were also asked whether cohorts were used in AMAO 2 calculations, and similarly, if yes, to share the characteristics used in the calculations. A total of 26 states responded to this question. Of the states that responded, most (N=20) did not use cohorts in AMAO 2 calculations (see Figure 16). Of the six states that included cohorts, similar characteristics to those used for AMAO 1 were reported (e.g., time in the state's schools and years in program). Two states reported that cohorts were established based on the time ELs spend in English language programming. One cohort represented fewer than five years of programming. The second cohort included students who had more than five years of programming.
Figure 16. Use of Cohorts in AMAO 2 Calculations
This question asked whether cohorts were used in the calculation of AMAO 3. Similar to Questions 17 and 18, if states answered yes, they were prompted to provide the characteristics used. A total of 26 states answered this question. Of these states, only five states reported using cohorts for AMAO 3 calculations (see Figure 17). The remaining 21 states did not use cohorts for this calculation. Three states provided additional information on the characteristics used. All three states reported using graduation rates.
Figure 17. Use of Cohorts in AMAO 3 Calculations
In this question, states were asked whether performance expectations were adjusted by grade. For example, a state might have different performance expectations for students in younger grades in certain domains. A total of 24 states responded to this question. Of those states, only two states made these adjustments (see Figure 18). In these two states, one state explained that unlike other grades, kindergarten was not "tiered." In the other state using this approach, it was reported that only scores in listening and speaking were used to calculate proficiency in grades K-1. For all other grades, proficiency was calculated with scores in all four domains. The remaining 22 states reported not making these adjustments in performance expectations by grade.
Figure 18. State Adjustments in Performance Expectations by Grade
States were asked whether alternate English language proficiency assessments were used as part of the calculations for AMAO 1 and AMAO 2. A total of 25 states responded to this question. A total of nine states reported using this alternate assessment in calculations; the remaining 16 states did not use alternate assessments in their calculations (see Figure 19). All nine states answering affirmatively reported using the WIDA Alt-ACCESS in their calculations. One state noted that "we have not yet determined expected growth and proficiency scores, so the Alternate scores contribute positively to our participation rate but drag down our accountability scores as they show no growth and cannot meet proficiency as of yet."
Figure 19. Use of Alternate English Language Proficiency Assessments in AMAO 1 and 2 Calculations
States were asked whether they had the authority to override a student's ELPA score designation. A total of 23 states responded to this question. A majority of the states (N=20) reported that they did not have this authority (see Figure 20). Three states reported having this authority. In these cases, they mentioned that this override was used for accountability purposes if a student was caught cheating, in rare cases when there was a classification or reclassification error, or if an EL team agreed to classify a student as an active EL when the student would have otherwise been exited due to the ELPA score.
Figure 20. State Authority to Override a Student's ELPA Score Designation
This question asked states to share how they addressed student mobility in their AMAO calculations. A total of 24 states responded to this open-ended question. Four states answered that they did not address mobility in their calculations. Of the remaining states, a variety of approaches were employed including:
States were asked whether administrative recordkeeping issues (e.g., deadlines for receiving assessment records) had an impact on AMAOs. A total of 25 states responded to this question. Most states (N=18) reported that administrative recordkeeping issues did not impact AMAOs (see Figure 21). A total of 8 states reported that there was some impact on AMAO calculations. Two of these states noted that if materials were not returned by the deadline, those districts' scores were not included in calculations. Two other states reported that AMAO calculations might be delayed due to late or incomplete data.
Figure 21. Impact of Administrative Recordkeeping Issues on AMAOs
This question asked states how AMAO decisions were made for students when a parent refused to have the student tested. A total of 23 states responded to this open-ended question. Six states reported that they did not allow parental refusal of testing. Four states specified how these untested students were included in AMAO calculations. In one state, untested students were not included in AMAO 1 growth calculations, counted as non-proficient for AMAO 2, and counted against participation for AMAO 3. In another state, they were included in AMAO 1 if the student was enrolled during the assessment window. In a third state, the score was counted as a zero. In a fourth state, the student was included as part of the denominator. In two states, non-tested students were counted as non-participating, and the state worked with the districts to make sure the participation target of 95% was reached.
In this question, states were asked how they approached AMAO calculations when the student self-refused testing. A total of 23 states responded to this open-ended question. Four states reported that the student would not be included in AMAO calculations. Of the remaining states, similar approaches to those reported in Question 25 were mentioned: students were recorded as nonparticipants, students were included in the denominator, and students were recorded as receiving a score of zero. In two cases, states reported that this situation had not happened in their state.
This question asked how states made AMAO calculations for students who were not tested, either entirely or partially. A total of 23 states responded to this open-ended question. Two states reported that these students were not included in AMAO calculations. An additional two states reported that the students were not included in AMAO 1 calculations, but were included in other calculations. Two states noted that the student would receive a score of zero. Other states had different approaches, including the following:
This question asked states how AMAO decisions were made when students were excused due to a medical emergency. A total of 23 states responded to this open-ended question. In a total of eight states, when a student was excused due to a medical emergency, the student was not included in AMAO calculations. In two states, there was an approval process for excusing the student from testing. Other states had the same process in place for this group of students as they did for other students not tested: they might be included in some AMAO calculations, included in the denominator, or included as nonparticipants.
This question asked states how AMAO decisions were made for students who were withdrawn from the school or district. A total of 23 states responded to this open-ended question. Five states reported that these students were generally not included in AMAO calculations. In an additional six states, students were not counted if they were withdrawn before the testing window. Three states reported treating withdrawn students in the same manner as other non-tested students. One state noted having no policy on how to treat withdrawn students in AMAO calculations.
States were also asked how they made AMAO decisions with regard to students whose identification was not available on answer documents. Twenty-three states responded to this open-ended question. A total of 12 states reported not including unidentifiable students in their AMAO calculations. Five states reported that their data systems did not allow for unidentifiable students to take an assessment. Three states reported never having had this happen so they were unsure of how this situation would be treated.
This question asked states whether they had any unique approaches to calculating AMAO 1 for Title III that had not been mentioned in the previous questions. A total of 23 states responded to this question. Of the states responding, the majority (N=18) reported that they did not have any additional unique approaches to AMAO 1 calculations (see Figure 22). In the remaining five states that answered yes, four noted that they had implemented growth models or would be implementing them soon. In one state, EL service time was considered in AMAO 1 calculations.
Figure 22. Unique Approaches to AMAO 1 Calculations
In this question, states were asked whether they had any unique approaches to calculating AMAO 2 that had not been mentioned in previous questions. A total of 24 states responded to this question. A total of four states acknowledged having a unique approach (see Figure 23). Among these four states, a wide range of approaches was mentioned:
Figure 23. Unique Approaches to AMAO 2 Calculations
This question asked states whether they had any unique approaches to calculating AMAO 3 that had not been previously mentioned. A total of 24 states responded to this question. Of these states, four reported having a unique approach to calculating AMAO 3 (see Figure 24). These unique approaches included the following:
Using growth in English language arts and mathematics as part of the calculations.
Using a cumulative Progress and Performance Index and extra credit for districts achieving high percentage of ELs achieving their Student Growth Percentile on ACCESS.
Developing a statewide consortium of approximately 90 school districts that had low EL N-sizes; by aggregating these districts, they could be included in AMAO calculations.
Figure 24. Unique Approaches to Calculating AMAO 3
All states receiving federal funds for Title III are required to calculate AMAOs. This report illustrates that states take a wide range of approaches in how they calculate each of the AMAOs. Of particular note is the diversity of approaches states take in including or excluding students who may be considered nonparticipants for one reason or another (e.g., parent refusal, student refusal, or withdrawal from school). As states move toward the implementation of the Every Student Succeeds Act (ESSA), it will be important to continue to revisit this topic, since states are likely to continue to grapple with AMAO calculations going forward.
1. Respondent Contact Information
2. What ELPA does your state use for AMAO calculations? Please explain
3. Does your state use different exit criteria for services than the measure(s) used for AMAO 2? If yes, please explain.
No
Yes
Other (please specify)
4. Does the calculation of one AMAO figure into the calculation of another AMAO?
No
Yes
Other (please specify)
5. What specific data are used to calculate AMAO 3 in your state? Select ALL that apply.
Percent proficient for reading and math - for all students in the ELL subgroup Growth targets set for district or state level
Specific graduation rate targets for all students or specific to ELLs
Other (please specify)
6. Does your state use alternate assessments in calculating AMAO 3? If yes, please describe.
No
Yes
Please describe
7. Does your state have policy on excluding students from being counted in AMAO 1? If yes, please describe.
No
Yes
Please describe
8. Does your state have policy on excluding students from being counted in AMAO 2? If yes, please describe.
No
Yes
Please describe
9. Does your state have policy on excluding students from being counted in AMAO 3? If yes, please describe.
No
Yes
Please describe
10. Does your state use confidence intervals in determining AMAO calculations? If yes, please describe.
No
Yes
Please describe
11. Does your state use a minimum N for Title III for AMAO 1?
No
Yes
Other (please specify)
12. Does your state use a minimum N for Title III for AMAO 2?
No
Yes
Other (please specify)
13. Does your state use a minimum N for Title III for AMAO 3?
No
Yes
Other (please specify)
14. Does your state have a minimum N for ELLs for AMAO 1 in a district to obtain Title III funds? If so, what is it?
No
Yes
What is it?
15. Does your state have a minimum N for ELLs for AMAO 2 in a school or district to obtain Title III funds? If so, what is it? No Yes What is it?
No
Yes
What is it?
16. Does your state have a minimum N for ELLs for AMAO 3 in a district to obtain Title III funds? If so, what is it?
No
Yes
What is it?
17. Does your state use cohorts in calculating AMAO 1? If so, what characteristics do you use?
No
Yes
What characteristics do you use?
18. Does your state use cohorts in calculating AMAO 2? If so, what characteristics do you use?
No
Yes
What characteristics do you use?
19. Does your state use cohorts in calculating AMAO 3? If so, what characteristics do you use?
No
Yes
What characteristics do you use?
20. Does your state use different performance expectations by grade, such as differences in measuring students in younger grades (e.g., Kindergarten) in certain domain areas?
No
Yes
Other (please specify)
21. Does your state include alternate ELPAs in its AMAO 1 and 2 calculations? If yes, please explain.
No
Yes
Please explain
22. Does your state have the authority to override a student's ELPA score designation? If so, when is this allowed?
No
Yes
When is it allowed?
23. How does your state address student mobility in AMAO calculations? How are AMAO calculations affected?
24. In your state, do administrative recordkeeping issues impact AMAOs (e.g., deadline for receiving assessment records)? If yes, how are AMAO calculations affected (e.g., are scores counted)?
No
Yes
How are AMAO calculations affected?
25. How does your state make AMAO decisions for students with regard to parent refusal of testing?
26. How does your state make AMAO decisions for students with regard to student self-refusal of testing?
27. How does your state make AMAO decisions for students with regard to not testing (entire or partial)?
28. How does your state make AMAO decisions for students with regard to excused for medical emergency?
29. How does your state make AMAO decisions for students with regard to withdrawn students?
30. How does your state make AMAO decisions for students with regard to unidentifiable students on answer document, etc., (e.g., are scores counted)? Please specify
31. Does your state have any unique approaches to calculating AMAO 1 for Title III not mentioned in previous questions?
No
Yes
Please specify
32. Does your state have any unique approaches to calculating AMAO 2 for Title III not mentioned in previous questions?
No
Yes
Please specify
33. Does your state have any unique approaches to calculating AMAO 3 for Title III not mentioned in previous questions?
No
Yes
Please specify
34. Please provide links to any state resources related to the questions in this survey.
35. Is there anything else you would like to add?