Nearing the Target in Disaggregated Subgroup Reporting to the Public on 2004-2005 Assessment Results

Technical Report 46

Gretchen R. VanGetson • Martha L. Thurlow

All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

VanGetson, G. R., & Thurlow, M. L. (2007). Nearing the target in disaggregated subgroup reporting to the public on 2004-2005 assessment results (Technical Report 46). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Tech46/

Table of Contents

Executive Summary
Overview
Method
Results
Assessment Participation in 2004-2005
Assessment Performance in 2004-2005
Assessment Performance: Trends
Other Information Collected for 2004-2005
Discussion
References
Appendices

This report marks the ninth analysis conducted by the National Center on Educational Outcomes (NCEO) of the public reporting of state assessment results for students with disabilities. This is the fourth analysis that NCEO has conducted since the passage of the federal No Child Left Behind Act of 2001 (NCLB). Overall, all 50 states and 5 unique states reported some disaggregated assessment results for students with disabilities for the 2004-2005 school year. Thirty-six states reported participation and performance data for all their general assessments, 12 reported participation and performance data for some of their general assessments, and 2 reported only performance data. When considering only assessments that were part of states’ NCLB accountability systems, more states reported all assessment data publicly: 44 states reported participation and performance data for all these assessments, 4 reported participation and performance data for some of these assessments, and 2 reported only performance data for all of these assessments. For alternate assessments, 47 states reported some disaggregated assessment results for students with disabilities. Forty-one states reported both participation and performance data for their alternate assessments, 2 states reported these data for some of their alternate assessments, 3 states reported only participation data, and 1 state reported only performance data for their alternate assessments. Three states did not report participation or performance data about their alternate assessment.

When reporting general assessment participation data for students with disabilities, 43 states reported the number of students tested, the most common way of reporting participation information, while 35 states reported either the percent of students tested or not tested for at least one of their assessments. For states’ alternate assessments, the most common way of reporting participation information was to give the number of students tested; this was the approach of 41 states. Thirty-eight states gave a percentage (either a percent tested or percent not tested).

In terms of reporting general assessment performance data, all 50 states reported on the percent of students with disabilities who were proficient, and 40 states reported the number of students with disabilities in each achievement level. For states’ alternate assessments, 42 states reported on the percent of students with disabilities who were proficient, while 27 states reported the number of students with disabilities in each achievement level. When we examined student performance on general assessments, we found that sizeable and variable gaps existed between students with disabilities and general education students. Gaps in the percentage of participating students reported as proficient tended to be larger at higher grades; however, meaningful comparison of these gaps across grade levels is limited by differences across grade levels in both the composition of the compared subgroups (Bielinski & Ysseldyke, 2000) and by differences in the assessments themselves at different grades. This difference by grade level in the gap in rate of proficiency was not present in the alternate assessment performance data. Percentages of students with disabilities achieving proficiency on the alternate assessment were typically higher than on the general assessment. In examining seven years of data from states that had publicly reported information using the same assessment (11 states), we found that, in general, there was a trend in both reading and math toward higher rates of proficiency among participating students with disabilities.

Overall, this report reinforces what was found in 2003-2004. States continue to improve their public reporting practices, especially for alternate assessments, but it is gradual, and there are still improvements to be made. This report discusses the results of the study and provides recommendations for how states can continue to improve their public reporting practices.

Overview

The end of the twentieth century saw education shift toward holding all students to high standards, and holding all schools and school districts accountable for students’ progress toward those standards (Goertz & Duffy, 2003). For example, Title I of the Improving America’s Schools Act (IASA) of 1994 required states to create assessment systems aligned with high standards to measure student progress at least once in elementary, middle, and high school in both reading and mathematics (Goertz & Duffy, 2003). Also in 1994, the Elementary and Secondary Education Act (ESEA) required that students with disabilities be included in standards-based assessment systems, answering a call from researchers that stressed the importance of including all students (McGrew, Thurlow, Shriner, & Spiegel, 1992; Zlatos, 1994).

It was not until 2001, when ESEA was reauthorized as the No Child Left Behind Act (NCLB), that standards-based assessment and accountability solidified its presence in the educational landscape. Under NCLB, by the 2005-2006 school year, any school accepting federal dollars must demonstrate adequate yearly progress in the number of students meeting proficiency standards in the areas of reading and mathematics in grades 3 through 8 and once between grades 10 and 12 (Peterson & Young, 2004). The ultimate goal is to have all students achieve proficiency by 2013-2014 (Goertz & Duffy, 2003). Further, these results must be disaggregated by student group (i.e., gender, disability, limited English proficient, ethnicity, economic status, migrant status); included in the state’s accountability system; and publicly reported in a clear, timely, and useful manner (Peterson & Young, 2004). “Public reporting is the most basic form of accountability” (Goertz & Duffy, 2003, p.6) in that it allows for public knowledge of and school responsibility for student achievement. Public reporting of state assessment participation and performance information for students with disabilities has been tracked by the National Center on Educational Outcomes (NCEO) since 1997.

Not surprisingly, since the passage of NCLB, there has been an increase in the number of states that publicly reported participation and performance data for all of their assessments; there was a jump following implementation from 28 in 2000-2001 to 35 in 2001-2002. This reporting hit a plateau in 2002-2003 and 2003-2004 (36 and 35 states reporting, respectively) (Klein, Wiley, & Thurlow, 2006; Thurlow & Wiley, 2004; Thurlow, Wiley, & Bielinski, 2003; Wiley, Thurlow, & Klein, 2005). The number of states that reported these data for all of their alternate assessments has also shown considerable improvement. For example, while 22 states reported this information in 2001-2002, 33 states reported it in 2003-2004 (Klein et al., 2006; Thurlow & Wiley, 2004).

The 2004-2005 school year was the third year that states were required to report on the performance of students with disabilities on standards-based assessments, and the last year before states were required to test in all grades 3 through 8, and once between grades 10 and 12. This report marks the ninth in a line of NCEO reports that document state public reporting practices, and pays particular attention to present and upcoming federal requirements in public reporting. Additional analyses were conducted as well, including an examination of accommodations reporting, a look at disaggregated student group reporting, and a Web site analysis to determine the accessibility of states’ public reporting of participation and performance data.

Method

We began our search for information by reviewing every state’s Department of Education Web site as well as the Web sites for the 11 unique states (i.e., American Samoa, Bureau of Indian Affairs, Commonwealth of Northern Marianna Islands, Department of Defense Education Activity, District of Columbia, Federated States of Micronesia, Guam, Palau, Puerto Rico, Republic of the Marshall Islands, Virgin Islands). We began collecting data in October 2005 and collected information for the 2004-2005 school year. We recorded the names and other information about the assessments that were administered, documented whether participation and performance information were reported for students with disabilities, and noted any other subgroups that were included in disaggregated reporting. We also examined the way in which participation and performance were reported, whether participation and performance information were reported for students who took the test with accommodations, and how many “clicks” it took to arrive at disaggregated assessment results from the Department of Education Web site homepage. In our initial review, a little over half of the states had already posted their 2004-2005 assessment data online in a way that made the data easy to locate and understand.

On February 10, 2006 we mailed a letter to each state director of assessment (see Appendix A) outlining our findings from the state’s Web site. We asked directors to review our findings, correct any misinformation, and provide the public document or Web page in which the correct information was available. We asked that they send us these changes by March 3, 2006. We received replies via fax, e-mail, letters, or phone calls. Many states directed us to a Web page that we had not found in our search. While a few sent paper copies of information, we were able to find this information on state Web sites as well. A few states gave us dates by which they expected their disaggregated assessment results to be posted. Overall, we received responses from 22 directors of assessment.

To ensure that our findings were as accurate as possible, we followed up these efforts with a letter to each state’s director of special education (see Appendix B). These letters were mailed on March 24, 2006. The letters asked the directors to review our findings and make any changes by April 14, 2006. For the 17 states from which we had already received a response from the director of assessment, we noted that in the letter by stating that “these tables have been verified by your state’s Assessment Director; please notify me if you have anything to add.” For states from which we did not hear from the director of assessment, we sent the same letter to the director of special education as we had sent to the director of assessment. Of the 50 states and 11 unique states to which we sent letters, 32 responded with either corrections or to verify that the information was correct.

Finally, there were still 10 states and 7 unique states for which we had heard back from neither the director of assessment nor the director of special education. For 9 of these states we found information on students with disabilities for all of their general and alternate assessments. For the remaining state, we reported all of the information we could find on its Web site. The unique states were not included in the full analysis. Also, a few states directed us to their Part B State Performance Plan (SPP) for information. The 2004 reauthorization of the Individuals with Disabilities Education Act (IDEA; PL 108-446) required states to create a long-term special education performance plan and:

Make the State’s performance plan available through public means, including by posting on the website of the State educational agency, distribution to the media, and distribution through public agencies. [Section 616©(ii)(I)]

Indicator number three of Part B includes the participation and performance of children with disabilities on statewide assessments, providing a form of reporting of these data. In this report, data from SPPs are provided only for states that specifically asked us to review their SPP.

Results

Characteristics of State Assessment Systems

Appendix C lists all the 2004-2005 state mandated general assessments that we identified for the 50 states and the 11 unique states. This list includes the state, the name of the test, the grades and content areas tested, whether the state had publicly available disaggregated participation and performance data for students with disabilities, and whether the assessment results were used for state accountability purposes. For the 50 states, we identified 97 different statewide assessments and 107 assessments total. (Ten are repeat assessments – three Terra Novas and seven ITBSs.) Thirty-three states had more than one general assessment. For the 11 unique states, six different assessments and eight total assessment systems were found; only one of these unique states gave more than one general assessment.

Because few unique states publicly reported complete assessment data, the following results include only information from the 50 regular states. Figure 1 breaks down the 107 testing systems by type: norm-referenced tests (NRT), criterion-referenced tests (CRT), exit tests used as a gate for graduation or earning a particular type of diploma (EXIT), and hybrid tests that combined standardized NRTs with additional state-developed test items (NRT/CRT). Exit exams were included in a separate category only if the state had a distinct test designed specifically for high school completion requirements. However, it should be noted that some states’ CRTs or NRTs also include an EXIT component.

Figure 1. Types of General Assessments

Criterion-referenced tests (CRTs) comprised 63% of all the assessments that states administered in 2004-2005. In fact, only seven states (Arizona, Delaware, Florida, Iowa, Missouri, North Dakota, and South Dakota) did not administer a separate CRT, though six of those states administered a NRT/CRT hybrid test (only Iowa administered solely an NRT). Norm-referenced tests comprised 17% of the assessments, exit exams comprised 13%, and 7% were a NRT/CRT hybrid. These numbers are similar to the 2003-2004 assessment pattern in which 61% of the assessments were CRTs, 16% were NRTs, 16% were exit exams, and 7% were hybrids (Klein et al., 2006).

States That Reported Disaggregated Regular Assessment Data for Students with Disabilities

Figure 2 summarizes the different ways in which regular assessment data were reported in all 50 states. Overall, 72% percent of states reported disaggregated participation and performance data for students with disabilities for all their assessments, 4% percent reported performance data for all assessments (but not participation data), and 24% percent reported participation and performance data for some, but not all, of their assessments. Figure 3 indicates how each of the 50 states reported their disaggregated participation and performance data for students with disabilities.

Figure 2. States that Disaggregated Assessment Results for Students with Disabilities

Figure 3. States that Reported 2004-2005 Disaggregated Regular Assessment Results for Students with Disabilities

Figure 4 shows the number of states that reported participation and performance data for the assessments that were included in their statewide accountability systems. In many states, only a subset of assessments was part of their NCLB accountability system. When we examined just the NCLB assessments, we found that 44 states reported participation and performance data for students with disabilities on all of these assessments. The states that reported disaggregated data for their regular accountability assessments did so regardless of whether they had just one assessment or multiple assessments (i.e., 21 of the 44 had more than one assessment included in their accountability system), and regardless of whether they tested in just a few grades or in as many as 10 grades. Of the four states that reported participation and performance information for some of their accountability assessments, Mississippi, Montana, and Utah were missing participation data for at least one test.

Figure 4. States that Reported 2004-2005 Disaggregated Results for Students with Disabilities in their NCLB Accountability Systems

Unique States That Reported Disaggregated Assessment Data for Students with Disabilities

This report is only the second to include unique states in the analysis of publicly reported data for students with disabilities. Because many of the unique states did not have information that was found to be publicly reported, we will only briefly mention them here. Table 1 contains a summary of the unique states and whether they reported participation or performance data for students with disabilities. More detailed information is included in Appendix C for general assessments and Appendix D for alternate assessments. Of the 11 unique states, 5 states publicly reported disaggregated data on the participation and performance of students with disabilities on statewide assessments: American Samoa, Bureau of Indian Affairs, Commonwealth of the Northern Mariana Islands, District of Columbia, and Guam. For all of these unique states, with the exception of Guam, these data were gleaned from the SPPs.

Almost all of the unique states administered an NRT, such as the Stanford Achievement Test, 9th Edition (District of Columbia) or 10th Edition (American Samoa, the Commonwealth of the Northern Mariana Islands, Guam). The only unique state to administer a CRT was the Commonwealth of Northern Mariana Islands. In looking at the reporting practices of the five unique states that reported participation and performance data for students with disabilities, all reported the number and percent of students with disabilities who participated in statewide assessments. American Samoa, the Commonwealth of Northern Mariana Islands, and Guam all reported both the number and percentage of students with disabilities who were proficient, while the Bureau of Indian Affairs and the District of Columbia reported only a percent. Further, American Samoa, the Commonwealth of Northern Mariana Islands, the District of Columbia, and Guam all administered an alternate assessment for students with disabilities covering the same content areas and grade levels as the regular assessment.

Table 1. Unique States that Reported Disaggregated Participation and Performance Data for Students with Disabilities

States That Reported Disaggregated Alternate Assessment Data for Students with Disabilities

Appendix D lists all the 2004-2005 state mandated alternate assessments that we identified. A total of 47 states reported some data for students with disabilities’ participation or performance on an alternate assessment. There were a total of 59 different alternate assessments administered in 2004-2005; six states administered more than one alternate assessment. As shown in Figure 5, results from our Web searches and mailings revealed that 41 regular states publicly reported both participation and performance data at the state-level for their alternate assessment. An additional three states reported participation only, one state reported only performance data, and two states reported some participation or performance data. Three states (i.e., 6% of all states) did not report any type of information about their alternate assessment. However, 80% of states did report both participation and performance for their alternate assessment, which is an increase over 66% in the 2003-2004 school year. Figure 6 illustrates how each state reported alternate assessment participation and performance data.

Figure 5. States That Disaggregated Alternate Assessment Results for Students with Disabilities

Figure 6. States That Reported 2004-2005 Disaggregated Alternate Assessment Results for Students with Disabilities

Assessment Participation in 2004-2005

General Assessment Disaggregated Participation Results for Students with Disabilities

Among the states identified as providing participation data for students with disabilities, the way in which this information was reported varied (see Appendix E). Figure 7 illustrates the number of assessments with disaggregated participation data and how those participation data were reported. Information is presented in terms of the number of assessments for which participation data were available, not in terms of the number of states. For example, in Alabama there are four assessments and each is counted separately. We used this approach because not all states report participation in the same way across assessments. For instance, one state might report only a count of students tested for one assessment, but for another assessment it might report a count tested, a percent tested, and a percent not tested.

Figure 7. Participation Reporting Approaches for General Assessments

Thirty-seven states (59 assessments total) reported either the percent of students tested or the percent not tested for at least one of their assessments, which is a slight increase from the thirty-four that reported rates in 2003-2004. For 50 of those assessments, the percent of students tested was given, and for 30 assessments the percent of students not tested was given. For 21 assessments, both percent tested and percent not tested were provided. Forty-three states (73 assessments) provided the number of students tested, making this the most frequent way of reporting participation data. Sixteen states (21 assessments) provided the number of students not tested. The number or percent of students who were exempt or excluded from assessments was given in seven states (9 assessments) and the number or percent of students absent was given in 12 states (18 assessments).

Figure 8 illustrates the participation rates reported in those states for which this information was clearly reported. It is important that states report the percentage of students tested, in addition to just a count, because this presents a more accurate picture of how many students are participating. To summarize participation rate information, we selected one grade, 8th grade, and one subject, math, to portray in Figure 8. This grade and subject area were chosen to remain consistent with past years’ reports. In one state (Kansas) the math assessment was administered in 7th grade; that participation rate was used here. However, not all states provided data broken down in this way. Some states provided math assessment participation data but multiple grade levels were aggregated; or a rate was provided but it was a percent of all students tested who were students with disabilities, rather than a percent of all students with disabilities who were tested. Some states that otherwise reported clear participation rates for students with disabilities did not administer a middle school level math test. States which for any of these reasons did not report clear participation rates for the 8th grade math test are not included in Figure 8. During the 2004-2005 academic year, participation rates ranged from 83% to 100%; 10 of the 20 states had participation rates of 95% or higher.

Figure 8. Percentages of Students with Disabilities Participating in Middle School Regular Math Assessments in Those States with Clear Reporting of Participation Rates

Alternate Assessment Disaggregated Participation Results for Students with Disabilities

Figure 9 illustrates how states reported participation for their alternate assessments. Many more states provided participation information this year compared to the previous testing year, 2003-2004, in which only 35 states provided participation data. In 2004-2005, 46 states provided participation information for some or all of their alternate assessments, for a total of 53 assessments. Appendix F outlines in more detail the methods used in reporting this information.

Figure 9. Participation Reporting Approaches for Alternate Assessments

Similar to reporting for the regular assessment, the most common way of reporting participation information for the alternate assessment was to give the number of students tested. This was done by 41 states on their 46 alternate assessments and the number of students not tested was reported by 15 states on their 16 alternate assessments. Thirty-eight states gave a percentage, which was either the percent of students tested (reported for 39 alternate assessments), not tested (reported for 19 alternate assessments), or both. Five states provided the number or percent of students who were exempt or excluded on their five alternate assessments, and seven states provided either the number or percent of students who were absent for their seven alternate assessments.

Thirty-eight states provided a percentage of either students tested or not tested. However, not all of these states reported the data clearly for each grade and content area, either aggregating across grade or content, or presenting the percent of all students tested who were students taking the alternate assessment. Figure 10 displays the participation rates for students with disabilities for the 8th grade math test in those states that provided clear participation rates The nine states shown provided information on the percent of students with disabilities who participated in the alternate assessment out of all students with disabilities. (Note: Washington’s data were from a 7th grade assessment).

Figure 10. Percentages of Students with Disabilities Assessed with the Alternate Assessment out of the Total Number of Students with Disabilities

Assessment Performance in 2004-2005

Regular Assessment Performance Results

As with participation data, the way in which states provided performance data for students with disabilities varied (see Appendix G). Figure 11 illustrates the number of assessments with disaggregated performance data and how those performance data were reported. Information is again presented in terms of the number of assessments for which performance data were available, not in terms of the number of states.

Figure 11. Performance Reporting Approaches for Regular Assessments

All 50 states (88 assessments total) reported a rate of proficiency, such as percent of students with disabilities who are proficient, which was also the most common performance reporting method. Reporting the percent of students with disabilities in each achievement level was the next most common method, used by 40 states (70 assessments). The number proficient, used by 20 states (31 assessments), and number not proficient, used by 18 states (25 assessments),were much less widely used as a reporting method. The NRT reporting methods, representing percentile ranks rather than proficiency status, were the least frequently used, probably because only 7% of the state assessments were NRTs.

For all states that clearly disaggregated performance data for students with disabilities, we examined both the performance of general education students and students with disabilities. When examining performance across states, it is important to remember that state assessments are different, in terms of both content and proficiency levels. The assessments may emphasize different standards and are likely to differ in difficulty. In addition, there is variability across states in the percentage of students with disabilities whose scores are reported. Thus, it is not appropriate to compare performance across states. It is possible, however, to examine the differences in percent of students with and without disabilities achieving proficiency within each state, although it should be noted that this gap is also affected by variability between the participation rates of the two groups. Clearly, comparisons of the gaps across states are also inappropriate.

Performance results are reported for both reading and math assessments because these content domains are the ones assessed by most states and are the first content areas required by NCLB to be assessed, reported, and included in accountability. We included English language arts assessments as reading if the state did not have a specific reading assessment. All of the assessments were CRTs except in Iowa, which used an NRT. For 2004-2005, we do not report performance on exit exams because the distinct exit exams do not include all of the exams used to hold students accountable for graduation requirements. Many of the other graduation requirement exams are also used for NCLB accountability requirements, and we have reported results along with the regular statewide exams used for accountability.

We separated grade levels into three categories: elementary (3-5), middle (6-8), and high school (9-12). For our summary, we chose to present only one grade for each level. When available, 4th grade was used to represent the elementary level, 8th grade to represent the middle school level, and 10th grade to represent the high school level. These grades were chosen because they are the grades at which the most states test students. If data from those grades were not available, the next grades used were 5, 7, and 11. The 3rd grade assessment was used for Missouri’s elementary level reading performance data. Additionally, some of the high school assessments did not specify grade level; these results were included as 10th grade. (See Appendix K for the actual grade used.)

Reading Performance. Figures 12-14 present the reading performance of students. The performance of students with disabilities in reading is generally much lower than the performance of general education students. At the elementary level, New York had the largest difference in percent proficient (49 percentage points) between general education students and students with disabilities. At the middle school level, Alabama had the largest difference in rates of proficiency (58 percentage points). At the high school level, the largest difference in rate (45 percentage points) was shown by Alabama. The states with the smallest difference in percent of students with and without disabilities achieving proficiency were Texas at the elementary level (11 percentage points), Nebraska and Texas at the middle school level (23 percentage points), and Pennsylvania at the high school level (16 percentage points). In general, the reported gaps in student proficiency rates are variable with few evident patterns. One evident pattern is that the gaps increase with grade level. Such a trend must be interpreted with caution, given differences in composition of the two compared groups across increasing grade level (Bielinski & Ysseldyke, 2000). Additionally, there are observably lower proficiency rates in reading at the middle school level than at elementary; however, the assessments at different grade levels may differ in terms of both content and proficiency levels, making comparisons of proficiency rate across grades inappropriate.

Figure 12. Elementary School Reading Performance on the Regular Assessment

Figure 13. Middle School Reading Performance on the Regular Assessment

Figure 14. High School Reading Performance on the Regular Assessment

Mathematics Performance. Performance of general education students and students with disabilities on states’ 2004-2005 mathematics assessments is shown in Figures 15-17. In general, the gaps in proficiency rates between students with disabilities and general education students on math assessments were quite similar to the gaps found for reading assessments; they vary considerably from state to state. In elementary grades, the largest difference in the reported percent of students achieving proficiency was 45 percentage points in Alabama. In middle school, the largest difference was 51 percentage points, shared by Alabama and Wisconsin; in high school it was 60 percentage points in Alabama. The states with the smallest differences in percentage of students with and without disabilities reported as proficient were Texas at the elementary level (10 percentage points), Missouri at the middle school level (12 percentage points), and Missouri at the high school level (17 percentage points). As on the reading assessments, the percent of students both with and without disabilities who achieve proficiency is observably lower at middle school than an elementary; interpretation of this trend is again limited by differences in the assessment used at different grade levels.

Figure 15. Elementary School Mathematics Performance on the Regular Assessment

Figure 16. Middle School Mathematics Performance on the Regular Assessment

Figure 17: High School Mathematics Performance on the Regular Assessment

Alternate Assessment Performance Results

Figure 18 illustrates the number of alternate assessments with disaggregated performance data and how those performance data were reported. Information is again presented in terms of the number of assessments for which performance data were available, not in terms of the number of states. Appendix H provides details about the assessments and reporting for each of the alternate assessments.

Figure 18. Performance Reporting Approaches for Alternate Assessments (Number of Tests = 59)

Forty-two states (48 assessments) reported a rate of proficiency—such as percent of students with disabilities who are proficient—which was also the most common performance reporting method. Reporting the percent of students with disabilities in each achievement level was the next most common method for 27 states (33 assessments). The number proficient (20 assessments) and number not proficient (17 assessments) were much less widely used as a reporting method. There were no NRT alternate assessments.

We examined alternate assessment performance data for all states that clearly disaggregated (i.e., by grade level and content area) these data for students with disabilities. Figures representing reading and math performance on the alternate assessment are included in Appendix I. On average, students with disabilities achieved higher rates of proficiency on the alternate assessment than on the general assessment. On the alternate assessment, there were no observable general differences in rates of proficiency across grade levels.

Assessment Performance: Trends

In an earlier analysis (Thurlow et al., 2003), the performance of students with disabilities for states that had at least three years of publicly reported data was examined. In 2000-2001, 13 states had publicly reported information on their statewide assessments for the past three years (California, Delaware, Indiana, Kentucky, Louisiana, Maryland, Minnesota, Missouri, New Jersey, New York, Texas, Utah, and Washington) for both math and reading. Colorado reported three years of information only for the reading assessment and Kansas reported information only for the math assessment. For the current period ending with 2004-2005, 11 states have seven years of publicly reported information on their statewide assessments (California, Colorado, Delaware, Indiana, Kansas, Kentucky, Louisiana, Missouri, New Jersey, New York, and Washington). For Colorado, full data for this period is available for the reading assessment only, and for Kansas for the math assessment only. States included in the trend analysis for 2000-2001 but excluded from the current analysis were the following: Texas and Maryland, which have since changed their assessments and cannot be included in the analysis; Minnesota, which did not report performance information in 2002-2003 (Wiley et al., 2005); and Utah, which aggregated its performance information across all grades in 2003-2004 (Klein et al., 2006).

Reading Tests. Figures 19-21 show the percentage of participating students with disabilities who achieved proficiency on statewide reading assessments at the elementary, middle, and high school levels over the past seven years. In general, the rate of proficiency for participating elementary school students with disabilities continues to increase in reading; 80% of the states included in this analysis have shown such an increase compared to the 2003-2004 school year and most of the states show an overall trend toward greater rates of proficiency for participating students. A few states have spikes or drops in the data across time (California, Louisiana), which may reflect altering assessments for NCLB. At the middle school level, all of the states reported a higher percentage of participating students achieving proficiency than in 2003-2004. While there are evident spikes in 2001-2002 for most states, and other years in California and Louisiana, several states show a trend across these years of increasing rates of proficiency among participating middle school students with disabilities. At the high school level, each of the three states with complete data demonstrate a trend across years toward higher rates of proficiency among those participating.

Figure 19. Seven-Year Trends of the Percentage of Elementary Students with Disabilities who Achieved Proficiency on Statewide Reading Exams

Figure 20. Seven-Year Trends of the Percentage of Middle School Students with Disabilities who Achieved Proficiency on Statewide Reading Exams

Figure 21. Seven-Year Trends of the Percentage of High School Students with Disabilities who Achieved Proficiency on Statewide Reading Exams

Math Tests. Figures 22-24 show the percentage of participating students with disabilities who achieved proficiency on statewide assessments at the elementary, middle, and high school levels for math over the past seven years. The math trends are similar to the trends noticed in reading performance. In general, it appears that rates of proficiency for participating elementary school students with disabilities continue to increase in reading: 90% of the states included in this analysis report higher rates of proficiency than in 2003-2004, and most of the states show a trend across these years of higher rates of proficiency for participating students. Again, data from some states show spikes or drops (California, Delaware, and Louisiana) that do not support a trend. At the middle school level, 70% of the states had a higher percentage of participating students achieving proficiency than in 2003-2004, and most states show an overall trend across these years of increasing rates of proficiency among participating students with disabilities. At the high school level, all four states with complete data show an overall trend across years of increasing rates of proficiency among participants.

Figure 22. Seven-Year Trends of the Percentage of Elementary Students with Disabilities who Achieved Proficiency on Statewide Mathematics Exams

Figure 23. Seven-Year Trends of the Percentage of Middle School Students with Disabilities who Achieved Proficiency on Statewide Mathematics Exams

Figure 24. Seven-Year Trends of the Percentage of High School Students with Disabilities who Achieved Proficiency on Statewide Mathematics Exams

Other Information Collected for 2004-2005

In our analysis of state reporting for 2004-2005, we looked at additional characteristics of states’ publicly reported information. Specifically, we looked at information available on accommodations used and how states publicly reported disaggregated student group assessment data. We also conducted a “click” analysis to determine the ease of access to Web-based reporting.

Accommodations

Sixteen states provided state-level information about students who took the general assessment with an accommodation. The same number of states provided accommodations information in 2003-2004. In some cases, states reported on standard accommodations (those considered appropriate and not ones that change the constructs measured by the assessment). In other cases they reported on nonstandard accommodations (which generally were considered to change the constructs measured—and might be referred to as “non-allowed” accommodations or “modifications”—although IEP teams could select them). Last, some states reported on both or did not specify which.

Table 2 summarizes the information the 16 states provided. Appendix J contains additional information about the data provided by these states, with details about the participation and performance of students. Five states reported student participation and performance by specific accommodation (e.g., directions read orally, braille, extended time), and three states indicated that this information was available in their SPP.

Table 2. States that Reported State-Level Information about Accommodations

¹ Report by specific accommodation (e.g., Braille version, Scribe, Assistive communication device, etc.).
² From State Performance Plan.
³ Numbers are provided, but without a denominator.

Assessment Data Disaggregated By Student Group

In addition to the six required reporting categories under NCLB—ethnicity, gender, limited English proficiency (LEP), migrant, poverty/low-income, and special education—some states provided assessment data about other disaggregated student groups. Figure 25 displays the number of states that provided participation or performance data for each disaggregated student group. All 50 states reported on four of the NCLB categories: ethnicity, LEP, poverty, and special education. One state did not report by gender, and six states did not report by migrant status. The most common other disaggregated student groups on which states reported assessment results were accommodations (16 states), Title I (13 states), and gifted and talented (12 states).

Figure 25. Number of States that Provided Disaggregated Assessment Data for Each Student Group

“Click” Analysis of Web-based Reporting

As we analyzed the participation and performance reporting on states’ Department of Education Web sites, it became evident that some states’ data were easier to find than others. Because the Web is used to provide publicly accessible assessment data in most states, it is crucial that these data be clear and easy to access; we therefore examined the ease-of-access to these data. It is important to note, however, that because Web sites are frequently updated, it is possible that some of our findings no longer hold true. It took an average of 3.4 mouse clicks to navigate from the states’ Department of Education homepage to actual data on students with disabilities’ participation and performance on state assessments. Overall, we arrived at disaggregated assessment data in three clicks or less for 41 states. Figure 26 shows the numbers of states in each “click” category.

Figure 26. Number of States in Each “Click” Category

Discussion

Overall, all 50 states reported some disaggregated assessment results for students with disabilities, which was an increase from 48 in 2003-2004. Thirty-six states reported participation and performance data for all their general assessments, 12 reported participation and performance data for some of their general assessments, and 2 reported only performance data. When considering only assessments that were part of states’ NCLB accountability systems, more states reported all assessment data publicly: 44 states reported participation and performance data for all these assessments, 4 reported participation and performance data for some of these assessments, and 2 reported only performance data for all of these assessments. There were no states that failed to report some form of disaggregated assessment data for students with disabilities, which was an improvement over two states that did not report these data in 2003-2004. For alternate assessments, 47 states reported some disaggregated assessment results for students with disabilities, a large jump from 36 states in 2003-2004. Forty-one states reported both participation and performance data for their alternate assessments, 2 states reported these data for some of their alternate assessments, 3 states reported only participation data, and 1 state reported only performance data for their alternate assessments. Three states did not report participation or performance data about their alternate assessment, much fewer than the 14 states that did not do so in 2003-2004.

When reporting general assessment participation data for students with disabilities, 43 states reported the number of students tested, the most common way of reporting participation information, while 35 states reported either the percent of students tested or not tested for at least one of their assessments. These numbers did not differ significantly from 2003-2004. However, there was increase in alternate assessment participation data reporting in 2004-2005. The most common way of reporting participation information was to give the number of students tested; this was the approach of 41 states compared to 32 states in 2003-2004. Thirty-eight states gave a percentage (either a percent tested or percent not tested) compared to 20 in 2003-2004.

In terms of reporting regular assessment performance data, all 50 states reported on the percent of students with disabilities who were proficient, and 40 states reported the number of students with disabilities in each achievement level. For states’ alternate assessments, 42 states reported on the percent of students with disabilities who were proficient, while 27 states reported the number of students with disabilities in each achievement level. When we examined student performance on regular assessments, we found that sizeable and variable gaps existed between students with disabilities and general education students. Gaps in percentage of participating students reported as proficient tended to be larger at higher grades; however, meaningful comparison of these gaps across grade levels is limited by differences across grade levels in both the composition of the compared subgroups (Bielinski & Ysseldyke, 2000) and by differences in the assessments themselves at different grades. This difference by grade level in the gap in rate of proficiency was not present in the alternate assessment performance data. Percentages of students with disabilities achieving proficiency on the alternate assessment were typically higher than on the general assessment. In examining seven years of data from states that had publicly reported information using the same assessment (11 states), we found that, in general, there was a trend in both reading and math towards higher rates of proficiency among participating students with disabilities.

Recommendations for Reporting

With the push to provide assessment results in a clear, accessible, and timely manner, Web-based reporting has clearly become the primary vehicle for sharing data with the public. Based on our analyses of publicly reported assessment data, especially disaggregated results for students with disabilities, we make the following recommendations:

1. Report results clearly for each test, subject area, and grade level. One of the most common issues we encountered was locating disaggregated assessment data which was not disaggregated to a level where it was useful. This was especially true for states’ alternate assessments. For example, one state provided disaggregated assessment results for students with disabilities for reading and math, but the results were collapsed across all grade levels. We could not use these data in our analyses, and we assume that these data were of little use to educators, parents, and other stakeholders as well. For this report, we categorized the assessment data as disaggregated if the results for students with disabilities were separated out in any manner, but in future years the results must be clearly disaggregated not only by the student group “students with disabilities” but also by test, subject, and grade level to be considered fully disaggregated.

2. Report the percentage of students tested in each student group. Reporting a percentage provides a more accurate picture of participation in the state assessment system than simply reporting on the number of students tested. However, for a group such as students with disabilities with a relatively small number of students, using total grade level enrollment as the denominator provides a very small and less useful percentage. States improve the utility of their data when they provide a percentage with the total subgroup grade level enrollment as the denominator: the percent of 3rd grade students with disabilities tested out of the total number of 3rd grade students with disabilities, for example. Reporting that 95% of students with disabilities participated in the state assessment system is more useful than reporting that 6% of students who participated in the state assessment system were students with disabilities.

3. Clearly report proficiency levels. States vary in their terminology for levels of proficiency. Some states are very clear as to which levels indicate proficiency and which levels indicate non-proficiency, while some states use terminology that makes it difficult to determine this distinction. We recommend clearly labeling each level as “proficient” or “not proficient.” States could add additional levels to this dichotomous system (e.g., highly proficient, well below proficient) as long as the “proficient” and “not proficient” distinctions are obvious.

4. Report the number and percent of students with disabilities using accommodations. This recommendation remains from the 2003-2004 report as no progress was noted in this area. Many students with disabilities are not able to take the general assessment in the standard format, and thus are provided with accommodations. Many states consider the scores of some of these accommodated assessments either to not count or to count as “not-proficient” because they are non-standard accommodations. In some states, the number of students participating using non-standard accommodations is quite high. If these numbers are not reported, then the picture painted of how all students are doing will be inaccurate. It is important to know the extent to which students are using accommodations, and specifically those accommodations that result in the removal of their scores.

5. Clearly label state assessment results on the states’ Department of Education home-page. To ensure easily accessible Web-reported state assessment data, states should have a clearly labeled link to these data from the states’ Department of Education homepage. For example, the link on the homepage could read “2004-2005 State Assessment Results” and lead to the assessment results page, which in turn could have a clear link to disaggregated assessment results for students with disabilities (e.g., “Assessment Results for Students with Disabilities”).

6. Report on all statewide assessments. Some states purposefully asked us not to include certain non-accountability assessments in our analysis. Though this request may have been made for a number of reasons, the most obvious reason was that there was little or no publicly reported data for these assessments. We recommend that states publicly report all statewide assessment data, whether or not the assessment is part of the accountability system. These data may be very useful for educators, administrators, parents, and researchers, and should be readily available.

References

Bielinski, J., & Ysseldyke, J. (2000). Interpreting trends in the performance of special education students (Technical Report 27). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Goertz, M., & Duffy, M. (2003). Mapping the landscape of high-stakes testing and accountability programs. Theory Into Practice, 42(1), 4-11.

Klein, J.A., Wiley, H.I., & Thurlow, M.L. (2006). Uneven transparency: NCLB tests take precedence in public assessment reporting for students with disabilities (Technical Report 43). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K.S., Thurlow, M.L., Shriner, J.G., & Spiegel, A.N. (1992). Inclusion of students with disabilities in national and state data collection programs (Technical Report 2). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Petersen, G.J., & Young, M.D. (2004). The No Child Left Behind Act and its influence on current and future district leaders. Journal of Law and Education, 33(3), 343-363.

Thurlow, M. L., & Wiley, H. I. (2004). Almost there in public reporting of assessment results for students with disabilities (Technical Report 39). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thurlow, M. L., Wiley, H. I., & Bielinski, J. (2003). Going public: What 2000-2001 reports tell us about the performance of students with disabilities (Technical Report 35). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Wiley, H. I., Thurlow, M. L., & Klein, J. A. (2005). Steady progress: State public reporting practices for students with disabilities after the first year of NCLB (2002-2003). (Technical Report 40). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Zlatos, B (1994). Don’t test, don’t tell: Is "academic red-shirting" skewing the way we rank our schools? The American School Board Journal (181), 24-28.

Appendix A

Verification Letter to State Assessment Director

The National Center on Educational Outcomes is examining states’ public reports on 2004-2005 school year assessment results. Our goal is to (a) identify all components of each state’s testing system, (b) determine whether each state reports disaggregated test results for students with disabilities, (c) describe the way participation and performance information is presented, and (d) describe how states report results for students who took the test with accommodations or modifications.

We have reviewed your Web site for test information, including both participation and performance data on your statewide assessments. Enclose are tables highlighting our findings from that review. A blank field indicates that we did not find information in that area. Please verify all included information and provide us with information that we could not find on your Web site. Also, if there is publicly reported information available for your state, please provide us with the public document and/or website that contains the accurate information. Address your responses to Gretchen VanGetson at the above address or via fax at (612) 624-0879.

If you have any questions about our request, please call Gretchen VanGetson at (612) 626-0658 or email: vang0603@umn.edu. If we do not hear from you by March 3, 2006, we will assume there is no additional publicly available information.

Thank you for taking the time to provide this information.

Sincerely,
 

Gretchen VanGetson
Graduate Research Assistant

Martha Thurlow
Director

Table 1: Tests Administered and Results Found

Please review this table for its accuracy, make any changes (if necessary), and fill in any blank fields.

*AAS=based on alternate achievement standards
*GLAS=based on grade level achievement standards
*Other

Table 2: Participation Information for Students with Disabilities

Please review this table, which describes the way in which participation data are publicly reported in your state. A “Y” indicates information is reported in this way. Please add a “Y” if you know of any other method of participation reporting, and please provide us with the information that is reported in that way (either a hard copy or a Web-link).

Table 3: Performance Information for Students with Disabilities

Please review this table, which describes the way in which performance data are publicly reported in your state. A “Y” indicates information is reported in this way. Please add a “Y” if you know of any other method of performance reporting, and please provide us with the information that is reported in that way (either a hard copy or a Web-link).

*=Percentile Rank

Table 4: Accommodations

We are interested in examining if and how states report information about students who take assessments using accommodations. Please change our responses (if necessary) to reflect information that is reported for your state. If you do make changes, please provide us with the information (either a hard-copy or a Web-link).

If yes, complete Table 5

Table 5: Participation and Performance for Students Tested with Accommodations

If there are any “Yes” responses in Table 4, please review this table for its accuracy and make any changes (if necessary).

Figure 1: Disaggregated Reporting Groups by State

This figure includes the different student groups for which your state publicly reports disaggregated data. Please change our responses (if necessary) to reflect information that is reported for your state. If you do make changes, please provide us with the information (either a hard-copy or a Web-link).

YES     Special Education or Disability
YES     Gender
YES     Ethnicity
YES     Migrant/Enrollment Mobility/Less than One Year
YES     Limited English Proficiency
YES     Poverty, Low-Income, Economically Disadvantaged, Free/Reduced Lunch
NO      Gifted and Talented
NO      Accommodations
NO      Title I
NO      Neglected or Delinquent
NO      Homeless
NO      Extended School Services
NO      Parent Education
NO      Section 504 Plan
NO      At-Risk/Targeted Assistance
NO      High School Vocational/Career/Technology

Appendix B

Letters to State Directors of Special Education

(Two forms depending on input from Assessment Director. Example here is if letter was verified by the Assessment Director. If no verification, letter was the same as in Appendix A. Also, the table format is the same as in Appendix A.)

 The National Center on Educational Outcomes is examining states’ public reports on 2004-2005 school year assessment results. Our goal is to (a) identify all components of each state’s testing system, (b) determine whether each state reports disaggregated test results for students with disabilities, (c) describe the way participation and performance information is presented, and (d) describe how states report results for students who took the test with accommodations or modifications. We have reviewed your state’s Web site for test information, including both participation and performance data on your statewide assessments. Enclosed are tables highlighting our findings from that review. A blank field indicates that we did not find information in that area. These tables have been verified by your state’s Assessment Director; please notify me if you have anything to add.

 Please verify all included information and provide us with information that we could not find on your Web site. Also, if there is additional publicly reported information available for your state, please provide us with the public document and/or Web site that contains the accurate information. Address your responses to Gretchen VanGetson at the above address or via fax at (612) 624-0879.

 If you have any questions about our request, please call Gretchen VanGetson at (612) 626-0658 or email: vang0603@umn.edu. If we do not hear from you by April 14, 2006, we will assume there is no additional publicly available information. Thank you for taking the time to provide this information.

Sincerely,

Gretchen VanGetson
Graduate Research Assistant

Martha Thurlow
Director

Appendix C

2004-2005 State Assessment Systems and Status of Disaggregated Data
 

^a  Data are presented for 2003-04 and 2004-05 combined, and not disaggregated in a way that they can be reported for 2004-05 only.
 

Unique States

Appendix D

2004-2005 Alternate Assessments

^{*AAS = based on alternate achievement standards; GLAS = based on grade level achievement standards; Other = may include a combination of AAS and GLAS or different form, such as an adaptive assessment.}
 

Unique States

 ^{*AAS = based on alternate achievement standards; GLAS = based on grade level achievement standards; Other}

Appendix E

Disaggregated Participation Information (Given for State-Level Data)*

*Shaded cells indicate unclear (e.g., aggregated grade level or subject level) reporting.

Appendix F

Disaggregated Alternate Assessment Participation Information (Given for State-Level Data)*