Krentz, J., Thurlow, M., & Callender, S. (2000). Accountability systems and counting students with disabilities (TechnicalReport 29). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Technical29.htm

This report is the first by the National Center on Educational Outcomes (NCEO) to examine the extent to which students with disabilities are included in accountability systems of states in which there are high stakes for educational systems. Conducted through a Web search of states’ education Web sites and direct links to their Web sites, we examined the indicators on which states collect information (e.g., student indicators, staff indicators, program indicators, and use of resources indicators) as well as the consequences provided by the states (e.g., rewards, assistance, or sanctions).

Of the 280 indicators listed for these states according to the Education Commission of the States (ECS), NCEO was able to find documentation of 178 of them on the states’ Web sites or direct links. Furthermore, NCEO found an additional 216 indicators not identified by ECS. However, it was difficult to determine the extent to which any of these indicators were used in making determinations for consequences. Information for many indicators was collected and reported but not included in the actual formulas for determining rewards, assistance, or sanctions. Most often, assessment scores served as the only indicator used for making this determination.

Five states had accountability formulas that could be identified in information found on state Web sites. Louisiana, Mississippi, New York, North Carolina, and Ohio listed their indicators and criteria as well as how they are used to determine the consequences for states. From the information found, other states may have some of these components of an accountability system, but did not provide the level of detail on their Web sites needed to determine how their systems were implemented.

This study highlighted that not only was it difficult to determine exactly what indicators factored into overall accountability systems, but it was also difficult to find evidence that students with disabilities were included in accountability systems. Although students with disabilities may be included in assessments, their scores may not be included in the accountability systems, and most states did not specifically state whether students with disabilities were included when defining the indicators used.

Overview

Educational accountability is gaining attention and influence as state and local policymakers implement sweeping accountability reforms. States are increasingly relying on large-scale assessments to measure student performance and to determine sanctions and rewards (Education Commission of the States, 1999; Olson, Bond, & Andrews, 1999). The Web sites of major organizations, such as the Council of Chief State School Officers (CCSSO: http://www.ccsso.org) and the Education Commission of the States (http://www.ecs.org) reflect this emphasis in content on their Web pages and their on-line reports.

Approximately half the states now use, or will soon use, tests in making decisions about student promotion (see Quenemoen, Lehr, Thurlow, Thompson, & Bolt, 2000) and high school graduation (see Guy, Shin, Lee, & Thurlow, 1999). All but two states have statewide assessments in place, many with more than one assessment (Olson, Bond, & Andrews, 1999). In the 2000 legislative sessions, over 300 bills were introduced in 41 states that dealt with statewide assessments (Coleman, Heller, & McNeil, 2000; see also the Web site for the National Conference of State Legislatures, http://www.ncsl.org/programs/educ/k12link.htm).

Not only are states increasingly attaching high stakes consequences to students for their performance on assessments (e.g., no diploma, no promotion to the next grade), but states are also attaching significant consequences to various aspects of the educational system (e.g., to teachers, schools, and administrators). This approach, known as system accountability, occurs when educators, administrators, schools, or districts are held responsible and consequences are assigned to them for various student performance indicators. For example, school accreditation may be based on test scores; teachers may receive cash rewards for student performance; or, administrators may be either sanctioned or rewarded based on school improvement, which in turn is measured to a large extent by student performance.

Within system accountability, there are different levels of consequences – referred to as the “stakes” of the assessment. At one end is simply publishing test scores – usually considered to be low stakes. While it might be considered “positive feedback” to have a school that performed well on test scores, or conversely, might prove an embarrassment for schools or districts that performed poorly, these consequences are relatively minor. In contrast, if teachers are assigned to “less prestigious” schools or receive cash bonuses based on student performance, the stakes are considerably higher. Similarly, if principals receive increased school funds or lose their jobs based on how students in the school perform, the stakes are very high.

In addition to obvious “rewards” and “punishments,” there is another type of consequence currently used in accountability systems. This type of consequence involves assistance to the system and is generally considered to be a much less significant consequence.  An example of assistance might include the provision of additional resources, such as monetary grants or specially trained personnel to schools that receive a “warning.” The distinction between “assistance” and “sanctions” is often subtle, at best. The “assistance” phase sometimes serves as a probationary period prior to more intense intervention or reconstitution of a school or district.

The Education Commission of the States (1999) has documented the specific indicators that states are using in their accountability systems. Although an array of indicators is listed for most states, it is unclear how they are used. Many questions remain about current and proposed accountability systems. Which indicators are truly high stakes? Which are simply included in a “report card” for a school or district? Are the indicators weighted in some manner or incorporated in a formula to develop an accountability measure? Are there positive rewards for successful schools and systems, assistance for struggling schools and systems, or sanctions for those who are not performing as expected?

In addition to these basic questions about state accountability systems, there are also questions that focus on specific groups of students. With the reauthorization of the Individuals with Disabilities Education Act in 1997 (IDEA 97), states and districts must include students with disabilities in their assessments. They are also required to report on the performance of students with disabilities, both aggregated with other students, and disaggregated.

Several analyses from before and after the enactment of IDEA 97 indicated that states have started to include students with disabilities in their assessments only recently (McGrew, Thurlow, Shriner, & Spiegel, 1992; Thompson & Thurlow, 1999), and that many states are not yet reporting the results (Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1998; Thurlow, Nelson, Teelucksingh, & Ysseldyke, 2000) even though they are required to do so by federal law. It is a logical follow-up question to ask about the extent to which the increasing participation of students with disabilities in assessments translates into their inclusion in accountability systems.

Of course, there are many reasons for including students with disabilities in accountability systems (Thurlow, Elliott, & Ysseldyke, 1998). If these students are not included in accountability systems, it means that roughly 10% of the student population is excluded from accountability. Students who are left out of assessments and accountability systems tend to not be considered when reform efforts are being designed and implemented. Furthermore, the education system does not obtain an accurate picture of education if a significant portion of students is excluded, nor is it able to make “fair” comparisons. Participation of students in accountability systems seems to promote higher expectations for those students who are included, and consequently, may be assumed to exclude from high expectations those students who are not included.

The purpose of this research was to examine the extent to which students with disabilities are included in accountability indicators in those states in which there are high stakes for systems. To do this, we had to first decipher the accountability systems that states use – this involved identifying both the indicators that are part of the accountability system and the “formula” that is used in each state to combine the indicators to determine an accountability score. We restricted our search to accountability indicators and formula used to determine system-level consequences and did not search for formula developed exclusively to meet Title I requirements for adequate yearly progress. From the indicators and formula identified, we examined the extent to which students with disabilities were included in the state’s accountability system.

Method

Initially, we used the document Education Accountability Systems in 50 States (Education Commission of the States/ECS, 1999) to determine which states we would include in our analysis. We selected states that ECS had listed as having a “statute” or “regulation” in the Rewards or Sanctions column of its Table I. There were 38 states that met this criterion. The ECS document also included information about which indicators were used by the various states (see ECS Table 5). We divided in half the list of states that had high stakes accountability systems for schools or districts, then two of us examined Web sites to verify and expand on the information that was included in the ECS report.

We used the Achieve Web site (http://www.achieve.org) “State Links” page as a quick link to the Web site for the Department of Education of each state. From this site, we also located other state offices (e.g., accountability, assessment, legislature).

We also used the Council of Chief State School Officers (CCSSO) Web site (http://www.ccsso.org) section on “Council Projects” where we located the Accountability Systems Profiles. (The direct Web address is http://www.ccsso.org/introprofile.html). According to this site, the Resource Center on Educational Equity of the Council of Chief State School Officers (CCSSO), in collaboration with the New England Comprehensive Assistance Center, is developing profiles of several state’s education accountability systems.

When we checked the site, there was information about the following 13 states: Alabama, Connecticut, Georgia, Illinois, Kentucky, Maryland, Massachusetts, New Jersey, North Carolina, Oregon, Rhode Island, Texas, and Vermont. The profiles included information about each state’s content and performance standards, statewide student assessment, and criteria used to determine whether schools or districts are eligible for rewards or are in need of improvement. The site also contained information about how school and district performance is publicized, as well as details about the provision of rewards, assistance, and sanctions. There was often helpful information about the inclusion of students with disabilities in the assessment systems, as well as information about provisions for English Language Learners.

The Resource Center on Educational Equity page of the CCSSO Web site indicated that the information for the 13 profiled states was gathered from various sources, including CCSSO reports, state statutes and regulations, and information posted on state education agency Web sites. The Center’s initial drafts were reviewed and revised by state education agency staff. We used the information from CCSSO’s state profiles only after independently verifying it on state Web sites.

At each state’s Web site we reviewed state accountability reports, report cards, policy guidelines, and other relevant items to identify indicators, consequences, and accountability formula. This review was conducted in 1999-2000, with all reviews completed by July 2000. After we completed our initial examinations of Web site information on states, the two lists of states were traded, so that the information could be verified. In some instances, there was little change from what the first examination had indicated. In other cases there was completely new and sometimes contradictory information obtained during the second examination of the Web site. This report contains the more recent version of those sites.

Results

It became very evident during our verification process that the information we sought was a moving target. It changed rapidly due to ongoing legislative processes, subsequent rule-making procedures that follow legislative sessions (in order to implement the legislation just passed), and states’ action taken in response to federal legislation. Our focus was on Web-based information, and we found a significant disparity between the level of sophistication and quality of information found on various states’ Web sites. Some seemed to have frequent additions or modifications, and others seemed to be fairly rudimentary. Links to other Web sites were prominent and user-friendly on some sites, and unavailable or very obscure on others; sites were frequently “down” or “temporarily under construction.” For example, important documents found during the first review were often moved or unavailable during the second review. Sometimes Web sites would change dramatically within a few days.

We collected information on the indicators used for accountability purposes, the formulas for calculation of school or district scores for accountability purposes, the consequences of the accountability systems (e.g., rewards, assistance, or sanctions), and the inclusion of students with disabilities in the accountability systems’ indicators and calculations.

Accountability Indicators

Table 1 summarizes the numbers of indicators that were identified in the ECS document and in our Web search. Of the 280 specific indicators listed in the ECS document, we were able to verify 178 of them when examining the State Departments of Education Web sites and direct link sites. In other words, we found, on average, only 64% of the indicators that ECS found using written reports. (Values ranged from 15% for Massachusetts to 100% for Alaska, Colorado, Kansas, Kentucky, New Jersey, Oklahoma, Rhode Island, South Carolina, Tennessee, Vermont, and West Virginia.)

Table 1. Indicators in State Accountability Systems

We also discovered on State Department Web sites 216 additional indicators used by states for their accountability systems. While the majority of indicators listed were for reporting purposes only (i.e., they are documented in school and district report cards), some of these indicators are used in accountability formulas to calculate school or district scores to determine access to rewards or assistance or the receipt of sanctions.

Although a state may have indicated on its Web site that it uses a large number of indicators, we found that it was often the case that the majority of the indicators were of the same type (e.g., all assessment scores). Thus, looking at only the number of indicators does not reflect the nature of the accountability system. To get a different picture, we looked at the spread of types of indicators, and compared these to the types found in the ECS analysis. The results of this comparison are provided in Appendix A. As is evident, relatively few different types of indicators are being used by states, and the consistency between the types our search revealed and those ECS identified are relatively similar.

Accountability Formula

Five states (Louisiana, Mississippi, North Carolina, New York, and Ohio) had accountability formulas using indicators to calculate school or district scores listed on their Web sites. The formulas are quite diverse, from those that simply divide the number of students reaching a given proficiency level by the total number of students taking the test, to those that have included factors for regression to the mean.

Other states have information on their Web sites that were close to being formulas, but are too vague to actually determine how accountability is calculated. For example, Indiana noted that a school receives a monetary incentive if the school has demonstrated improvement over the average of its scores for the past three years in two or more areas (with the areas defined as attendance rates, math scores, language arts scores, and ISTEP + total battery scores). Similarly, Pennsylvania indicates on its Web site that it gives rewards based on improvements in achievement rates over the past two year average, or higher graduation rates or attendance rates over a criteria set each year (but the criteria were not provided). The state also indicated that dropout rates are collected but not counted in the reward system. Finally, Kentucky’s Web site clearly indicates that it has an accountability index that is a statistic that combines a school’s academic, nonacademic, and other “appropriate data,” but how these elements are combined is not evident. Dropout criteria also must be met, although the specific criteria are not stated. Kentucky also indicates that it uses a regression model to compare expected growth with actual growth in determining when a reward is given.

Of the five states’ formulas, only one specifically mentions students with disabilities (and another mentions English Language Learners). In the four states that do not mention students with disabilities, the implication is that all students are included in the same way. The one formula that did identify students with disabilities actually factored in a weight that decreases the impact of the scores of students with disabilities on the resulting accountability index. The rationale for this approach is that special education students might take longer to achieve standards because of lost opportunities to learn. Detailed descriptions of the formula are provided in Appendix B.

Consequences

Table 2 summarizes the nature of consequences of school or district performance in each of the states examined. It is important to note that this table is a point-in-time snapshot. Several states indicated that they are currently changing their existing accountability system or are implementing new components to their accountability systems. Currently under development are reward systems for five states, assistance programs for two states, and sanctions for three states.

Table 2. Nature of Consequences in State Accountability Systems

Insufficient information was available on some of the Web sites to determine whether any rewards, assistance, or sanctions were given. Eleven states did not include any information about rewards for high achieving schools or districts. Seven states did not include any information about assistance programs for low achieving schools or districts on their Web sites. Eight states did not include any information about sanctions for chronically low achieving schools or districts.

Of the 38 states listed as having high stakes system accountability, information on consequences could be found on the Web sites of 34 of them. Of the 34 states with some consequence in place for performance, 17 had all three components: rewards, assistance, and sanctions. An additional seven states had both assistance and sanctions. Of the 34 states that provided information on their accountability components, seven had no form of sanctions, seven had no form of assistance, and nine had no form of reward.

Rewards

Of the states that provided information on their accountability components, 22 states indicated that they provided rewards for schools or districts with high achievement (based on their indicators). Table 3 summarizes the types of rewards offered by these states. Details of the rewards provided by each state are available in Appendix C.

Table 3. Rewards for High Achieving Schools

^a Decisions are made at the school or district level or on an individual basis.
^b High Performing schools are designated through a Quality Performance Accreditation.

Most states provide some form of tangible or monetary rewards. For example, Nebraska provides Quality Education Incentive Payments and Georgia provides Pay for Performance awards. These monetary rewards are most commonly used to purchase equipment, pay for professional development, or make possible other improvements in the schools’ educational processes.

A majority of states also have some process for formal recognition or praise of schools or for districts with superior performance. Examples include banners, certificates, and plaques. In all but two of the states providing recognition or praise, there is also the use of tangible or monetary rewards. Only Mississippi and Virginia use recognition or praise without tangible or monetary rewards.

Six states provide some form of additional decision-making control or exemption from policy guidelines for districts or schools that demonstrate high achievement. For example, Florida states that schools rated high are eligible for deregulated status and increased budgetary control, while Kentucky allows high-performing schools to be exempted from certain reports, paperwork requirements, and administrative regulations.

Two states are coded as “Other” in Table 3 because of their unique situations.  In Colorado, decisions about rewards are made at the district level. Kansas uses its indicators to determine school accreditation; schools with high student achievement and continual academic improvement receive a special accreditation. For five states (Arkansas, Delaware, Massachusetts, New York, and Washington), the process for determining rewards for high achieving schools is under development.

Assistance

Twenty-six states indicated that they had assistance for schools or districts that had low achievement according to their indicators. Table 4 summarizes the types of assistance offered by the states. Details on the assistance provided by each state are presented in Appendix D.

Table 4: Assistance for Low Achieving Schools

^a Decisions are made at the school or district level or on an individual basis.

Most states that provide assistance either require schools or districts to develop improvement plans or the states provide assistance in developing these improvement plans. For example, Oregon districts and schools identified for improvement are required to develop Improvement Plans with descriptions of their assessment of problem areas, proposed activities to be undertaken, and a timeline for achieving these goals. New Mexico selects the lowest performing schools to receive assistance from the State Board of Education in creating a School Improvement Plan. Often these plans include outcome criteria against which a school or district’s performance is measured, or they may contain binding agreements for changes to be implemented.

The same number of states (but not necessarily the same states) that require improvement plans provide some form of non-monetary assistance. Seventeen states have both improvement plans and non-monetary assistance. Examples of non-monetary assistance include state assistance teams that conduct site visits and provide recommendations, technical assistance with data analysis, staff training and consultation/mentoring by specifically designated teachers, administrators, or teams of educators.

Nine states provide some form of monetary assistance, which may be coupled with improvement plans, non-monetary assistance, or both. Ohio provides Continuous Improvement Development/Implementation Grants for districts to design and implement comprehensive plans in conjunction with their continuous improvement plans. South Carolina offers targeted grants for professional development, alternative schools, homework centers, and modified school days (i.e., extended year or day schools).

Colorado is coded as “Other” in Table 4 because decisions about whether a school or site will receive some form of monetary or non-monetary assistance are made locally. In Delaware and Massachusetts, the process of determining rewards, assistance, or sanctions is under development.

Sanctions

Twenty-six states indicated that they had sanctions for schools or districts with chronic low achievement. Table 5 summarizes the types of sanctions administered by these states. Appendix E gives details of the various sanctions.

Table 5: Sanctions for Low Achieving Schools

^a Decisions are made at the school or district level or on an individual basis.
^b The state may increase the school system’s local fair share to finance necessary corrective actions, require the local school system to raise funds from local revenue sources to finance the corrective actions, or file a civil action in county court to determine if any local school board member or local superintendent has prevented or delayed the corrective action plan implementation.
^c Although sanctions were not specified, indicators are used in determining a school’s accreditation.

Twenty-one states reported using some process to notify the public of low achieving schools. While many more states may identify schools as low achieving or provide information on schools’ poor performance, public notification as coded in Table 5 consisted of a more formal process whereby a label or level was assigned to the school that clearly identified it as low achieving. For example, Alabama labels schools and systems as Academic Alert Levels 1, 2, or 3, Arkansas labels schools as High Priority, Alert Status, Low Performing Status and Academic Distress, and Texas labels campuses as Low-Performing and districts as Academically Unacceptable. In addition, states that have a formal notification process were coded in this category of sanctions. For example, in Kansas each school issues a report of Accreditation Deficiencies that includes its plans to correct the deficiencies; these are reported to the communities in their primary languages.

Fifteen states reported enforcing state mandates or assuming some managerial control of low achieving districts or schools. Examples include mandatory assistance, administrative orders from Commissioners, and state takeovers of schools. In all but two of these states (New Jersey and Oklahoma), mandates or management sanctions accompanied the public notification.

Twelve states reported the possibility of reorganizing staff or restructuring schools to remedy chronically low achieving schools. For example, Tennessee may remove the superintendent and board members from office, New Jersey may dismiss or demote school staff, and Connecticut may close and reconstitute schools.

Ten states require schools to pay for the transfer of their students to better performing schools or the state pulls the funding for students. For example, Nevada requires schools to pay for students to be transferred to the nearest fully accredited school.

Six states will pull the official status of low achieving schools. Mississippi may withdraw the accreditation status of chronically low performing schools, while Illinois may “non-recognize” a school or district that may then be dissolved and realigned with other districts.

Four states are coded as “Other” in Table 5. Both Colorado and Oregon make decisions either locally or on an individual basis. Georgia uses a variety of sanctions involving either schools raising funds to support corrective actions or the state filing a lawsuit. New Mexico identifies schools with accreditation deficiencies, which must be publicly reported.

Students with Disabilities

The inclusion of the assessment scores of students with disabilities in accountability systems of various states is represented in Table 6. More detailed description of the inclusion/exclusion policies of individual states can be found in Appendix F.

Table 6: Inclusion of Assessment Scores for Students with Disabilities in Accountability Systems

^a This accountability system is currently in place for grades K-8. Grades 9-12 will be added in 2001.
^b One accreditation criterion requires a 95% participation rate allowing exclusions of 5% of students.
^c Scores may be excluded from students who are expected to eventually meet the same basic curriculum objectives, but due to their educational delays, are not expected to meet them for that school year.
^d Students with disabilities are counted in the participation rates for a subtest even if they are excluded from taking it but do participate in the other subtests.
^e Reasons allowed for exempting students with disabilities include parent request, IEP status, LEP status, extended absence, withdrawal, of other reasons.

Seven states seem to include all students with disabilities in their accountability systems. One state says all are included, but there was no confirming information on the Web site to validate or contradict that information. Of the remaining six states, five have documented evidence that indeed all students with disabilities count. They are Illinois, Kentucky, Missouri, New Mexico, and New York. For example, Illinois clearly indicates that if a student (IEP or non-IEP) takes a state assessment and passes the “attemptedness” criterion, that student’s score will be included in the school, sub-district, district, and state averages and data computations. Beginning in 1999, the option to “bubble out” a student’s score (allowing the student to take the state assessment and excluding that score from the school, district, and state aggregations) was no longer allowed. The seventh state, Louisiana, has a system of inclusion of all students with disabilities currently in place for grades K-8 and indicates that grades 9-12 are scheduled to be added in 2001.

Two states (Alabama and Tennessee) indicate that students with disabilities are excluded from their accountability systems. In Alabama, the Web site emphasized the inclusion of students with disabilities in assessments, but went on to indicate that special education students’ scores were not included in classifying a school or system as Academic Clear, Academic Caution, or Academic Alert. On the Web site for Tennessee, there was no definitive information, but there was a related link which revealed the fact that any student who is eligible for special education services under federal law would not be included as a part of its value-added assessment system.

More commonly, a defined subset of students with disabilities is excluded. For example, based on information on the Web it appears that seven states use non-standard accommodations as the determining factor of whether a student with disabilities is included in the accountability system. Arizona, Delaware, Indiana, Nevada and New Jersey use non-standard accommodations as the sole criteria. In addition to using non-standard accommodations as a reason for exemption, Mississippi also permits scores to be excluded for students who are expected to eventually meet the same basic curriculum objectives, but due to their educational delays are not expected to meet them for that school year. Oregon mentioned non-standard accommodations as a reason for exclusion, but was unclear as to whether there were additional criteria for exclusion as well.

Three states (Michigan, Mississippi, and Pennsylvania) use various other factors to determine inclusion or exclusion. An example of this is Pennsylvania, which may exclude students with disabilities for a number of reasons including parent request, IEP status, LEP status, extended absence, and withdrawal from school.

Ten states exclude a subset of students with disabilities, but it is unclear in information provided on the Web site of how that subset is determined. In Florida, for example, schools are required to assess at least ninety percent of all eligible students, but it is not clear how they determine which students are excluded. In Georgia, students are excluded if their assessments are coded in a particular manner. In South Carolina, students who are included in statewide testing are included in the accountability system. However, the exemption rates for students with disabilities in 1998 exceeded six percent of the student population and represented approximately 60% of the special education students.

In 12 states it simply was not possible to determine whether all students with disabilities are included. For example, Colorado makes no direct reference to the inclusion of special needs students. Massachusetts indicates that it will include all public school students in the Massachusetts Comprehensive Assessments (MCA) and the grade 3 reading test, but its accountability system is still listed as under development on the state Web site, to be implemented in 2001.

 Discussion

It became very evident during our data collection efforts that high stakes educational accountability systems are difficult to accurately evaluate from the information available on state Web sites. While we intentionally chose to search for Web-based information as an indication of the quantity and quality of information readily available to the public at large, it became apparent that there is a wide discrepancy among the level of sophistication and degree of thoroughness and accuracy of this information from state to state. Additionally, the fact that these policies often change due to legislative action, or rulemaking following legislative action, would seem to lend itself to updating more frequently on the Web, yet this did not necessarily prove to be the case. Some sites seemed to be updated often, and others seemed to be essentially unchanged from one visit to another, even though several weeks may have passed in the interim. In some instances, the changes were not always improvements. On occasion we would find an important document on-line during one visit, and then check the same Web site a few days later, and the document would no longer be available, even by a related link.

It is possible that states indeed have clear guidelines with indicators and formula for their accountability systems, but that they do not make them available to the public through their Web sites, whether by conscious decision, or simple omission. For example, some states indicated that they did collect school and student data in order to make accountability decisions about consequences for schools and districts, but they did not explain their process on the Web. Others indicated how they were reporting scores for students with disabilities to the public, but did not specify whether or how these scores were included in their accountability systems.

The complexity of the systems that determine accountability may add to the difficulty of making information available to the public. Some states face the challenge of duplication of jurisdiction of various departments or divisions with overlapping responsibilities for holding schools or systems accountable. Some states have created distinct accountability departments or divisions that may act independently from state departments of education. Furthermore, states often have overlapping systems of accountability. There may be accountability systems, accreditation systems, and report card systems, which may or may not be related to each other, or may be driven or directed by distinct departments.

One of our biggest challenges in studying state accountability systems was determining whether specific indicators were actually included in the systems. Many states reported that they collected data on multiple indicators, but in fact, relied solely on statewide testing data in determining the consequences for districts or schools. When other data were used, they were most often reported for comparison purposes rather than applied to a formula used to calculate a school’s or district’s score or index.

Although several states included policy information about student participation in testing and the reporting of those scores to the public, they did not include specific information about the inclusion of scores in the accountability formula. States often did not specify whether students who were excluded from statewide testing were included in accountability systems in some other manner, such as by entering zeros for their testing data, or using scores from an alternate assessment aligned with the statewide testing, or whether they were entirely omitted from the system. One notable exception to this was Kentucky where scores from alternate portfolios are included in the academic indices. This allows every student to contribute the same weight to the academic component of the accountability index, regardless of whether they participated in the regular components of the assessment system or in the alternate portfolio assessment.

Our research also highlighted the difficulty in distinguishing between various types of consequences, particularly between “assistance” and “sanctions.” Though many states classified their interventions as being part of an assistance program, there was often a fine line between an intervention that was assistance and one that actually crossed the threshold and became a sanction. There was little consistency between states on this determination. What was considered assistance in one state might be classified to be a sanction in another. Some states used assistance-like interventions and sanction-like interventions in reverse order. For example, Nevada mentioned punitive results for school failure, including effects on funding and school administration that might precede interventions of assistance; assistance would be rendered only after three years of failure.

Despite our original intent to gather information on the extent to which students with disabilities were included in school and district accountability systems, we were not very successful in doing so. To a great extent, our failure was our desire to obtain publicly available information, which was in itself not clear at all. What we did find raises concerns about the extent to which states are, or even intend to, include students with disabilities in their accountability calculations. We were alarmed to find a state formula that specifically reduced the impact of students with disabilities on accountability scores. While the notion of some type of phase-in of requirements might seem appropriate for student accountability (i.e., give students with disabilities extra time to meet graduation test requirements), it does not seem appropriate for system accountability, where the approach, instead, might be to increase the impact of these students on accountability scores so that the students get the opportunities that have not been provided to them in the past. Title I requires that all students be included in accountability measures of adequate yearly progress. These measures are to be in place by 2001. Our analysis, which did not seek out these adequate yearly progress measures, indicated that existing state accountability systems are far from the Title I ideal. It appears that education has a significant way to go before it is truly accountable for the learning of all students.

 References

Coleman, A., Heller, H., & McNeil, L. (2000, July). High stakes tests: Legal and political issues. Handout presented at the Annual Meeting of the National Conference of State Legislatures, Chicago, IL.

Education Commission of the States. (1999). Education accountability systems in 50 states. Denver, CO: Author.

Guy, B., Shin, H., Lee, S. Y., & Thurlow, M. (1999). State graduation requirements for students with and without disabilities (Technical Report 24). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Thurlow, M. L., Shriner, J. G., & Spiegel, A. N. (1992). Inclusion of students with disabilities in national and state data collection programs (Technical Report 2). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Olson, J. F., Bond, L. & Andrews, C. (1999). Data from the annual survey: State student assessment programs: Data on 1997-1998 statewide student assessment programs. Washington, D. C.: Council of Chief State School Officers

Quenemoen, R., Lehr, C. A., Thurlow, M., Thompson, S. J., & Bolt, S. (2000). Social promotion and students with disabilities: Issues and challenges in developing state policies (Synthesis Report 34). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thompson, S. J., & Thurlow, M. L. (1999). 1999 State special education outcomes: A report on state activities at the end of the century. Minneapolis, MN: University of Minnesota.

Thurlow, M. L., Elliott, J. L., & Ysseldyke, J. E. (1998). Testing students with disabilities: Practical strategies for complying with district and state requirements. Thousand Oaks, CA: Corwin Press, Inc.

Thurlow, M. L., Langenfeld, K. L., Nelson, J. R., Shin, H., & Coleman, J. E. (1998). State accountability reports: What are states saying about students with disabilities? (Technical Report 20). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thurlow, M. L., Nelson, J. R., Teelucksingh, E., & Ysseldyke, J. E. (2000). Where’s Waldo? A third search for students with disabilities in state accountability reports (Technical Report 25). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Appendix A

Types of Accountability Indicators

Appendix B

Accountability Formulas

Appendix C

Rewards for High Achieving Schools

Appendix D

Assistance for Low Achieving Schools

Appendix E

Sanctions for Low Achieving Schools

Appendix F

Indicators Including Students with Disabilities