NCEO Policy Directions

Published by the National Center on Educational Outcomes
Number 13 / June 2002

Including Alternate Assessment Results in Accountability Decisions

Prepared by Rachel Quenemoen and Martha Thurlow

This document has been archived by NCEO because some of the information it contains is out of date.

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Quenemoen, R., & Thurlow, M., (2002). Including alternate assessment results in accountability decisions (Policy Directions No. 13). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Policy13.htm

Alternate assessments provide a mechanism for students with the most complex disabilities to be included in assessment systems. Like regular assessments, the purpose of alternate assessments is to provide valid and reliable assessment data that accurately reflect the state’s learning standards, and that indicate how a school, district, or state is doing in terms of overall student performance.

An integral part of maximizing the benefits of assessing students is to include the results of those assessments in school accountability. This applies to students with disabilities who are assessed through alternate assessments.

The purpose of this Policy Directions is to address policy options for including the results of alternate assessments in school accountability systems. Examples of approaches used in several states are provided.

 

Background

Two policy issues have emerged with respect to alternate assessments – (1) can alternate assessments be included in school accountability systems? and (2) what is the best way to include alternate assessment results in accountability systems? The first question has been debated, with arguments for including these results focusing on the need to count all students and arguments against including these results focusing on technical difficulties in determining how to count alternate assessment results. These arguments are summarized in Table 1.

Table 1.  Pro and Con Arguments for Including Alternate Assessment Results in Accountability

Pro Arguments

Con Arguments

Each student’s assessment score is valued.

Scores of alternate assessment can improve.

Scores of alternate assessment are not a negative factor for school accountability index; thus, inclusion is not discouraged.

Does not value the “real” level of proficiency that must be reached by most students.

Could encourage inappropriate placement of students in the alternate assessment.

 

 

The objective of including alternate assessment results in school accountability is to ensure that the students who participate in the alternate assessment are included in the opportunities to learn that are generated by accountability systems, and ultimately in the improved learning that accompanies them. Schools can use accountability information based on all students to make policy, administrative, and instructional decisions that improve schooling practices so that all students, including students in the alternate assessment, are successful.

Starting Point: Good Alternate Assessments

Alternate assessments are part of the standards-based reform initiative designed to ensure that all students attain high standards of learning. Thus, they are one part of the broad effort to:

Despite being part of the standards-based system, alternate assessments involve different assessment approaches from those used in most general assessment systems. Thus, questions are raised about how results from alternate assessments can be combined with results from general assessments for accountability purposes.

To address questions about ways to include alternate assessment results in accountability systems, it is essential that the alternate assessment be well-developed (see Table 2 for some of the characteristics of well-developed alternate assessments). Once this has been accomplished, it is much easier to identify ways in which the results of these assessments can be included in accountability systems.

 

Table 2.  Characteristics of Good Alternate Assessments

There has been careful stakeholder and policymaker development and definition of desired student outcomes for the population, reflecting the best understanding of research and practice.

Assessment methods have been carefully developed, tested, and refined.

Professionally accepted standards are used to score evidence (e.g., adequate training, dual scoring, third party tie breakers, reliability tests and rechecks of scorer competence).

An accepted standards-setting process has been used so that results can be included in reporting and accountability.

The assessment process is continuously reviewed and improved.

 

Accountability Options

Although there are many ways in which alternate assessment results can be reported (see Bechard in Resources), these approaches can be merged into two basic options for accountability (see Hill in Resources). The two options reflect different ways of counting alternate assessment results – either as a score that can cover a range of values and thus can be improved in accountability, or as a score that is always at the lowest level.

These options have been described in terms of how the alternate assessment results will be scaled relative to the results of the general assessment (see Hill in Resources):

Option 1: Scale the results so that the value awarded for achievement levels on the alternate are the same or similar to the value awarded for achievement levels on the general assessment.

This approach is based on a belief that achievement on the alternate assessment is valued just as much as achievement on the general assessment. The alternate assessment, like the general assessment, must be aligned to the state’s content standards (usually by extending those standards). The alternate assessment also has a defined alternate achievement standard to complement the desired achievement standard for students in the general assessment.

In this option, improvements in alternate assessment scores are just as important as are improvements in general assessment scores. A student in the alternate assessment can achieve proficient status as defined for this small population of students, just as can a student in the general assessment. For this approach to work, it is important that the population of students for whom the alternate assessment is appropriate be clearly defined – defined in such a way that the number of alternate assessment students is stable from one year to the next.

The major objection to Option 1 is that awarding the same number of points or the same proficiency label for successful performance on the alternate assessment may devalue proficient performance on the regular assessment. It also could promote the inclusion of some students in the alternate assessment who should be in the general assessment. Both of these objections can be addressed through policy.

Option 2: Scale the results so that the achievement levels on the alternate assessment are at the lower end of the scale and achievement levels on the general assessment are at the upper end of the scale.

This approach is based on a belief that there must be an absolute level of achievement that is the target for all students, regardless of the significance of their disabilities. A student taking the alternate assessment with this approach can never achieve proficient status, but instead can only achieve the lowest achievement level possible—just because the student participates in the alternate assessment.

This approach discourages the inappropriate assignment of students to the alternate assessment because the scores a student could receive are automatically at the lower end of the achievement scale. On the other hand, there is nothing that can be done to improve the score of a student in the alternate assessment because the student will always earn the lowest achievement level possible regardless of actual improvements in achievement.

The major objection to Option 2 is that the presence of students who are appropriately assigned to the alternate assessment is likely to bring down school achievement indices. As a result, there is little incentive for schools to want these students in their buildings, thereby discouraging inclusive educational settings for alternate assessment students.

Comparison of Accountability Options

The selection of one accountability option over another is a policy decision – one that should be reached through stakeholder involvement. The decision also should be based on evidence from states that have adopted one approach or another, and by consideration of the results of simulation research.

The effects of adopting Option 1 have been demonstrated in simulation research conducted by Richard Hill from the Center for the Improvement of Educational Assessment (see Resources). Three conclusions were supported when he used state data in simulated formulas for accountability:

  1. The impact of including alternate assessment scores on school gains is trivial if the number of alternate assessment participants remains fairly constant from year to year.

  2. Making gains on the alternate assessment that are comparable to gains on the general assessment introduced little additional measurement error.

  3. Including alternate assessment results in accountability appears to lead to better outcomes for the students who participate in the alternate assessment.

If a school or district is looking for gains in achievement for all students, the school should be entitled to rewards for gains in the alternate that are equivalent to rewards for gains in general assessments.

Hill concludes that states should address several essential questions as they consider options for including alternate assessment scores in accountability: What is fair? What will encourage the greatest improvement for every student? What seems reasonable? Thus far, states have answered these questions in different ways.

Examples of State Approaches to Accountability Option 1

To some extent, the application of the accountability options in states will reflect the differences in the assessment systems on which accountability is based. The three approaches described here begin with the assumption that all students count; the question of whether results can be included has already been answered: yes they can. The approaches differ in the specific ways that results are included.

Approach 1: Alternate Assessment with Different Labels and Counting Rules

In this approach, achievement levels for the general assessment and alternate assessment are different, as are the descriptors of the levels. For example, one state refers to the four achievement levels of the general assessment as novice, partially proficient, proficient, and advanced. In contrast, for the alternate assessment, there are three achievement levels, with the labels beginning, partially skilled, and skilled.

In this first approach to including alternate assessment results in accountability, a set of decision rules is used to determine when the alternate assessment results are considered. The decision rules reflect the importance placed on appropriate participation in assessments. Thus, alternate assessment results are included when there is no evidence of sufficient participation in the general assessment, and when adequate improvement in general assessment scores relative to a long-range target is not evident.

In this approach, any school that has a general assessment participation rate that is less than 95% automatically goes into school improvement. For those schools without adequate gains in general assessment performance, other factors are considered as well (for example, participation rates for the alternate assessment, grade 1 and 2 assessment results, progress of Title I students, reductions in percentage of students in the novice level).

Schools gain or lose points based on a combination of participation and progress on the alternate assessment. Specific points added or subtracted from school scores are as follows:

Obviously, the specifics of the way in which the alternate assessment factors into the accountability system can be altered, but the key element of this approach is that the alternate assessment is counted in some way.

Advantages. An emphasis on getting all students into the system is likely to result in high participation rates. Schools can improve their status through full participation and through improved alternate assessment performance.

Disadvantages. Different labels for achievement levels and use of alternate assessment results only when general assessment participation and performance is not considered adequate essentially gives less weight to the alternate assessment. For schools making adequate gains with their general assessment students but inadequate gains (or no gains) with their alternate assessment students, there is no recognition of the need to improve programs for those students with the most significant disabilities.

 

Approach 2: Two Alternate Assessments Counted Differently

In this approach there are two alternate assessments, one of which is intended for students with significant disabilities (generally an alternate portfolio), and the other of which is for students “who may not be able to take the general assessment but who are not eligible for the alternate portfolio” (generally some type of lower grade-level assessment or an academic inventory). Each alternate assessment is included in accountability, but in different ways.

The academic assessment, which is most like a general assessment, expands the academic achievement levels of the general assessment. A common approach is to add levels to each of the existing achievement levels. For example, there may be Novice 1, 2, 3, and 4; Apprentice 1, 2, 3, and 4; Proficient 1, 2, 3, and 4; and Distinguished 1, 2, 3, and 4. This essentially provides more steps for demonstrating progress. These scores can be included in accountability in the same way that general assessment scores are – generally one number for each content area (reading, math, and others).

The alternate portfolio is included in this accountability approach in a different way from the alternate academic inventory. While the same labels are given to the alternate portfolio achievement levels (novice, apprentice, proficient, distinguished) as are given to the general assessment achievement levels (without the extra levels 1, 2, 3, and 4 given to the academic alternate), each student receives only one score rather than the multiple scores in the general assessment and the alternate academic inventory. Thus, the general assessment counts several times, as does the alternate academic inventory, while the alternate portfolio counts just once.

Advantages. All students receive scores and all scores count. The use of the same labels conveys a message that students with the most significant disabilities can become proficient and distinguished, just as other students can.

Disadvantages. Scores of students in the alternate portfolio count less than the scores of other students. This approach essentially averages scores rather than averaging students, and some students count more than others. Schools may earn awards without any impact on students in the alternate portfolio. This may lead to less concern about improving programs and instruction for these students.

 

Approach 3: Alternate Assessment with the Same Labels and Same Counting

This approach also uses the same labels for the achievement levels assigned to the general assessment and the alternate portfolio assessment – novice, apprentice, proficient, and distinguished. The belief that students who demonstrate proficient performance within the structure of the alternate assessment should make the same contribution to the school accountability index is carried into the numbers of the accountability system. The same point values employed for the general assessment are used for the alternate portfolio assessment. In addition, students in the alternate assessment are required to demonstrate achievement within the same multiple content areas as are students in the general assessment, and scores are assigned accordingly for each content area.

To further support the commitment to inclusion of all students in the accountability system, this approach generally has a couple of additional aspects. One is to assign the lowest score possible to any student who is not in either the general or alternate assessment. To guard against inappropriately including students in the alternate assessment, an audit point is identified (generally 1–2% of all students) to indicate that the level of participation in the alternate assessment needs to the checked because it might be too high.

Advantages. Scores of students in the alternate assessment count just the same as scores of students in the general assessment. This approach essentially averages students, rather than test scores, so that each student receives equal weight in the school accountability index. With the scores of all students counting (even those not in either assessment), participation is encouraged. Further, schools are unlikely to earn awards without having an impact on students in the alternate portfolio. This may lead to more concern about improving programs and instruction for these students.

Disadvantages. Without certain additions, this approach could result in students being inappropriately placed in the alternate assessment. Additions, such as assigning the lowest score possible to non-participating students and monitoring participation rates through an audit point, can result in an approach with no obvious disadvantages.

Summary

The imperative to include all students in school accountability systems and the decisions that are made based on these systems is clear. Yet, there are multiple ways in which scores can be included, and the effects of each approach needs to be carefully considered. The two most basic options (1) to count achievement on the alternate assessment comparable to achievement on the general assessment, or (2) to count achievement on the alternate assessment as automatically lower than achievement on the general assessment, is the starting point for conversations about which approach to take for accountability.

There are many approaches that can be taken within each option. In this document we have highlighted those approaches that are consistent with the first option (where alternate assessment scores have the same or similar value as general assessment scores), the option that has been supported by simulation research on the technical, motivational, and policy effects of different approaches. As implementation of different approaches proceeds, it will be important to monitor the actual effects of various approaches.

Resources

The Impact of Including Special Education Students in Accountability Systems. Hill, R. (2001). Portsmouth, NH: National Center for the Improvement of Educational Assessment. http://www.nciea.org/cgi-bin/pubspage.cgi?sortby=pub_date

Models for Reporting the Results of Alternate Assessments within State Accountability Systems (Synthesis Report 39). Bechard, S. (2001). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Use of Alternate Assessment Results in Reporting and Accountability Systems: Conditions for Use Based on Research and Practice (Synthesis Report 43). Quenemoen, R., Rigney, S., & Thurlow, M. (2002). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Top of page