NCEO Synthesis Report 27

Issues and Considerations in Alternate Assessments

by James E. Ysseldyke, NCEO, University of Minnesota, College of Education and Human Development

Ken Olsen, Mid-South Regional Resource Center, University of Kentucky, Human Development Institute

Martha Thurlow, NCEO, University of Minnesota, College of Education and Human Development

Published by the National Center on Educational Outcomes

January 1997

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Ysseldyke, J. E., Olsen, K. R., & Thurlow, M. L. (1997). Issues and considerations in alternate assessments (Synthesis Report No. 27). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis27.htm

Executive Summary

Accountability systems that include all students must have a way to measure the performance of students who cannot be assessed through typical testing procedures. Alternate assessments are designed to meet this need, which is estimated to apply to approximately 1 to 1-1/2% of the student population. There are many issues that need to be considered as states and districts begin to design and implement alternate assessments. Thirteen issues described and discussed in this report are:

Issue 1: Eligibility–Who Takes Alternate Assessments?

Issue 2: Who Makes Eligibility Decisions?

Issue 3: How Do We Maintain a Unified Educational System?

Issue 4: How Do We Avoid Overuse of Alternate Assessments?

Issue 5: How Do We Assess in Reliable and Valid Ways?

Issue 6: Should Data be Aggregated or Reported Separately?

Issue 7: How Should Results be Communicated?

Issue 8: Linkage: Do We Test What We Teach, and Do Students Have an Opportunity to Learn What We Test?

Issue 9: What are the Cost Benefits?

Issue 10: How Will Data be Collected?

Issue 11: What Kind(s) of Data are to be Collected?

Issue 12: How Do We Maintain Confidentiality?

Issue 13: How Do We Ensure that School Personnel are Trained to Administer, Score, Interpret and Use the Results of Alternate Tests?

Alternate Assessments in Accountability

National, state, and local educational personnel increasingly are engaged in efforts to document the extent to which education is working for students. State and national legislatures have passed laws, rules or guidelines requiring that states set educational standards and assess the extent to which students are making progress toward those standards. In an increasing number of states students must pass tests in order to receive a high school diploma (Bond, Braskamp, & Roeber, 1996; Thurlow, Ysseldyke, & Anderson, 1995). The 1990s clearly is a decade of accountability in education, and at least part of the information used to demonstrate accountability comes from assessments.

It is important at the outset of this paper to distinguish between assessment and accountability. Assessment is a process of collecting information, one form of which may be test information, for the purpose of making decisions about students. Accountability is a system activity designed to assure those inside and outside the educational system that schools are moving in desired directions (Center for Policy Options, 1994). Information obtained from assessments typically is used to demonstrate accountability. Accountability involves reaching consensus on clear goals for an organization or person, reliably assessing achievement toward those goals, and attaching consequences to the success or failure to achieve goals.

A major challenge in education is to demonstrate accountability for all students. National and state education legislation (e.g., Goals 2000, Improving America’s Schools Act, and the pending versions of the Individuals with Disabilities Education Act) includes language specifying that accountability applies to all students. Educational goals and standards are for all students. States and school districts are to report on the performance and progress of all of their students. “All” includes students with disabilities and students with limited English proficiency. About 85% of students with disabilities have relatively mild or moderate disabilities and can take state and national large scale assessments, either with or without accommodations (like large print, testing in a separate setting, or extended time). Yet, there is a group of students with disabilities for whom current tests are inappropriate, and who, therefore, are excluded from district, state and national assessments. These typically are students with severe cognitive delays or multiple disabilities or students who require such significant accommodations that the validity of the test would be compromised. In this paper, we address issues and considerations in assessing this group of students. These students can be assessed through a practice that has become known as “alternate assessment.” Alternate assessments are used when students do not “fit” within the regular assessment program, or when the tests typically used do not “fit” a segment of the school population.

What is Alternate Assessment?

In Table 1, we show the three kinds of tests used with students with disabilities: general state tests, general state tests with accommodations, and alternate assessments. We also provide estimates of percentages of students expected to participate in each kind of test. An alternate assessment is a substitute way of gathering information on the performance and progress of students who do not participate in the typical state assessments used with the majority of students who attend schools. Typical state assessments involve use of criterion-referenced assessments, standardized norm-referenced multiple choice tests (like the Iowa Tests of Basic Skills), performance tests, and sometimes portfolios. Alternate assessments can be the same kinds of assessments (e.g., performance measures or portfolios), but they probably differ in format, content, or level from the assessments that are used with the majority of students. We have chosen to use the term alternate assessment rather than alternative assessment. The term alternative assessment has multiple meanings, and anything that is an alternative to a standardized multiple choice test has become known as an “alternative.” So, throughout this paper we use the term alternate assessment.


Table 1. Three Kinds of Assessments for Students with Disabilities

Examples of Alternate Assessment Practices

Only two states currently have alternate assessment programs: Maryland and Kentucky. Personnel in Texas are developing a blueprint for an alternate assessment, and are expected to complete it in early 1997. Many other states are exploring a variety of options to begin development of such a program.

Maryland
Maryland has several kinds of tests within its assessment programs. The Maryland Functional Testing Program includes four basic minimum competency tests: three multiple choice tests in reading, mathematics and citizenship, and a modified-holistically scored direct writing assessment of both narrative and explanatory writing skills. All of the tests are untimed. The Maryland Functional Testing Program (MFT) is a graduation requirement program that now is required to be completed by the end of eighth grade. A high school test is being developed in the same content areas to replace the MFT, but to reflect higher standards. Students will be required to pass this test in order to graduate.

The Maryland School Performance Assessment Program (MSPAP), currently administered at the elementary and middle school levels, measures higher order thinking processes and the application of knowledge and skills to real world situations. It is a single test covering mathematics, reading, writing, science, language usage, and social studies.

Maryland’s performance assessment for populations with severe disabilities is called the Independence Mastery Assessment Program (IMAP). IMAP is currently being field tested in approximately one-third of the school systems in Maryland. It reflects an alternative set of educational outcomes, more life-skills oriented, that were identified and developed by the state under the direction of the IMAP Advisory Committee (see Table 2). Students with severe disabilities participate in either MSPAP or IMAP, depending on which outcomes they are pursuing. In the pilot districts, students with disabilities participate either in (a) MSPAP with no accommodations, (b) MSPAP with accommodations, or (c) IMAP.

Table 2. Examples of Maryland’s Content Domain Outcomes for Students in Independence Mastery Assessment Program

Kentucky
Until the current year, four types of assessment tasks comprised the Kentucky Assessment System: assessment tasks involving portfolios, assessment tasks involving performance events, assessment tasks involving open-ended questions, and assessment tasks involving machine-scorable questions. Use of performance event scores for accountability was dropped during the 1997 school year because of questions about the reliability of these scores. Kentucky also monitors school progress in terms of non-cognitive indicators such as school attendance rates, dropout and retention rates, reduction in physical and mental health barriers to learning, and the proportion of students who make a successful transition to work, postsecondary education, or the military.

Kentucky has the Alternate Portfolio Assessment for use with students who have severe cognitive disabilities. The following are key concepts of the Alternate Portfolio:

• An Alternate Portfolio Advisory Committee, charged with the task of identifying the Academic Expectations to be assessed within the Alternate Portfolio, first looked at the critical functions of each of the Academic Expectations, and determined the extent to which each could be demonstrated by students eligible to participate in the Alternate Portfolio Process. Twenty-eight academic expectations were identified as critical (see Table 3 for examples of some of these expectations and indicators of them).
• Scores of students participating in the Alternate Portfolio are weighted equally with those of students participating in the regular assessment for the school’s accountability purposes.
• The student’s portfolio must include seven to ten entries related to the state’s academic expectations, and included among the entries must be:
  • a student activity schedule or routine;
  • a resume of job experiences; and
  • a sample of the student’s present mode(s) of communication.

Table 3. Examples of Kentucky’s Academic Expectations for Students in the Alternate Portfolio System

Issues in Alternate Assessment

Many groups and individuals engaged in state and national assessments have raised issues about alternate assessments. We have derived the list of issues described below from multiple forums with diverse sets of school personnel; focus groups with school, community, legislators and business people; interviews; meetings with working groups; feedback at workshops and conferences; and our own experience.

Issue 1: Eligibility–Who Takes Alternate Assessments?
Decisions about who participates in alternate assessments have significant ramifications. Some contend that alternate assessments are for students who cannot participate in typical state tests. Others contend that alternate assessments are for students who are working toward meeting a set of separate standards. Still others contend that alternate assessments are intended for students who are not working toward a “real” high school diploma. And, still others contend that alternate assessments are only for students who are working toward separate standards and who are not working toward a typical high school diploma. Therefore, the establishment of criteria for inclusion in an alternate assessment has implications for diploma awarding, graduation, and even for definitions of the purpose of education.

The complexity of this issue is enhanced by the fact that standards are specified at various levels. When states specify broad standards, like “students who complete school will solve math problems with sufficient accuracy to be successful in their next environment,” there are multiple levels of expected performance, and multiple ways in which the standard can be assessed. When states specify narrow standards, like “All students will complete school with working knowledge of mathematics three years beyond ninth grade algebra,” there is one level of expected performance, and limited ways in which to assess the extent to which it has been met. Resolution of this issue is fundamental. School personnel must decide whether there will be more than one set of standards, decide how broad those standards will be, and then decide the extent to which students will work toward separate standards. How this decision relates to the type of diploma earned must be determined as well.

The way in which the issue of who takes the alternate assessment is resolved will ultimately affect many of the other issues stated below. For example, if assessment personnel take the position that alternate assessments are for students who cannot take typical state assessments, how will potential overuse of the alternate assessment be limited? If, on the other hand, alternate assessments are limited to students who are not working toward the same standards as other students, then how do we produce an alternate assessment that provides valid and reliable scores that can be used in an accountability system?

Issue 2: Who Makes Eligibility Decisions?
Generally, it is thought that individuals who know a student well should decide in which kinds of assessments the student will participate. In many states and districts, this decision is made by IEP teams, while, in others, the decision is made by teachers or building principals. Should IEP teams decide who participates in what kinds of assessments? If so, how much training or re-training will be needed to enable them to make appropriate decisions? Should parents decide whether their children will take tests and the kinds of tests they will take?

Issue 3: How Do We Maintain a Unified Educational System?
If school personnel either hold students with disabilities responsible for meeting separate standards, or assess them using a separate set of tests, they run the risk of creating or enhancing separate (often set-aside) rather than unified educational structures. When separate groups (even categorical groups) are provided educational opportunities directed toward meeting separate goals or standards, we end up with special education for students with disabilities, Title I education, bilingual education, technical education, education for students at risk, and “general” or “regular” education. Where does the fractionation of the educational system end? In an era in which there is a tremendous push for unified educational systems, such separatism typically is discouraged.

Issue 4: How Do We Avoid Overuse of Alternate Assessments?
Decisions must be made about who participates in typical assessments and who participates in an alternate assessment. One danger in having available an alternate assessment is that it may encourage school personnel to have large numbers of students participate in the alternate assessment, thereby lowering rates of participation in typical assessments. This danger is especially relevant if there are less stringent consequences for the schools when students take the alternate assessment. It is expected that few students (between one-half of one percent and two percent of the school age population) will need alternate assessments. Most professionals agree that the decision about participation should not be based on expectations of poor performance on the general education assessment. In Kentucky and Maryland, students are considered eligible for alternate assessments when they are working on a different curriculum from most students, one that may not lead to a diploma. And, of course, the breadth of the standards toward which students are working will often help to define the kind of assessment system in which they participate (see Issue 1).

Issue 5: How Do We Assess in Reliable and Valid Ways?
Clearly, it is very difficult to develop technically adequate measures of skill development for some kinds of students in some domains. For example, it is difficult to get reliable and valid indices of academic and functional literacy for students with severe cognitive or communication impairments, students with multiple disabilities, or students who refuse to respond to tests, tasks, or interviews. It also may be difficult (and this is true for both typical and alternate assessment) to get technically adequate indices of student development in the domains of citizenship and contribution, responsibility and independence, and social-emotional functioning.

Issue 6: Should Data be Aggregated or Reported Separately?
Assessors regularly argue about the extent to which it is wise to aggregate data across multiple assessment systems (e.g., typical state assessments and alternate assessments), including multiple alternate assessment systems. In the recent past, there were many educators who believed firmly that students with disabilities should not be part of accountability systems. Today, most agree with the contention that accountability systems should be inclusive. The debate now seems more focused on how to report than on who to assess. And, some contend that data should be reported in the aggregate rather than separately.

Because nearly all students who participate in alternate assessments will do so because they are not working toward general education goals or standards, some might assume that all assessments of these students will be individual appraisals of the extent to which they are achieving IEP objectives. The primary difficulty with this approach is that IEP goals are not equivalent across students, and attainment of IEP goals cannot easily be aggregated for accountability purposes. There are common domains of goals or a common core of learning that all students who are unable to participate in the general education assessment still need to be working toward. The state must work to link the alternate assessment to this common core.

Issue 7: How Should Results be Communicated?
When states implement both typical tests and alternate tests as parts of their accountability systems, clear communication of the results is a major challenge. The challenge is intensified when some students take typical tests with accommodations. Assessment personnel must be able to communicate specifically who took what kinds of tests under what kinds of conditions, and give clear information about the level of performance of the students.

Issue 8: Linkage: Do We Test What We Teach, and Do Students Have an Opportunity to Learn What We Test?
Curriculum alignment is an important issue for all kinds of assessments used for educational accountability. It is critical that educators test what they teach. Given the diversity of curricula in which students are enrolled, a major challenge is one of making certain that what is tested is actually taught. Students with severe disabilities are often enrolled in curricula for which the goals are attainment of life skills, such as self-care, mobility and development of positive social interactions. It is important that alternate assessments include these areas of curricular content. Defining these areas of the curriculum probably has already occurred, but may need to be revisited as an alternate assessment is developed.

Students must also have an opportunity to learn and be tested on the outcomes of schooling that are valued for all students. Balance is critical here. Teachers in Kentucky report that statewide testing pushed them to focus on key intended outcomes (essential elements) of instruction. We do not want to return to the days of overly limited expectations for students with severe disabilities. An alternate assessment system should be designed to assess achievement toward pre-determined standards. Preferably, these would be the same standards measured by typical assessments. In any event, alternate assessments should measure high standards and target the goals for the student.

Issue 9: What are the Cost Benefits?
States incur considerable costs when they develop or revise their general assessment programs. Likewise, costs will be incurred when they develop alternate assessments for students. The cost per child can be very high given the low incidence of some disabilities. It is not expedient for states to construct separate tests for each kind of student with a disability, yet the very purpose of the alternate assessment is to develop more appropriate measures. The state must strike a balance among development cost, time, and responsiveness to idiosyncrasies.

It can take considerable time to develop/design alternate assessments. It also takes time to administer the assessments. Efforts must be made to keep time commitments reasonably low.

Issue 10: How Will Data be Collected?
Assessors must decide how they will gather information on student achievement and development. Ysseldyke and Olsen (1996) have described many ways in which data may be collected, relying on the methodologies described earlier by Salvia and Ysseldyke (1995). Data may be gathered through observation, recollection (interview, completion of checklists, self-report), testing, and record review. Assessors must make choices among these options, trying to balance cost, time, and reliability/validity. For example, use of multiple measures (e.g., performance event and parent interview) will increase reliability, but will also increase cost.

Issue 11: What Kind(s) of Data are to be Collected?
Assessment is a process of collecting data for the purpose of making decisions about students and/or schools. Assessment personnel regularly debate the kinds of data to be collected. To the extent that standards are broad and can encompass all students, then it makes sense to measure the same content in the alternate assessment as is measured in typical assessments. It is necessary to modify how information is collected, but not what is collected. Thus, for example, data might be collected on mastery of science goals by means of an alternate portfolio rather than by means of performance events.

Issue 12: How Do We Maintain Confidentiality?
Alternate assessments typically are given to only a few students. While not unique to alternate assessment, confidentiality is an issue when the results of the alternate assessment are reported. Because there typically are only a few students in a district or region who take alternate assessments, their individual performances could be easily identifiable. The state will have to find ways of public reporting that make it difficult to track performance to individual students.

Issue 13: How Do We Ensure that School Personnel are Trained to Administer, Score, Interpret and Use the Results of Alternate Tests?
Alternate assessments involve assessment practices and new ways of collecting data for which school personnel have limited training. They require new training designed to bring people up to speed with the assessment practices, and training in curriculum and instructional approaches designed to achieve outcomes specified and measured by alternate tests. Providing this training can be costly and time consuming, yet not providing training runs the risk of long term negative effects. Kentucky and Maryland have elected to involve their teachers of students with disabilities in development and scoring of the assessments. This involvement has had the benefit of merging training, development, and implementation in a cost-effective manner.

Summary

National and most state legislation includes language specifying that standards, assessments, and accountability systems are for all students, including students with disabilities. States face significant challenges in their efforts to develop fully inclusive accountability systems. It is clear that alternate assessments will be needed for a small number (probably less than one percent) of students. States considering alternate assessments must take into account the 13 issues specified in this paper. The content, magnitude, and method of alternate assessment and the ways in which data are reported and used will have a significant effect on states’ development or maintenance of inclusive educational accountability systems. Failure to adequately address these issues could lead to standards for some, restricted policies, separate educational systems, divided accountability systems, and diminished educational outcomes for a large number of students.

References

Bond, L., Braskamp, D., & Roeber, E. (1996). The status report of assessment programs in the United States. Oak Brook, IL: North Central Regional Education Laboratory.

Center for Policy Options. (1994). Issues and options in outcomes-based accountability for students with disabilities. College Park, MD: Center for Policy Options in Special Education.

Salvia, J. & Ysseldyke, J. (1995). Assessment (5th ed.). Boston: Houghton Mifflin.

Thurlow, M.L., Ysseldyke, J.E., & Anderson, C. (1995). High school graduation requirements: What’s happening for students with disabilities? (Synthesis Report 20). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J.E. & Olsen, K. (1997). Putting alternate assessments into practice: Possible sources of data (Synthesis Report 28). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Top of page