Accountability for the Results of Educating Students with Disabilities

Assessment Conference Report on the New Assessment Provisions of the 1997 Amendments to the Individuals with Disabilities Education Act

Published by the National Center on Educational Outcomes in collaboration with:
Council of Chief State School Officers (CCSO)
National Association of State Directors of Special Education (NASDSE)

by James E. Ysseldyke and Martha L. Thurlow (NCEO); Elizabeth Kozleski (University of Colorado); Daniel Reschly (Iowa State University)

April, 1998

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Ysseldyke, J. E., Thurlow, M. L., Kozleski, E., & Reschly, D. (1998). Accountability for the results of educating students with disabilities: Assessment conference report on the new assessment provisions of the 1997 admendments to the Individuals with Disabilities Education Act. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://cehd.umn.edu/NCEO/OnlinePubs/awgfinal.html

Foreward

On June 4, 1997, President Clinton signed into law the Individuals with Disabilities Education Act Amendments of 1997 (P.L. 105-17). This new law marked a major milestone in the education of children with disabilities, characterized by a shift in emphasis from "access to education" to "improving results." The law recognized that high expectations, access to the general curriculum, and public reporting and accountability were key elements in improving results for these children. Thus, the law introduced new requirements to complement such previous requirements as "free appropriate public education," "individualized educational programs (IEP)," and "least restrictive environment," which have reshaped special education in the past two decades.

New requirements call for states to develop and report on performance goals and indicators, including the performance of students with disabilities on assessments. They also call for students with disabilities to be included in general State and district-wide assessments, with appropriate accommodations where necessary. Alternate assessments are required for students who cannot participate in the State or district-wide assessments. State education agencies are required to report on the participation and performance of students with disabilities on regular and alternate assessments. And, IEPs are required to include statements about individual modifications to State or district-wide assessments, or participation in alternate assessments.

Obviously, a number of challenges must be faced if the benefits of these new requirements are to be achieved. Recognizing these challenges, the U.S. Department of Education’s Office of Special Education and Rehabilitative Services (OSERS), and its component Office of Special Education Programs (OSEP) requested that the National Center on Educational Outcomes (NCEO), a project funded by OSEP, convene a working conference in Washington DC to define issues and develop recommendations related to State and district-wide assessments and accountability. The following report describes the findings of this conference.

The conference took place on January 7 through 9, 1998. The Department of Education had published for public comment, October 22, 1997, a set of proposed regulations to implement the new law, and the comment period was still in effect during the working conference. However, while the conference participants knew about the proposed regulations and discussed them, the meeting was not intended to respond to or propose revisions to the regulations.

This report contains a body of insightful and useful information contributed by the conference participants, and it will be an important resource to inform decision making and planning in the U.S. Department of Education. It is being distributed outside the Department not as an expression of Departmental policy or plans, which it is not, but rather as a resource for other organizations and individuals with an interest in these issues, in the hope that it will contribute to the sorts of collaborative endeavors and informed public discourse that are needed to attain the benefits of the new IDEA amendments and achieve better results for children with disabilities.

Executive Summary

The 1997 amendments to the Individuals with Disabilities Education Act (IDEA) reflect a concern about the standards to which students with disabilities are held, and about the extent to which they participate in state and district assessments, the primary means that education has used to demonstrate educational results. The assessment provisions in the amendments specifically require that students with disabilities be included in state and district assessment programs, with accommodations where necessary, and that their scores on these assessments be reported in the same detail and with the same frequency as the scores of other students. The amendments also require that states develop alternate assessments for those students unable to participate in the regular state or district assessment programs, that guidelines be developed for determining who should be included in alternate assessments, and that the performance of students taking alternate assessments be reported.

While the assessment provisions have been applauded by policymakers, they produce many challenges for states and districts. It is important to take advantage of the considerable expertise that does exist and that might help inform the next steps to be taken—to look at the challenges and generate recommendations for the future to ensure that the letter and intent of the law are met. A meeting was held January 7-9, 1998 to bring together a diverse group of knowledgeable individuals to review the provisions, and then to identify key issues and make recommendations for assessment practices, research and development, technical assistance, professional development, and monitoring.

Sixteen critical issues were identified as ones that will affect the extent to which the letter and intent of the assessment provisions in the 1997 amendments to IDEA are met. These issues fall within three broad areas as follows:

1.  Lack of Consistency or Consensus

• Definitional confusion

• Confusion about the requirements of the law

• Absence of consensus on a conceptual framework

• Variability among states and districts within states

• Competing conceptions of accountability

• Lack of broad stakeholder involvement

• Lack of consensus about standards for students with disabilities

• Divergent purposes

2.  Assessment Design and Administration

• Lack of knowledge about assessment

• Lack of participation of students with disabilities in test development and standardization

• Questions about validity of accommodations

• Questions about validity of alternate assessments

• Lack of data on how decisions are made about participation and accommodations

• Differential higher education training

• Consequences of IDEA assessment provisions for students and systems

• Compliance for rights versus compliance for outcomes

Recommendations

Recommendations were developed in the areas of: (1) assessment practices, (2) research and development, (3) technical assistance, (4) professional development, and (5) monitoring.

Assessment Practices

Assessment practices discussed here are limited to those related to large-scale assessments (e.g., state and district assessments) and alternate assessments. Assessments used to determine eligibility for special education services are not the focus of these recommendations. Eight recommendations are proposed for assessment practices:

AP-1 Develop clear definitions and guidelines.
AP-2 Establish expert panel.
AP-3 Collect test data by age not grade.
AP-4 Track consequences of 1997 IDEA assessment provisions.
AP-5 Hold forum to promote participation during assessment development.
AP-6 Describe workable accountability systems.
AP-7 Enlist public support for new requirements.
AP-8 Promote state leadership.

Research and Development

The research and development activities addressed include all types of research (e.g., empirical, policy, experimental) and related development activities. Three recommendations are proposed for research and development activities:

RD-1 Develop collaborative research efforts.
RD-2 Develop model demonstration projects for alternate assessments.
RD-3 Conduct research on accommodations, alternate assessments, and related topics.

Technical Assistance

Technical assistance is defined to include the provision of system-level training and supports designed to enable the system to conduct its activities in the area of concern (e.g., accountability assessments, alternate assessments, etc.). Seven recommendations are proposed for technical assistance activities:

TA-1 Create a technical assistance planning team.
TA-2 Document and evaluate current assessment efforts.
TA-3 Develop technical assistance materials.
TA-4 Create a forum for information dissemination.
TA-5 Establish state teams and provide training on alternate assessments.
TA-6 Establish a single point of contact for alternate assessment information.
TA-7 Provide a cadre of experts to provide startup technical assistance to states.

Professional Development

Professional development refers to training provided to individuals (or groups of individuals), both preservice and inservice. There are many potential audiences for professional development activities, including school-based staff (teachers and related services personnel), policymakers at the federal, state, and local levels, state and district administrators, test developers, and faculty in institutions of higher education (IHEs). All play key roles in supporting and implementing IDEA, but the kind and amount of information disseminated to each audience needs to be tailored to the needs of that audience. Faculty members in IHEs (both general and special education) are considered to have the greatest need for professional development, reflecting the pivotal role they have in preparing educational professionals of all kinds, coupled with the concern that many faculty members who train educational professionals are not up to date on the law and on accountability and assessment systems as they involve students with disabilities. In reality, however, all of the potential audiences have substantial needs for professional development on accountability and assessment systems.

Seven recommendations are proposed for professional development activities:

PD-1 Require competencies in large-scale assessment.
PD-2 Develop core training materials that allow for adaptations.
PD-3 Develop a coordinated professional development plan.
PD-4 Target IEP teams for immediate training.
PD-5 Increase OSERS voice on the U.S. Department of Education’s initiative team for Goal Five on well-trained teachers.
PD-6 Develop and disseminate information for staff in parent organizations.
PD-7 Establish a federal priority for funding on alternate assessments.

Monitoring

Compliance monitoring includes those activities undertaken by the federal education agency with state education agencies, or by states with their local education agencies, to ensure that the letter and intent of special education law are being met. Seven recommendations are proposed for monitoring activities related to the assessment provisions of the law:

M-1 Redefine the purpose of monitoring.
M-2 Integrate NASDSE and NCEO models to guide monitoring efforts.
M-3 Monitor for consistency, comprehensiveness, and progress on state improvement plans.
M-4 Use a standard formula to analyze participation.
M-5 Change the consequences of monitoring.
M-6 Involve stakeholders in monitoring process.
M-7 Monitor goal alignment for students taking the alternate assessment.

Next Steps

These recommendations provide a basis for action by numerous groups. Initially, the recommendations are intended for use by the U.S. Department of Education, to plan ways to ensure that the letter and intent of the new assessment provisions in the 1997 reauthorization of IDEA are met. Beyond this, the recommendations (and background issues) are useful to states, districts, and their constituencies (including administrators, teachers, parents, and the general community) as they think about what it takes to implement the new requirements.

A good follow-up activity to the January 7–9 meeting, at which stakeholders generated the issues and recommendations, is to prioritize the recommendations. This could be accomplished within the U.S. Department of Education, or by involving a broader group of stakeholders. The prioritized recommendations then could be used as the basis for developing action plans that define specific avenues to be followed for meeting the letter and intent of the assessment provisions in the 1997 amendments to IDEA.

Overview

In 1975, Public Law 94-142 gave school-age youngsters with disabilities the right to a free and appropriate public education. Before the passage of the law, children often were relegated to institutions or they stayed at home because schools were not required to accept them.

It has been more than 20 years since that law changed the lives of youngsters with disabilities, their families, and the communities in which they live. In fact, over time many more students have been identified as having disabilities and needing special education services. And, the importance of providing services within general education settings to the maximum extent possible has led to the recognition of the importance of participation in the general education curriculum for most students with disabilities. Adherence to the law has been monitored over the years by focusing on the numbers of students with disabilities, their educational placements, and the extent to which due process procedures assured by the law have been followed.

Within the past decade, numerous voices have expressed the concern that it is time to attend to more than just whether students with disabilities are receiving a free and appropriate education. These voices argue that it is time to look at the results of the students’ education. This message has been expressed by the disability community and by legislators who are also responding to the need to justify the costs of special education with evidence of concrete results for students and for the communities in which they live.

On June 4, 1997, President Clinton signed into law amendments to the Individuals with Disabilities Education Act (IDEA) that respond in many way to these voices. The 1997 amendments reflect a concern about the standards to which students with disabilities are held, and about the extent to which they participate in state and district assessments, the primary means that education has used to demonstrate educational results. The assessment provisions in the amendments specifically require that students with disabilities be included in state and district assessment programs, with accommodations where necessary, and that their scores on these assessments be reported in the same detail and with the same frequency as the scores of other students.

The amendments also require that states develop alternate assessments for those students unable to participate in the regular state or district assessment programs, that guidelines be developed for determining who should be included in alternate assessments, and that the performance of students taking alternate assessments be reported.

Among the reasons for the assessment provisions in the 1997 amendments were findings that students with disabilities were being excluded from many state and district assessments, and that there were significant negative consequences of this exclusion, both for the system and for the individual student. For example, exclusion from the assessment system often resulted in students being excluded from the curriculum or from reform initiatives designed to improve student performance. Exclusion from system-level high stakes assessments (e.g., tests that determine whether schools will receive rewards or be reconstituted) also resulted in increases in the rates of referral to special education.

Perhaps the primary reason for concern about the exclusion of students with disabilities from state and district assessments was the lack of accountability for the results of education for these students. Intentional exclusion of students, either from testing or from reporting, meant that there were no data available on the results of education for students with disabilities.

The assessment provisions in the 1997 amendments have been applauded by policymakers. Yet, the provisions produce many challenges for states and districts. It is important to take advantage of the considerable expertise that does exist and that might be relied on in determining the next steps to be taken—to look at the challenges and generate recommendations for the future to ensure that the letter and intent of the law are met.

One way to take advantage of the expertise that exists is to bring knowledgeable individuals together to review the provisions, and then to identify key issues and make recommendations related to assessment practices, research and development, technical assistance, professional development, and monitoring. A meeting to do just that was held January 7-9, 1998 in Washington, DC. It included 44 people from various stakeholder groups, including state assessment directors, state directors of special education, researchers, technical assistance providers, parents, representatives of educational advocacy groups and professional organizations, and U.S. Department of Education staff (see Appendix A for lists of meeting participants and others who provided critical reviews of this report). The meeting was convened by the National Center on Educational Outcomes (NCEO) in coordination with the Council of Chief State School Officers (CCSSO) and the National Association of State Directors of Special Education (NASDSE). The meeting agenda and procedures used during the meeting to ensure complete coverage of the issues and generation of recommendations are summarized in Appendix B.

Critical Issues

There are numerous issues that surround the assessment provisions included in the 1997 IDEA amendments. We believe that 16 critical issues will affect the extent to which the letter and intent of the law are met. These issues fall within three broad areas as follows:

1.  Lack of Consistency or Consensus

• Definitional confusion

• Confusion about the requirements of the law

• Absence of consensus on a conceptual framework

• Variability among states and districts within states

• Competing conceptions of accountability

• Lack of broad stakeholder involvement

• Lack of consensus about standards for students with disabilities

• Divergent purposes

2.  Assessment Design and Administration

• Lack of knowledge about assessment

• Lack of participation of students with disabilities in test development and standardization

• Questions about validity of accommodations

• Questions about validity of alternate assessments

• Lack of data on how decisions are made about participation and accommodations

• Differential higher education training

3.  Consequences

• Consequences of IDEA assessment provisions for students and systems.

• Compliance for rights versus compliance for outcomes

Each of these issues is discussed in brief.

Lack of Consistency or Consensus

Definitional Confusion

There is considerable confusion about the meaning of many of the terms used in the law. Different people have different interpretations of what the terms mean, and states define the terms differently in their official policy documents as well as in less formal communication. In fact, communication during the meeting was hindered by definitional differences.

There is a pressing need to define and differentiate terms such as accountability systems, assessment systems, alternate assessments, disaggregated scores, accommodations and modifications. Clear understanding of terms will aid practice as well as improve monitoring, technical assistance, and research efforts.

Confusion about the Requirements of the Law

There is much confusion among policymakers and practitioners about the requirements of the assessment provisions. For example, there is confusion about the requirement that students with disabilities participate in state and district-wide assessment programs. Are students required to participate in district-wide assessments only when a state does not have a state assessment? Also, must students with disabilities participate in all assessments given by states (some states have as many as six different state-run assessment programs), or does participation in a single assessment meet the letter and intent of the law?

There is confusion about eligibility for alternate assessments. In general, it is agreed that alternate assessments are an option to be made available only to students with the most significant support needs. Both the Assistant Secretary of the Office of Special Education and Rehabilitative Services and the Assistant Secretary of the Office of Civil Rights have indicated in a public letter (September 29, 1997) that the alternate assessment is for "the small number of students whose IEPs specify that they should be excluded from regular assessments, including some students with significant cognitive disabilities." Yet, the issue of who will be allowed to take alternate assessments is the beginning point for all discussions about alternate assessment. The discussion is linked integrally to the discussion of the extent to which students will need accommodations in typical assessments. The greater the band of accommodation flexibility in typical assessments (recognizing that most, but not all, accommodations produce valid scores), the smaller the number of individuals who will be involved in an alternate assessment system.

Reporting provisions in the law also are a source of confusion. For example, when is it statistically unsound to report on the performance of students with disabilities? How many students are required to protect the anonymity of students?

Lack of clarity about these matters will impede implementation of the law, and will impede implementation of inclusive accountability systems, monitoring, technical assistance, and research efforts. Confusion could lead to exaggerations and distortions of legal requirements that could, in turn, lead to diminished support among educators and the general public for education of students with disabilities. The concern is that substantial agreement about the requirements of the law is needed. Once agreement is achieved, it should serve as the foundation for massive professional development at the state and local levels.

Absence of Consensus on a Conceptual Framework

An overriding issue in the design of inclusive assessment and accountability systems is the absence of consensus on a conceptual framework to guide the process. This is the case for accountability systems in general, and especially true in efforts to design alternate assessment systems. There are only three or four state models of alternate assessments, with full implementation occurring in only one state. State assessment and accountability systems vary considerably, and there is little consensus both on what ought to be measured and how students with disabilities should be included and reported on in the systems.

A conceptual framework would include specification of outcomes along with input and process indicators, and potential sources of data on the indicators. And, the design of both typical and alternate assessments should be grounded in a conceptual framework that entails attainment of essential outcomes (content standards) and high expectations (performance standards) for all students. When thinking about the development of a conceptual framework, it is important to give considerable thought both to standards and the purposes of assessment.

Variability Among States and Districts within States

States are at very different points in the development and implementation of their assessment and accountability systems. Some have much farther to go than others to achieve compliance with the law. There may be a need for differentiated or tiered technical assistance, depending on where states are in the development and implementation process. There also may be a need to use different monitoring standards, depending on a state’s or district’s point in the development and implementation process.

Competing Conceptions of Accountability

Different states operate with different conceptions of accountability. These, in turn, influence perceptions about the feasibility of including students with disabilities in accountability systems. For example, one approach to accountability sets specific terminal goals that are to be reached at specific times (e.g., 8th grade, end of school), and expects variable improvement in performance over baseline. Schools that start at different points must make different rates of progress to be on target toward meeting the terminal goals.

A second approach requires a specific percentage of improvement (e.g., 10%) over baseline each year; all schools must make the same rate of improvement, regardless of where they start. Schools that start at different points and make the same rates of improvement end at different points.

A third approach is to exempt from requirements for improvement those schools or districts already reaching the target goals; this approach recognizes the difficulty of having 100% of the students pass any given assessment.

The participation of students with disabilities in accountability systems may be viewed as easier in the second and third approaches than in the first approach. The approach that a state adopts may inherently encourage or discourage commitment to the participation of students with disabilities.

Lack of Broad Stakeholder Involvement

The limited involvement of representative stakeholders in the conceptualization, design, development, implementation, use, and evaluation stages of assessment and accountability systems is a concern. Trusting relationships between schools and families often are lacking, and concerns exist that performance data will help only select groups.

Lack of Consensus about Standards for Students with Disabilities

One of the purposes of assessment and accountability systems is to provide data about the quality and impact of educational services on the student/consumer. Without some common expectations for content and performance standards, it is difficult to design a methodology to measure and compare results. Comparison probably requires comparable curricular standards.

In special education the notion of individualized education plans and programs has meant that there is little consensus about a set of curricular expectations for students with disabilities at the school, district, state, or national levels. Are there alternate standards, modified standards, or expanded standards for students with disabilities? Should there be? If so, what do these look like and for whom are they appropriate? To date, few people with expertise in educating students with disabilities have been involved in the formulation of standards, or in extending or expanding standards that will serve as the basis for alternate assessments.

Divergent Purposes

The purposes of district and state assessment systems are varied. Assessment for accountability purposes often is not differentiated from assessment of educational growth. Further, accountability systems may serve multiple agendas. For instance, accountability systems that are used to determine graduation from public K-12 education have much higher stakes for individuals than accountability systems that primarily inform policymakers at local, state, and national levels. In many states, practitioners use the same assessments for multiple purposes. The degree to which the design, purpose and use of alternate assessments mirrors the design, purpose and use of typical assessments has consequences for use of data in making policy decisions. To the extent that there is variation in design, purpose and use, there may be differential treatment of students who participate in the two systems.

Teams of educators and related services personnel have used assessment to decide who is eligible to receive special education services. Eligibility and placement decisions involve identifying specific conditions (like mental retardation or deafness) and the degree of educational need. Special educators also have used tests for determining the instructional needs of individual students. In contrast, general educators have focused on using the results of assessment for the purposes of system accountability and system-level improvement. The groups of people who are going to have to work together on implementation of assessment and accountability systems (including alternate assessments) have a tradition of viewing and talking about assessment in different ways.

Assessment Design and Administration

Lack of Knowledge about Assessment

Many of those who make decisions about implementation of accountability systems, who educate teachers, and who monitor systems have limited knowledge about assessment or about the relationships among assessment, use of data, and instruction. This includes state and federal policymakers, general and special education teachers and administrators, faculty in personnel preparation institutions, and parents.

Knowledge about assessment includes knowledge about norming, technical adequacy (reliability, validity, and scaling), accommodations, data aggregation, test development and standardization, and formats for reporting. In some cases the knowledge base is inadequate and research findings are needed; in other instances the knowledge is available, but not well disseminated or applied to practitioners.

Many state and district assessment personnel lack the knowledge needed to implement alternate assessments. They lack knowledge in all areas, from setting guidelines about who participates in alternate assessments to actually implementing the assessments. Knowledge also is lacking on how to gather, score, aggregate, interpret and report data from alternate assessments.

There currently is an absence of a broad approach to technical assistance and professional development. Training tends to occur at points or levels of testing (e.g., 4th and 8th grade) rather than for all personnel in the system. It also tends to focus on a single area of concern, such as training in team decision making, or how to use the results of tests, rather than taking a comprehensive approach.

Lack of Participation of Students with Disabilities in Test Development and Standardization

There are many concerns about how tests are developed and standardized. For example, tests often include very restricted ranges of skills; many students have skills outside the range assessed by the test. The issue sometimes is referred to as the lack of a basal, or as limited behavior sampling. In some states this issue is being addressed by having students take out-of-level tests, even though this approach is usually thought to be inconsistent with a standards-based philosophy.

Other concerns include the exclusion of students with disabilities from norming groups, and the failure of test developers to administer tests with accommodations when they standardize their measures. Exclusion from norming often means that users must interpolate to get indices of performance. Failure to include accommodations during the standardization process produces inappropriate norm groups and the exclusion of many students from the groups using assessments after they have been standardized.

Questions about Validity of Accommodations

There is clear and considerable concern about the extent to which the provision of accommodations to students with disabilities affects the technical adequacy of tests, and about whether the provision of accommodations changes what is measured. There is discussion about whether accommodations provide students with disabilities with an unfair advantage during testing. State data are perceived by some as "more valid" if students with disabilities are excluded or are not given accommodations than if they are included or provided with accommodations. Yet others question the validity of assessments given without including students with disabilities, or given without providing accommodations for those students with disabilities needing them.

Specific issues arise for each disability type, or combinations of disabilities, and for each specific accommodation. There is considerably more rhetoric and opinion than sound empirical evidence about the validity of specific accommodations. The knowledge base about the effects of accommodations is not adequate to address many practical, everyday questions, nor is it in a form that is readily accessible to or easily understood by personnel in states and districts.

The congruence between instructional accommodations and assessment accommodations is another important validity consideration. Anecdotal evidence suggests that assessment accommodations often are out of sync with accommodations provided in instruction. This occurs when students experience accommodations for the first time when they take a test. It also occurs when the accommodations typically provided during instruction are not available during assessment. There are also questions about the appropriateness of certain accommodations for assessment even though they might be used during instruction (such as reading a reading passage to the student during an assessment of decoding skills).

Questions about Validity of Alternate Assessments

Alternate assessments are assumed to require more performance-based measures, creating concerns about several technical aspects of these assessments: comparability with regular assessment measures; generalizability across assessment tasks; generalizability across students assessed using alternate assessments; inter-rater reliability; validity; and whether performance measures adequately represent educational standards. If performance measures are required, how many performance indicators will be designed for each educational standard? As the number of performance indicators expands, the cost of administration of the alternate assessment increases. At some point, issues of validity are necessarily traded off against issues of cost and other indices of feasibility such as the amount of time spent in assessment and instruction. The care with which these decisions are made will have an impact on confidence in the assessment results.

Lack of Data on How Decisions are Made about Participation and Accommodations

We do not yet know much about how decisions are made or about how they should be made regarding the assessments students take, the kinds of accommodations they use, and how their scores are reported and used. It is anticipated that criteria are needed to guide the making of decisions about assessment accommodations, and about who participates in alternate assessments. Should criteria be set at the local, state, or national level? If criteria for accommodations or eligibility for alternate assessments are set at the local level, decision making may vary greatly from district to district, making the results difficult to evaluate. Similarly, when criteria are set at state levels, it is difficult to create a national description of the performance of students with disabilities.

Differential Higher Education Training

A major issue facing professional development, especially as it relates to alternate assessments, is the nature of higher education training educators receive. Teacher training continues to be departmentalized, with the same topics taught in different ways and covering different information. Special education teachers almost always are prepared separate from general education teachers, reinforcing the notion that there are separate general and special education systems. Yet, knowledge of the general education curriculum by special educators is seen by many as the key to including students with disabilities in assessments and accountability systems.

General and special educators also obtain different information on assessment, and neither group receives a complete picture of assessment. This issue is particularly challenging for the development and implementation of alternate assessments because of the belief that collaboration among general and special educators, as well as with test developers and other professionals, is a critical component of a successful alternate assessment. Without collaboration, the alternate assessment will be seen as a special education assessment, rather than as a part of the broader accountability system.

Consequences

Consequences of IDEA Assessment Provisions on Students and Systems

We do not know much about what will happen to students and schools when the assessment provisions of the new IDEA are implemented. In the past, implementation of high stakes accountability systems has resulted in increased referral of individuals for special education evaluations, and increased retention of students at grade level. Alternate assessments could become dumping grounds for large numbers of students who experience difficulties with or do not pass state assessments; this could cause an increase in special education referrals.

It is intended that states and districts will use data from large-scale accountability systems to improve instruction and educational results for students. We do not yet know the extent to which this will happen, although some preliminary data suggest there is more attention to curriculum and more ownership for the results of all students. We also do not know how businesses, communities, and agencies will interpret and use the results of inclusive accountability systems.

Compliance for Rights versus Compliance for Outcomes

There is a need for a reasonable balance between monitoring states and districts for compliance with the law and monitoring them for achieving quality results for students with disabilities. At the same time, there needs to be guidance about the kinds of data states or districts need to substantiate quality outcomes.

Federal compliance monitoring has an enormous impact on state and district activities and personnel. Monitoring often is viewed as punitive; it is the source of considerable anxiety, and it drives to a significant extent the priorities of states and districts. The process should be less punitive. It should become more collaborative, such as by having monitors work in partnership with those they monitor. Whatever the approach, it should be designed to improve educational services and outcomes for students with disabilities.

Recommendations

Recommendations were developed in the areas of (1) assessment practices, (2) research and development, (3) technical assistance, (4) professional development, and (5) monitoring. While they are based on the identified issues, there is not a one-to-one correspondence between issues and recommendations. Furthermore, the recommendations in the five areas are not mutually exclusive, and the recommendations within one area sometimes are dependent on recommendations being carried out in another area. In each of the five areas, the recommendations are presented along with clarification of the meaning and intent of each recommendation.

Assessment Practices

Assessment practices discussed here are limited to those related to large-scale assessments (e.g., state and district assessments) and alternate assessments. Assessments used to determine eligibility for special education services are not the focus of these recommendations. Eight recommendations are proposed for large-scale assessment practices:

AP-1 Develop clear definitions and guidelines.
AP-2 Establish expert panel.
AP-3 Collect test data by age not grade.
AP-4 Track consequences of 1997 IDEA assessment provisions.
AP-5 Hold forum to promote participation during assessment development.
AP-6 Describe workable accountability systems.
AP-7 Enlist public support for new requirements.
AP-8 Promote state leadership.

AP-1: Develop Clear Definitions and Guidelines

There is a need for the U.S. Department of Education to develop clear definitions and guidelines on the requirements of the 1997 amendments to IDEA and the extent to which states have flexibility in the way in which they meet the requirements of the law.

The Department of Education should develop and disseminate at least three types of guidelines documents. First is a set of clear definitions and distinctions among terms, including but not limited to: accommodation, alternate assessment, assessment for accountability versus assessment to determine eligibility or individual student progress. The effort to make sure that terminology is clear and consistent also is a first step in setting up a good monitoring system. Starting a national dialogue on the topic is a good approach, with the ultimate goal being to improve the quality and consistency of data collection, and public reporting of data, across states.

Second, the U.S. Department of Education should develop a set of guidelines for states and districts on IDEA 1997 implementation, including critical topics such as (1) guidelines for IEP team decision making about participation in assessments, accommodations, and documenting those decisions, (2) technical issues surrounding alternate assessment development, and (3) design processes for states to follow in developing alternate assessments.

At a minimum, the following topics about technical issues should be addressed to assist states and districts:

• Ways to ensure acceptable reliability and validity coefficients

• Possible approaches to ensuring equivalency of the alternate assessment and the general assessment

• Acceptable and workable item development procedures

And, the following topics should be addressed about design processes:

• Accountability system purposes

• Valuing the education of students for whom alternate assessments are deemed necessary

• Determining who may be included in alternate assessments

• Standards for information obtained from alternate assessments

The third type of guidance document should be guidelines that address the extent to which there may be flexibility in meeting the requirements of the law. Attention will need to be given to how states at different points in their assessment practices can move toward compliance with the law.

AP-2: Establish Expert Panel

A panel of experts should be established to address numerous issues related to the requirements of the 1997 amendments to IDEA. Specifically, OSEP should identify a group of assessment and disability experts to delineate and disseminate: (a) guiding principles, (b) exemplary practices for general and alternate assessment issues, and (c) considerations for policy development. Specific activities of the panel could include the development of a Web site, a set of policy papers, and policy guidance for OSEP.

It is often difficult for states and districts to get access to current sound information about ongoing and best assessment and accountability practices. Several potential solutions to this challenge were identified.

One approach is to develop and maintain an Internet Web site where states could get information about best practices in assessment and accountability. There are many documents in which guidelines or exemplary practices are specified. Since judgments need to be made about the principles that could guide development and implementation of accountability systems, exemplary practices, and considerations for policy development, a group of assessment and disability experts should meet on a regular basis to update the database. The following kinds of information could be reviewed and put on the Web site:

• Guidelines for IEP teams to use in making decisions about who takes what kinds of assessments, the nature of accommodations to permit, and documenting those decisions

• Common terminology

• Philosophy underlying inclusive assessment and accountability systems

• Important considerations (or questions to ask) in developing an accountability system that includes all students

• Considerations for inclusive test development

• Ways to align standards, curriculum, instructional practices, instructional accommo-dations, assessments, and assessment accommodations

• Considerations in interpreting, reporting, and using large-scale assessment information

• Important considerations in making graduation, promotion, and retention decisions

• How to evaluate the effectiveness of your accountability system

The Web site should provide different levels and kinds of information directed to different audiences. For example, on the law, there might be four different options at the Web site: (a) one page with key provisions, (b) two to three pages with information describing the intent of the law and the changes that are likely to result from implementation of the law, (c ) a lengthy discussion of the different levels of implementation of the IDEA amendments on accountability and assessment, and (d) a compilation of commentaries and case law regarding accountability and assessment.

The Web sites should be available through the standard Internet technology, thus making it available in virtually every school in the country as well as on CD ROM for those who do not have Internet access. The user should have a menu of options that provides a wide choice of topics (e.g., accommodations, kinds of, nature of, advantages and disadvantages of each kind, student characteristics and accommodations, effects on results, best practices statements, research reviews, recently published research). The menu also should provide different levels of information on each topic (e.g., basic, intermediate, and advanced). Information at the site should be updated regularly (weekly or monthly). The existence of the Web site should be widely publicized and consumer reactions systematically studied as a means to increase accessibility and usefulness.

A set of policy papers should be developed on a conceptual framework, design principles, and policy issues for alternate assessments. Alternate assessment designs should be grounded in a value base that supports Least Restrictive Environment (LRE) and full participation of all students in standards-based reform. Because much is unknown about the unintended consequences of assessment for accountability, it is vital to remind assessment designers and policymakers that these principles should be inherent in any accountability system.

Alternate assessments must be designed to provide information relative to key performance indicators that represent the most essential features of the educational experience of students with disabilities, including opportunity-to-learn variables. Further, these assessments should be based on clear performance expectations that result in both the identification of curriculum gaps as well as the measurement of student performance. Alternate assessments should include mechanisms or procedures that allow assessment information to be used both for program planning and system accountability.

An expert panel should develop a policy paper that provides a conceptual framework for alternate assessment, as well as define and provide guidelines for developing and implementing alternate assessments. This paper must address technical issues, including how to approach test content, reliability, and validity issues. Through the panel, a set of key performance indicators should be identified so that states and districts have a base for an alternate assessment system that they can modify to meet their own data needs.

AP-3: Collect Test Data by Age Not Grade

State assessments typically are grade-based. When thinking about the participation of students with disabilities in assessments, this becomes an issue because more of these students have been retained, and thus are tested (if included in the assessment) with students who are younger than they are. Furthermore, without strict criteria about testing students according to their age, there is increased likelihood that students may be retained in a non-tested grade so that their scores do not count.

It has been documented that students may be retained in a non-tested grade so that their scores do not count in an accountability system. School scores can change dramatically when students’ scores are included by age rather than grade. To provide consistency in accountability systems, the recommendation is to require that test data be collected by age not grade.

It is important to note, however, that this recommendation really applies to all students, not just to students with disabilities. Implementation of this recommendation should flow into the assessment system, starting with kindergartners, and adding a grade each year (in effect, "grandparenting" all students presently out of grade level).

AP-4: Track Consequences of 1997 IDEA Assessment Provisions

Both intended and unintended consequences of implementing the assessment and accountability provisions of the 1997 amendments to IDEA should be tracked. This activity is an integral part of assessment practice.

Although at this time we cannot anticipate what the consequences will be of implementing the assessment provisions of the new IDEA, tracking them is critical. Consequences might include such effects as the impact on referral rates, outcomes for students and systems, costs, and impact on least restrictive environment (LRE) placement and instruction.

AP-5: Hold Forum to Promote Participation During Assessment Development

A critical key to appropriate participation of students with disabilities in assessments and accountability systems is to have the systems developed with all students in mind, including students with disabilities. For this to happen, it is particularly important to involve test developers, researchers, and the disability community in identifying ways to include students with disabilities from the beginning, when tests and other measures are being developed.

What is needed is a forum in which test developers, researchers, and parents collaborate to identify ways to include students with disabilities in initial test development and standardization. This should enable identification of specific items or item-types that may be inappropriate for some students, consideration of the kinds of accommodations to be provided, including these in the test development and standardization process, and integration of alternate assessment as a integral part of an inclusive accountability system.

AP-6: Describe Workable Accountability Systems

The implementation of appropriate accountability and assessment practices is influenced by examples of existing practice. A good way to provide these examples to districts and states is to describe workable accountability systems. A good way to describe these systems is through case studies and exemplars.

Many states now have accountability systems in place. It is important to use these as case studies of how these systems operate and their effects, especially the effects of implementing specific conceptual frameworks. For example, Iowa has used the conceptual model of outcomes and indicators developed by the National Center on Educational Outcomes (NCEO) as the framework for its ISEE Results Accountability System. This type of effort and others like it should be documented so other districts and states can learn from them.

There is also a need for exemplars related to alternate assessments, including exemplars of alternate assessment guidelines and procedures that could create opportunities for cross-fertilization and in-depth discussion of alternate assessment issues. A compendium of alternate assessment models for national dissemination should be developed as soon as possible. An example of an approach to developing a compendium is the CyberSurvey initiated by NCEO to collect and continually update information on what states are doing to develop and implement alternate assessments.

AP-7: Enlist Public Support for New Requirements

For the new assessment provisions to be implemented as intended, there must be public understanding and support. In general, this does not exist now. Therefore, a concerted effort must be made to engage the public and enlist its support.

Earlier we noted that there is much misunderstanding about assessment and accountability systems, and about the intent of the systems. The letter and intent of the law may only be met if stakeholders are engaged and their support is enlisted. Existing resources, such as the Regional Resource Centers (RRCs) and the National Center on Educational Outcomes (NCEO) could be used to support this effort.

In addition, assessment and accountability system issues should be repeatedly brought to the public’s attention. For example, current reporting mechanisms could be used to include indicators on the extent to which students with disabilities participate in assessments and the status of states in developing and implementing alternate assessments. Two avenues for doing this are: (1) Education Week’s annual "Quality Counts" publication, and (2) Fairtest’s Testing Our Children: A Report Card on State Assessment Systems.

AP-8: Promote State Leadership

Responsibility for the successful implementation of the 1997 amendments to IDEA rests not just with federal initiatives, but also in leadership at the state level. Leadership at the state level will help local and state policymakers make needed improvements in their educational systems. The state leadership needs to surround itself with an infrastructure for managing implementation of the assessment provisions in the amendments.

In addition to national-level efforts, there is a need for state leadership in meeting the letter and intent of the 1997 IDEA amendments. Leadership at the state level can ensure that assessment and accountability are congruent with the intent of 1997 IDEA. For the alternate assessment requirement, state activities might include:

• Revise state assessment systems to reflect the alternate assessment requirements of IDEA.

• Specify clear guidelines and procedures for making decisions about whether students will participate in the alternate or general state assessment, including information on the roles of the IEP team, parents, and staff in determining which students will be given the alternate assessment.

• Identify the objective factors that will be considered in determining whether a student will be given an alternate assessment.

• Report the design, purpose, and use of the alternate assessment as well as the percentage and characteristics of the students who participate in the assessment.

• Hold all districts and local educational agencies to the same criteria for technical quality for all assessments (e.g., valid, reliable, and based on documented research).

State leadership must be bolstered by a supportive infrastructure. Without this, other key activities that need to be undertaken, such as consistent and ongoing professional development, will be very difficult to achieve.

Research and Development

The research and development activities addressed here include all types of research (e.g., empirical, policy, experimental) and related development activities. Three recommendations are proposed for research and development activities:

RD-1 Develop collaborative research efforts.
RD-2 Develop model demonstration projects for alternate assessments.
RD-3 Conduct research on accommodations, alternate assessments, and related topics.

RD-1: Develop Collaborative Research Efforts

Research should not be isolated in the special education community because when it is it usually does not have an impact on the right constituencies. Therefore, there must be collaboration among test companies, the U.S. Department of Education, and state departments of education (assessment and special education units) on any research on inclusive test and measurement practices.

Research results hopefully will influence the practices of test companies, the U.S. Department of Education, and state education departments. These should be the entities involved in conducting the research on inclusive test and measurement practices (norm sampling, benchmarking, establishing cut scores, etc.). They also should be the actors in efforts to expand assessment practices to make testing and measuring more accessible for all students.

Envisioned here is a collaborative effort to improve tests, especially norm-referenced tests, particularly with regard to their use with students with disabilities. The federally-funded SCASS Technical Guidelines for Performance Assessment project illustrates well this collaboration between university-based researchers, testing companies, and state education agency staff.

RD-2: Develop Model Demonstration Projects for Alternate Assessments

Because alternate assessments are so new, there is a tremendous need not only to study existing examples of alternate assessments, but also to develop models of these assessments. These models then can be subjected to examination.

The U.S. Department of Education should create a set of incentives for state and local districts to develop model demonstration projects that target the development of alternate assessment models. These projects must produce well-researched and grounded alternative assessment models that are generalizable to other situations. It makes sense for these model projects to start at the local level, with the goal being to design systems that are scaleable to larger units. It also makes sense for the U.S. Department of Education to encourage collaboration among developers, to speed dissemination of workable practice.

RD-3: Conduct Research on Accommodations, Alternate Assessments, and Related Topics

There are many important topics suggested by the assessment provisions in the 1997 amendments to IDEA that require answers obtained from careful research. Five primary topics need to be addressed: (a) accommodations, (b) alternate assessments (definition, design, implementation), (c) decision making, (d) data analysis and reporting, (e) impact and consequences.

For each of the five topics for which there is a need for research, there are general themes that need to be addressed, as well as a series of specific questions. For example, research on accommodations needs to be experimental in nature, and designed to address the perception that the use of accommodations may invalidate a test. Experimental research goes beyond simply examining the performance of students who use accommodations and comparing it to the performance of students who do not use accommodations by providing appropriate controls. With experimental studies, we hope to answer questions about the extent to which accommodations provide more reliable and accurate pictures of students’ knowledge and skills. Several of the currently funded assessment projects represent a step in this direction, but many are not conducting the experimental investigations that are needed.

Given the limited experience that educators have had with alternate assessments, there is considerable need for research in this area. The research should focus on the definition, design, and implementation of alternate assessments. Several of the questions that might be addressed in research on alternate assessments are listed in Table 1.

Since appropriate decision making is critical to several aspects of the assessment provisions of the 1997 amendments to IDEA, it is also important to conduct research in this area. Understanding how decisions are made about the use of accommodations and modifications is important to being able to make recommendations for practice and training.

A number of questions about how decisions are made now remain unanswered. For example, how do IEP teams make decisions about participation in assessments? What types of information do they most rely on? What types of information are most needed? How do teams determine what accommodations are needed, and which are appropriate for use in certain types of assessments? What are the decision-making processes that result in setting standards and deciding what the consequences of assessment systems might be? How do teams determine how test results will be used? Answering these questions will form a foundation for not only best assessment practices, but also for technical assistance and professional development.

Research also should address questions about data analysis and reporting. These topics, especially with regard to the participation of students with disabilities, are relatively uncharted areas. Among the questions that should be targeted are those listed in Table 2.

Long-term investigations need to be conducted on the impact and consequences of implementing new assessment systems, particularly with respect to their effects on students with disabilities, For example, researchers could document changes in referral rates, progress toward expected results or standards, cost, etc. Since people are concerned about the extent to which participation in assessment systems results in negative experiences for students with disabilities, the effects of participation should be studied. Research questions on the impact and consequences of implementing alternate assessments are listed in Table 3. Many of these questions apply just as well to regular assessments.

Technical Assistance

Technical assistance is defined to include the provision of system-level training and supports designed to enable the system to conduct its activities in the area of concern (e.g., accountability assessments, alternate assessments, etc.). Seven recommendations are proposed for technical assistance activities:

TA-1 Create a technical assistance planning team.
TA-2 Document and evaluate current assessment efforts.
TA-3 Develop technical assistance materials.
TA-4 Create a forum for information dissemination.
TA-5 Establish state teams and provide training on alternate assessments.
TA-6 Establish a single point of contact for alternate assessment information.
TA-7 Provide a cadre of experts to provide startup technical assistance to states.

TA-1: Create a Technical Assistance Planning Team

It is urgent for a network to be established on new policies, and in particular, the requirements related to the development and implementation of alternate assessments. The best approach to establishing the kind of network needed is to begin by creating a team to plan technical assistance activities.

Given the lack of expertise that exists and the limited number of models of alternate assessments on which TA providers can build, in combination with the lack of detailed policy on alternate assessments, it is urgent that a network be established and that a TA planning team be formed. The network would include representatives of all groups to which states and districts might go to obtain assistance regarding the alternate assessment requirements. These groups would include, at a minimum, representatives from the Council of Chief State School Officers (CCSSO), the National Association of State Directors of Special Education (NASDSE), the National Center on Educational Outcomes (NCEO), and Regional Resource Centers (RRCs). It might also include representatives of Regional Educational Laboratories and Comprehensive Assistance Centers. The creation of this network would help to ensure that all TA providers would have access to all the expertise on assessment and accountability systems, and alternate assessments in particular, that does exist as well as all the models that have been generated, appropriate aggregation procedures, and how best to report results. A TA planning team would help ensure that efforts of the various groups were coordinated.

TA-2: Document and Evaluate Current Assessment Efforts

Current practices should be documented and evaluated as a basis for technical assistance. At the same time, there should be formative evaluation, in which current approaches are implemented in varying contexts, and the effects of the implementation systematically evaluated.

Good examples of current practices are needed. These would be extremely useful in technical assistance. For example, a paper should be developed that describes different levels of implementation of the accountability and assessment provisions, similar to the OSEP-supported monographs on due process. Different levels of implementation are inevitable, especially over the next few years, because states were at very different stages of implementation when the law was established. These different levels need to be documented and progress toward implementation evaluated.

Best practice examples and guidelines are needed especially in the difficult areas of choosing among curricular options (e.g., basic skills, general academic curriculum, social competencies, job-related skills) for students with disabilities. This information needs to be followed by discussions about accountability and assessment decisions that are consistent with the curricular decisions. Also requiring attention is collaborative decision making, particularly with parents and students as full partners. Alternate assessment is a challenging area for technical assistance, one that will require much attention.

TA-3: Develop TA Materials

Regardless of specific approaches to providing technical assistance, they will not effectively and efficiently meet existing needs unless supporting informational materials are developed. Materials are needed on a wide range of topics, including the law itself, basic concepts and definitions, accommodations, alternate assessments, and decision making. The materials must be designed for a variety of audiences.

Technical assistance materials need to consist not only of "content," but also of needs assessment tools for use by TA providers. They also need to include suggestions about the procedures and process for providing technical assistance. Furthermore, the technical assistance materials should emphasize the use of common language, and a consistent philosophy, vision, and goals. There is also a need for the materials to foster procedures for team building and collaboration.

More specifically, it is recommended that some materials should focus on the law, with commentaries about the spirit and intent of the law tailored to different audiences (and with appropriate disclaimers that the commentaries do not reflect official OSEP policy). It is also suggested that a paper should be developed to explore possible legal interpretations of the IDEA amendments, including the legal evolution of the accountability and assessment requirements through case law. Since such evolution is inevitable, a discussion paper may play some role in guiding that evolution, as well as in preparing policymakers for future developments. This discussion and resulting printed document should be available to all, but directed specifically to policymakers at the federal, state, and local levels.

Information on accommodations also needs to be developed for technical assistance materials. This information would focus on reasons for accommodations, the nature and kinds of accommodations, effects of accommodations in improving the meaning of assessment results, and the desirable interaction between assessment and instructional accommodations. There is a significant need for materials that develop a consistent understanding and use of accommodations and alternate assessments.

User-friendly self-study guide materials also should be developed and made widely available to enhance the literacy of significant decision makers (e.g., teachers, parents) about accountability and assessment for students with disabilities. A separate self-study guide on alternate assessments is needed. Top priorities include checklists and guidelines designed, for example, to increase participation, match accommodations to the needs of specific students, and improve consistency in instruction and assessment. The message and materials should always return to the bottom-line goal of improving instruction and outcomes for students with disabilities

TA-4: Create a Forum for Information Dissemination

The need for communication to further technical assistance efforts is critical, yet challenging if we rely on traditional avenues of disseminating information. Developing a forum for the dissemination of information related to assessment in general, and especially related to alternate assessments, is essential to effective and timely technical assistance.

A forum should be created to disseminate and share information on assessment in general, and to give particular attention to alternate assessments. The most appropriate venue for such a forum of diverse stakeholders, including test developers, might be a national listserv.

TA-5: Establish State Teams and Provide Training on Alternate Assessments

It is extremely important to establish state teams to develop and implement the alternate assessment requirements of the reauthorized IDEA. These teams need to include general educators, special educators, test developers, curriculum personnel, and assessment personnel, all of whom should take joint responsibility for the implementation of the alternate assessment requirements. The teams should receive intensive training on the fundamental issues related to alternate assessments. Furthermore, a mechanism should be established to provide ongoing support to the teams.

The U.S. Secretary of Education should request (perhaps require) states to establish teams and also convey the message that technical assistance is available to states. Having a letter with this message come from the current U.S. Secretary of Education emphasizes that this is not a "special education" message. Among the available TA providers would be those participating in the network identified above (e.g., CCSSO, NASDSE, NCEO, RRCs, Ed Labs, and CACs). (A letter to states with this message from the Secretary of Education should be copied to general and special education TA providers, and to other national organizations.)

State teams cannot be expected to know how to provide technical assistance to local education agencies unless they are trained. Furthermore, without a unified approach to training, we can have no hope for consistency in approaches across states. Ways to carry out this recommendation should be established by the TA planning team and the identified alternate assessment network.

Support to the state teams must be consistent and ongoing. It should be provided by various TA providers connected via the TA planning team, the single point of contact that is coordinating and brokering information, and the Web-based and listserv systems.

TA-6: Establish a Single Point of Contact for Alternate Assessment Information

A single point of contact will need to be established to facilitate the most efficient collection, analysis, and dissemination of information on alternate assessments. This contact should be assessment-based, rather than special education-based.

A new project (or a consortium of projects) should be funded to serve as the single point of contact. The purpose of the point of contact is both to coordinate all development, implementation, and evaluation efforts related to alternate assessments, and to broker these activities.

TA-7: Provide a Cadre of Experts to Provide Startup Technical Assistance to States

It is a challenge to begin providing technical assistance (TA) to states, especially on the topic of alternate assessments, when few providers have the information needed to provide technical assistance on this topic. Clearly, there is a need for experts to provide training for state-level TA providers.

A cadre of experts could provide immediate startup technical assistance to state-level TA providers, and a training process could follow for state TA planning teams. Hopefully the startup TA will occur at the same time that some of the other mechanisms reflected in these recommendations are getting underway.

Professional Development

Professional development refers to training provided to individuals (or groups of individuals), both preservice and inservice. There are many potential audiences for professional development activities, including school-based staff (teachers and related services personnel), policymakers at the federal, state, and local levels, state and district administrators, test developers, and faculty in institutions of higher education (IHEs). All play key roles in supporting and implementing IDEA, but the kind and amount of information disseminated to each audience needs to be tailored to the needs of that audience. For example, all of these audiences need to know about the law, but the depth of information about the law should vary significantly for different audiences.

Faculty members in IHEs (both general and special education) are considered to have the greatest need for professional development, reflecting the pivotal role they have in preparing educational professionals of all kinds, coupled with the concern that many faculty members who train educational professionals are not up to date on the law and on accountability and assessment systems as they involve students with disabilities. In reality, however, all of the potential audiences have substantial needs for professional development on accountability and assessment systems.

Seven recommendations are proposed for professional development activities:

PD-1 Require competencies in large-scale assessment.
PD-2 Develop core training materials that allow for adaptations.
PD-3 Develop a coordinated professional development plan.
PD-4 Target IEP teams for immediate training.
PD-5 Increase OSERS voice on the U.S. Department of Education’s initiative team for Goal Five on well-trained teachers.
PD-6 Develop and disseminate information for staff in parent organizations.
PD-7 Establish a federal priority for funding on alternate assessments.

PD-1: Require Competences in Large-Scale Assessment

There is a striking lack of training for teachers and other educators on the topic of large-scale assessments, including the concept of alternate assessments. A requirement for knowledge about large-scale assessments should be incorporated into standards that teachers and others must meet for certification.

Agencies involved in assessing prospective teachers should include these competencies in their assessments. In addition, the topic of alternate assessments should be included as one target of standards and assessments. Among the groups dealing with standards that specifically should be targeted are the Council for Exceptional Children (CEC), the National Council on the Accreditation of Teacher Education (NCATE), the National Association of School Psychologists (NASP), Educational Testing Service (ETS), the National Board of Professional Teacher Standards, and the Interstate New Teacher Assessment and Support Consortium (INTASC).

State standards for the approval of teacher education programs also should incorporate this content into their standards for the approval of teacher, support service, and administrator preparation programs. Greater coordination and sharing of resources is needed in the continuing education programs provided by different agencies (e.g., SEA, professional associations) and for different professionals (e.g., teachers, administrators, counselors) to incorporate essential content regarding students with disabilities for all educational professionals. Expanded opportunities to train different professionals in the same content should be provided as well as greater sharing of resources.

PD-2: Develop Core Training Materials That Allow for Adaptations

Core training materials that can be adapted are viewed as extremely important to professional development efforts. These materials must include curriculum and training modules, and address a wide range of topics.

There is a need for both training materials, and state flexibility in providing training. This suggests that core training materials that include curriculum and training modules should be designed to allow for state specific adaptations. Among ideas for needed content of training materials are the following:

• Provisions of the law regarding accountability and assessment

• Accountability systems as a means to improve instruction and outcomes for students with disabilities

• Options in the design and implementation of accountability systems

• Accountability systems and assessment provisions for students with widely varying levels of functioning, including those for whom a traditional academic curriculum is not appropriate

• Decision making about when and under what conditions students with disabilities should participate in general education accountability and assessment systems without accommodations, with accommodations, or through an alternate assessment system

• Information for assessment professionals and test developers on how to broaden the scope of assessment techniques to accommodate a broader range of abilities and disabilities, and alternative methods to assess the achievement progress of students with disabilities

• Strengths, limitations, and alternatives associated with different approaches to assessment, and applications to students with disabilities

• Decision-making about the kind and nature of accommodations, and about matching accommodations to the curricular objectives and capabilities of individual students

• Competencies related to interpreting assessment results with and without accommo-dations and skills in estimating the effects of different accommodations

• Means to encourage parents and students to participate in decisions about accountability and assessment, including training on the implications of choosing or rejecting alter-natives to the standard practices for general education students

• Use of assessment results to make formative and summative programmatic decisions for individual students and groups of students with disabilities

• Training packages that can be used with districts to improve accountability and assessment systems

A variety of vehicles can be used to present these core materials, including the Web site proposed as an Assessment Practice activity (see AP-2).

PD-3: Develop a Coordinated Professional Development Plan

In each state there will need to be a coordinated professional development plan that incorporates all assessment requirements. These include Title I requirements, as well as those in Goals 2000 efforts.

It is important to recognize that a number of programs now have requirements related to assessment. For example, Goals 2000 and the Improving America’s Schools Act (which funds Title I programs), in particular, have requirements that state assessments include all students. These requirements are melded in consolidated plans, which most states seeking federal funds have proposed. It is critical for professional development to reflect all of these requirements, in a coordinated and systematic way. This effort can be supported through one or more of the current federally-funded school reform and comprehensive professional development programs.

PD-4: Target IEP Teams for Immediate Training

Immediate implementation of the amendments to IDEA requires immediate training of those most directly involved in making decisions: IEP teams. In the past, variability in implementation of state guidelines has been attributed to different levels of understanding among IEP teams. In all aspects of the new law, IEP teams are critical to carrying out the letter and intent of the law.

Nearly every new requirement related to assessment and accountability in the 1997 amendments to IDEA is dependent on good decision making by IEP teams. Yet these teams have not been trained to make these decisions. In many cases, they have not even had sufficient training to know what the new requirements are. A strong, widespread training effort must be undertaken with IEP teams, and it must be done immediately. Furthermore, the focus should be on practical strategies to incorporate the additional requirements without breaking what is often viewed as an already overloaded and time-consuming process.

PD-5: Increase OSERS Voice on the U.S. Department of Education's Initiative Team for Goal Five on Well-Trained Teachers

Goal Five of our national education goals seeks to ensure that there is a well-trained teacher in every classroom. It is imperative that "well-trained" includes being versed in assessment issues and procedures, particularly as they relate to students with disabilities.

A logical point from which professional development related to assessment and students with disabilities can be launched is the current national education goal of ensuring that there is a well-trained teacher in every classroom. Teachers are going to be a key component in ensuring workable assessments for students with disabilities, from making decisions about participation in assessments and determining appropriate accommodations, to implementing alternate assessments. Messages about the need for all students to be included in assessments, and the teacher training implications of this new requirement, should be integrated within planning at the top-most levels. Thus, the initiative teams established by the U.S. Secretary of Education, particularly the Initiative Team on Ensuring a Well-Trained Teacher in Every Classroom, are probably the place to start.

PD-6: Develop and Disseminate Information for Staff in Parent Organizations

There is a tremendous need for parent information and support as the new IDEA requirements are implemented. Because of the belief that this is an important professional development concern, training information needs to be developed for staff in the organizations that serve parents.

Informed parents are also an important aspect of successful change in programs for students with disabilities. This will apply to both large-scale assessments and alternate assessments. There is a particular need to reach such parent-oriented groups as Parent Training Institutes (PTIs), Goals 2000 Parent Centers, and other parent-focused organizations.

PD-7: Establish a Federal Priority for Funding on Alternate Assessments

Because of their newness, and the short timeline within which alternate assessments must be implemented, funding for professional development on these assessments is critical. It must be a federal priority. This is an immediate need.

Although funding is not the only solution to many of the professional development needs related to alternate assessments, a comprehensive funding effort is a critical component of successful professional development related to alternate assessments. This funding effort must be comprehensive and coordinated. Thus, the priority for funding related to alternate assessments must include components that support research, development, dissemination, training, technical assistance, and professional development.

 Monitoring

Compliance monitoring includes those activities undertaken by the federal education agency with state education agencies, or by states with their local education agencies, to ensure that the letter and intent of special education law are being met. Seven recommendations are proposed for monitoring activities related to the assessment provisions of the law:

M-1 Redefine the purpose of monitoring.
M-2 Integrate NASDSE and NCEO models to guide monitoring efforts.
M-3 Monitor for consistency, comprehensiveness, and progress on state improvement plans.
M-4 Use a standard formula to analyze participation.
M-5 Change the consequences of monitoring.
M-6 Involve stakeholders in monitoring process.
M-7 Monitor goal alignment for students taking the alternate assessment.

M-1: Redefine the Purpose of Monitoring

Federal compliance monitoring has an enormous impact on state and district officials, including being a significant source of anxiety and priority setting. It is important to decrease the punitive perception of compliance monitoring and to build a greater sense of partnership among agencies for improving educational services to students with disabilities.

Continuous improvement. Federal compliance monitoring activities should focus on continuous improvement of state services to students with disabilities, addressing primarily the development of close ties among state monitoring, technical assistance to districts, resource allocation, connections among agencies and people, and state improvement plans. Federal and state monitoring should form the basis for problem solving by identifying problems in services to students with disabilities, collecting data on the problems, summarizing results, disseminating results to key parties, and formulating and implementing plans for resolution. This implies monitoring the progress and success of those state improvement plans and revising the plans as needed.

Data reporting and use. Monitoring also should focus on how assessment data are used. States should use data to review the current status of education and to make improvements. A measure of compliance would be whether data are used to make appropriate improvements. Federal and state monitoring procedures should combine quantitative and qualitative data, in many cases supplementing quantitative with qualitative data. For example, quantitative data that showed a high use of separate schools could be examined through qualitative data such as interviews or focus groups with persons who make placement decisions.

Federal IDEA monitoring should consider the broad accountability system and state improvement plans for all students and all schools, including overall results on statewide assessments, the extent to which students with disabilities are included in the general accountability system, and the extent to which districts and individual schools are succeeding with various groups of students (e.g., Title I, students with disabilities, at risk).

A graduated and individualized schedule is recommended for monitoring compliance with the accountability and assessment provisions of IDEA. Different expectations for states cannot, of course, exist indefinitely. Expectations should be increased systematically and without excessive delays for all states. Discussions involving NASDSE and OSEP should focus on a realistic schedule for full implementation of the accountability and assessment provisions in all states. New monitoring schedules will provide more emphasis on assessing the achievement of positive outcomes, while still recognizing the need to monitor compliance with the civil rights aspects of IDEA.

M-2: Integrate NASDSE and NCEO Models to Guide Monitoring Efforts

It is important to monitor more than just the academic results of education, especially for the alternate assessment. If monitoring focuses just on what states produce for assessments, other important areas might be ignored.

Important areas that might be ignored if monitoring focuses just on academic results include other outcomes, such as those delineated in the NCEO model (e.g., citizenship, independence, responsibility, physical health), as well as components of a balanced educational accountability system, as identified in the NASDSE accountability model (e.g., system-level accountability; individual student learning; inputs, processes, and rights). There is a need to integrate the NASDSE and NCEO models, and to use these to provide the framework for developing practices for state and federal monitoring. Essentially, this approach would ensure that monitoring will extend beyond limitations that might be imposed by narrow assessments developed by states or local education agencies.

M-3: Monitor for Consistency, Comprehensiveness, and Progress on State Improvement Plans

The newness of the IDEA 1997 requirements on participation in assessments and the development and implementation of alternate assessments presents a need for continuity in checking for implementation consistency, and to ensure that the comprehensive assessment system includes the alternate assessment. Furthermore, the addition of a requirement for State Improvement Plans in the 1997 amendments provides an opportunity for the federal monitoring agency to work with states to develop ways to monitor progress on these plans.

Monitoring procedures should ensure consistency in the implementation of state policy on participation in the alternate assessment. Such monitoring should include reviews at the individual level (i.e., examine files for students with disabilities who are identified as eligible to participate in the alternate assessment).

For a comprehensive assessment system to include the alternate assessment, there should be evidence that the alternate assessment was developed through a collaborative effort of general assessment personnel and special educators. Also, there should be evidence that collaboration continues into the administration of the alternate assessment, and that the scoring and reporting of these assessment results is part of the larger comprehensive assessment system. The exact indicators that need to be monitored will have to be delineated.

The changed nature of State Improvement Plans merits a new approach to monitoring. An approach developed by OSEP and the states working together probably is most desirable. This approach should include, at a minimum, goal setting, indicator tracking, and improvement activities. The approach might also be multi-level, with diminishing amounts of data collected from local data collection to district collection, to state collection. Performance indicators needed for program evaluation and program planning might be most numerous at the local level, with a smaller subset of such indicators reported to the state, and yet a smaller subset of key performance indicators reported to the U.S. Department of Education.

M-4: Use a Standard Formula to Analyze Participation

Because participation rates can be defined in so many ways, it is imperative to have a standard formula for all states to use. This standard formula should be used in monitoring the participation of students with disabilities in statewide assessments.

The difficulties in defining rates of participation in assessments have been documented. For example, the work of NCEO has shown the current tendency of states to use different formulas when calculating participation rates in regular assessments. Although the 1997 IDEA requires only that states report numbers of students in state and district-wide assessments, monitoring should analyze participation rates, but use a standard formula across states when performing the analysis.

M-5: Change the Consequences of Monitoring

The consequences attached to monitoring have not been tied to what happens to students. Monitoring has been viewed largely as serving a policing function, one that helps constrain "inappropriate" activities rather than foster appropriate activities. Tying the results of monitoring to technical assistance makes more sense and turns monitoring into more of a teaching function.

Federal monitoring has been viewed too often with contempt by states. State monitoring has been viewed similarly by districts. An approach to changing past practices is to attach different consequences to monitoring. The goal of the change should be to make monitoring more relevant to state or local needs and to have meaningful consequences. Another approach that is viewed as very positive and proactive is the provision of technical assistance to those who need it.

M-6: Involve Stakeholders in Monitoring Process

Monitoring is carried out by individuals employed by the federal monitoring agency or by the state. These individuals may not reflect all constituencies that need to be involved in conducting the compliance monitoring activities. It is strongly recommended that trained parents and current special education directors be added as participants in some aspects of federal and state monitoring activities.

Monitoring has been conducted primarily by groups of individuals employed by the agency conducting the monitoring. In some instances, the individuals might have provided a very narrow perspective for a team meant to assure that the letter and intent of IDEA are met. As such, other important stakeholders should become participants in some of the monitoring activities. While these added individuals may require some special training to join the monitoring team, it is an extremely worthwhile investment. However, restrictions should be placed on who these individuals can be. For example, trained parents should only be on teams monitoring districts and/or states other than the one in which they live. District directors should only be on teams visiting districts and/or states other than their own. Logistics and potential legalities such as funding for the training and travel would have to be addressed.

M-7: Monitor Goal Alignment for Students Taking the Alternate Assessment

Because there may be some unintended incentives for having students take alternate assessments rather than regular assessments, there is a need to inquire about the alignment of goals, standards, and measures for students who are not taking the regular assessment. This should be one focus of monitoring.

The alignment of goals, standards, and assessments for individual students is extremely important. The potential exists for students to be pushed into an alternate assessment even though they may be striving toward the same goals and standards as the majority of students in the system. This is inappropriate. It should be possible to avoid this tendency by making the connection among goals, standards, and assessments a target of monitoring.

Next Steps

The recommendations presented in this report provide the basis for action by numerous groups. Ideally, they will first be used by the U.S. Department of Education, to plan ways to ensure that the letter and intent of the new assessment provisions in the 1997 reauthorization of IDEA are met. Beyond this, the recommendations (and background issues) are useful to states, districts, and their constituencies (including administrators, teachers, parents, and the general community) as they think about what it takes to implement the new requirements.

An essential follow-up activity to the January 7-9 meeting, at which stakeholders generated the issues and recommendations presented here, is to prioritize the recommendations. This could be accomplished within the U.S. Department of Education, or by involving a broader group of stakeholders. The prioritized recommendations then could be used as the basis for developing action plans that define specific avenues to be followed for meeting the letter and intent of the assessment provisions in the 1997 amendments to IDEA.

Appendix A
Meeting Participants and Other Reviewers

Meeting Participants

Sue Bechard, Colorado Department of Education

Gail Bornfield, National Information Center for Children and Youth with Disabilities

Marsha Brauen, Westat

Martha Brooks, Delaware Department of Education

John Corpolongo, Oklahoma Department of Education

Deborah Crockett, National Association of School Psychologists

Louis Danielson, Office of Special Educational Programs

Lizanne DeStefano, University of Illinois

Pasquale DeVito, Rhode Island Department of Education

Kevin Dwyer, National Association of School Psychologists

Loren Faibisch, National Center on Educational Outcomes

Martha Fields, National Association of State Directors of Special Education

Bill Frey, Westat

Jim Friedebach, Missouri Department of Education

Arnold Goldstein, National Center for Education Statistics

Marcia Harding, Arkansas Department of Education

Gerrie Hawkins, Office of Special Education Programs

Thomas Hehir, Office of Special Education Programs

Judith Heumann, Office of Special Education and Rehabilitative Services

Sarah Kennedy, University of Kentucky

Elizabeth Kozleski, National Institute for Urban School Improvement, University of Colorado

Milagros Lanauze, Office of Bilingual Education and Minority Languages Affairs

Patrice Linehan, National Association of State Directors of Special Education

Jay McIntire, Council for Exceptional Children

David Malouf, Office of Special Education Programs

Wayne Martin, Council of Chief State School Officers

Margaret McLaughlin, University of Maryland College Park

Kenneth Olsen, Mid-South Regional Resource Center

Lucian Parshall, Michigan Department of Education

Daniel Reschly, Iowa State University

Larry Ringer, Office of Special Education Programs

Ed Roeber, Council of Chief State School Officers

Michele Rovins, Federal Resource Center

Allison Seyfarth, National Center on Educational Outcomes

Sharif Shakrani, National Assessment Governing Board

David Sweet, Office of Educational Research and Improvement

Martha Thurlow, National Center on Educational Outcomes

Gerald Tindal, University of Oregon

Scott Trimble, Kentucky Department of Education

James Viola, New York State Education Department

Janet Vohs, Federation for Children with Special Needs

Deborah Voltz, University of Louisville

Edward Wilkens, Northeast Regional Resource Center

Phoebe Winter, Council of Chief State School Officers

James Ysseldyke, National Center on Educational Outcomes

Other Reviewers*

Eileen Ahearn, National Association of State Directors of Special Education

Judith Ball, Eastern Instruction Support Center, Pennsylvania

James Canfield, Louisiana Department of Education

Judith Elliott, National Center on Educational Outcomes

Ronald Erickson, National Center on Educational Outcomes

Lynn Fuchs, Vanderbilt University

Joseph Kovaleski, Pennsylvania Department of Education

Richard Smiley, Alaska Department of Education

Michael Vanderwood, Iowa State Department of Education

Don Watson, Colorado Department of Education

* Other reviewers included members of the Assessing Special Education Students SCASS (State Collaborative on Assessment and Student Standards).

Meeting Procedures

Two activities occurred before the actual convening of the meeting. First, when individuals were invited to participate in the meeting they were asked to review the new assessment provisions in the 1997 amendments to IDEA. These were summarized in a document that included information on the language in the amendments, proposed regulations directly related to the provisions, and comments.

The second activity in which participants engaged before the meeting was to complete a survey sent to them by NCEO. For the survey, participants identified issues and recommendations for each of the focus areas within two strands (assessment/accountability and alternate assessment). Information from this survey was compiled by NCEO staff prior to the Working Group Conference, so that the information could provide a basis for initiating discussions.

The meeting agenda (see Figure B-1) directed a systematic process for identifying issues and recommendations, and for gaining consensus from all participants about the important issues and recommendations for meeting the language and intent of the law. First, background information was presented to participants in a general session. During this session, a panel that included representatives of the U.S. Department of Education and NCEO provided information on the law and on what is known about current practice related to: (a) participation, (b) accommodations, (c) alternate assessments, (d) reporting of results, and (e) implementation of assessments and accountability systems.

Following the general session, participants were divided into the two strands and then into subgroups:

Accountability Strand

Group 1 – assessment practice; research and development

Group 2 – technical assistance; professional development; monitoring

Alternate Assessment Strand

Group 1 – assessment practice; research and development

Group 2 – technical assistance; professional development; monitoring

The goal of each subgroup was to develop consensus on a set of issues and recommendations. Next, the two subgroups within each strand reviewed each other’s issues and recommendations for overlap and omissions. Then subgroups focusing on the same topic areas convened, again to identify overlap and omissions.

Following the meeting, group facilitators wrote up the results of their discussions. These were merged with each other by staff at NCEO to produce a summary of overall issues that provide challenges to meeting the language and intent of the IDEA amendments, and specific recommendations for (a) assessment practices, (b) research and development, (c) technical assistance, (d) professional development, and (e) monitoring. NCEO also merged any additional recommendations that were generated as a result of the larger group sessions. A draft of the report was reviewed by all meeting participants before dissemination to the public.

Council of Chief State School Officers

One Massachusetts Avenue

Washington, DC

January 7-9, 1998

Wednesday: January 7, 1998

11:30 – 1:00 Lunch (Reckford Room, 1st Floor)

Greetings from OSERS and OSEP

Thomas Hehir, Director, Office of Special Education Programs

Judith Heumann, Assistant Secretary, Office of Special Education and Rehabilitative Services

Participant Introductions

James Ysseldyke, Director, National Center on Educational Outcomes

1:00 – 2:30 IDEA 1997: Panel Discussion

Louis Danielson (moderator), Director, Research to Practice Division, Office of Special Education Programs

Joleta Reynolds, Senior Policy Advisor, Office of Special Education Programs

Larry Ringer, Special Assistant to the Director, Monitoring and State Improvement Planning Division, Office of Special Education Programs

James Ysseldyke, Director, National Center on Educational Outcomes

Martha Thurlow, Associate Director, National Center on Educational Outcomes

2:30 – 2:45 Break

2:45 – 3:30 Overview of Charge to Participants of Assessment Working Group Conference

Procedures and Products

Group Assignments (see separate list with participant names)

• Strand A: Accountability Systems

Group 1: (Large Conference Room)

Assessment practices (including classroom assessment); research and development

Group 2: (3rd Floor Conference Room)

Professional development; technical assistance; monitoring

• Strand B: Alternate Assessment

Group 3: (Reckford Room, 1st Floor)

Assessment practices (including classroom assessment); research and development

Group 4: (Reckford A, 1st Floor)

Professional development; technical assistance; monitoring

3:30 – 5:00 Group Discussions (4 Groups)

Issues (Implications and Challenges)

Recommendations

5:00 – 5:30 Reconvene as Large Group to Check Agenda

(i.e., is what we are doing working?) (Reckford Room)

 

 

Thursday: January 8, 1998

8:30 – 9:00 Continental Breakfast and Review of Task

9:00 – 11:30 Group Discussions (4 Groups continue)

(Break as needed)

Issues (Implications and Challenges)

Recommendations

11:30 – 12:30 Lunch

12:30 – 1:30 Group Discussions (4 Groups finish)

Issues (Implications and Challenges)

Recommendations

1:30 – 5:00 Combined Groups Show, Tell, Clarify, and Discuss

(Break as needed)

Strand A Groups 1 and 2 (Large Conference Room)

Issues and Recommendations for:

Assessment Practices

Research and Development

Professional Development

Technical Assistance

Monitoring

Strand B Groups 3 and 4 (Reckford Room)

Issues and Recommendations for:

Assessment Practices

Research and Development

Professional Development

Technical Assistance

Monitoring

5:00 – 5:30 Summary of Day’s Activities (Reckford Room)

Plans for Tomorrow

Friday: January 9, 1998

8:30 – 9:00 Continental Breakfast

9:00 – 10:30 Summary and Integration of Work Groups (Reckford Room)

Assessment Practices

Strand A

Strand B

Research and Development

Strand A

Strand B

10:30 – 10:45 Break

10:45 – 12:00 Summary and Integration of Work Groups (Reckford Room)

Professional Development

Strand A

Strand B

Technical Assistance

Strand A

Strand B

Monitoring

Strand A

Strand B

12:00 – 12:30 Next Steps

12:30 – 1:30 Lunch and Good-byes

 

Top of page