NCEO has now published four reports on standards setting for alternate assessments. Three reports describe actual standard-setting processes used in states. The fourth report is an overview description of standard-setting approaches.

NCEO’s most recent report is authored by Nancy Arnold (Washington Alternate Assessment System: Technical Report on Standard Setting for the 2002 Portfolio, Synthesis Report 52). It documents the standard-setting procedure for alternate assessments in the state of Washington, which is based on Washington’s Essential Academic Learning Requirements (EALRs) in the areas of Communication, Reading, Writing, Mathematics, and Science. The Washington Alternate Assessment System (WAAS) is a portfolio assessment designed for a small percentage of the total school population for whom traditional assessments, even with accommodations, would not be appropriate.

The portfolio is comprised of samples of the student’s work. Each entry in the portfolio must document student progress on IEP skills linked to the EALRs and the ability of the student to generalize those skills. Evidence of the student’s work demonstrates participation in and progress toward IEP goals aligned to state standards.

A 14-member standard-setting panel was selected to set the standards for the Portfolio. Each person participating in the process was chosen for his or her qualifications as a judge of student performance based on various factors. The panel included teachers and subject area experts, balanced by representatives from various counties, urban areas, those who work with students with significant disabilities, and members from schools serving various sizes of populations, gender, and race/ethnicity. The panelists’ task was to classify student work into one of several performance categories designed to express levels of performance.

The report covers the initial procedure, training, and seven-step process including evaluation. This report is available online at http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis52.html.

Another report on alternate assessment standard-setting is written by Dan Wiener (Massachusetts: One State’s Approach to Setting Performance Levels on the Alternate Assessment, Synthesis Report 48). It describes the debates and decisions that formed the basis for Massachusetts’ portfolio method of alternate assessment, and decisions that shaped the method used to set standards for the assessment. Massachusetts decided to use an analytical rubric to convert raw scores to performance levels. The reasoning behind the Massachusetts approach and the methods used to produce an overall performance level are described. This report is available at http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis48.html.

The third standard-setting report is by Barbara Olson, Ronald Mead, and David Payne (A Report of a Standard Setting Method for Alternate Assessments for Students with Significant Disabilities, Synthesis Report 47). It describes a body of work standard-setting approach that was used for a standards-based portfolio alternate assessment. The authors identify time and resource constraints with this method, and emphasize the importance of range-finding and pin-pointing phases for the body of work approach. This report is available at http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis47.html.

Ed Roeber (Setting Standards on Alternate Assessments, Synthesis Report 42) summarizes a variety of standard-setting approaches that can be applied to alternate assessments. He cautions that both technical and practical considerations must be weighed while choosing an approach. This report is available at http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis42.html.

NCEO’s report, Improving Validity of Large-Scale Tests: Universal Design and Student Performance (Technical Report 37), shares the results of a Student Initiated Grant Study funded by the U.S. Department of Education, Office of Special Education Programs. This study looked at a sample of 231 sixth grade students from traditionally under-performing schools and populations, and compared the results they received on two tests. Students took a traditionally designed assessment developed from released large-scale assessment items presented in a standard format, as well as a second test created using the constructs of the traditionally designed test, but including universal design features.

In developing the second test, a mathematics content area expert determined item constructs. Experts on Universal Design in testing and an advisory group examined the items for design flaws and possible improvements to each item that would not change the constructs.

An experimental design was used in this study, with all students taking both the traditional and the universally designed assessment. Groups of students were randomly assigned to the order of the tests.

Results of this research demonstrated that students scored, on average, significantly higher on the universally designed test. Interviews with students following the testing confirmed that they perceived that they scored higher on the universally designed test, and that they preferred the tests with Universal Design features. They also indicated that they recalled the universally designed tests better and that they had been taught the information on the universally-designed tests. They perceived the tests to be different even though the content of the tests was the same. This report can be found at http://cehd.umn.edu/NCEO/OnlinePubs/Technical37.htm.

NCEO and other researchers are presenting at the annual conference of the American Educational Research Association, April 12-16 in San Diego, CA. Interesting presentations include:

April 12:
States’ Procedures for Ensuring the Precision and Accuracy of Out-of-Level Test Scores (1-1:40p) Roundtable 37.
Presenters: Minnema, Thurlow, Moen

Test Accommodations and Students with Disabilities: An Analysis of the Interaction Hypothesis (1-1:40p) Roundtable 37. Presenters: Li, Scarpati, Sireci

Reliability and Validity of the Oregon Extended Writing Assessment (1-1:40p) Roundtable 2. Presenters: Crawford and Tindal

The Impact of Reform: The No Child Left Behind Act and Students with Disabilities (4:05-4:45p) Roundtable 3. Presenter: Harvey-Koelpin

April 13:
Effects of Statewide Testing on ESL Curriculum and Instruction: What’s Intended and Unintended? (12:25-1:55p)
Presenter: Anderson

Attitudinal Issues in the Math Assessment of English Language Learners (12:25-
1:55p). Presenter: Abedi

April 15:
Assessing Progress of English Language Learners Under NCLB Title III: Challenges and Choices (10:35-12:05)
Discussant: Abedi

I Say Potato, You Say Potahto: The Assessment-Speak Gap Between General and Alternate Assessment Experts
(12:25-1:55p). Presenters: Quenemoen, Ryan, Thurlow

Reporting Scores of ELLs in NAEP and the States (4:05-6:05p). Presenters: Rivera, Collum, Thurlow, Albus, Liu, Goldstein, DeVito

April 16:
What is the Role of Research in Accommodation Decision-Making for English Language Learners? (8:05-
10:15a) Symposium. Presenters: Shakrani, Maihoff, Goldstein, Thurlow, Abedi. Discussants: Tindal, Leos, Mele-McCarthy