Published by the National
Center on Educational Outcomes
August, 2003
On August 25, 2003, 30 states and 9 organizations participated in the National Center on Educational Outcomes’ (NCEO) teleconference, marking the end of the second year of quarterly teleconferences on inclusive assessment and accountability. The topic of universally-designed assessments was introduced by Sandra Thompson of NCEO, who defined what is meant by "universally-designed assessments," and then described the research that NCEO is conducting to help determine when an assessment is universally designed. Defining universally designed assessments as "assessments that are designed and developed to allow participation of the widest range of students, resulting in valid inferences about their performance," she clarified that the concept is not about lowering achievement standards.
Sandy Thompson noted that NCEO has two research projects underway on universal design – one a student initiated project (awarded to Christopher Johnstone), and one a field-initiated project awarded to Thompson. Various approaches to research were highlighted, including experimental comparisons and analyses of differential item functioning. Much of the discussion during the teleconference focused on the use of a form (available on the NCEO Web site) called "Considerations for Reviewing Items" (and similar forms for reviewing tests and RFPs), based on the elements of universal design (see NCEO’s Synthesis Report 44, Universal Design Applied to Large-Scale Assessment). Five basic considerations were discussed for items:
(1) Does the item meet general criteria for measuring what it is intended to measure?
(2) Is the item free of textual or pictorial clutter?
(3) Is there clear text format, pictures, and graphics?
(4) Is the text concise and readable?
(5) Does the item allow changes to its format without changing its meaning or difficulty?
Sandy Thompson noted that the "think aloud" process that has been used in the universal design research is one of the most exciting ways to explore how students interact with items, which often is very different from how adults expect that the students will interact with them.
Lorraine Petuski and Susan Izard represented the New England Compact, a group of four small states in the Northeast (Maine, New Hampshire, Vermont, and Rhode Island) working together to establish common priority standards, a common test blueprint, some exemplars, and to develop teacher expertise. The states decided to move forward with universal design, specifically asking whether items are acceptable for all students from the beginning. When thinking about test specifications, they had to think about accessibility and the elements that had to be included. Discussion of exemplars for use led to a decision to test out some accommodations using experimental designs. Finally, professional development is occurring on awareness of grade level expectations, use of universal design, deeper understanding of accommodations, and training lead trainers. Later, there are plans to disseminate information to other states. The New England Compact also has included language about universal design in the RFP.
Gerald Tindal, another researcher who has received funding to conduct research on universally designed assessments, noted that there are many ways that an item could function the same even though it did not look the same. He suggested that there are five points that should be thought about in the pursuit of universal design:
(1) How standardized do things have to be?
(2) Is the purpose of testing just to get a score, or is it to understand what makes performance successful or not successful?
(3) Is it a problem when something works for everyone?
(4) What happens when group research designs are used to make decisions about individual students?
(5) What are ways to look at performance at the item level?
A link to Jerry Tindal’s Web site is posted on the teleconference message board at http://education. umn.edu/NCEO/NCEOdiscuss_toc.htm.
Robert Dolan and Jacqui Kearns spoke about a research project in Kentucky in which assessments are presented via computer. Bob Dolan noted that universal design improves both learning and assessment for all students, not just for students with disabilities. Several examples of universal design for learning at the state level were given, including Kentucky’s digital text in the classroom. This past spring, Kentucky’s testing system went on-line. Students could use text-to-speech, change the font, change the screen, etc.
Jacqui Kearns spoke about the research questions that were examined as part of the research study. For example, focus groups of students liked the independence that the system gave them and how it worked; still, there was some question about the extent to which the technology was available for their regular classroom work. Also, while some features promoted accessibility (for example, multiple means of representation), there were some problems with layout, the flexibility of navigation, and speech correctness. On the whole, however, the results from students were very positive. The project’s Web site is posted on the teleconference resource page at http://education. umn/NCEO/Presentations/tele6.htm.
Federal partners noted the support for the universal design concept in federal regulations and through the funding of projects. Several states asked specific questions related to universal design. Conversations can be continued on the NCEO Web site at http://cehd.umn.edu/NCEO/NCEOdiscuss_toc.htm.
Mark your calendars for the next teleconference on November 3, 2003: How testing, teaching, and learning for all students change under standards-based reform.
Results from NCEO’s 2003 State Survey of Special Education Directors has been completed and will soon be released. It addresses the topics of assessment participation and performance, accommodations, out-of-level testing, alternate assessments, universally designed assessments, computer-based assessments, accountability, and assessment consequences. States were asked to identify emerging issues as well as their technical assistance needs.
Most state directors of special education reported an increase in academic expectations and in the use of appropriate accommodations for students with disabilities. They also reported increased participation of special educators in training on standards and assessments, and increased networking between general and special educators.
More states are able to document assessment accommodations use and trends in assessment performance of students with disabilities than were able to do so in 2001. Nearly all states reported that alternate assessments are now aligned with the state’s academic content standards.
Some priority issues identified by state directors of special education were a lack of instructional strategies that positively impact student performance, familiarizing special education teachers and administrators with the standards, helping IEP teams understand access to the general education curriculum, concerns about too many students participating in alternate assessments, decreasing numbers of students with disabilities graduating with a standard diploma, and an increase in dropout rates.