Published by the National
Center on Educational Outcomes
Number 5 / May 2002
Summary of
Teleconference on Technology Based Assessments
On
April 29, 2002, 37 states and
10 organizations participated in NCEO’s third teleconference. Panelists from the
University of Minnesota and University of Oregon, State education staff from Virginia and
North Carolina, and Federal staff from the Office of Special Education Programs, Title I,
and the National Center for Education Statistics shared information, followed by questions
and comments from all participants. (Materials provided by NCEO and the presenters are
still posted at http://cehd.umn.edu/NCEO/Presentations/presentations, click on “Teleconference
3.”) Highlights of some of the presentations are given here.
Research
Findings
Sandy
Thompson from NCEO presented an overview of a recently completed NCEO paper that
synthesizes the opportunities and challenges presented to all students by computer-based
assessments. The paper also explores research on the use of accommodations on these
assessments. Sandy summarized a process that states can use to ensure that all students
are included in the design and implementation of computer-based assessments.
Jerry
Tindal from the University of Oregon briefly reviewed seemingly contradictory research on
the effects of selected computer-based accommodations, concluding that these mixed results
may relate to different uses of computers in testing. He suggested that if the computer is
used as a communications device, simply offering alternative response options to paper and
pencil, the use of accommodations generally is straightforward. However, when the computer
is used as an accommodation tool in the assessment (for example, as a word processor,
spreadsheet, or database), then its use may introduce variables other than the constructs
being tested. Dr. Tindal suggested caution as we explore the use of the computer as a tool
instead of simply as a communications approach. His research Web site is linked to the
NCEO Web site above, and includes a link to his presentation summary points.
Report
from Virginia
Virginia
is moving its end of course exams from paper and pencil to an online approach for use by
all students. These are multiple choice tests only. The State is tailoring its existing
accommodations policies to fit the online environment in partnership with a stakeholder
group of special education and LEP teachers, assistive technology staff, parents, and
assessment staff. With this group, the State has identified where problem-solving is
needed to ensure that all students can access the online exams. Areas of concern range
from the simple mechanics of providing the accommodations through existing technology to
software or hardware conflicts that may limit use of necessary accommodations such as
screen readers, requiring longer term planning and solutions.
Report
from North Carolina
North
Carolina is in the first year of full implementation of a computerized adaptive testing
approach. It developed this option as a way to solve issues of access to the assessment
system for a small number of students who are not appropriately assessed in the alternate
assessment for students with significant cognitive disabilities, but who also cannot
participate in the general assessment in a meaningful way, even with available
accommodations. According to the presenters, if this approach works as it should, the
adaptive nature of the test should allow more precise measurement for these students, with
fewer test questions and greater flexibility.
The
presenters identified several issues, including concern about whether they are changing
the constructs being measured, whether subscore reporting is reliable given fewer and
varied items, and whether the blend of items in the item pool is distributed appropriately
across grade levels and content. One issue of great concern is the capacity of IEP teams
to make appropriate decisions on how each student should participate. For example, they
anticipated that a “few thousand” students would be caught in a gap between the
alternate assessment and the general assessment with accommodations. Instead, this year
over 30,000 students were identified as needing the computerized adaptive test. The
presenters concluded that many of these students are inappropriately placed in the
adaptive option, suggesting that increased training for IEP teams is needed.
Next
Teleconference
Be
sure to reserve September 30, 2002, for the first teleconference of the 2002-2003 school
year at 11:30 Eastern, 10:30 Central, 9:30 Mountain, and 8:30 Pacific time. Notices of the
topic and registration information will arrive in August. Contact Rachel Quenemoen at
quene003@umn.edu if you have suggestions for topics or presenters for future
teleconferences.
Alternate
Assessment Results in Accountability
NCEO
has produced Synthesis Report 43 and Policy Directions 13 on the topic of including
alternate assessment results in accountability decisions. These products reflect the
content of the conversation in the October 29, 2001 teleconference on alternate
assessments in reporting and accountability. The characteristics of good alternate
assessments are highlighted as a starting point for the discussion of two accountability
options for counting alternate assessment results: (1) scale the results so that the value
awarded for the achievement levels on the alternate are the same or similar to the value
awarded for achievement levels on the general assessment, and (2) scale the results so
that the achievement levels on the alternate assessment are at the lower end of the scale
and achievement levels on the general assessment are at the upper end of the scale. These
approaches are compared and actual state examples of ways to implement the first option
are highlighted.
Two
new products are available from NCEO that address the application of universal design
principles to assessment. Synthesis Report 44 builds on universal design principles first
used in architecture by identifying seven elements of universally designed assessments—(1)
inclusive assessment population, (2) precisely defined constructs, (3) accessible,
non-biased items, (4) amenable to accommodations, (5) simple, clear, and intuitive
instructions and procedures, (6) maximum readability and comprehensibility, and (7)
maximum legibility. The meaning of each of these elements is described, and examples
provided. Universally designed assessments are a promising approach to providing
appropriate assessment conditions for all students, giving each student the opportunity to
demonstrate achievement of standards being tested.
Policy
Directions 14 is a quick overview of the concept of universally designed assessments. It
is designed to communicate to policymakers and others who need a quick overview of
universally designed assessments.
A
literature review on the opportunities and challenges of computer-based testing for
students with disabilities is the topic of NCEO’s Synthesis Report 45. This review
examines computer-based testing in light of the needs of students with disabilities. It
pays particular attention to considerations that must be made for accommodations when a
test is computer-based.
Starting from
the assumption that the elements of universally designed assessments are in place, the
paper identifies five steps that should be taken to ensure a good trans-formation from
paper and pencil tests to computer-based testing for students with disabilities: (1)
assemble a group of experts to guide the transformation, (2) decide how each accommodation
will be incorporated into the computer-based test, (3) consider each accommodation or
assessment feature in light of the constructs being tested, (4) consider the feasibility
of incorporating the accommodation into computer-based tests, and (5) consider training
implications for staff and students.