In an effort to validate the findings of experts, a series of items can be examined by students themselves using cognitive lab, or think-aloud methods.
Think aloud methods were first used in the 1940s and have since been used for a variety of "end user" studies in the fields of ergonomics, psychology, and technology. In the case of statewide assessments, the end users are students who will take tests. Think aloud methods tap into the short-term memory of students who complete assessment items while they verbalize. The utterances produced by students are the data that researchers or states can use to better understand items.
The verbalizations produced in think aloud studies provide excellent information because they are not yet in the long-term memory. Once experiences enter our long-term memory, they may be tainted by personal interpretations. Therefore, an excellent way of determining whether design issues really do exist for students, is to have students try out items themselves in "live time."
NCEO typically videotapes all think aloud activities, but states can also either audiotape or have several observers review field notes. Inter-rater agreement is important for making decisions based on think aloud activities, so some strategy for confirming what is viewed or heard during think aloud activities should be undertaken. In addition, it is useful to include students who achieve at a variety of levels on statewide achievement tests. To this end, a sample population might include students without disabilities and majority culture children as well as students with disabilities, English language learners, and students from low socioeconomic status. A NCEO recent research report on think aloud methods can be accessed at http://education.umn.edu/nceo/OnlinePubs/Tech44/
An example of a process for selecting and conducting think aloud studies is described in Vignette #1.Next: Universal Design Step 5