Models for Understanding Task Comparability
in Accommodated Testing

The 1997 amendments to the Individuals with Disabilities Act (IDEA) contain a strong directive for the participation of students with disabilities in large-scale testing. Yet, the format of the assessments themselves is not specified. For example, the language of the reauthorized legislation includes the following statement: "Children with disabilities are included in general State and district-wide assessment programs, with appropriate accommodations, where necessary" [IDEA, section 612 (a)(17)(A)]. The question then, is what constitutes an "appropriate" accommodation?

1. The measures used within any given eligibility area must be technically adequate (have established reliability and validity). For example, when students are found eligible to receive special education services, we need to consistently and truthfully document a disability and its adverse educational impact.
   
2. To the greatest degree possible, students should be able to adapt to the standard testing situation, and if any changes are made they should be only minor ones. That is, changes should not be made if they don’t need to be made.
   
3. The skill being tested should be the same regardless of any changes made in the way the test is given or taken. Changes in testing must be made that are limited to the removal of "irrelevant sources of difficulty but still measure the same construct" (Thurlow, Ysseldyke, & Silverstein, 1995, p. 264).
   
4. The meaning of scores should be the same regardless of any changes being made in the manner in which the test is given or taken. As in the third point above, not only should the skill remain the same, but it’s meaning and the implications for using the scores to make decisions should not be different with changes in the test administration or response.
   
5. The accommodation should not have the potential for benefit for students without disabilities. Any change must, therefore, reflect differential performance among different students: Some students should be affected positively by the change and others should be unaffected.

This approach to determining appropriate accommodations is controversial for two reasons: (a) it is uncertain what the best comparison groups are - all students in general education, just low performers, or just average performers; and (b) it is not clear how students with disabilities should be sorted for comparisons - by disability category or by area in which services are provided (e.g., reading decoding, language processing, etc.). The finding of differential effects implies that there has been differential access when no accommodations are used. Equal access is achieved, for example, when an accommodation improves the performance of a student in that area where the student receives special education assistance, but does not improve the performance of average students not on IEPs.

A Continuum of Testing Options

A framework can be constructed for placing accommodations on a continuum with the following definition used to create four groups: An accommodation is a change that (a) provides unique and differential access (to performance) so certain students may complete the tests and tasks without other confounding influences but (b) does not change the nature of the construct being tested. Such changes typically are designed for specific individuals and for particular purposes.

On the left side of the continuum (Figure 1 above) is a standard test and an accommodated test in which minor changes are made in the way it is given or taken and may be considered a "standard" assessment, and reflect the same construct. The purpose of making accommodations is to provide access to participation in an assessment program with a primary test or instrument. The changes may be implemented within or across special education populations, referring to the unique needs of students in a manner that may or may not be disability oriented and/or related to an adverse educational impact. Because the construct has not changed, scores can be aggregated.

Changes also can be made for specific students that are substantial and modify the construct being tested. Although the assessment still focuses on documenting performance as part of an assessment program with a primary instrument or test, substantial changes are made in the administration-response of it. The net effect is that different tasks and noncomparable scores are created compared to those generated in the standard-accommodated assessment. As a result, some type of dissagregated reporting system may need to be considered.

At the most extreme end of the continuum, changes in test administration and responses lead to alternate assessments. For changes made with students with severe disabilities and with unique needs for whom the primary measure is not appropriate, such alternate assessments may be needed. Rather than documenting performance on the primary instrument or test, one might sample behavior using tasks uniquely created to meet the needs of the student. The tasks and the scores would be noncomparable to those reflected in the primary measures and a different set of measures as well as dissaggregated reports may be needed. These changes in the assessment program are depicted in the figure above as falling on a continuum that quite likely reflects both decreasing numbers of students and increasing amounts of change in moving from left to right.

As can be seen in the table of accommodations provided by NCEO (see Table 1), tests can be administered in a variety of ways (e.g., individually or in whole class, in one session or in multiple sessions); such changes also can reflect the way in which the test is taken or behavior is sampled (e.g., orally responding for a scribe to write rather than writing). Four general classes of changes in testing practices have been used by various state departments: (a) timing/scheduling, (b) setting, (c) response, and (d) presentation (Thurlow, Ysseldyke, & Silverstein, 1995). Furthermore, responses can be changed through different test formats or with the use of assistive devices. Examples of the more obvious response accommodations include increasing the spacing on the test, using graph paper, using wider lines and/or wider margins, giving the response orally, using paper in an alternative format (word or line processed, Braille, etc.), and allowing the student to mark responses in a booklet instead of on an answer sheet. Likewise, within presentation changes, a distinction is made between changes to the test directions and the use of assistive devices or support changes. These authors also list a number of examples in which the test presentation format is changed by (a) using Braille, magnifying equipment, or large print; (b) signing directions; (c) interpreting directions; and finally, (d) orally reading the directions.

Table 1. Assessment Accommodations NCEO/1996-Working Draft

Although these sets of accommodations appear to make considerable sense in the practical world of large-scale assessment and while many states have adopted some of these and others (Siskind, 1993b), it is difficult to justify or explain why they are not uniformly adopted: Some states allow use of some of these accommodations and others do not allow them to be used (Thurlow, Scott, & Ysseldyke, 1995; Thurlow, Seyfarth, Scott, & Ysseldyke, 1997).

Models for Making Decisions about Task Comparability

In this paper, three models are presented for determining task comparability, reflecting both the current state of decision-making and providing a strategy for enhancing current practices. The first model is descriptive. It focuses on policy presentation, interpretation, and analysis. This approach is similar to that used in setting standards, in which judgments are made for passing scores. When setting standards, the judgment is about the cut-score; in identifying accommodations, the judgment is about task and response (score) comparability. The second major model in determining task comparability is comparative. This approach provides empirical, post-hoc evaluation information on implementation of accommodations to help develop or revise policies. The third model is experimental. It provides a more formal data collection system to control threats to validity (internal and external). This model implies control over the selection of participants rather than the use of intact groups and the assignment of subjects to conditions rather than post-hoc evaluations; both features help establish cause-effect relationships. Two types of experimental designs are available for studying groups and single cases.

In all three models, the common focus is on determining whether the construct being measured is modified when testing conditions or tasks are changed. Although differences exist among models in the emphasis given to data to inform the decision-making process, all three models or combinations of them can be used to make decisions on task comparability. Therefore, the models should not be viewed as completely separate. For example, with a policy model, judgments are made through simple reference to the policy to a more formal analysis. From a post-hoc evaluation perspective, decisions about accommodations are based on multiple data sources and the relationship among variables, ranging from process-implementation data to student outcome data, though in all instances the data are within extant structures and procedures. Finally, with an experimental approach, task comparability rests on well-controlled designs for collecting data and making inferences from findings. For both types of experiments, group or single case, threats to internal validity are minimized so that cause-effect statements can be made. When groups of students are studied, the designs call for matching treatments (types of accommodations), students, and outcomes. When single cases are studied, a functional analysis is used to either hypothesize and/or verify important distinctions in tasks or student responses.

In Figure 2 below, these models have been placed on a continuum from less to more data-based decisions. On the left, policy and data to inform policy are endlessly intertwined; moving to the right, data are used to justify, explicate, and validate policy, first being created from within policy and then being generated outside of the policy with increasing levels of sophistication and use.

This process for making decisions should help schools fulfill the mandates of the newly reauthorized IDEA legislation. In particular, it should help states ensure inclusion, provide accommodations or modifications in large-scale assessment programs, and report outcomes. Schools now are required to include students with disabilities in all district and state testing programs, as well as provide appropriate accommodation when necessary; therefore, systematic procedures are needed for classifying accommodations. Furthermore, for students exempted from taking such tests, not only must this decision be explained but alternate, modified measures need to be provided; these measures should be sensitive to individual student needs. Finally, performance must be reported for students with disabilities, whether participating in accommodated or modified testing programs. Performances will have to be reported both aggregated with other students, and disaggregated. If performance cannot be reported in the aggregate, then it still needs to be disaggregated and reported.

In Figure 2, the standard for making judgments appears as a continuum with an increasing scale of evidence. Furthermore, as states formulate policy, a very uneven mix of evidence may co-exist across the decision areas of inclusion, accommodation-modification, and reporting outcomes. Some states may have a better evidentiary base in some areas than others. Furthermore, with states changing rapidly in their ramping up to meet the legislative mandates, a quickly changing landscape is in the offing. Therefore, in the examples noted below, actual state policy may be different now; nevertheless, at the time to which it is referenced, it provides a good example of the six types of evidence within the three models.

Descriptive Model

Three types of evidence are used in a descriptive model. They rely on current policy for decision-making. They range from a simple presentation of policy with no other information, to a justified presentation, and finally an analysis of policy. They all use a common language that informs others through policy, with varying degrees of explanation or justification. Little external information, however, is presented in the policies to ascertain the worth of either the judgments or the policies, hence the term descriptive. Examples of external information include presentation of information outside of the policy itself (e.g., from other states’ policies).

Policy Presentation

In this type of evidence, decisions about task comparability are made by referring to policy. Only slight variations in the degree of justification for the decisions are made. In one extreme of the descriptive model is a simple list of allowable accommodations that are assumed to define task comparability.

Some states simply present policy to define comparability, with little justification provided. In general, the major guiding premise has been that the accommodation be explicitly listed as allowable in state or district policy or test publisher’s manual and that it be described on the IEP, therefore making it part of the daily instruction and test situations. In the other extreme are policy directives with explanatory notes or interpretations. In these cases, the reasoning behind assumed comparability or noncomparability is explained in the policy.

Policy Interpretation

When policy is interpreted, not just documented, explanations are provided that help guide the decision-making process. Although little external information is presented outside of the policy, the judgment process is clearly more transparent and can be internally cross-referenced. The two examples of such systems are from Texas and Maryland. Policy interpretation may anticipate implementation strategies, for example, with a list of allowable accommodations to be distributed to parents, students, and school personnel; training and technical support may be provided then to help educators select appropriate accommodations. Interpretations may help teachers understand which accommodations are appropriate for students: how to base them on the needs of individual students and how to ensure that accommodations have been used during instruction prior to testing.

The use of case studies to explicate the decision-making process provides an interesting array and range of different examples by focusing on the minimal differences (only slight nuances that distinguish the two examples from each other, one of which reflects a positive instance and the other of which represents a negative instance). This strategy is an extremely efficient and smart way to explicate the central feature of a concept (i.e., accommodation versus modification) because it is possible to ignore other (irrelevant) features of the concept except those that are minimally different.

Policy Implementation Analysis

At some level, task comparability may be considered by tracking the implementation of various accommodations. In conducting this kind of analysis, data are collected to understand factors associated with the use of accommodations. The focus of this perspective is to understand the degree to which accommodations are proposed and/or used within the context of other (student) mitigating factors so that one could begin to explain why some accommodations tend to be selected.

By analyzing the degree to which students arrive at testing situations with accompanying problems such as those noted above, it should be possible to understand why some accommodations are recommended while others are not recommended. For example, rather than simply noting that accommodations include separate testing with flexible scheduling, it would be possible to justify an accommodation based on the student’s capacity to work in a group, work independently, and remain on-task for 45-60 minutes. For students who cannot be engaged in this manner, it can be inferred that the performance is adversely affected and the test is not measuring what it should be measuring. This information, documented by the IEP team, would help teachers make decisions about accommodations.

In reporting accommodations data, meaning and the basis for understanding is provided in the relationships among the variables. For example, the same data reported for two groups of students may lead to an understanding why an accommodation is appropriate or permitted some of the time for some students and not at other times for different students. In the data reported by CCSSO, considerable differences exist in the number of states allowing various accommodations for students with disabilities versus those with limited English proficiency (see Table 3, which reflects accommodations permitted by various states, ordered from most to least frequent for students with disabilities).

Further distinctions can be made comparing types of decisions (individual accountability versus group evaluation) or types of test instruments (published, norm-referenced versus state-specific, standards-based) to help inform policy about implementation, particularly the implications. In effect, task comparability is likely a function of the students being tested, the decision being made, and the type of test being used.

Clearly, policy implementation analysis begins to move task comparability judgments into a comparative realm, with data collected at various levels of the testing program. In the examples cited earlier (see Policy Implementation Analysis Example Two), the primary information was the number of states permitting an accommodation for various types of students. Another strategy would be to collect data on the actual frequency of use as done in Missouri (see Policy Implementation Analysis Example Three).

Other perception and survey data also could have been collected. For example, how important are accommodation for helping students participate and succeed (as judged by students, parents, and teachers), and to what degree are there differences among various groups of people? How much better did certain students do when provided accommodations than comparable students who were not provided the same or similar accommodations? When questions like this are asked, judging comparability begins to move toward a comparative model involving post-hoc analyses. The shift is from collecting singular data within systems and making inferences across them to collecting multiple data within a system and co-relating the information together. In this shift, policy analysis moves to post-hoc evaluation.

Comparative Model

The comparative model shifts the definition of task comparability to include multiple data sources within a system. This model incorporates frequency data on use of accommodations, judgments about their appropriateness, and outcomes data on performance when accommodations are used. With these data sources, relational statements can be made about process and performance.

As the next example illustrates, one of the limitations of post-hoc evaluations is the sample used, in particular its representativeness to other samples. Furthermore, little information is known about how the accommodation was related to previous instructional programs and other sources of influence that may have been operational at the time of testing. Therefore, it is difficult to determine the exact cause of performance differences with and without accommodation.

Although these outcomes provide the field with a rich data source for considering accommodation, they represent post-hoc evaluation data that is confounded with several other variables. For example, the research is limited to college admission testing, all of which represents a limited group of tests for students with disabilities (e.g., those who are both secondary students and who are college bound). The proportions of those with disabilities who participate in such tests are very small and may not be representative of the larger group. And, the tests are unlike those used in statewide accountability systems.

As can be seen with these three examples, a post-hoc evaluation perspective provides considerable data on both process and outcome. Generally, the results can be organized so that the findings may be somewhat conclusive though the explanations for them are not as certain. The major problems with this approach for examining task comparability is that cause-effect relationships are difficult to make using post-hoc evaluation data. For example, by using intact groups, many threats to internal validity exist about selection of subjects, as well as the mortality of subjects (those who drop out or appear with incomplete data), historical events that occur during the study period, and subject maturation (physical or psychological changes). Furthermore, subject selection may interact with any of these latter threats. For example, in many school systems, differential drop out rates occur for students from various ethnic backgrounds, reflecting a problem in which subject selection interacts with mortality. Another major limitation is the manner in which subjects are assigned to treatments (accommodation): It often is not clear and the representativeness of the subjects is uncertain. Finally, in most evaluations, no control or comparison groups are used to determine the differential effect of a treatment (accommodation). For example, while the two studies from Florida (Beattie, Grise, & Algozzine, 1983; Grise, Beattie, & Algozzine, 1982) provide important initial findings about the comparability of tasks, no general education students received the modified tests.

Experimental Model

In the experimental model, a research design is established before data are collected. Appropriate controls are implemented in the manner in which data are collected to ensure that conclusions have integrity. Two components to an experimental model include (a) a research design for organizing the investigation and controlling threats to the validity of the findings, and (b) a technically adequate measurement system to scale behavior so that inferences can be made from the outcomes.

Either group designs or single-subject designs can be used within the experimental model. Both designs should address four specific threats to validity identified by Cook & Campbell (1979): (a) internal, to allow appropriate inferences about cause and effect; (b) statistical conclusion, to ensure that data are appropriately analyzed; (c) external, to consider other populations and settings for applying the findings; and (d) construct, to explain the theoretical network in which the findings are placed. Group designs for assessing the effects of accommodations typically involve comparing students with disabilities using and not using accommodations to either themselves or another group of students. Single subject designs most often use the same student, comparing performance when an accommodation is used to performance when the accommodation is not used.

Group Designs

Three factors need to be considered in conducting a group design: (a) students, (b) treatments (accommodations), and (c) outcomes (measures). These three factors may be either crossed with each other (all factors are presented with each other) or nested (only some factors are presented with each other). Below is a study that depicts each of these relationships for two factors: students and treatment accommodations.

With an experimental model, not only must a research design be used to investigate the relationship among factors (students, treatments, and measures), but a valid measurement system must be used. Validation, however, is not of the tests or measures but of the inferences made from them (Messick, 1989). Messick employs a two-by-two matrix for understanding the construct validity of score meaning and social values. One facet is the source of justification for testing, which may consider evidence to understand score meaning or values to understand consequences. The other facet is the function or outcome from testing, which addresses how the test is to be interpreted or used. Crossing these two facets creates a four-celled table highlighting a unitary view of validity integrating meaning and values. The matrix is designed to build understanding in a progressive manner so that any construct being measured is neither underrepresented nor with irrelevant variance.

1. The evidential basis of test interpretation reflects construct validity (CV) in addressing convergent-discriminant evidence; the focus of interpretation is primarily scientific and empirical.
2. The evidential basis of test use focuses on the construct validity (CV) of performance in applied settings with the benefits of testing considered in relation to costs and relevance/utility (R/U).
3. The consequential basis of test interpretation is comprised of construct validity (CV) with reference to broad theories and philosophical views, all of which address value implications (VI) and become embedded within score meaning. This block often "triggers" score-based actions.
4. The consequential basis of test use considers construct validity (CV), relevance/utility (R/U) and value implications (VI), and potential as well as social consequences (SC) in applied settings, focusing on equity and fairness along with many other broad social interpretations, in a sense the functional worth of the test.

The validation process in this progressive matrix is never definitive but proceeds in an iterative manner with evidence interacting with values to create score meaning and interpretations. The validation process includes both evidence and value in an evaluative rhetoric of argumentation. Finally, validation is interpretative, addressing both the meaning of scores and the use of them to make decisions in applied settings (Messick, 1995).

In summary, group design experimental studies need to rely not only on systematic and prospective data collection procedures but also on technically adequate measurement systems. By engaging in a prospective design, explanations of outcomes are less confounded and contain fewer threats to validity. Furthermore, experimental investigations with groups of students need to reflect a program of research in which studies are linked together not just to generate findings but also to inform decision-making. Tests need to be used and interpreted with classroom implications; they need to be validated in the context of both evidence and consequences in this interpretation and use. In the final analysis, however, group designs cannot be used to make predictions for individuals. Although the average (mean) performance may have been higher or lower in either of these groups, not all students may have been equally affected.

Single Case Studies

For individual decisions, only a behavioral approach, utilizing single case studies, can be considered. When the experimental research is aimed at judging task comparability for individual students, the research paradigm is anchored to the unique needs of the student and maximizes the relationship between the process and the outcomes. Using group designs an accommodation may be found to be effective for students with disabilities and not for students without disabilities. Nevertheless, this general finding cannot be used to make predictions of effect or lack of effect for every specific student. All effects are based on group averages and reflect a likelihood for finding that effect with individual students.

Functional analysis focuses on the function of various environmental variables in controlling behavior within the context of reinforcement contingencies. In a functional analysis, the single case is used to define the comparability of a task or response. Generalizations across individual cases can proceed only tentatively, because the hallmark of a functional analysis is the focus on the individual case. When the function of behavior is experimentally analyzed, the proof of comparability is documented consistently within and across phases in which tasks have been changed. Task comparability can be considered from two perspectives with a functional analysis, either in (a) defining a response class or (b) to infer the function of the behavior and response class.

Within a functional analysis approach, contingencies are considered in terms of specific reinforcement paradigms (i.e., positive and negative reinforcement, escape, avoidance, etc.). Basically, a class of responses has a common effect on the environment (the definition of an operant) and is therefore premised upon the assessment of environmental variables that influence the behavior or class of behaviors. Two caveats need to be considered in this perspective: (a) reinforcers are defined in terms of the probability of increasing or decreasing a behavior, and (b) a history of reinforcement "in turn, influences how an individual responds to current environmental contingencies" (Mace, 1994, p. 385): The same "reinforcers" may not be considered uniformly across individuals.

Response class analysis. In determining task comparability, it is important to entertain the possibility that different specific behaviors may function as part of the same response classes. This provides a useful way to think about slightly different behaviors that are maintained in a similar manner in the environment. Response classes are used in behavior analysis to describe discrete behaviors that (a) may have different topographies but serve the same function (i.e., are controlled by the same reinforcement paradigm) or (b) may have similar topographies but serve different functions (are controlled by different reinforcement paradigms). Individual behaviors may belong to more than one response class (i.e., may have multiple controlling contingencies).

Behaviors that form response classes are easy to describe with social behaviors, for example, many behaviors that reflect "compliance" include attending, making eye contact, initiating behaviors upon request, responding with speed and accuracy, etc., all of which have the same function: They reflect a connection between the "mand" (command or demand) and the response. In the case of students who "comply" when interacting with others, they perform upon request and with attention to the task; for those who do not "comply," they delay their response or fail to fully perform it.

Behaviors that apply to academic situations are less common. In this kind of functional analysis, the focus needs to be on the manner in which the student generates a response that serves the same function as other behaviors. For example, with students having a physical disability, a paper-pencil task may be impossible because of a lack of limbs or digits. If this student can read the problems and choices on a multiple-choice test, but cannot fill in the bubbles because of this physical disability, the test may include an accommodation in which the student presses a keypad on a communication board or signals the correct response verbally. Few people would have trouble in considering this response comparable to the physical act of filling a bubble on a response sheet.

But what if the test is designed to measure a broad construct with a number of possible responses used to provide a solution, as is the case with many performance tasks? Then, it is critical to make decisions about task comparability in relation to the outcomes, not the indicators. By distinguishing between outcomes and indicators, it is possible to consider tasks as comparable at one level (as a reflection of an outcome) when obvious differences exist in their topography (as reflections of different indicators).

Use a variety of reading strategies to increase comprehension and learning. Locate information and clarify meaning by skimming, scanning, close reading and other reading strategies.

At the outcome level, several different behaviors form a response class, all which are different topographically. Phillips (1994) suggests this approach when she states: "To determine whether an accommodation affects test validity, one must examine the test objectives as they are currently written...if test objectives are to communicate accurately to test users, the skills tested must be measured as specified in the objective so that the inferences made from the test scores will be the same for everyone" (Phillips, 1994, p. 99).

By considering whether test behaviors function within the same response classes, it is possible to both differentiate indicators from outcomes and ascertain the degree to which they are maintained by similar contingencies. With the above reading outcome defined in Oregon, a number of different behaviors can be used to show reading, essentially providing a relatively robust response class. For example, consider a person who is visually impaired to such a degree that only Braille or print-to-voice transcription can be used to acquire information from text. In Oregon, the person would be considered to be receiving a comparable task if it can be demonstrated that the behavior results in the same effect upon the environment as the regular reading task does for those with sight. For example, the verbs within the various content standards include locating, skimming, identifying, determining, analyzing, and evaluating. All of these behaviors essentially belong to a large response class in which the person being tested interacts with printed text, though no limitation is made for how that text is accessed. The outcome is interacting with text and task comparability is determined by analyzing the objectives of the test, not the indicators reflecting their attainment. In this case, many different indicators could reflect effective reading using a variety of tasks, all of which are comparable at some level dealing with the effect they have on the environment.

In contrast, if the outcomes and indicators are confused, then task comparability is difficult to analyze by reference to response classes. For example, in states that mandate passing a specific norm-referenced test as the outcome, then any administration of the test in any manner different from that used in developing the normative standard, would be a different task, making it noncomparable.

The issue of behavior-maintaining contingencies, which is considered in the next section, can only be considered after clarity is brought to the issue of response classes in defining the critical behaviors for determining success. In the next section, it is assumed that specific behaviors are clearly defined in terms of outcomes and indicators; obviously, if confusion exists, these subsequent strategies may be less than useful.

Functional analysis through interviews, observations, and behavioral manipulation

Assuming that specific behaviors are defined and their meaning has been qualified in terms of outcomes and indicators, three methodologies for conducting a functional analysis can be used in ascertaining task comparability and developing accommodations for students. First, interviews can be conducted of key individuals who would be knowledgeable about the student. Second, direct observations can be made of students to determine what contingencies maintain the occurrence of specific behaviors in various settings. Third, behavioral manipulations and interventions can be made to determine how behavior fluctuates with the presence-administration of various reinforcers-punishers. All three approaches, however, are aimed at understanding how the behavior functions in the environment and is maintained in level and rate (O’Neil, Horner, Albin, Storey, & Sprague, 1990).

Interviews. With interviews, the following issues are addressed: (a) dimensions of behavior (topography, frequency, duration, and intensity), (b) ecological events that have a potential impact on behavior (e.g., medications, sleep cycles, eating routines, or activity schedules), (c) events and situations that predict the occurrence of behaviors (e.g., time of day, setting, social control, or activity), (d) the functions of undesirable behaviors and a history of their occurrence, (e) the efficiency of undesirable behaviors (e.g., physical effort, along with likelihood of "payoff" and delay), (f) primary mode of communication, (g) events, actions, and objects that are perceived as positive, and (h) the presence of "functional alternative" behaviors. As can be anticipated, with answers to such questions, it should be possible to formulate the degree to which various behaviors interfere with or facilitate appropriate social and academic functioning.

Direct observations. With direct observations, similar information is collected on the ecological context, as well as antecedent-consequent functions, of various behaviors. With this format for collecting information, the following dimensions are considered: (a) the setting to observe, (b), the time of day, (c) specific behaviors, (d) setting events that appear to serve as discriminative stimuli, (e) the perceived functions of behavior (i.e., whether to obtain something or avoid-escape something), (f) the actual consequences that occur, and (g) any anecdotal reactions noted by the observer. In addition to helping members of IEP teams collect useful information for developing appropriate accommodations, this kind of information, when collected across various settings, can be used to begin explaining why some behaviors tend to occur more or less than other behaviors. The advantage of collecting direct observational data over interview data, is that the information is more objective (verifiable) and contemporary.

Behavioral manipulation. Finally, systematic manipulations of reinforcement contingencies can be implemented, based on either the interview or the observational data, to determine whether specific behavioral changes can be made. With minimal inferences made about cause and effect, this method of conducting a functional analysis can provide the most persuasive information about why specific behaviors are present and persist in relation to various environmental events. With this system for conducting a functional analysis, several designs are available within two broad classes of withdrawal-reversal and multiple-baseline designs.

When functional analysis is used to consider whether behavior is being maintained under different conditions, "the function of a behavior may be affected by the context in which the behavior occurs" (Day, Horner, & O’Neill, 1994, p. 279). With a functional analysis, the focus is on the larger investigation of the maintaining contingencies, their identification and manipulation of variables that control behavior.

Figure 5. Academic Performance under Varying Accommodations: Words Correct Per Minute

Figure 6. Academic Performance under Varying Accommodations: Error Count and Prosody Rating

Figure 7 - On-Task Behavior under Varying Accommodations

Summary of single case approaches. In summary, functional analyses of individual students provides a basis for determining whether tasks are comparable for accommodated and standard administrations. In two major ways, tasks may be judged as comparable: (a) by classifying test behaviors within comparable response classes and (b) by identifying potential variables that appear to be maintaining behavior in various settings. When considering tested behaviors as part of a response class, accommodations may focus on either the adjunct behaviors in which the student accesses the test-taking situation or as part of the tested behavior itself. When considering task comparability through a functional analysis, interviews, observation, and direct manipulation may be used to provide a line of evidence. Obviously, fewer inferences are made as one proceeds first through the use of interviews (identifying plausible explanations), then through the direct observation of students in various settings (comparing actual contingencies), and finally through behavioral manipulation of contingencies (providing confirmation of cause-effect relationships).

One caveat should be noted in using a functional analysis to determine task comparability when accommodations are used. In many instances, students are performing on tests and in testing situations with a complex arrangement of contingencies maintaining their behavior. Generally, these contingencies have been developed over a long time and involve many individuals, from parents to teachers and peers. In considering accommodations to maximize student participation in and successful completion of large-scale tests, the accommodations are not considered prosthetics for life. Rather, they are an arrangement of the environment representing a next step toward independent behavior management (that is, self-managed behavior). This view of accommodations is very different than those in which a prosthetic is being used that is an ever-present part of that student’s life (for example, Braille is not a temporary accommodation for students with vision impairments). Yet, a consistent reinforcement program aimed to get students to attend, listen, read, and respond, may well be needed for only a short while and eventually be faded into the more standard approach in which a few short, motivating statements are made at the beginning of the test for students to do their best.

Summary and Conclusions

Six approaches for determining the comparability of tasks have been presented in this paper, organized into three models: descriptive, comparative, and experimental. This evidential continuum was developed to reflect an increase in both the amount and systematicity of data-information that is collected to support the decision-making process. By referring to data-information, the overall purpose is to make consistent decisions about the acceptability of variations in tasks, and therefore decisions about whether they can be considered accommodations, or whether they are modifications that change the nature of the construct being measured.

The continuum began with three types of descriptive data-information, in which task comparability is simply presented, with decisions presented in relative isolation within policies. No other evidence would appear, with policy serving as a reference to be followed. For educators to use this form of decision making, they simply have to know the policy. The advantage to this strategy is ease of use and clarity of direction: Either tasks appear on the list of accepted accommodations or they do not appear on the list. The disadvantage is that the list is finite, when in fact, a nearly infinite number of variations in assessment construction and administration actually is possible: When an accommodation is proposed and it is not on the list, little guidance is provided for determining whether it is an acceptable one.

Further along on the evidential continuum, task comparability was determined by reference to policy that was elaborated and interpreted. By having information that explains policies, educators can make decisions about the use of various accommodations and also can apply a similar logic to instances not specifically addressed in the policy. Through the use of examples and nonexamples, minimum differences can be highlighted, providing educators a frame of reference for making decisions that not only supports a list of accepted accommodations but the rationale for their placement into either acceptable or unacceptable changes. The advantages clearly are the breadth of instances that can be addressed, moving well beyond a finite list. Yet, in this breadth, it is possible that rules and rationales become misinterpreted and misapplied, reflecting a disadvantage in the confusion and lack of standardization that likely is to result.

The last approach within the descriptive model was a policy analysis perspective, in which the decision to consider tasks as comparable was made by reference to data-information within the assessment system. This perspective provides educators a frame of reference for not only interpreting but understanding the implications from any decisions that are made. Although viewed with singular data sources that are non-relational, this information can be extremely valuable in providing outcomes that contextualize any decisions. Obviously, the advantage to this perspective is the possibility of better understanding what it means when a task is deemed comparable; educators thus have the ability to make predictions. The major disadvantage is the expense in data collection needed to implement this perspective. This perspective was considered descriptive because the data-information were singular and not relational. In contrast, the remaining perspectives can be viewed in a relational manner, in which understanding an accommodation is dependent upon several other variables that can be inter-related.

In a comparative strategy, post-hoc evaluations are used to make judgments of task comparability, with several variables considered in the process. Truly relational statements can be made in which implications can be framed and analyzed to not only make decisions but also test predictions, with follow-up data collected to formatively evaluate the effects of those decisions. This perspective has great appeal in providing an empirical basis for change and therefore presents educators with a significant advantage of being data-based; furthermore, over time decisions can continue to be racked in outcomes and implications. Nevertheless, this perspective has a disadvantage in being post-hoc with intact groups of students and lack of control from many threats to validity. Therefore, although results may be data-based, cause-effect statements may be made inadvertently, potentially creating more harm than good.

Finally, experimental strategies may be used to organize either a group or single case study. Both of these approaches help improve the decision-making process with not only data-information, but with many threats to validity controlled. For example, rather than only considering intact groups, studies are devised to include groups that represent important constituencies for comparison. In addition, resources are strategically allocated so that cause-effect statements can be made with more certainty. Rather than simply reporting outcomes and postulating inferences, these approaches provide information leading to explanations. While group designs are used to study task comparability with well-defined collectives of students having common demographics, they cannot be used to understand task comparability for individuals; therefore, single case designs are used to extend the logic of this scientific approach to individual students with unique needs. The clear advantage of experimental strategies is the integrity of understanding what accrues and the strength of predictions that can be made in similar situations. Yet, this certainty also is attained with considerable investment in collecting data-information. Furthermore, in many educational circumstances, it is difficult to both manage programs as they are currently configured and develop alternative ones concurrently. Therefore, such strategies are less likely to be implemented without external funds.

In the end, the continuum is meant to be broad and encompassing so that any educational agency can place itself on it. Furthermore, it is displayed as a true continuum that increases in the amount and quality of information used in making decisions about task comparability. As the 1997 IDEA mandates are implemented, it may be important to begin the process of taking inventory not on the issue of task comparability itself, but on the manner in which such decisions are made.

References

Benderson, A. (Ed.) (1988). Testing, equality, and handicapped people. Focus (ETS Publication; Princeton, NJ), 21 (23).

Bennett, R. E., Rock, D. A., & Kaplan, B. A. (1985, November). The psychometric characteristics of the SAT for nine handicapped groups (ETS Research Report RR-85-49). Princeton, NJ: Educational Testing Service.

Bennett, R. E., Rock, D. A., & Kaplan, B. A. (1987). SAT differential item performance for nine handicapped groups. Journal of Educational Measurement, 24(1), 44-55.

Bennett, R. E., Rock, D. A., & Kaplan, B. A. (1988). Level reliability and speededness of SAT scores for nine handicapped groups. Special Services in the Schools, (3/4), 37-54.

Cook, D., and Campbell, D. (1979). Experimental and quasi-experimental research designs. Boston: Rand McNally.

Helwig, R., Stieber, S., Tindal, G., Hollenbeck, K., Heath, W., & Almond, P. (1997). A comparison of factor analyses of handwritten and word-processd writing of middle school students. Eugene, OR: University of Oregon Research Consultation, and Teaching Program.

Hollenbeck, K., Tindal, G., Heath, W., & Almond, P. (1997). Reliability and decision consistency: An analysis of writing mode at two times on a statewide test. Eugene, OR: University of Oregon Research Consultation, and Teaching Program.

Koretz, D. (1996). The assessment of students with disabilities in Kentucky.CSE Technical Report No. 431. Los Angeles: CRESST/Rand Institute on Education and Training.

Messick, S. (1989). Validity. In R. Linn (Ed.), Educational Measurement (3rd Edition), pp. 13-105. New York: MacMillan.

Messick, S. (1995). Validity of psychological assessment: Validations of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749.

National Center on Educational Outcomes (1993). NCEO finds states active in ‘outcomes’ efforts. Outcomes, 2(2), p.2. Mpls., University of Minnesota: Author.

Nolen, S. B., Haladyna, T. M., & Haas, N. S. (1992). Uses and abuses of achievement test scores. Educational Measurement: Issues and Practice, 11(2), 9-15.

O’Neil, R., Horner, R., Albin, R., Storey, K., & Sprague, G. (1990). Functional Analysis of Problem Behavior. Sycamore, IL: Sycamore Publishing Company.

O’Sullivan, R. G., & Chalnick, M. K. (1991). Measurement-related course work requirements for teacher certification and recertification. Educational Measurement: Issues and Practice, 10(1), 17-19, 23.

Phillips, S. E. (1994). High stakes testing accommodations: validity versus disabled rights. Applied Measurement in Education, 7(2), 93-120.

Robinson, G. L., & Miles, J. (1987). The use of coloured overlays to improve visual processing - A preliminary survey. The Exceptional Child, 34(1), 63-70.

Siskind, T. G. (1993a). Modifications in statewide criterion-referenced testing programs to accommodate pupils with disabilities. Diagnostique, 18(3), 233-249.

Siskind, T. G. (1993b). Teachers’ knowledge about test modifications for students with disabilities. Diagnostique, 18(2), 145-157.

Thurlow, M. L., Ysseldyke, J. E., & Silverstein, B. (1995). Testing accommodations for students with disabilities. Remedial and Special Education, 16(5), 260-270.

Thurlow, M., L., Scott, D. L., & Ysseldyke, J. E. (1995). A compilation of states’ guidelines for accommodations in assessments for students with disabilities, [Synthesis Report No. 18]. Mpls., MN: University of Minnesota National Center on Educational Outcomes.

Tindal, G., Hollenbeck, K., Heath, W., & Almond, P. (1997). The effect of using computers as an accommodation in a statewide writing test. Eugene, OR: University of Oregon Research Consultation, and Teaching Program.

Tindal, G., Hollenbeck, K., Heath, W., & Stieber, S. (1997). Trait differences on handwritten versus word-processed compositions: Do judges rate them differently? Eugene, OR: University of Oregon Research Consultation, and Teaching Program.

Return to Accommodations Topic Area

Top of page