Educators' Opinions About Out-of-Level Testing: Moving Beyond Perceptions

Out-of-Level Testing Project Report 15

Published by the National Center on Educational Outcomes

Prepared by Jane E. Minnema • Martha L. Thurlow • Gretchen R. VanGetson

September 2004

This document has been archived by NCEO because some of the information it contains may be out of date.

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Minnema, J.E., Thurlow, M.L., & VanGetson, G.R. (2004). Educators' opinions about out-of-level testing: Moving beyond perceptions (Out-of-Level Testing Project Report 15). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/OOLT15.html

Background

Under pressure to include more students in large-scale assessment programs, especially those students with disabilities, states have used a variety of testing approaches. Several states have introduced out-of-level testing as one option for measuring academic proficiency. Out-of-level testing, which is most commonly defined as the administration of a test at a grade level below the grade a student is enrolled in school, has taken on multiple forms. Of the 17 states that used out-of-level tests in school year 2003–2004, some states tested students with disabilities at any grade level necessary to reach a student’s level of instruction (VanGetson, Minnema, & Thurlow, 2004). Other states chose to limit out-of-level testing to only those grade levels available in the regular large-scale assessment. States also differed in the test used for below grade level testing. Both norm-referenced and criterion-referenced tests were used as well as norm-referenced tests with additional criterion-referenced components that augmented the original instrument.

States’ out-of-level testing policies have been similar in the contentiousness that has surrounded them. Proponents of out-of-level testing generally contend that there are three benefits for students with disabilities: (1) undue test frustration is avoided, (2) test measurement accuracy is improved, and (3) test items are better matched to students’ current educational goals and instructional level (Thurlow, Elliott, &Ysseldyke, 1999). It is often thought that testing students with disabilities on the grade level of enrollment in school is actually unfair. Taken to its extreme, students can be traumatized by being forced to dwell on test items for which they do not have the academic skills.

On the other hand, opponents of out-of-level testing contend the following: (1) testing students below their grade of enrollment does not match the system accountability purpose of a statewide assessment, (2) lower grade level testing reflects teachers’, family members’, and students’ lower expectations for academic learning, and (3) the resulting instruction over time will focus on lower-level standards than those that the student could be striving to acquire (Thurlow et al., 1999). By continuing to test a student out of level from one school year to the next, a student may lose important opportunities to learn so that the student’s true learning potential may never be known.

Only recently have the many issues that surround out-of-level testing begun to be sorted out in the literature. Concerns have arisen about states’ reporting practices since out-level test scores are not readily available in states’ data reports that are used for local school improvement planning (Minnema & Thurlow, 2003). Determining the prevalence of students with disabilities who had been tested out of level on a state by state basis has proved to be difficult (Thurlow, Minnema, Bielinski, & Guven, 2003). A focused research study on states’ out-of-level test score reporting yielded no state with publicly reported out-of-level test results that were clearly labeled and readily identifiable unless state educational agencies produced special data runs of those test data (Minnema & Thurlow, 2003). Recent research findings have demonstrated negative instructional effects of testing students with disabilities below grade level (Minnema, Thurlow, & Warren, 2004a and 2004b).

Given the emphasis in current standards-based educational reform efforts for all students to receive challenging, grade-level standards-based instruction, these research findings are disconcerting. The research suggests that a relatively large subgroup of students—students with disabilities—is receiving instruction that is not on grade level.

Over the past few years, arguments similar to those raised in the late 1990s have continued to circulate in practice among educators and policymakers. The purpose of this research was to document the pro and con arguments that surround out-of-level testing. We conducted a survey to describe teachers’ and administrators’ perceptions and opinions about out-of-level testing when testing students with disabilities in states’ large-scale assessment programs that are used for accountability purposes.

Method

We created a one page, doubled sided written survey that contained closed and open-ended items to describe educators’ perceptions and opinions about out-of-level testing (see Appendix A for a copy of the survey instrument). The 16 closed survey items were in the form of opinion statements requiring respondents to indicate “agree,” “disagree,” “don’t know,” or “no opinion” on a 4-point scale. We decided to use this type of scale rather than a Likert scale so that we could learn whether respondents knew about out-of-level testing rather than having a measure of the strength with which they agreed with the statements in the survey items. The first survey item requested a “yes” or “no” response as to whether the participant was familiar with out-of-level testing. We also gathered demographic data on four closed survey items and additional opinion data on three open-ended items.

The survey was distributed in four states at state-level teacher and administrator meetings and training sessions. We mailed packets of surveys and written directions for completing the surveys as well as written distribution and collection procedures. One contact person in each state educational agency was responsible for distributing and collecting the surveys and returning the finished surveys in addressed, pre-paid mailings. In State 1, special education teachers (n = 53) and administrators (n = 81) completed surveys. In State 2, only administrators (n = 52) participated in the survey. In another state, State 3, both special education teachers (n = 29) and district test coordinators (n = 18) returned surveys. The final state, State 4, had special education teachers (n = 48), regular education teachers (n = 19), administrators (n = 89), and assessment coordinators (n = 54) participating in the survey. We did not calculate a return rate because our sample was a purposive sample where all training attendees were invited to participate, but not required to complete a survey. Since attendance was not necessarily recorded at all sessions, we had no estimate of the exact number of surveys distributed but not returned.

All numeric data were entered into spreadsheets and checked for data entry accuracy. We used descriptive statistics to analyze these data. The narrative data were transcribed and types of responses were tallied for categories of results. A second researcher analyzed one-fifth of the narrative data set to ensure accuracy of findings.

Only descriptive statistics could be used for data analysis of numeric data. Because of this, we set decision rules for interpreting our numeric data to draw final conclusions. For those items where one category received 40% or more with the other three categories receiving 25% or less of the responses, we called this occurrence “a clear majority.” We considered a response pattern to be “without a clear majority” when all four categories received 25% or less of the responses. 

Results

We present our survey results on a state by state basis by treating each subgroup of our sample as a separate data set. For each state, we first describe the state’s sample with various descriptive statistics. Second, the numeric survey results for each sample subgroup are presented in the next tables. Depending on the number of sample subgroups that participated, the next tables contain the actual survey items as created for the survey instruments with the states’ test names replaced with “state test” to protect the confidentiality of those participating in the research study. We also provide a column in our tables to report on the number of “no responses” per survey item, which was not part of our original survey instrument (see Appendix A for copies of the instruments). Other demographic results are also included in the next few tables to conclude the state’s report on numeric findings. Third, as a final section for each state’s survey results, we present the results of open-ended survey items.

Following our presentation of each state’s survey results, we take a step back from our data to glean global thematic conclusions that consider all states’ data as a composite whole. While these conclusions draw from states with approaches to testing students with disabilities that vary widely, our summative ideas are relevant in a general sense for educators and policymakers who are pondering the value of this approach to standards-based assessment.

State 1

State 1 is a large western state where only students with disabilities are tested below the grade in which they are enrolled in school. The state test is an augmented norm-referenced test that contains criterion-referenced items that are aligned to state content standards. State policy allows that the State 1 assessment be administered at any level below grade level that is necessary to match a student’s level of instruction, but only one or two levels below grade level is considered to be a standard test administration. These results are entered into accountability indices at the lowest proficiency level at the student’s grade of enrollment. Further below grade level is treated as a nonstandard test presentation, which cannot be entered in accountability indices.

Sample – State 1
A total of 134 surveys were distributed and returned in State 1. The subgroups in the purposive sample from State 1 contained special education teachers (n = 53) and school administrators (n = 81) who returned the sample either completed or with an indication that they were not familiar with out-of-level testing. Of the participants who returned the surveys, 91% (n = 48) of special education teachers and 93% (n = 75) of school administrators responded affirmatively to the question about familiarity with out-of-level testing. When asked whether teachers in the respondents’ school districts tested students with disabilities out of level, 58% (n = 31) of the teachers said “yes,” with 9 teachers not responding. For school administrators, 68% (n = 55) indicated that students with disabilities were tested out of level in their school district while 15 did not respond to this survey item. There were five special education teachers and six school administrators who did not respond to the item pertaining to familiarity with out-of-level testing, but completed all of the remaining survey items. Overall, we were satisfied that the educators in State 1 who responded to our survey were familiar enough with testing students with disabilities out of level in large-scale assessments to knowledgeably answer the survey items.     

Other State 1 sample demographic features are presented in Table 1. These data indicated that sample participants were well experienced educators. Nearly 75% of teachers and 70% of administrators had at least 11 years or more of experience in schools. In terms of educational background, the sample was also well educated. Eleven percent of teachers and 7% of administrators had Bachelor’s degrees. Almost 90% of the special education teachers and over 90% of school administrators held advanced degrees with the majority of both groups holding Master’s degrees. Please note that missing responses were not included in this table due to space constraints. In terms of years of experience, seven teachers did not complete this survey with 11 school administrators not responding. For educational degree information, eight teachers and 11 administrators did not complete the item.

Table 1. State 1 Sample - Participant Levels of Experience and Education

^a Non-responses to this item for 7 teachers and 11 school administrators.

^b Non-responses to this item for 8 teachers and 11 school administrators.

Numeric Results – State 1
The numeric results for the two subgroups in State 1 are presented in tables that show all survey items, on the order presented in the survey. Generally speaking, special education teachers either agreed or disagreed with the survey items (sentences). By looking for those items with which at least 51% agreed and those items with which at least 51% disagreed, we identified 11 out of 16 closed items (items a-d, f-I, k, n-o). All of these items had relatively low rates of respondents indicating no opinion, don’t know, or giving no response. The content of the items suggested a pattern in the data of overall favorable opinions about testing students with disabilities out of level in states’ large-scale assessments. For instance, nearly two-thirds of the participants agreed that out-of-level testing was a good way to include students with disabilities in statewide testing (item a; 60%). Nearly two-thirds of the respondents disagreed that out-of-level testing had negative consequences for all students (item b; 60%) and negative consequences for students with disabilities (item c; 58%). In terms of the accuracy of out-of-level test results, 67% of the respondents agreed that test data were more accurate when students with disabilities were tested below their grade of enrollment in school (item d).  

On those survey items without clear agreement or disagreement (items e, j, l, m, p) the indecision was more often about content that was neutral or positive toward out-of-level testing. For example, item e, which suggested that students who participated in out of level tests were more motivated test takers, showed 43% agreeing, 12% disagreeing, 15% not knowing, and a total of 21% either not responding or having no opinion. Special education teachers were also not clearly decisive about whether Individualized Education Program (IEP) teams actually do select students accurately for out-of-level tests (item j), whether IEP teams considered future effects of out-of-level testing when deciding whether to test out of level or not (item l), whether parents understood the future effects of testing out of level (item m), and whether out-of-level testing was easily administered (item p).

Table 2. State 1 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

School administrators’ responses also indicated either agreement or disagreement with the survey items (see Table 3). Most agreement or disagreement percentages ranged from 56% to 78%, although there were more of them with a clear majority (13 of 16 items). Administrators’ responses in State 1 mirrored those of the special education teacher responses in State 1 in that the pattern in the responses again revealed favorable opinions about out-of-level testing. This was true for all but three of the survey items where there was no clear pattern in the data. School administrators’ responses were more evenly spread across response categories as to whether IEP teams did select students for out-of-level testing appropriately (item j), whether parents understood the future consequences of out-of-level testing for their children (item m), and whether out-of-level tests were easy to administer (item p).

Table 3. State 1 - School Administrator Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

In addition to the educator opinion and demographic data that we gathered with our written survey, we also asked what information IEP team members used to select an appropriate grade level at which to administer an out-of-level test. Participants were given a list of options with the directions to select all types of information that applied to their schools. Participants’ responses are displayed in Table 4. The most frequently identified type of student information used to select an out-of-level test level was the grade level on which core content instruction was delivered. About one third of the participants, both teachers and administrators, indicated that students’ previous test scores were used as indicators of appropriate out-of-level test levels while approximately 15% of teachers’ and administrators’ responses suggested that standardized tests are used. Approximately one-fifth of the responses showed that IEP team members compare test item content to students’ curricular content to determine a test grade level. Less than 5% of all participants responded that a locator test was administered to determine the student’s academic grade level at which an out-of-level test should be administered. Approximately 10% of respondents indicated another method used to select the out-of-level test grade level. These methods included considering student abilities, following eligibility criteria, measuring against IEP goals, portfolio assessments, relying on teacher expertise, or following parent requests.

Table 4. State 1 - Information Used to Choose Out-of-Level Test Grade Levels

Narrative Results – State 1
Participants in State 1 also responded to three open-ended survey items. The subgroups were combined across the state for this qualitative analysis. Each of these three items are presented in Table 5 with a result statement derived from the most frequent narrative responses and supportive participant quotes.

Table 5. State 1 - Narrative Response to Open-ended Survey Items

State 2

State 2 is a mid-size southern state that eliminated out-of-level testing in 2003-2004. Prior to eliminating out-of-level testing, State 2 allowed out-of-level testing for students with disabilities who met participation criteria. The state test in State 2 has both criterion-referenced and norm-referenced components, but the out-of-level test used only an extended version of the norm-referenced component in lieu of the criterion-referenced test. In State 2, students with disabilities testing out of level were required to take the reading or math test at least three grade levels below the student’s grade of enrollment, and were allowed to take different test levels in different content areas. Out-of-level test results in State 2 were aggregated at the student’s grade of enrollment at the lowest proficiency level in accountability indices.

Sample – State 2
Only school administrators participated in State 2 (n = 52), and 88% of those participants (n = 46) indicated that they were familiar with out-of-level testing. When asked if teachers in their district tested students with disabilities out of level on the state tests, 73% (n = 38) answered “yes” while eight school administrators did not respond to this item. Further demographic information is presented in Table 6. The majority of the participating school administrators had extensive experience, with 65% having 21 or more years of experience. Also, all of the participants hold advanced degrees, with the majority (54%) holding Master’s degrees. Missing responses were not included in this table; five administrators did not respond to the years of experience question, and four did not respond to the educational degree question.

Table 6. State 2 - Participant Levels of Experience and Education

Numeric Results – State 2
The numeric results for the only subgroup in State 2, school administrators, are presented in Table 7. Participants showed relative agreement on eight of the survey items in which at least 50% either agreed or disagreed with the item (items b-c, I, k, n-p). They generally disagreed that out-of-level testing has negative consequence for all students (50%) or for students with disabilities (54%). In terms of accuracy, 65% said that the out-of-level test results are more accurate than on-level test results for students with disabilities. Additionally, participants responded that students with disabilities who are tested out of level cannot (65%) and do not (73%) meet grade of enrollment standards. Finally, 71% answered that administering out-of-level tests is not easy to do. The pattern that emerges in these data depicts questionable opinions of out-of-level testing.

Table 7. State 2 – School Administrators Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

On eight of the items, participants showed no clear pattern of response. This included school administrators’ opinion on whether out-of-level testing is a good way to include students with disabilities in the state tests (item a). There was striking disagreement between respondents on whether IEP teams accurately select students who need to test out of level, with 38% agreeing, 42% disagreeing, 6% not knowing, and 14% with no opinion or not responding (item j), or whether IEP teams consider the future consequence of testing out of level when making this decision, with 44% agreeing, 37% disagreeing, 4% not knowing, and 16% with no opinion or not responding (item l). Also, school administrators were not visibly decisive about students’ motivation (item e), frustration (item f), amount of guessing (item g), and amount of questions answered when they are tested out of level (item h), or about whether parents understand the future consequences for their child who is tested out of level (item m).

We also asked school administrators in State 2 what information IEP teams use to select the appropriate test grade level at which to administer an out-of-level test to a student with a disability. Participants were given a list of options with the directions to select all types of information that applied to their school. Responses are displayed in Table 8. School administrators indicated that the student’s instructional level and the student’s previous test scores are the most prevalent methods for selecting the out-of-level test grade level. Some participants (21%) provided another method of selection not included in the list. These methods included considering the student’s age, grade level, prior IEP, present IEP, academic performance and ability level, as well as the state eligibility criteria or a combination of all of the listed methods.

Table 8. State 2 - Student Information Used to Choose Out-of-Level Test Grade Levels

Narrative Results – State 2

The school administrators in State 2 also responded to three open-ended survey items. Those results are in Table 9.

Table 9. State 2 - Narrative Response to Open-ended Survey Items

State 3

State 3 is a small eastern state that also eliminated out-of-level testing in 2003-2004. The state’s testing program included a combination of both criterion-referenced and norm-referenced exams. Only students with disabilities in grades 5, 8, and 10 were eligible to test out of level in the state’s testing program, and were allowed to take only a grade 3, 5, or 8 test. Like State 2, State 3 included out-of-level test results at the student’s grade of enrollment at the lowest proficiency in accountability indices.

Sample – State 3
Two subgroups returned a total of 47 surveys in State 3: special education teachers (n = 29) and district test coordinators (n = 18). The number of respondents in this state was lower than in other states. It should be noted that only 18 district test coordinators participated in this study in State 3. Because of the low number of test coordinators’ in particular, caution must be exercised in interpreting these findings. Special education teachers either received their surveys at the training session (n = 9) or in the mail (n = 20). No differences were found between the two data sets when analyzed separately; therefore, the results were combined to form one subgroup of special education teachers. Of the special education teachers, 93% (n = 27) were familiar with out-of-level testing; 100% (n = 18) of the district test coordinators were familiar with out-of-level testing. The two special education teacher participants who were not familiar with out-of-level testing did not complete the rest of the survey. Only 26% (n = 7) of the special education teachers and 11% (n = 2) of the district test coordinators indicated that teachers in their districts use out-of-level tests to assess students with disabilities in the state tests. (Three special education teachers and seven district test coordinators did not respond.)

Table 10 shows the years of experience and educational degree of the participants. The special education teachers and district test coordinators were both very experienced in the field, with 76% of special education teachers and 72% of district test coordinators having 11 or more years of experience. Also, 100% percent of both subgroups have advanced degrees, with the majority (69% of special education teachers and 56% of district test coordinators) holding Master’s degrees. There were four nonresponses to the years of experience item and three nonresponses to the educational degree item for the special education teachers. There were four nonresponses for both items for the district test coordinators.

Table 10. State 3 - Participant Levels of Experience and Education

Numeric Results – State 3
The numeric results for the special education teachers and district test coordinators in State 3 are presented in Tables 11 and 12, respectively. Special education teachers were in relative agreement with each other on half (eight) of the survey items. For instance, more than two-thirds of these participants agreed that students are less frustrated when allowed to take an out-of-level test (item f; 66%), IEP teams can accurately select students in need of testing out of level (item i; 66%), students who take out-of-level tests do not meet their grade of enrollment standards (item o; 69%), and administering out-of-level tests is not easy to do (item p; 72%). Special education teachers indicated less agreement on the other half of the items. Specifically, there was disagreement on whether out-of-level testing is a good way to include students with disabilities in states’ tests (item a), provides more accurate test scores than an on-level test (item d), or increases student test performance motivation (item e). The lack of agreement on some of the items may have been a result of many participants not knowing the answer to the question. A relatively high number of participants answered “Don’t know” on items g (31%), h (34%), and m (31%).

Table 11. State 3 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

The district test coordinators appeared to show more of a pattern of agreement than did the special education teachers (see Table 12). On eight of the items, the majority of the district test coordinators either agreed or disagreed with the item; for five of those items there was over 60% agreement or disagreement, depending on the item. Participants did not agree that testing out of level has negative consequences for all students (item b; 67%). They agreed that students testing out of level are less frustrated (item f; 61%) and guess less frequently (item g; 61%) than if they took an on-level test, and they agreed that IEP teams can appropriately select students in need of testing out of level (item i; 67%) and parents help make this decision (item k; 61%). Additionally, on two of the items (h and o), there was unanimous agreement or disagreement among those who responded and had an opinion. The district test coordinators agreed that students answer more questions when tested out of level as opposed to on level, and disagreed that students who are tested out of level do meet their grade of enrollment standards. It should be noted that the percentage of respondents who did not know the answer to these two questions was relatively high.

There was no pattern of agreement or disagreement among the district test coordinators on six of the items. Most notably, participants as a group were unsure about the accuracy of out-of-level test scores in comparison to on-level test scores (item d), the student’s motivation to perform on out-of-level tests as opposed to on-level tests (item e), whether parents understand the consequences of testing their child out of level (item m), and whether administering out-of-level tests is easy to do (item p). It should be noted that only 18 district test coordinators participated in this study in State 3; the low number of participants should be kept in mind when interpreting these data.

Special education teachers and district test coordinators in State 3 were also asked to provide what information IEP teams in their school district use to select the appropriate test grade level at which to administer an out-of-level test to a student with a disability. Participants were given a list of options with the directions to select all types of information that applied to their school district. Responses are displayed in Table 13. Both subgroups indicated that the student’s instructional level and the student’s previous test scores are the most prevalent methods for selecting the out-of-level test grade level. Test item and curricular comparison was also used in the participants’ schools. Some participants provided another method of selection not included in the list. These methods included psychological and achievement testing, considering the overall functioning of the student and classroom achievement, weighing the consequences of testing out of level for that student, relying on state mandated criteria, and considering individual student needs.

Table 12. State 3 – District Test Coordinators Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

Table 13. State 3 - Student Information Used to Choose Out-of-Level Test Grade Levels

Narrative Results – State 3
Participants in State 3 also responded to open-ended survey items. The subgroups were combined across the state for this qualitative analysis. Results from our content analysis are in Table 14.

Table 14. Narrative Response to Open-ended Survey Items for State 3

State 4

State 4 is a small New England state where any student may take the state tests out of level as long as that student receives approval from the Student Support Team, which is located in the state educational agency. The state uses only criterion-referenced exams in its state tests. Students testing out of level are only allowed to test at the grade levels offered in the regular assessment (grade 4, 8, and 10). Out-of-level test scores are transformed into an on-grade level equivalent using score transformation rules, but out-of-level test scores are automatically entered at the lowest proficiency level at the student’s grade of enrollment for accountability indices.

Sample – State 4
State 4 had special education teachers (n = 48), regular education teachers (n = 19), administrators (n = 89), and assessment coordinators (n = 54) participate in the survey for a total of 210 responses. A total of 88% (n = 42) of the special education teachers, 68% (n = 13) of the regular education teachers, 91% (n = 81) of the administrators, and 85% (n = 46) of the assessment coordinators indicated that they were familiar with out-of-level testing. When asked whether teachers in their school districts used out-of-level testing, 75% (n = 36, 9 nonresponses) of the special education teachers, 37% (n = 7, 9 nonresponses) of the regular education teachers, 82% (n = 73, 12 nonresponses) of the administrators, and 76% (n = 41, 9 nonresponses) of the assessment coordinators answered “yes.”

We asked participants for demographic information pertaining to their experience as educators and the highest educational degree obtained. This information is presented in Table 15. Overall, special education and regular education teachers were less experienced than assessment coordinators and school administrators. The largest respondent group for both the special education teachers (35%) and the regular education teachers (32%) had 10 years or less of teaching experience. The largest respondent group for the assessment coordinators had 11-20 years of experience (33%), indicating slightly more experience than the teachers. And the largest respondent group for the school administrators had 21-30 years of experience (39%). However, school administrators had the most experience of all of the groups, with the majority (57%) having 21 or more years of experience in education. The most common degree held for all groups was a Master’s degree, with 50% of the special education teachers, 37% of the regular education teachers, 67% of the school administrators, and 50% of the assessment coordinators holding this degree. Nonresponses were not included in the table due to space limitations; five special education teachers, six regular education teachers, and 11 school administrators did not respond to these two items. Additionally, 10 assessment coordinators did not respond to the years of experience item, and 13 assessment coordinators did not respond to the educational degree item.

Table 15. State 4 - Participant Levels of Experience and Education
 

Numeric Results – State 4
The special education teachers showed exceptional consistency in their responses, which are displayed in Table 16. The respondents came to a majority consensus (over 52% of respondents answered in the same manner) on 14 of the 16 items (all but items l and m). For the most part, special education teachers in State 4 had a very positive opinion of out-of-level testing, as evidenced by 83% agreeing that out-of-level testing is a good way to include students with disabilities in the state tests. The only negative sentiment emerged when questioned about meeting grade of enrollment standards; participants responded that students cannot and do not achieve proficiency on grade of enrollment standards (items n and o). The two items on which the special education teachers had no clear response pattern focused on IEP teams considering the future consequences of testing out of level when making that decision for a student (item l), and parents understanding the consequences of testing their child out of level (item m).

Table 16. State 4 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

The regular education teachers’ responses showed less of a consistent pattern than did the special education teachers’ responses. As illustrated in Table 17, only six items elicited a clear response pattern from the regular education teachers. They answered that out-of-level testing is a good way to include students with disabilities in the state tests (item a; 68%), does not have negative consequences for students (item b; 68%) or only students with disabilities (item c; 58%), yields more accurate scores than on-level test scores (item d; 53%), and produces less student frustration than on-level tests (item f; 53%). They also responded that IEP teams can accurately select students in need of out-of-level testing (item i; 68%). There were some items on which regular education teachers exhibited a pattern of response (e.g., item j) without a majority response either in agreement or disagreement with the survey items. There was a high no response rate (32%) in this already small group of participants, and this may have had an effect on the response frequencies for these items. The only two items on which the regular education teachers in State 4 were undecided were item m (if parents understand the consequences of testing their child out of level) and item p (if administering an out-of-level test is easy to do).

Table 17. State 4 – Regular Education Teacher Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

Much like the special education teachers, the school administrators (see Table 18) and assessment coordinators (see Table 19) showed a strong pattern of consistency in answering the survey items. For the school administrators, a clear pattern of response was noted in 14 of the 16 items; this was true for 13 of the 16 items for the assessment coordinators, almost all with positive opinions of out-of-level testing. Similar to the special education teachers, the school administrators and assessment coordinators responded that students cannot and do not achieve proficiency on grade of enrollment standards (item n). And, they did not arrive at a group consensus about IEP teams considering the future consequences of testing out of level when making that decision for a student (item l) or parents understanding the consequences of testing their child out of level (item m). The assessment coordinators also were unsure whether administering out-of-level tests is an easy thing to do (item p).

Table 18. State 4 –School Administrator Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

Table 19. State 4 –Assessment Coordinator Opinions on Testing Students with Disabilities Out of Level in Large-Scale Assessments

We also asked all of the participant subgroups in State 4 what information IEP teams use to select the appropriate test grade level at which to administer an out-of-level test to a student with a disability. Participants were given a list of options with the directions to select all types of information that applied to their school. Responses are displayed in Table 20. For all of the groups, the student’s instructional level was the most prevalent method for selecting the out-of-level test grade level, followed by reviewing the student’s previous test scores and comparing the test items to the curriculum the student is being taught in the classroom. Along with a small number of participants indicating that their district uses the test company’s locator test scores or the student’s standardized achievement test scores, some participants provided another method of selection not included in the list. These methods included considering individual student needs, the student’s IEP or 504 plan, student performance on other assessments or in the classroom, or resulting from a team discussion including special education teachers or case managers.

Table 20. State 4 - Student Information Used to Choose Out-of-Level Test Grade Levels

Narrative Results – State 4

Three open-ended survey items were completed by the participants in State 4. The subgroups were combined across the state for this qualitative analysis with the results presented in Table 21.

Table 21. State 4 - Narrative Response to Open-ended Survey Items

Discussion

The purpose of our discussion of these written survey results is two-fold. First, we consider the responses across the four states from the open-ended survey items holistically. The purpose of this look at our data is to confirm the variety of opinions held by educators in practice that have little consistency or uniformity. In other words, the viewpoints do not align easily into categories either by educator role or state membership. After we have confirmed these types of assumptions about out-of-level testing, we once again take a step back from our data to make item by item comparisons to see how the four states that participated in our survey differed on these dimensions. Each survey item that garnered numeric data addresses opinions that are commonly held in educational practice, but to date have not been grounded in data-based information. We follow our discussion of conclusions with a discussion of the explanation of the patterns in our data base.

Narrative Findings – Open-ended Survey Items

We interpreted our narrative thematic findings across states to draw overarching conclusions that deal with all states’ narrative responses.

There was wide disparity in participants’ opinions about the benefits and disadvantages of out-of-level testing.
In terms of the narrative findings from this survey project, there was no pattern to our data that would suggest that some subgroups in our sample held beliefs consistently across the group. Rather, there was wide variability in what educators thought were benefits or disadvantages to out-of-level testing. For instance, when asked about the positive aspects of testing students with disabilities below their grade of enrollment in school, some participants indicated that there were no benefits to out-of-level testing while others believed that out-of-level testing is a positive means of including more students with disabilities in statewide testing in a less frustrating test situation. With decreased stress, some participants thought that students experience more success and were better motivated test takers. When tested at the student’s instructional level, the resulting test scores were a more valid measure of what students know and can do.

On the other hand, participants identified negative social experiences for students who are embarrassed to be tested below their grade of enrollment when asked about the drawbacks to out-of-level testing. Also, learning expectations are lower for students with disabilities who are tested out of level that in turn puts them at risk for not receiving a regular high school diploma. Others were also concerned that out-of-level testing did not foster achieving grade level content standards. Yet, other participants indicated that there are no negative aspects to out-of-level testing.

To look at the patterns of responding by educator role and state membership, there is some overlap. Again, our results do not point to a clear consensus of opinion. In terms of positive aspects of out-of-level testing, all four states indicated most frequently that reducing test taking frustration was the overall benefit of testing out of level. Beyond that, there was agreement about the type of benefit but the frequency at which the benefits were named varied widely. Also, State 1 listed a benefit, practice test taking skills that no other state named. State 4 listed two benefits, better test performance and increased test score validity, which were not named by any other state.

When comparing states’ responses about the negative aspects of out-of-level testing, the results are even more irregular than when naming the benefits of out-of-level testing. States 1 and 4 most frequently indicated that negative impacts on students’ self esteem was the biggest drawback to out-of-level testing, while States 2 and 3 selected the negative future effects most frequently. The second most frequent response for State 1 and 2 was that out-of-level test scores were not useful while States 3 and 4 did not mention the lack of usefulness at all. Other factors identified as negative aspects were not achieving grade level standards, lowered learning expectations, and problems in administering the tests. States 1 and 2 indicated that there were no negative aspects to out-of-level testing, but did so the least frequently.

If out-of-level test scores are useful, they are used for instructional planning.
Thinking generally about respondents’ ideas about how out-of-level test scores are used, again there was wide variability both within and across states. Some participants indicated that out-of-level test data were useful for instructional decision making while others thought that out-of-level test data were useful developing students’ IEPs. There were also frequent responses that out-of-level test scores were neither useful nor used in educational practice.

The specific responses that respondents generated across states shows some overlap in responses, although again a pattern in responding is difficult to discern. States 3 and 4 indicated most frequently that out-of-level test scores are used to monitor students’ academic achievement while the most frequent response for State 1 was that the scores were not used and for State 2 to increase participation in statewide testing. Both State 1 and State 2 did mention progress monitoring, but not as frequently as the other states. The only response shared by all four states, although not at the level of frequency, was that respondents did not know how out-of-level test scores were used.

Narrative Findings – Closed Survey Items

In addition to the open-ended items that garnered narrative responses, we also presented items that required participants to select among a list of narrative responses that were provided for them. The item requested participants to select all options that are used in their school district to choose the appropriate level for an out-of-level test. Options available were considering a student’s instructional level, considering previous test scores, administering a test company’s locator test, administering a standardized achievement test, or comparing test items and a student’s curriculum.

Considering a student’s instructional level was the most frequently selected option used to determine a test level at which an out-of-level test would be administered.
In comparing the four states on this item, there is more agreement across states and subgroups than in the responses to other survey items. All four states indicated most frequently that a student’s instructional level is the determining factor in selecting a test level at which to administer an out-of-level test. As a second most frequent response, States 2 and 3 along with special education teachers in State 4 responded that a student’s previous test scores are used to set a test level. Otherwise, all other possible selections for this item did not yield frequent enough responses to be able to determine a clear majority. In other words, states and subgroup samples were unclear about using the remaining factors for deciding on the test level for an out-of-level test.

Numeric Findings – Closed Survey Items

Our second aim in discussing these survey results is to make comparisons on an item by item basis to see how states’ results compared and contrasted. We employed our decision rules to determine instances where there was a clear majority of opinion and where there was no clear majority, from which we then created overarching themes that crossed all states.

Most participants across states agreed that out-of-level testing was an advantageous way to include students with disabilities in statewide testing.
We first consider item a, whether out-of-level testing is a good way to include students with disabilities in statewide testing. With 60% in agreement for State 1 and 83% agreement for State 4, these two states overwhelmingly affirmed the use of out-of-level testing. While State 3 was not in complete agreement with this item, the district test subgroup did agree that out-of-level testing was a positive approach to statewide testing. In contrast, State 2 and the special educators from State 3 did not agree that out-of-level testing was a good option within a statewide testing program. This finding is not surprising given the results of the open-ended survey items that pointed to varying opinions as well.

There were high amounts of disagreement across states as to whether out-of-level testing was disadvantageous for students.
Thinking now about items b and c, whether out-of-level testing has negative consequences for all students or just students with disabilities, States 1, 2, 3, and 4 disagreed. Response percentages of disagreement ranged from 45% to 76% for item b and 50% to 75% for item c indicating relatively high amounts of disagreement among participant subgroups across states. Given the relatively high level of agreement about item a, the benefits of out-of-level testing as a statewide testing option, this finding too is not particularly surprising. The findings are relatively straightforward in that out-of-level testing is thought to be a “good” testing option that does not have negative consequences for either all students or just students with disabilities.

There was little agreement across states as to whether out-of-level testing is beneficial for students.
Yet, when our survey items began teasing apart the aspects that comprise the first three survey items, the consistency in our findings began to breakdown. Survey items d through h deal with the various aspects of out-of-level testing thought to be beneficial such as increased test score accuracy, better test motivation, reduced frustration, less guessing, and increased test item answering. However, the responses to these items are not necessarily consistent with the seemingly overriding affirmation of out-of-level testing evident in the first three survey items. Except for States 1 and 4 where respondents agreed with each benefit most frequently, the other two states showed variable responding that contradicted the results of items a, b, and c.  State 2 responded most frequently that out-of-level test scores were not more accurate than on-grade level test scores. There was no clear majority of responses for the remaining items indicating that there was no clear agreement or disagreement about the benefits of out-of-level testing. State 3 also did not demonstrate clear agreement or disagreement for items d through h, except that respondents did reply most frequently that out-of-level testing did reduce student test taking frustration.

Participants agreed that IEP teams can make accurate decisions about testing students out of level, but one state thought that they did so.
Items i and j requested participants to indicate whether IEP teams can and do select students with disabilities appropriately for out-of-level testing. States 1,  2, and 3 overwhelmingly agreed that IEP teams can select students appropriately for out-of-level testing, but were uncertain as to whether that was actually happening in practice. These results are not as contradictory as would first appear when out-of-level testing context is considered. All four states have indicated that they provide various training formats on out-of-level testing criteria so that practitioners understand how to select students appropriately (Thurlow & Minnema, 2001). Only State 4 has specific monitoring practices in place so that all applications for out-of-level testing are reviewed by state level personnel. It seems reasonable that only this state would be able to respond confidently that IEP teams do select students appropriately.

States believed that parents help make decisions to test out of level in ways that consider the future consequences for students, but that parents do not necessarily understand the long term ramifications.
Three items, items k, l, and m, pertained to decision making about student selection for out-of-level testing. Items k and m addressed parent participation in particular, for which we found unique patterns in responding. In terms of helping to make the decision to test a student with disabilities out of level, all participants in each state indicated that parents helped make the decision. When it came to understanding the future consequences of testing students out of level, there was no clear pattern in opinion. In State 1, opinions varied by educator role. School administrators agreed that IEP teams considered the future consequences when deciding to test out of level, but special education teachers were not sure whether parents understood the ramifications. State 2 was not clear about either IEP team considerations or parent understandings for future consequences. State 3 was similar to State 1 in that responses indicated that IEP teams consider future consequences of out-of-level testing when selecting students, but were not certain that parents understood those consequences. In State 3 though, responses did not vary by educator as they did in State 1. State 4 also thought that parents helped make decisions about out-of-level testing and the future consequences were considered, but these respondents did not know whether parents understood the consequences or not.

Across all four states, participants believed that students with disabilities who were tested out of level could neither be capable of meeting nor do meet grade-level content standards.
For items n and o, which address whether students with disabilities can and do meet grade level content standards, there was complete consistency within the responses of States 1, 2, and 3. None of these states indicated that they thought that students with disabilities who are tested out of level are capable of achieving grade-level content standards as indicated by grade level proficiency standards. These states also agreed that students with disabilities who were tested out of level did not meet grade level standards in their actual classroom learning experience.

There was no consistent answer in terms of whether administering out-of-level tests is easy to do.
The last closed item in our survey, item p, asked participants to indicate whether out-of-level tests were easy to administer. Opinions were not necessarily clearly delineated. State 1 and one subgroup of participants in State 3 (district test coordinators) were not sure whether the process of testing students with disabilities out of level was easy to do. On the other hand, State 2 indicated that administering out-of-level tests was difficult to do while State 4 indicated that the administration was easy to do.

Conclusions

To conclude the reporting of this survey research, we discuss three broad themes that emerge from our consideration of the narrative and numeric data as a composite whole. These themes are written generally enough to provide thoughtful considerations for all stakeholders, but cannot be generalized specifically to other educator subgroups or states that test out of level because we used a purposive sample from which to collect our data.

Educators’ positive perceptions of out-of-level testing change when the testing option is thought about with greater detail.
Taken all together, our findings suggested that when educators are thinking generally about out-of-level testing, the option appears to be a reasonable solution for including students with disabilities in statewide testing. When asked if out-of-level testing was an advantageous way to include students with disabilities in large-scale assessments, a majority of participants indicated this to be true. Also, when asked if out-of-level testing had negative consequences for students with and without disabilities, participants basically disagreed.

However, when prompted to think about out-of-level testing on a deeper level by considering specific aspects of this approach to testing, inconsistencies began to emerge. For instance, while all states agreed overall that out-of-level testing did reduce student frustration during test taking, other benefits of out-of-level testing were not as readily agreed on. Within the results for the remaining benefits, there was wide variability in terms of both agreement and disagreement.

Even more confusing is the conflicting responses on different survey items. For example, the response pattern clearly indicated that participants’ agreed that students with disabilities who were tested out of level could not meet grade level content standards. In looking at the thematic results from the narrative data, participants in two states indicated that students who are tested out of level are unable to receive a high school diploma. This negative aspect of testing students below the grade in which they were enrolled in school seems to be in direct contrast to the fact that these students cannot achieve on-grade level. As another example, all participant subgroups disagreed that out-of-level testing has negative consequences for students with disabilities (item c) with responses ranging from 50% to 81% disagreement.  When asked in an open-ended item to generate negative aspects or impacts of out-of-level testing on students with disabilities, all subgroups across all four states produced multiple negative aspects or impacts. This narrative finding that prompted respondents to think more deeply about out-of-level testing is in direct contrast to the results from the closed item, which was written more generally.

Educators believe that students with disabilities who are tested out of level cannot meet content standards set for their grade of enrollment.
Possibly the most discouraging aspect to our survey findings was the uniformity in responses regarding students with disabilities who have been tested out of level and their learning expectations. All sample subgroups, which included special and general educators, administrators, and district test coordinators, indicated that those who participate in out-of-level testing could not and do not meet grade level standards. This finding is particularly alarming given the federal mandate in NCLB that all students meet grade-level proficiency on states’ content standards.

In today’s accountability minded environment, it is essential that schools figure out ways that all students can achieve at optimum levels. Looking at instructional issues, test instrument access, or accommodations use is a place at which planning can begin. The consequences are much larger than learning and test taking decisions that occur on a daily basis. Large-scale assessment data that are used for accountability purposes have far reaching implications for students, teachers, administrators, and schools. A subgroup of a school’s population loses valuable opportunities to learn while teachers and administrators may be blamed for lack of demonstrating proficiency on grade level content standards. School improvement planners are not given a true picture of how all students are performing so that their decisions do not improve learning for every student.

Educators’ opinions about out-of-level testing vary widely so that consensus is rare.
The analysis of both numeric and narrative data from our survey yielded results that were highly variable across states and subgroups of educators. Agreement occurred about using students’ instructional levels to determine the test level of an out-of-level test, that IEP teams can select students accurately for out-of-level testing, that parents help make the decision to test below grade level, and that students with disabilities who are tested out of level cannot and do not meet the content standards for their grade of enrollment. The often disparate opinions held by practitioners suggest that they have not coalesced on a common set of viewpoints about testing students with disabilities out of level in large-scale assessment programs that are used for accountability purposes. Nevertheless, their general impressions are clear – out-of-level testing is more frequently seen in a positive light than a negative one.

The debate will undoubtedly continue concerning the value of testing students with disabilities below their grade of enrollment in school. Given our recent findings (Minnema et al., 2004a; Minnema et al., 23004b; Minnema & Thurlow, 2003; Thurlow et al., 2003), it behooves all stakeholders to understand that the passion with which opinions are sometimes expressed may not represent all educators’ opinions. For each discussion of out-of-level testing, we recommend thinking through the issues from multiple perspectives. It is through critical thought that we understand the deeper issues that surround contentious educational conundrums such as out-of-level testing.

References

Minnema, J., & Thurlow, M. (2003). Reporting Out-of-level test scores: Are these students included in accountability programs? (Out-of-Level Testing Report 10). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT10.html

Minnema, J.E., Thurlow, M., & Warren, S. (2004a). Understanding out-of-level testing in local schools: A first case study of policy implementation and effects. (Out-of-Level Testing Report 11). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT11.html

Minnema, J.E., Thurlow, M., & Warren, S. (2004b). Understanding out-of-level testing in local schools: A second  case study of policy implementation and effects. (Out-of-Level Testing Report 12). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT12.html

Thurlow, M., Elliott, J., & Ysseldyke, J. (1999). Out-of-level testing: Pros and cons (Policy Directions No. 9). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at  http://education.umn.edu/NCEO/OnlinePubs/Policy9.htm

Thurlow, M., & Minnema, J. (2001). States’ out-of-level testing policies (Out-of-Level Testing Report 4). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT4.html

Thurlow, M., Minnema, J., Bielinski, J., & Guven, K. (2003). Testing students with disabilities out of level: State prevalence and performance results (Out-of-Level Testing Report 9). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT9.html

VanGetson, G., Minnema, J., & Thurlow, M. (2004). Rapid changes, repeated challenges: States’ out of level testing policies for 2003-2004 (Out-of-Level Testing Report 13). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Available at http://education.umn.edu/NCEO/OnlinePubs/OOLT13.html

Appendix A

Survey of Administrators, Teachers, and District Test Coordinators

SURVEY OF [educator role]

Out-of-Level Testing in [state name]

Definition: “Out-of-level testing” means giving a student the [state test name] at a level lower than the student’s assigned grade level.

Q1. Are you familiar with out-of-level testing?

1. Yes (Please continue with the survey.)
2. No (Your survey is complete. Please return it to the registration table.)

Q2. To what extent do you agree or disagree with the following statements? (Circle one answer for each item.)

Q3.          Which of the following do teachers in your district use to choose the appropriate level for an out-of-level test?  (Circle all that apply.)

         a.     Consider a student's instructional level

         b.     Consider a student's previous test scores

c.        Administer a test company's locator test

d.        Administer a standardized achievement test

e.        Compare test items and a student's curriculum

         f.      Another way (Please specify)________________________________________________________

Q4.      How does your district use out-of-level test scores?

Q5.      What are the positive aspects or impacts of testing students out of level in the [state test name]?

Q6.      What are the negative aspects or impacts of testing students out of level in the [state test name]?

Q7.      Do teachers in your district test students with disabilities out of level in the [state test name]?

            1.         Yes

            2.         No

Q8.      How long have you worked in the field of education?    __________ years

Q9.      What is the highest degree you have obtained?  (Circle one.)

1.         Bachelor’s degree

2.         Master’s degree

3.         Specialist’s degree

4.         Ph.D.

 Q10.  What is your position in your school system?  ______________________________

Please return your survey to the registration table to receive a report from the National Center on Educational Outcomes!

Top of page