Overview

The Individuals with Disabilities Education Act of 2004 and the Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001, require that all students participate in statewide assessments used for accountability purposes. Most students with disabilities participate in the regular test with or without accommodations. A few with the most significant disabilities participate in an alternate assessment based on alternate achievement standards (AA-AAS). A few states also have an Alternate Assessment based on Grade-level Achievement Standards (AA-GLAS) for students with disabilities who need testing formats or procedures that are not included in the regular assessment and are not addressed with the use of accommodations.

In 2007, federal regulations allowed states to develop another assessment, an Alternate Assessment based on Modified Academic Achievement Standards (AA-MAS). Students who participate in this test may be from any disability category. They must have access to grade-level content but be unlikely to achieve grade-level proficiency within the time period covered by their Individualized Education Programs (IEPs). For accountability purposes states may count the proficient and advanced scores of students with disabilities participating in an AA-MAS subject to a two percent cap of all students enrolled in the tested grades. This test is optional, and states are not required to offer it (U.S. Department of Education, 2007).

During the 2010-11 school year 14 states had an operational AA-MAS (Price, Hodgson, Lazarus, & Thurlow, 2011). Since Spring 2010 the state of Tennessee has had an operational AA-MAS called the Tennessee Comprehensive Assessment Program-Modified Academic Achievement Standards (TCAP-MAAS). The Tennessee Department of Education provided training to teachers in the state on how to make and implement participation and accommodations decisions for the TCAP-MAAS assessments. They also provided training on test administration.

Following the first year of implementing the TCAP-MAAS, the state wanted to learn more about teacher perceptions of the assessment, including what worked well and what challenges they observed. The state had questions such as: Do teachers understand the selection criteria and requirements as they decide which of their students will be tested on the TCAP-MAAS? And, do they understand how to prepare for and administer the new test, including seamlessly incorporating accommodations into the testing situation?

Teachers' knowledge, beliefs, and decisions set the tone for how students participate in a test (Altman, Cormier, & Crone, 2010; Ketterlin-Geller, Alonzo, Braun-Monegan, & Tindal, 2007). As state departments of education initiate new policies, activities, or decisions that affect instruction and assessment in classrooms, it can be a challenge to reach the entire state's population of educators with information vital to a new initiative. Previous studies (Altman, Cormier, Lazarus, et al., 2010; Lazarus, Thompson, & Thurlow, 2006) explored teachers' accommodations decision making and use in other states. This report presents the results of a survey of Tennessee educators intended to measure their experience with the TCAP-MAAS that was recently implemented in the state. We also (a) asked questions about how the teachers received training on this assessment, (b) sought to measure the extent that information about the new TCAP-MAAS had reached teachers across the state, and (c) measured accommodations decision making.

The Tennessee State Department of Education, with support from the National Center on Educational Outcomes (NCEO), conducted the survey of special education teachers in Tennessee in summer 2010 following the state's first use of the TCAP-MAAS. Specifically, the survey was conducted to answer the following three questions:

1. How effective was the teacher training on the TCAP-MAAS?
2. Are there differences in teachers' perceptions of teacher and student experiences between the new TCAP-MAAS and the regular assessment taken by the student the previous year?
3. Which accommodations are used most often and which factors influence decision making and implementation?
   

Top of Page | Table of Contents

---

Procedure

Survey Development

The Tennessee State Department of Education, with support from NCEO, developed a survey for special education teachers in Tennessee. We aimed the survey specifically at teachers of one or more students who took the TCAP-MAAS in spring 2010. The purpose of the survey was to gain information about teacher perceptions of the new state assessment option, the TCAP-MAAS. Specifically we sought to learn whether teachers perceived that the assessment improved student educational outcomes on key indicators. Several questions were also asked about accommodations decision making. See Appendix A for a copy of the survey.

 

Data Collection

The survey was made available online through a survey tool, and was disseminated to teachers through administrators of local education agencies via e-mail. The state provided a list of e-mail addresses of special education coordinators/directors for each district. An e-mail was sent to each of these contacts. The contact was asked to forward a message containing the link to the survey to all special educators in the district. It should be noted that some special educators did not teach a student who took the TCAP-MAAS and therefore were unable to participate in the survey. There were 191 respondents to the survey; 133 respondents completed the entire survey and 58 indicated that they did not have any students who participated in the TCAP-MAAS. Those who did not have any students were thanked, directed out of the survey, and did not complete the remaining questions.

The survey contained 10 true-false questions that were designed to assess teacher knowledge of the TCAP-MAAS (see Table 1). Summary scores were created for teachers based on the number of correct responses out of a possible ten points.

Table 1. True-False Questions in the Teacher Knowledge Section of the Survey

The TCAP-MAAS is an assessment that measures achievement in the general education curriculum, but the standard or level of achievement has been modified. [true] The TCAP-MAAS is a test that has questions that are less difficult than the regular TCAP achievement assessment. [true] The TCAP-MAAS has the same number of test items as the regular TCAP assessment. [false] A child's teacher is the only person who can make the recommendation for a child to take the TCAP-MAAS. [false] Parents must give permission for their child to take the TCAP-MAAS. [true] Students taking the TCAP-MAAS must be learning the general education curriculum. [true] Students taking the TCAP-MAAS cannot receive a regular high school diploma. [false] A student who takes the TCAP-MAAS must have IEP goals based on the curriculum standards for her/his grade. [true] On the TCAP-MAAS students may use all of the accommodations used on the regular TCAP. [true] The TCAP-MAAS can only be given to students with IEPs. [true]

Top of Page | Table of Contents

---

Results

Research Question #1: How effective was the teacher training on the TCAP-MAAS?

Teachers obtained a range of scores on the teacher knowledge portion of the survey. As shown in Table 2, 133 teachers completed the survey. Scores ranged from zero (the respondent answered "I don't know" to each question) to ten correct answers out of a possible ten, with three teachers answering every question correctly. The most difficult question was "The TCAP-MAAS has the same number of test items as the regular TCAP assessment." Only 33 of the 133 teachers provided a correct answer. The easiest item, which 127 of the 133 respondents answered "true" correctly, was "On the TCAP-MAAS students may use all of the accommodations used on the regular TCAP."

Educators received training on the new TCAP-MAAS through a number of channels during the 2009-2010 school year and summer prior to the first year's live testing using this assessment. Participants were asked how they received this training (e.g., state staff, district-level training, training by school administrator, online training, self-training from independent source, other). As seen in Table 2, there does not appear to be a relationship between number of training sources accessed and teacher scores on the knowledge assessment.

Figure 1 shows the relationship between teacher experience and teacher knowledge on the TCAP-MAAS. Teachers were divided into two experience groups-less than 10 years of experience (n = 52), and 10 years or more of experience (n = 79). The results were similar for both of these groups. Fifty-two percent of the teachers with less than 10 years of experience scored in the low or moderate knowledge range (i.e., 10% in the low range plus 42% in the moderate range), whereas 51% of the teachers with ten or more years of experience scored at the low or moderate level (i.e., 8% in the low range plus 43% in the moderate range).

Figure 1. Teacher Knowledge Levels by Years of Teaching Experience

 

Note: Two respondents who completed the knowledge question section of the survey did not indicate how many years of teaching experience they had, and are not included in the results presented in this figure.

 

Research Question #2: Are there differences in teachers' perceptions of teacher and student experiences between the new TCAP-MAAS and the regular assessment taken by the student the previous year?

Since the TCAP-MAAS test was a new option in the state of Tennessee, we wanted to learn more about whether teachers thought the test would lead to improved instructional practices for students with disabilities who participated in this assessment option. The survey asked questions designed to learn more about whether respondents associated the new TCAP-MAAS with changes in the instructional preparation of the participating students. As seen in Figure 2, the majority of teachers indicated that students had about the same level of opportunity to learn grade-level content when compared to the previous year. Depending on the content area, 27% to 35% of the teachers indicated that students either had "more" or "much more" opportunity to learn grade-level content. Approximately 6% of all respondents indicated that they did not know whether there was any change in instruction for the students taking the new assessment.

Figure 2. Teacher Reported Student Opportunity to Learn Grade-Level Content Compared to Previous Year

 

We also wanted to determine whether teachers perceived a change in student motivation to take the test as compared to their experiences in previous years taking the regular TCAP. As shown in Figure 3, only about 23% of respondents thought students were "more" or "much more" motivated to take the math and reading assessments when then took the TCAP-MAAS. Across all content areas (i.e., reading, math, science, social studies), more than 60% of the teachers thought that student motivation was about "the same" as the previous year when they took the regular assessment. Approximately one in ten participants responded "I don't know" across these survey items.

Figure 3. Teacher Reported Level of Student Motivation to Take the TCAP-MAAS Compared to Previous Year's Regular Test by Content Area

 

Table 3. Teachers' Perceived Experiences with the TCAP-MAAS Compared to Their Previous Experiences

Survey Item	Percentage of Responses by Response Category
	Much Less	Less	The Same	More	Much More	I Don't Know
Was TCAP-MAAS testing a more or less positive experience for students this year as compared to regular TCAP testing the previous year?	0%	2%	7%	46%	39%	6%
Were TCAP-MAAS students more or less involved in making decisions about their education this year as compared to the previous year?	0%	6%	56%	22%	8%	8%
Was it more or less difficult to select accommodations for the TCAP-MAAS as compared to the regular TCAP?	2%	14%	62%	11%	6%	5%
Was it more or less difficult to match accommodations for instruction with TCAP-MAAS assessment accommodations as compared to the regular TCAP?	2%	17%	62%	10%	6%	4%
Was it more or less difficult to provide accommodations on test day for the TCAP-MAAS as compared to the regular TCAP?	2%	12%	61%	14%	8%	2%

Figure 4 shows the respondents' perceptions of parent and student involvement and experiences related to the TCAP-MAAS. Almost 60% believed that parent involvement in the decision-making process should be about "the same" when students take the TCAP-MAAS as when they take the TCAP. Seventy-one percent indicated the level of instructional support remained "the same" when students took the TCAP-MAAS; 69% said the level of assessment support was "the same."

Figure 4. Teachers' Perceptions of Parent and Student Experiences with the TCAP-MAAS

 

Table 4 shows that about 58% of respondents said they thought the TCAP-MAAS increased the likelihood of graduation for the students who participated in it, while 36% said that the likelihood was "the same." More than three-quarters of the educators indicated that the TCAP-MAAS was a valuable addition to the assessment system. About the same percentage of teachers believed that more students should be assessed with the TCAP-MAAS.

Table 4. Responses to Various Components Related to the TCAP-MAAS

Survey Item	Percentage of Responses by Response Category
	Much Less	Less	The Same	More	Much More	I Don't Know
Do you think that students now taking the TCAP-MAAS will be more or less likely to graduate with their classmates as compared to when they were taking the regular TCAP?	1%	2%	36%	42%	16%	3%
Do you think the TCAP-MAAS is a more or less valuable addition to the testing system as compared to your thoughts before the testing season began?	1%	4%	14%	44%	34%	4%
Do you think that more or less students should be assessed with the TCAP-MAAS?	2%	3%	15%	40%	38%	3%

Research Question #3: Which accommodations are used most often and which factors influence decision making and implementation?

The survey investigated issues related to accommodations decision making for the TCAP-MAAS and the regular TCAP, the frequency of use of specific accommodations, and common issues in carrying out accommodations decisions on test day for both assessments. Figure 5 shows that more than 80% of respondents indicated that they considered the following factors to be of equal importance when making decisions for the TCAP and the TCAP-MAAS: parental input, feasibility of providing the accommodation, student input, state policies and guidelines, and teacher input. Almost two-thirds of the respondents indicated that it was more important to consider whether an accommodation improved the student's chances of passing the test on the TCAP-MAAS than for the regular TCAP. Thirteen percent indicated that students were less likely to be fully included in the general education setting when they took the TCAP-MAAS.

Figure 5. Comparison Between the TCAP-MAAS and the TCAP on the Importance of Selected Accommodations Decision-Making Factors

 

The survey also asked teachers about which specific accommodations their students used. We asked respondents to rate their frequency of use for a number of accommodations from never to always (including rarely, occasionally, and frequently). As shown in Table 5, most teachers indicated that they "frequently," or "always" selected Read Aloud Instructions, Read Aloud Items, Calculator, and Flexible Setting for use by their students. More than three-quarters of the teachers indicated that their students never used Braille or Sign Instruction Verbatim.

Table 5. Frequency of Accommodation Use by Accommodation Type

Accommodation	Percentage of Responses by Response Category
	Never	Rarely	Occasionally	Frequently	Always
Read Aloud Instructions	1%	1%	1%	32%	65%
Read Aloud Items	2%	1%	1%	36%	61%
Calculator	2%	2%	2%	34%	60%
Flexible Setting	2%	2%	7%	35%	54%
Flexible Time of Day	8%	14%	23%	28%	27%
Student Reads Aloud	39%	21%	10%	12%	18%
Multiple Testing Sessions Within Day	20%	22%	24%	22%	12%
Student Answers in Test Booklet	21%	22%	28%	18%	11%
Student Answers on a Separate Sheet of Paper	53%	19%	14%	6%	8%
Visual/Tactile Aids	62%	17%	6%	9%	6%
Sign Oral Instructions Verbatim	79%	11%	2%	2%	5%
Auditory Aids	69%	17%	7%	3%	4%
Scribe (Excluding Writing Assessments)	42%	27%	20%	7%	3%
Braille	86%	12%	1%	0%	2%
Large Print	52%	29%	15%	2%	2%
Colored Overlay	74%	15%	8%	2%	1%
Magnifying Equipment	72%	18%	8%	1%	1%

Making quality decisions about which accommodations to use on statewide assessments and providing the necessary documentation does not always ensure that the test will be carried out in the intended manner on assessment day. Figure 6 presents responses to a survey question that asked teachers to identify test day issues that they may have encountered. Ten percent of the respondents indicated that the lack of staff availability determined the accommodations provided, and seven percent stated that accommodations that required a special test booklet were sometimes not provided. The most commonly selected response to this question was "other" suggesting that there is a great deal of variability in the issues teachers and other test administrators face on test day.

Figure 6. Test Day Issues Experienced by Teachers based on a List of Possible Responses

 

Top of Page | Table of Contents

---

Discussion

The perceptions of teachers whose students took the TCAP-MAAS provide insights into teacher and student experiences with this assessment option. The results of this study provide preliminary evidence that teachers thought that the TCAP-MAAS has sometimes led to improved student or teacher experiences. Eighty-five percent of the respondents believed that the TCAP-MAAS was either a more positive or much more positive testing experience for students compared to the students' testing experience the previous year. About 20% of respondents indicated that student motivation increased as a result of participation in the TCAP-MAAS, and more than half thought that long-term outcomes such as high school graduation could be positively influenced by a student's inclusion in the population tested with the new assessment.

The majority of respondents only scored at the low or moderate levels on a quiz of knowledge about the TCAP-MAAS. To make appropriate decisions for students who participate in this assessment, teachers must be knowledgeable about the TCAP-MAAS. This includes knowledge of test characteristics and participation guidelines. The survey results suggest that teachers in Tennessee may benefit from additional training about this assessment option.

The federal regulations that permit the use of an AA-MAS indicate that students who participate in this assessment option must have access to grade-level content. Most teachers responded that there was no change in opportunity for students to learn the grade-level content in core subject areas in the year they were tested on the TCAP-MAAS for the first time. This suggests, based on that one criterion, that the students were qualified to participate in the TCAP-MAAS, and had been receiving instruction in grade-level content for two consecutive years. However, approximately 20% of teachers indicated that students had more access to grade-level content in the first year they participated in the TCAP-MAAS, which suggests that their students may not have had the opportunity to learn grade-level content when they participated in the regular TCAP. This is a troubling finding because if these students lacked access to grade-level content when taking the regular test, it is impossible to tell how they would have performed on the regular test if educators had provided them with appropriate instruction and curriculum.

More than two out of three educators who completed the survey indicated that the increased likelihood of students with disabilities passing the TCAP-MAAS affected their decision making related to accommodations. Based on the survey questions, it was unclear whether students received accommodations on test day that they had perhaps not successfully used in prior testing situations or in the classroom. Future research should examine this phenomenon to ensure that best practices in accommodations selection and implementation are in place for all assessments.

According to the study results, more than 95% of the teachers reported that their students used the Calculator, Flexible Setting, Read Aloud Items, and Read Aloud Instructions accommodations at least occasionally. These results lead to the following question: Is the need for the Read Aloud Items or Calculator accommodation a key factor in TCAP-MAAS participation decisions? This question cannot be answered with the data collected from the survey, and future research is needed to investigate the reasons for the high occurrence of certain accommodations.

Only special education teachers who had at least one student who participated in the TCAP-MAAS could participate in this survey. The survey link was sent to LEA special education coordinators and they were asked to forward it on to the appropriate teachers. Tennessee does not disaggregate data in a way that we can tell what the total population of teachers who administered the AA-MAS was. There were only 133 respondents, which suggests that the response rate may have been very low; unfortunately we are unable to estimate a response rate.

The perceptions of teachers whose students took the TCAP-MAAS can provide insights into what is working well and where there were challenges. Other states can learn from Tennessee's experiences, and states should attend to the challenges mentioned by the participants in this study to improve student experiences and outcomes. According to Thurlow (2008) "attending to the nature of the assessment and to providing a strong instructional foundation are critical in avoiding unintended negative consequences (e.g., students falling further behind) and promoting positive consequences (e.g., improved instruction and greater progress)" (p. 138).

Top of Page | Table of Contents

---

References

Altman, J. R., Cormier, D., & Crone, M. (2010). Large scale assessment and high stakes decisions: Guidelines for educators. In A. Canter, L. Paige, & S. Shaw (Eds.), Helping children at home and school (Third Edition). Bethesda, MD: National Association of School Psychologists.

Altman, J. R., Cormier, D. C., Lazarus, S. S., Thurlow, M. L., Holbrook, M., Byers, M., Chambers, D., Moore, M., & Pence, N. (2010). Accommodations: Results of a survey of Alabama special education teachers (Synthesis Report 81). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Individuals with Disabilities Act. (2004). Public Law 108-446. Washington, DC: U. S. Government Printing Office.

Ketterlin-Geller, L. R., Alonzo, J., Braun-Monegan, J., & Tindal, G. (2007). Recommendations for accommodations: Implications of (in) consistency. Remedial and Special Education, 29(4), 194-206.

Lazarus, S. S., Thompson, S. J., & Thurlow, M. L. (2006). How students access accommodations in assessment and instruction: Results of a survey of special education teachers (Issue Brief 7). College Park MD: University of Maryland, Educational Policy Reform Research Institute.

Price, L. M., Hodgson, J. R., Lazarus, S. S., & Thurlow, M. L. (2011). Characteristics of states' alternate assessments based on modified academic achievement standards in 2010-2011 (Synthesis Report 85). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thurlow, M. L. (2008). Assessment and instructional implications of the alternate assessment based on modified academic achievement standards (AA-MAS). Journal of Disability Policy Studies. 19(3), 132-139.

U.S. Department of Education (2007, April 9). Final Rule 34 CFR Parts 200 and 300: Title I-Improving the academic achievement of the disadvantaged; Individuals with disabilities education act (IDEA). Federal Register: 72(67), Washington DC: Author. Retrieved from http://www.cehd.umn.edu/nceo/2percentR-Eg/FederalRegApril9TwoPercent.pdf

Top of Page | Table of Contents

---

Appendix A