Martha L. Thurlow, Andrew R. Hinkle, Sheryl S. Lazarus, and Kristin K. Liu
January 2024
All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Thurlow, M. L., Hinkle, A. R., Lazarus, S. S., & Liu, K. K. (2024). Revisiting the meaning of “reduced depth, breadth, and complexity” for AA-AAAS(NCEO Report 441). National Center on Educational Outcomes.
The terms “depth,” “breadth,” and “complexity” are often used to describe alternate assessments based on alternate academic achievement standards (AA-AAAS). Yet, the meaning of these terms often is not defined, nor is it clear how a state might incorporate these concepts into the development of its AA-AAAS or in the validity arguments that the state makes for its assessment.
This report was developed to provide (a) an overview of relevant history and legislation that address depth, breadth, and complexity of AA-AAAS; and (b) questions for states to consider as they revise or develop new AA-AAAS. The information also can be useful for states submitting evidence for an existing AA-AAAS to the U.S. Department of Education peer review process. The identified questions can produce descriptions that may be helpful in documenting evidence for the validity of results and interpretations of the AA-AAAS.
It is important for each state to have a clear understanding and strong rationale for the terms it uses to describe depth, breadth, or complexity, and how they are reflected in the design, development, and evidence of technical adequacy of its AA-AAAS. The questions provided in this report were created to help states clarify their understanding of the terms in their own AA-AAAS.
Alternate assessments aligned to alternate academic achievement standards (AA-AAAS) are designed and implemented for students with the most significant cognitive disabilities. Historically, these assessments have been described as reflecting reduced depth, breadth, and complexity. Yet, the meaning of these terms is often not defined. Further, it is unclear how a state’s approach to reducing depth, breadth, or complexity is incorporated into the development of a state’s AA-AAAS and the validity arguments that the state makes for its assessment.
The purpose of this report is to provide: (a) an overview of relevant history and legislation that address depth, breadth, and complexity of AA-AAAS; and (2) questions for states to consider as they revise or develop new AA-AAAS. This information also can be useful for states submitting evidence for an existing AA-AAAS to the U.S. Department of Education peer review process (U.S. Department of Education, 2018). The identified questions can produce descriptions that may be helpful in documenting evidence for the validity of results and interpretations of the AA-AAAS.
Federal laws, regulations, and guidance reflect a growing understanding of best practice in developing and implementing AA-AAAS. With the growing understanding, the terms used and the meaning of some terms changed.
The Individuals with Disabilities Education Act (IDEA) of 1997 was the first federal law to mention alternate assessments for students with disabilities who could not participate in general assessments even with accommodations. Following the enactment of the Elementary and Secondary Education Act (ESEA) in 2001 known as No Child Left Behind (NCLB), regulations and guidance clarified that assessments aligned to alternate achievement standards were for students with the most significant cognitive disabilities (see Appendices A and B for details). The term complexity(e.g., cognitive complexity) and terms that imply breadth (e.g., range, scope) appear in federal law, regulations, and guidance, but the term depth is not used. However, depth was used frequently in other documents about AA-AAAS. The field often continues to use the three terms and sometimes makes a clear distinction among the meanings of the three terms.
The first mention of one of the three terms in the regulations (U.S. Department of Education 2003) was the term complexity. They stated, “An alternate achievement standard is an expectation of performance that differs in complexity from a grade-level achievement standard” (p. 69699). The term “breadth” was implied by the U.S. Department of Education’s (2005) non-regulatory guidance. It stated that “An alternate assessment based on alternate achievement standards may cover a narrower range of content (cover fewer objectives under each content standard) and reflect a different set of expectations in the areas of reading/language arts, mathematics, and science than do regular assessments….” (p. 16). Complexity was addressed in the statement, “An alternate achievement standard sets an expectation of performance that differs in complexity from a grade-level achievement standard” (p. 20). It was also addressed in the statement, “When examined across grades, however, alternate achievement standards are not expected to show the same clearly defined differences in cognitive complexity as the grade-level achievement standards for the regular test” (p. 21).
State alternate assessments used for ESEA accountability purposes are required to undergo U.S. Department of Education peer reviews for technical adequacy (U.S. Department of Education, 2018). Guidance for peer reviewers describes the kinds of evidence that states should provide for their review (see Appendix C for excerpts from peer review guidance). The guidance notes that “Alternate academic achievement standards set expectations of performance that differ in scope and complexity.”
When states first developed their AA-AAAS, they decided how they wanted to reflect the state’s grade-level academic content standards. States sometimes prioritized those standards that were critical to success in the next grade. They also considered how much simplification (or reduction in complexity of the achievement of the standards) was reflected in the state’s AA-AAAS. Recently, researchers who have explored alignment of AA-AAAS have noted that expectations for students with the most significant cognitive disabilities have changed since NCLB (see Bechard et al., 2021) and that alignment considerations have grown out of the changed expectations. Specifically, Bechard and colleagues indicated that “students with significant cognitive disabilities are expected to engage in higher thinking and operations to accomplish more complex academic content” (p. 7). Related to this was the ESEA 2015 requirement that states’ AA-AAAS be coordinated with the requirements of the Workforce Innovation and Opportunity Act (WIOA) of 2014 and that “states demonstrate that students with the most significant cognitive disabilities who met the states’ alternate academic achievement standards were on track to pursue postsecondary education or competitive integrated employment” (Thurlow et al., 2019, p. 2). These kinds of shifts in perspectives over time means that it may be important for states to explore how they think about and document the reduction of depth, breadth, and complexity of their AA-AAAS.
The sections included here on depth, breadth, and complexity provide brief backgrounds on each term. Then, they pose possible questions that states could ask themselves about depth, breadth, and complexity of their AA-AAAS items and AA-AAAS overall.
ESEA assessment peer review guidance (U.S. Department of Education, 2018) does not use the term depth. Although the term complexity is used in the guidance, states providing evidence for it often refer to depth (see section on complexity). It is important for states to determine whether they will use both terms or just one term.
Depth in relation to assessments is usually used when describing items in an assessment. The term most often has been associated with the Norman Webb (1999) approach to alignment, which explores depth of knowledge (DOK) of items in addition to other criteria. Although DOK is used for each item, test alignment evaluations often judge whether an assessment has an appropriate distribution of DOK levels.
Webb (1999) and others typically refer to DOK as reflecting cognitive demand, which is described as how complex thinking has to be to complete an assessment item. For example, lower DOK may include thought processes like recall and reproduction, whereas higher DOK may include thought processes that require synthesizing from multiple sources or transferring information from one area to solve problems in another area.
Another interpretation of depth may be how much of a single skill is measured by an item. For example, a deeper level item may address both description and dialogue narrative techniques whereas an item of less depth may only address description.
DepthQuestions for States to Ask: For the AA-AAAS, a state may want to consider the following questions as it revises its AA-AAAS or develops a new AA-AAAS, or as it prepares for peer review:
|
ESEA’s assessment peer review guidance (U.S. Department of Education, 2018, Critical Element 2.1) requires states to describe the breadth of the grade-level academic content standards that the assessment is designed to measure. It specifically indicates that the state must describe “if the AA-AAAS is designed to cover a narrower range of content than the State’s general academic assessment” (p. 38).
ESEA’s assessment peer review guidance (U.S. Department of Education, 2018) implies that breadth refers to the scope or range of the content standards covered by the items in the AA-AAAS. In some cases, the AA-AAAS may cover the same content standards as the general assessment (perhaps, the essence of those standards). In other cases, the AA-AAAS may focus on a subset of standards that the state believes reflect the core content that is needed for students with the most significant cognitive disabilities to progress from one grade to the next. Another approach a state may take is to consider the range of skills required by a single standard, from the same grade-level academic achievement standard to a partial version of the standard. Extended standards and core content connectors reflect these approaches. The approach a state takes typically reflects its theory of action and can be justified by that and its assumptions about how children with the most significant cognitive disabilities learn and progress in school.
BreadthQuestions for States to Ask: For the AA-AAAS, a state may want to consider the following questions as it revises its AA-AAAS or develops a new AA-AAAS, or as it prepares for peer review:
|
ESEA peer review guidance (U.S. Department of Education, 2018) refers to the cognitive complexity (Critical Element 2.2) of the AA-AAAS and cognitively challenging (Critical Element 2.1) items. It does not define these terms.
Some states seem to use the terms depth and complexity interchangeably, whereas others use the term complexity to refer specifically to either the format of items (e.g., multiple choice with three response choices or four response choices) or an item’s connection to other disciplines. A state should be able to justify the approach it uses.
ComplexityQuestions for States to Ask: For the AA-AAAS, a state may want to consider the following questions as it revises its AA-AAAS or develops a new AA-AAAS:
|
States may use only the terms breadth and complexity, similar to the terms used in the U.S. Department of Education (2018) peer review guidance, or they may use and differentiate the three terms: depth, breadth, and complexity. Regardless of the approach, it is important for the state to have a clear understanding and strong rationale for the terms it uses and how they are reflected in the design, development, and evidence of technical adequacy of its AA-AAAS.
Bechard, S., Karvonen, M., & Erickson, K. (2021). Opportunities and challenges of applying cognitive process dimensions to map-based learning and alternate assessment. Frontiers in Education, 6.
Thurlow, M. L., Nye-Lengerman, K., & Lazarus, S. S. (2019). Suggestions for aligning alternate achievement standards with WIOA(NCEO Brief #16). National Center on Educational Outcomes.
U.S. Department of Education. (2003). 34 CFR Part 200, Title I – Improving the academic achievement of the disadvantaged, Final rule, 68(263).
U.S. Department of Education. (2005, August). Alternate achievement standards for students with the most significant cognitive disabilities. Non-regulatory guidance.
U. S Department of Education. (2018, September). A state’s guide to the U.S. Department of Education’s peer review process.
Webb, N. (1999). Alignment study in language arts, mathematics, science, and social studies of state standards and assessments for four states.Council of Chief State School Officers.
Alternate assessments were first required in the 1997 reauthorization of the Individuals with Disabilities Education Act (IDEA). The description of the alternate assessment in IDEA was that it is for students with disabilities unable to participate in general assessments even with accommodations.
The first references to alternate academic achievement standards, to students with the most significant cognitive disabilities, and to terms like complexity emerged during the development of regulations for the 2001 reauthorization of the Elementary and Secondary Education Act (ESEA) as the No Child Left Behind (NCLB) Act. The need for this kind of clarification was sought because of the accountability requirements of NCLB. Initially, the alternate assessment was not included in the accountability provisions of NCLB. After concerns were expressed by disability advocates, the inclusion of alternate assessment performance in counts of students who were proficient was included in 2003 regulations (U. S. Department of Education, 2003).
The 2003 regulations for NCLB introduced the idea of “alternate achievement standards” to contrast with the grade-level achievement standards included in the original NCLB. They also included a limit on the percentage of students who could be counted as proficient. The regulations used, for the first time, the term “students with the most significant cognitive disabilities.” The regulations stated:
An alternate achievement standard is an expectation of performance that differs in complexity from a grade-level achievement standard. These regulations clarify that a State is permitted to use alternate achievement standards to evaluate the performance of students with the most significant cognitive disabilities and to give equal weight to proficient and advanced performance based on the alternate standards in calculating school, district, and State AYP [adequate yearly progress], provided that the number of proficient and advanced scores based on the alternate achievement standards does not exceed 1.0 percent of all students in the grades tested at the State or LEA level. (U.S. Department of Education, 2003, p. 69699)
In 2005, the U.S. Department of Education provided non-regulatory guidance to further explain alternate academic achievement standards. This guidance introduced the idea of reduced breadth and less complexity. Question B-4 stated:
B-4. What are alternate assessments based on alternate achievement standards?
An alternate assessment based on alternate achievement standards may cover a narrower range of content (e.g., cover fewer objectives under each content standard) and reflect a different set of expectations in the areas of reading/language arts, mathematics, and science than do regular assessments or alternate assessments based on grade-level achievement standards. The questions on an alternate assessment might be simpler than those on a regular assessment or the expectations for how well students know particular content standards may be less complex but still challenging for students with the most significant cognitive disabilities. If a State chooses to use such assessments, it must establish alternate achievement standards through a documented standards-setting process; the assessments based on alternate achievement standards must yield separate results for reading/language arts, mathematics, and (beginning in the 2007-08 school year) science. Proficient and advanced scores in reading/language arts and mathematics from an alternate assessment based on alternate achievement standards may be used in AYP decisions in the same manner as any other scores, subject to the 1.0 percent cap at the LEA and State levels. (See Section F.) (pp. 16-17)
Other questions and answers provided additional clarification of these ideas:
These excerpts from federal law, regulations, and guidance use only the term “complexity,” although they imply “breadth” through the use of terms like “range” and “scope.”
C-1. What is an alternate achievement standard?
An alternate achievement standard sets an expectation of performance that differs in complexity from a grade-level achievement standard. The December 9, 2003 regulations clarify that a State is permitted to use alternate achievement standards to evaluate the performance of students with the most significant cognitive disabilities.
In general, alternate achievement standards must be aligned with a State’s academic content standards, promote access to the general curriculum, and reflect professional judgment of the highest achievement standards possible. (See 34 C.F.R. §200.1(d).)
The characteristics of an alternate achievement standard are the same as those described in the Title I assessment regulations for a grade-level achievement standard. That is, they are aligned with the State’s academic content standards (although they may reflect prerequisite skills rather than grade-level skills); describe at least three levels of attainment; include descriptions of the competencies associated with each achievement level; and include assessment scores (cut scores) that differentiate among the achievement levels and a description of the rationale and procedures use to determine each achievement level. These standards will be considered during the Department’s peer review of each State’s standards and assessment system under NCLB. (p. 20)
C-3. If a State chooses to develop such standards, how can it do that consistently with the requirements to have tests in grades 3-8 and high school by 2005-06?
If a State chooses to establish alternate achievement standards, such standards must be aligned with the State’s academic content standard for the grade in which the student is enrolled (or, in the case of students in ungraded classrooms, the grade level commensurate to the student’s age). (See section E-1 for further details.)
There must be a clearly stated definition of proficiency available for students assessed on the basis of alternate achievement standards, and their scores must be reported in relation to this standard. Because these students are often in ungraded classrooms, the idea of grade-by-grade alternate achievement standards for them is somewhat ambiguous. The alternate achievement standards must be challenging for students with the most significant cognitive disabilities and defined in a way that supports individual growth through a linkage to different content across grades. When examined across grades, however, alternate achievement standards are not expected to show the same clearly defined differences in cognitive complexity as the grade-level achievement standards set for the regular test.
A State may thus define alternate achievement standards for grade clusters (e.g., grades 3-5, 6-9 or 10-12) rather than for individual grades. Such standards, however, must reflect the professional judgment of the highest achievement standards possible for this group of students. For example, it is not acceptable for a State to develop a single test that employs a single rubric to define a single proficiency standard that is applied to all students enrolled in grades 3 through 12 who are tested on the basis of an alternate achievement standard. Such an arrangement fails to reflect the changes in content that would be expected across grades and cannot provide an appropriate challenge for older or more capable students. (p. 21)
C-5. What is meant by “professional judgment of the highest achievement standards possible?”
Title I requires that, for the general assessment, States establish challenging academic content standards that contain rigorous content and encourage the teaching of advanced skills, and challenging student achievement standards that determine how well students are mastering this content. States must create the achievement standards with all students in mind, so that they are realistic for a wide variety of individuals. The standards should represent a consensus among experienced teachers, parents, and other appropriate individuals regarding the performance expected after appropriate student effort in a challenging instructional program. Students with the most significant cognitive disabilities who participate in an alternate assessment based on alternate achievement standards are entitled to the same deliberate approach to defining achievement standards that represent a rigorous but realistic challenge for this heterogeneous group of students and a challenging long-range goal for their school and LEA. The term “highest achievement standards possible” is intended to reflect that the alternate achievement standards should be no less challenging for students with the most significant cognitive disabilities than the standards set for all other students. (p. 22)
E-1. What does it mean to have alternate achievement standards that are aligned with the State’s academic content standards?
Alternate achievement standards are substantially different expectations for student mastery of grade-level content, but they may not be defined as skills that are wholly independent of a State’s academic content standards. Setting alternate achievement standards is the final step in an assessment development process that includes consideration of the content to be assessed, the manner in which student understanding of that content will be demonstrated, the method for scoring student responses/products, and the manner in which student results will be reported. States will find it necessary to consider each component, beginning with the content on which students with the most significant cognitive disabilities will be assessed. This should be content that is clearly related to grade-level content, although it may be restricted in scope or complexity or take the form of introductory or pre-requisite skills. The task of defining alternate achievement standards in reading/language arts, mathematics, or science for students with the most significant cognitive disabilities should begin with consideration of the State’s academic content standards for the grade in which the student is enrolled, then adapting or “extending” those content standards to reflect instructional activities appropriate for this group of students. The next step should be designing an assessment that allows these students to show what they have learned along with a method of scoring the assessment. Finally, a group of experienced special educators and, as appropriate, parents and other individuals, should be convened to examine a sufficiently large sample of student responses and to determine the type of response(s) that is regarded as proficient for this group of students. (pp. 26-27)
E-2. How can alternate assessments based on alternate achievement standards be aligned with a State’s academic content standards?
In practice, alignment with the State’s academic content standards means that a State has defined clearly the connection between the instructional content appropriate for non-disabled students and the related knowledge and skills that serve as the basis for a definition of proficient achievement for students with the most significant cognitive disabilities.
One State, for example, has developed a curriculum framework for students with the most significant cognitive disabilities that moves from grade-level expectations to gradually less complex versions of the standard. This continuum provides a range of entry points at which a student with disabilities can access the content at an appropriately challenging level. For example, it lists the following skills for grades 3 through 4 content standards under Mathematics Operations: “Select, use and explain various meanings and models of multiplication and the division of whole numbers. Understand and use the inverse relationship between the two operations.” The State’s standards document also identifies the essence of the standard in several brief statements, for example, “understand the meaning of multiplication and division; and represent multiplication and division problems concretely.” The State then provides several illustrations of the knowledge and skills appropriate for use in the alternate assessment. These range from the less complex, “Illustrate the concept of multiplication using groups of objects,” to more complex knowledge that approaches grade-level expectations, such as “Identify the commutative property of multiplication using number sentences (3 x 5 = 5 x 3)….”
Alternate achievement standards may include prerequisite or enabling skills that are part of a continuum of skills that culminates in grade-level proficiency. The use of alternate achievement standards, however, must not result in inappropriate placements or assignment of students to a curriculum that does not include academic content. (p. 27)
1 U.S. Department of Education. (2005). Alternate achievement standards for students with the most significant cognitive disabilities non-regulatory guidance. Available at https://eric.ed.gov/?q=ED485842&id=ED485842
Peer review guidance from the U.S. Department of Education’s Office of Elementary and Secondary Education (OESE) provides additional details for the review of state assessments (U.S. Department of Education, 2018). In its introduction it states: “Alternate academic achievement standards set expectations of performance that differ in scope and complexity from grade-level achievement standards” (emphasis added, p. 23). For the AA-AAAS, evidence sought by peer reviewers includes, for example: