Guidance for Examining Disproportionality of Student Group Participation in Alternate Assessments
The purpose of this brief is to examine disproportionality with respect to student group participation in Alternate Assessments aligned with Alternate Academic Achievement Standards (AA-AAAS). This paper is a companion to another National Center on Educational Outcomes (NCEO) brief, Guidance for Examining District Alternate Assessment Participation Rates (Evans & Domaleski, 2018). The companion brief outlined a broad framework and general principles to guide the examination of AA-AAAS participation rates, particularly in light of the challenge of small n-sizes and associated uncertainty.
This brief provides a more detailed examination and illustration of a particular issue, disproportionality, which can be situated within the framework. We begin this brief by reviewing the framework with respect to its application to address disproportionality. This is followed by an illustration of specific analytic steps, and concludes by providing guidance for further inquiry that links back to the guiding principles in the framework.
In addition to the guidance found in both briefs, Guidance for Examining District Alternate Assessment Participation Rates (Evans & Domaleski, 2018) and this brief, Guidance for Examining Disproportionality of Student Group Participation in Alternate Assessments, NCEO and the National Center for the Improvement of Educational Assessment (NCIEA) produced a third companion resource, a video training module: Guidance for Examining Participation Rates and Disproportionality (Evans & Domaleski, 2019). The video training module covers the principles and process necessary to engage in a review of participation and disproportionality on the AA-AAAS. It also provides some examples to demonstrate analytic techniques in Excel with sample data. The video may be accessed at: https://nceo.info/Resources/videos.
Examining disproportionality with regard to alternate assessment participation is essentially an inquiry into whether certain groups are over- or under-identified as having a significant cognitive disability. This brief is intended to fill a gap created by the lack of an explicit definition or guidance on how to calculate disproportionality for students with significant cognitive disabilities who participate in an AA-AAAS. We distinguish this requirement for the alternate assessment from the definition and requirements under the Individuals with Disabilities Education Act (IDEA) that requires states to collect and examine data to determine whether significant disproportionality is occurring in the overall population of students receiving special education services (IDEA Section 618(d)).
Specifically, we address the following key questions:
The Every Student Succeeds Act (ESSA) of 2015 requires states to apply for a waiver prior to the testing window if they think they will go over the 1% participation rate cap for students with the most significant cognitive disabilities taking an AA-AAAS (34 CFR 200.6(c)(2)). Part of a state’s waiver application is verifying and addressing disproportionality in the identification of students with the most significant cognitive disabilities. Specifically, the state must provide evidence that it has verified that each district (Term used synonymously with Local Education Agency [LEA]) that the state anticipates will test more than 1% of its assessed students in any subject using an AA-AAAS:
(1) followed the state’s guidelines for participation in the AA-AAAS; and
(2) will address any disproportionality in the percentage of students in any subgroup under section 1111(c)(2)(A), (B), or (D) of the Act taking an AA-AAAS (34 CFR 200.6(d)), consistent with section 612(a)(16)(C) of the IDEA.
These student groups include seven racial and ethnic groups (White, Black or African American, Hispanic, Native American or Alaska Native, Asian, Pacific Islander, and Multiracial), socio-economic status (as determined by a students’ eligibility for Free and Reduced Price Meals), and English learners. The state must also provide a plan and timeline with clear, actionable steps and milestones for how the state will address any disproportionality in the percentage of students taking an AA-AAAS as identified through the data provided by districts (34 CFR 200.6(c)(4)(iv)).
As stated previously, the law does not explicitly define the term “disproportionality” or provide any guidance on methodology for analysis of disproportionality for students with the most significant cognitive disabilities taking an AA-AAAS.
A comprehensive investigation of disproportionality involves the use of both analytical techniques and other sources of evidence to corroborate findings whenever possible. We distinguish our definition of disproportionality from requirements under IDEA that require states to collect and examine data to determine whether significant disproportionality is occurring in the overall special education population with respect to the identification of students with disabilities, the placement of such students in particular educational settings, and the incidence, duration, and type of disciplinary actions of such students (IDEA Section 618(d)).
The previous NCEO brief (Evans & Domaleski, 2018) provided an overall framework for analyzing district alternate assessment participation rates. We start by highlighting some key points from that framework that are broadly applicable to examining disproportionality in alternate assessment participation. The framework consists of four guiding principles that characterize approaches likely to be more effective in identifying ‘exceptional’ versus ‘expected’ district alternate assessment participation rates and also disproportionality in the identification of certain groups of students with the most significant cognitive disabilities who take an AA-AAAS.
1. A comprehensive solution cannot be purely empirical due to small n-sizes.
Analyses and inferences are challenged by small n-sizes when examining overall district participation rates in alternate assessments. The issue with small n-sizes is exacerbated in this context because those small n-sizes are now to be broken down (disaggregated) for student groups. For this reason, it is important to be cautious about the analyses conducted and the inferences that can be made at each level. For example, an analysis that drills all the way down to student groups at the school level is going to significantly limit the claims or inferences in which one can have confidence. Moreover, minimum cell sizes and overall n-sizes are extremely important to consider in this situation for public reporting purposes to prevent individual students from being identified from their unique personal characteristics.
It is important for states to streamline and prioritize the targeted groups for their analyses given the constraints provided by small n-sizes and associated uncertainty. States should understand the limits of expanding the number of groups included, especially at certain levels (e.g., district, school). This indicates the need for careful examination of the data and use of multiple sources of evidence when attempting to determine disproportionality of certain groups taking an AA-AAAS. States should combine judicious data analyses with thoughtful inquiry that is tailored to the unique context of the state and districts in the state, and that also takes into account multiple years of data and other sources of evidence whenever possible.
2. It is important to detect atypical or “exceptional” values.
This principle draws our attention to the need to develop a baseline for expectations. In our earlier NCEO brief, we highlighted several research-based assumptions that can help detect when certain values are ‘expected’ versus ‘exceptional’ and therefore in need of additional scrutiny. We can employ a similar procedure here, although there is no evidence to suggest that the student groups required under federal law (race/ethnicity, free-and-reduced price lunch (FRL) status, English learner (EL) status) are more or less likely to have the most significant cognitive disabilities and take an AA-AAAS. We might assume that there should be similar proportions of students with significant cognitive disabilities from across race/ethnicity categories, FRL status, and EL status compared to the general population. The question is at what level should we make this assumption?
Given the even smaller n-sizes for group analyses it is likely advisable to set a baseline expectation at the highest unit of analysis possible with multiple years of data. This is especially important for smaller districts or districts with relatively few students in one or more groups. In the case of very small n-sizes, even multi-year analyses may not be sufficient to produce an appropriate n-size for an analysis that can reasonably support an inference about disproportionality. In such instances, it is particularly important to corroborate findings with other sources of information, such as data at the state level, data from other (especially comparable) districts, conversations with district special education administrators, etc.
When examining data for disproportionality in certain student groups taking an AA-AAAS, states should consider the potential for an interaction to occur between the under- or over-identification of certain groups (e.g., students from certain racial/ethnic groups with disabilities) with an examination of disproportionality. The under-identification of some student groups has been documented in the overall special education population (e.g., Morgan, Farkas, & Hillemeier, 2017), but not as well established in the population of students with the most significant cognitive disabilities. Given this lack of research evidence about the “true” distribution of students by group who take an alternate assessment, these errors can potentially create misleading assumptions about expected participation rates and confound interpretation of results.
3. Apply methods to address uncertainty.
As stated in the previous NCEO brief (Evans & Domaleski, 2018), uncertainty is a reality for any inferential statistical analyses where judgments are made about the likelihood that an observed difference might have happened by chance alone. Uncertainty is a critical issue for the AA-AAAS due to the even smaller n-sizes that occur during subgroup analyses. Methods discussed in the earlier brief included confidence intervals and multiple years of data. In the next section, we discuss these methods applied to AA-AAAS disproportionality calculations.
4. The culminating decision and subsequent actions based on the evidence is (a) a matter of degree, and (b) related to unique context and circumstances.
Making an error in one direction or the other can have real implications for students with the most significant cognitive disabilities. Type I error (false positive: an unwarranted claim of disproportionality) and Type II error (false negative: failure to detect disproportionality) apply here. Districts identified with ‘exceptional’ or atypical values in disproportionality for certain groups on an AA-AAAS will have to provide a clear description of not only how they followed the state’s guidelines for participation in the AA-AAAS, but also how the district will address the identified disproportionality. The consequence of a Type I error results in expenditure of time and resources that may be unwarranted. The consequence of a Type II error is a failure to detect and address a problem of possible inequitable practice.
Due to the extremely small n-sizes for certain subgroups, there is likely no single piece of evidence sufficient to decide with certainty about whether any identified disproportionality is warranted. We suggest evaluating the preponderance and collection of evidence in light of state and district trends over multiple years of data to describe a level of confidence along a continuum.
Analytic Approaches to Explore Disproportionality
Two analytic approaches from among a range of potentially broader approaches are recommended here to examine disproportionality in AA-AAAS participation. The central purpose of these analyses is to quantify the difference between observation and expectation. More specifically, the questions of interest are:
To address these questions, we propose the following process.
Step 1: Establish Participation Rate for Each Focal Group
Focal group refers to any student group that is being examined, such as students who are economically disadvantaged, English learners, students in a particular race/ethnic group, etc.
Establishing the participation rate for each focal group (i.e., the student group under analyses) is confounded by the problem of small n-sizes. One approach to address the small n-size challenge is to produce the rate based on multiple years of data. For example, Table 1 illustrates how three years of data may be combined to create a multi-year calculation for AA-AAAS participation for one group.
As illustrated in Table 1, the annual participation rates varied for the focal group from 28.7% to 36.9% over the period examined. The multi-year participation rate for the focal group during the period is 32.9%. This is simply calculated as the sum of focal group participants divided by the sum of all participants (148/450).
Table 1. Illustration of Calculating Participation Rates with Multi-Year Data
Step 2: Calculate the Test Statistic
Here we illustrate how to calculate a test statistic to quantify the difference between the AA-AAAS observed participation rate and expected participation rate. We illustrate this using two approaches: (a) difference in proportions, and (b) risk ratios.
Difference in proportions. The difference in proportions is simply the observed proportion of focal group AA-AAAS participants minus the “expected proportion.” Although there are many ways to establish the “expected proportion,” we suggest simply using the proportion of focal group members in the population for the unit of analysis (e.g., district). We present each proportion as a percent for ease of interpretation. For example, consider Table 2, which illustrates hypothetical participation rates for a district over multiple years.
Table 2. Illustration of AA-AAAS and Non-AA-AAAS Proportions
The focal group proportion is calculated in the same manner as illustrated in Table 1: the number of focal group AA-AAAS participants is divided by the total AA-AAAS participants, yielding a participation rate or proportion of 32.9% (148/450). Similarly, the proportion for the non-AA-AAAS students is calculated by dividing the focal group students who are not AA-AAAS participants by the total number of non-AA-AAAS participants (15,000/50,000)—a proportion of 30.0%.
The difference in proportions is produced by simply subtracting the focal group proportion for AA-AAAS participants from the proportion of focal group students who are not AA-AAAS participants (32.9% - 30.0%), which is a difference of 2.9%. In other words, there are about 3% more focal group students participating in the AA-AAAS than one would expect based on the proportion of focal group students in the general population. This procedure can be repeated for all focal groups of interest.
Risk ratio. Another way to describe the relationship of observed versus expected proportions is with a risk ratio, sometimes termed “relative risk.” This is simply calculated by dividing the proportion of focal AA-AAAS participants by the proportion of focal group students who are non-participants. Drawing from the previous example, the calculation is as follows:
A risk ratio of 1.0 indicates that the proportion of focal group students who are AA-AAAS participants is the same as the proportion of focal group students who are not AA-AAAS participants. In other words, there is no evidence of disproportionality when the risk ratio is at or very near 1. Values greater than 1 indicate that the probability or “risk” of a focal group student being an AA-AAAS participant is greater than expectation. For example, a risk ratio of 2.0 indicates that a student who is a member of the focal group is twice as likely to participate in the AA-AAAS. Values less than 1.0 indicate that a focal group student is less likely to be an AA-AAAS participant than expected. For example, a risk ratio of 0.5 indicates that students in the focal group are half as likely to be AA-AAAS participants.
Step 3: Determine if the Difference is Meaningful
The final and perhaps most challenging step is to make a determination of whether the difference in proportion or risk ratio is meaningful. That is, do the results provide evidence to support or refute a claim of disproportionality?
One way to address this question is by conducting a test of statistical significance on the test statistic computed in step 2. This is commonly accomplished by computing a confidence interval and then evaluating whether the test statistic is outside of the interval for a desired level of confidence (e.g., 95%). Confidence intervals can be produced for differences in proportions or risk ratios, the latter of which requires a transformation to a natural log. The technical details for producing these confidence intervals are beyond the scope of this paper (see, for example, Agresti, 1996 for details). However, the video module, produced as a companion to this paper, provides more technical details and illustrative examples.
If we assume a 95% confidence interval on the proportion of focal group students participating in an AA-AAAS in the previous example is +/- 4.34, the “true range” for the proportion would be from 28.6% to 37.2%; this takes into account variance due to the n-size. This range includes the 30% expectation, based on the proportion of focal group students who are not AA-AAAS participants. This would not be statistical evidence of disproportionality (or that the proportions are significantly different).
It is important to remember that statistical tests are always an incomplete inquiry into ‘meaningfulness,’ particularly in the present context given the challenge of low n-sizes. We caution against making a claim about disproportionality based strictly on significance testing. Whether a finding is considered meaningful should be informed by inquiries such as:
Ultimately, we refer back to the principles in the framework to guide decision making about findings. The decision should take into account qualitative as well as quantitative information; multiple sources of information should be evaluated to address uncertainty; and context should be considered. Finally, decisions are typically a matter of degree and require a blend of evidence with informed judgment.
The broad framework and general principles outlined in an earlier NCEO brief (Evans & Domaleski, 2018) can be applied to disproportionality and used by states to ensure that any examination of disproportionality for a student group participating in an AA-AAAS takes into account the unique challenges associated with disaggregating already small n-sizes. Due to the small n-size of the alternate assessment student population, it is difficult (if not impossible) to set solely empirical guidelines for when a district’s alternate assessment participation rate is disproportionate. We recommend a measured approach that combines appropriate quantitative analyses and qualitative inquiry at the highest unit of analysis possible (e.g., district or state), uses multiple years of data to increase the n-size, and makes judicious decisions about the selection of focal student groups.
We also urge caution when examining disproportionality in student groups at the district level, particularly when the total district alternate assessment population size is small. Differences in student group participation rates based on small n-size may appear as large differences in proportions or relative risk ratios. It is advisable to consider the differences in proportions or risk ratios alongside frequency counts to inform a better judgment about disproportionality and associated certainty.
Agresti, A. (1996). An introduction to categorical data analysis. New York, NY: Wiley & Sons.
Evans, C. M., & Domaleski, C. (2018). Guidance for examining district alternate assessment participation rates. Minneapolis, MN: National Center on Educational Outcomes (NCEO) and the National Center for the Improvement of Educational Assessment. Retrieved from https://rtc3.umn.edu/docs/OnlinePubs/NCEO1percentBrief.pdf
Morgan, P. L., Farkas, G., & Hillemeier, M. M. (2017). Replicated evidence of racial and ethnic disparities in disability identification in U.S. schools. Educational Researcher, 46(6), 305–322. Retrieved from https://doi.org/10.3102/0013189X17726282
NCEO Brief #18, August 2019
Contributors to the writing of this Brief were Carla Evans and Chris Domaleski
NCEO Director, Martha Thurlow; NCEO Associate Director, Sheryl Lazarus; NCEO Assistant Director, Kristin Liu.
All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Evans, C., & Domaleski, C. (2019, August). Guidance for examining disproportionality of student group participation in alternate assessments (NCEO Brief #18). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
NCEO is supported through a Cooperative Agreement (#H326G160001) with the Research to Practice Division, Office of Special Education Programs, U.S. Department of Education. The Center is affiliated with the Institute on Community Integration at the College of Education and Human Development, University of Minnesota. The contents of this report were developed under the Cooperative Agreement from the U.S. Department of Education, but does not necessarily represent the policy or opinions of the U.S. Department of Education or Offices within it. Readers should not assume endorsement by the federal government. Project Officer: David Egnor
NCEO works in collaboration with Applied Engineering Management (AEM), Council of Chief State School Officers (CCSSO), National Association of State Directors of Special Education (NASDSE), and WestEd.
National Center on Educational Outcomes