Why We Can't Say Much About the Status of Students with Disabilities During Education Reform

NCEO Synthesis Report 21

Published by the National Center on Educational Outcomes
in collaboration with St. Cloud State University and the National
Association of State Directors of Special Education

Prepared by:

Kevin S. McGrew, Mike L. Vanderwood, Martha L. Thurlow, and James E. Ysseldyke

August, 1995

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

McGrew, K. S., Vanderwood, M. L., Thurlow, M. L., & Ysseldyke, J. E (1995). Why we can't say much about the status of students with disabilities during education reform  (Synthesis Report No. 21). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://cehd.umn.edu/NCEO/OnlinePubs/Synthesis21.html

Abstract

Education reform initiatives throughout the entire educational system have focused attention on outcomes and quantifiable data.  With increasing frequency, the data needed to monitor and evaluate reform initiatives are being drawn from large-scale state and national data collection programs.  Although sufficient national level school completion outcome indicators and state goal statements suggest the potential for evaluating the outcomes for students with disabilities, secondary data analysis of state and/or national data bases is not feasible.  The exclusion of significant numbers of students with disabilities from both state and national data collection programs and/or the variable nonexistent disability-specific variables used to identify these students in these programs makes it all but impossible to describe the status of students with disabilities.  Recommendations are offered for improving the collection and reporting of information in state and national data collection programs.

Overview

"School reform has riveted national attention on the numbers" (Hanford & White, 1991). Reform initiatives throughout the entire educational system have shifted the focus toward outcomes and quantifiable data. With increasing frequency, the data needed to monitor and evaluate these reforms are being drawn from large-scale national data collection programs (e.g., National Assessment of Educational Progress - NAEP) and statewide assessments (e.g., state-wide testing).

The recent national goals and educational indicators movements produced a flurry of activity to identify databases that include indicators that could help monitor progress toward goal attainment. The Special Study Panel on Educational Indicators (1991) reported that the success of educational reform depends on the development of a "comprehensive education indicators information system capable of monitoring the health of the enterprise, identifying problems, and illuminating the road ahead" (p. 6). Similar calls for the improvement of the existing national education data system and the development of new components to include in the system have been echoed in reports by the National Education Goals Panel (1991) and the National Education Statistics Agenda Committee of the National Forum on Education Statistics (1990). Each of these groups has turned toward national data collection programs for indicators to monitor progress during the current wave of reform. Parallel activities have also occurred within states as they have grappled with documenting the effects of state and local education reform initiatives.

Even with the past year of upheaval and questioning about continuing Goals 2000, few have challenged the need to collect data on academic outcomes. There is a commitment to the need for data on the effects of education for students, particularly students with disabilities.

In addition to the general education reform movement, reform initiatives in special education (Skrtic, 1991) have produced increased interest in the analysis of existing national and state databases. Since the passage of PL 94-142 in 1975, there has been over a decade of evaluation studies that have focused primarily on the issue of educational access for students with disabilities and implementation of the processes embodied in the law. Increasingly the question of "where's the beef?" has been asked from both within and outside of special education. Focus has recently turned toward evaluating the outcomes of special education, or, "where's the data?" on effectiveness (DeStefano & Wagner, 1991).

Purpose of this Report

The National Center on Educational Outcomes (NCEO) was established in October, 1990 to work with state departments of education, national policy-making groups, and others to facilitate and enrich the development and use of indicators of educational outcomes for students with disabilities. It is believed that responsible use of such indicators will enable students with disabilities to achieve better results from their educational experiences. Students with a wide array of disabilities fit within this population, including those with learning disabilities, emotional disabilities, and speech and language impairments, those with sensory disabilities such as hearing impairments and visual impairments, and those with multiple and more severe disabilities, typically involving significant mental impairments.

One of the four major strategic goals of NCEO during the past five years has been "to enhance the availability and use of outcomes information in decision making at the federal and state level." Several activities have been subsumed under this goal. Two activities have dealt specifically with identifying and reviewing the characteristics of state and national data sets as a prelude to secondary analysis of the data. The purpose of these activities has focused on determining the feasibility of extracting policy-relevant information on the educational status and performance of students with disabilities.

The results of this critical review process have contributed to NCEO's strategic goals by:

Determining the extent to which the current state and national data collection programs consider students with disabilities when planning and implementing data collection.

Determining the extent to which students with disabilities are included or excluded in state and national data collection programs as a result of sample design and inclusion/exclusion procedures.

Determining the depth and breadth of outcomes included in state and national data collection programs and the extent to which these outcomes approximate a conceptual model for a comprehensive system of indicators.

Determining the means by which state and national data collection programs describe the functional characteristics of students with disabilities.

During the past five years, NCEO staff have systematically examined the extent to which students with disabilities are "at the table" when it comes to participation in state and national data collection activities. Given that nearly 5 million school-age youngsters with disabilities receive some form of special education services, services that are provided at significant expense to our educational system, it is imperative that we examine how these students are performing. This report synthesizes the results of the NCEO activities that focused on determining whether secondary data analysis of state and/or national data collection programs could produce policy-relevant reports on the status of children and youth with disabilities (McGrew, Spiegel, Thurlow, Ysseldyke, Bruininks, & Shriner, 1992).

The NCEO Approach

The NCEO approach to secondary data analysis of state and national data collection programs followed a five step plan. The five steps were: (1) identify a working list of potentially useful and relevant state and national data sets, (2) target specific data collection programs that include indicators relevant to NCEO's conceptual model of educational outcomes and indicators, or that are featured prominently in the national educational indicator dialogue, (3) conduct disability sensitivity reviews of the targeted databases, (4) analyze the target databases to determine the extent to which NCEO's conceptual model of outcomes and indicators are represented in state and national data collection programs, and (5) develop and complete a list of prioritized secondary data analyses. An outline of this process is presented in Figure 1.

This report summarizes the results and experiences of NCEO in following its five-step plan for producing reports on the status and performance on children and youth with disabilities.

Identification of a Working List of Databases

The identification of a preliminary list of national and state databases required two different sets of activities.

Figure 1.  NCEO Approach to Identifying and Conducting Secondary Data Analysis of State and National Databases

National data collection programs. The process of locating national data programs required the use of a mixture of formal and informal sources. Written, phone, and personal contacts were made by NCEO staff with a wide array of public and private agencies and organizations devoted to either the production (e.g., National Center for Education Statistics - NCES; National Center for Health Statistics - NCHS) or dissemination (e.g., National Technical Information Service - NTIS; Data Users Services Division of the Census Bureau) of database information. Policy oriented reports produced by organizations that utilized secondary analysis of data were reviewed for possible clues about additional data sets. In addition, critical reports by groups actively involved in recommending the use of data from the current national education data system (National Education Goals Panel; Special Study Panel on Educational Indicators; National Education Statistics Agenda Committee) were reviewed for references to important data sets. Articles devoted to the listing of data sets (e.g., Taeuber & Rockwell, 1982, "National social data series: A compendium of brief descriptions") were also found to be useful in the identification of additional data sets. Finally, informal sources were also used and consisted of referrals to sources from individuals contacted at various agencies or organizations and contacts with researchers who are visibly active in the utilization of data from the national education data system for policy relevant research. This process resulted in the development of a "working list" of over 70 national databases (McGrew et al., 1992).

State data collection programs: In the Spring of 1991, state directors of special education or their designees responded to the annual NCEO national survey of state special education outcomes activities (NCEO, 1992). This survey was used to gather information on state efforts in the areas of federally-reported data, assessment of outcomes, inclusion of students with disabilities in state assessments, state assessment needs and highlights, activities in selected outcome areas, and practices, programs, and plans related to outcomes.

In the initial annual survey, 49 of the 50 states reported that some students with disabilities took part in their general education large-scale achievement assessments. These state assessments typically varied from the administration of nationally-normed commercial achievement tests (e.g., Stanford Achievement Test) to state-developed norm-referenced or minimum competency exams. Slightly over half of the 50 states (n = 27; 54 %) indicated that students with disabilities could be identified in their data sets. In other words, some variable was present in the state database that indicated each student's special education status. These 27 states were the initial working list selected for inclusion in NCEO's secondary data analysis activities.

Identification of Target Databases

The process of targeting specific state and national data collection programs proceeded in the following manner:

National data collection programs. A major NCEO activity has been the development of a conceptual model of educational outcomes for children and youth with disabilities (Ysseldyke, Thurlow, Bruininks, Gilman, Deno, McGrew, & Shriner, 1992). As a result of this model development process, the major outcome domains were identified for the assessment of outcomes for all students at six key developmental points: Early Childhood-Age 3 (Ysseldyke, Thurlow, & Gilman, 1993a); Early Childhood-Age 6 (Ysseldyke, Thurlow, & Gilman, 1993b); Grade 4 (Ysseldyke, Thurlow, & Erickson, 1994a); Grade 8 (Ysseldyke, Thurlow, & Erickson, 1994b); School Completion (Ysseldyke, Thurlow, & Gilman, 1993d); and Post-School (Ysseldyke, Thurlow, & Gilman, 1993c). These were identified by key stakeholders (i.e., policy makers, teachers, parents, legislators, advocates, etc.) who participated in a consensus meeting process (Vanderwood, Ysseldyke, & Thurlow, 1993). The school completion version (Ysseldyke, Thurlow, & Gilman, 1993d) of this dynamic model is presented in Figure 2. National data collection programs that appeared to include indicators of the NCEO outcome domains were identified.

A second consideration focused on national education reform initiatives. The reports of the major national groups (e.g., National Education Goals Panel; National Education Statistics Agency Committee of the National Forum on Education Statistics; Special Study Panel on Educational Indicators) that were either (1) developing comprehensive systems of indicators, or (2) making recommendations on how to improve the national education data system were reviewed to identify national data collection programs that were receiving significant attention (McGrew et al., 1992). Based primarily on these two considerations, 28 national data collection programs were targeted. The list of targeted national data sets is presented in Table 1.

State data collection programs. Individual follow-up phone calls were made by NCEO staff to the 27 states that were identified (as a result of the NCEO state survey) as ones that might have usable data sets for secondary data analysis. Potential data sets were defined as those that included achievement data from large-scale achievement testing programs.

After a lengthy process (see description by McGrew, Spiegel, Thurlow, Shriner, & Ysseldyke, 1994 for description), only 6 of the 27 targeted state databases were acquired by NCEO staff. Seven general types of problems were encountered when trying to obtain the targeted data sets from 21 of the states. These problems included nonresponses to requests for data, concerns about confidentiality, computer files with suspect or unreadable data, excessive acquisition costs, and unreliable identification of students with disabilities in the databases.

Disability Sensitivity Reviews of Targeted Databases

Contacts were made with the appropriate state or national agency staff to request additional information on the targeted data collection programs. Requests were made for all relevant methodology and technical reports and manuals. For the state data, formal requests were made to secure copies of the computer files. All relevant materials for each targeted data set were subjected to a detailed review process (see description by McGrew et al., 1994) that focused on determining the extent to which each database might be suitable for secondary data analyses. Specific attention was focused on determining the extent to which students with disabilities were included and appropriately identified in the final database.

Inclusion of students with disabilities in national and state data collection programs. A review of state and national data collection programs found that "the ability to extract useful policy-relevant information on the outcomes of students with disabilities is hampered by the significant exclusion of portions of this population in a number of these data collection programs" (McGrew, Thurlow, & Spiegel, 1993, p. 327). NCEO found that most existing state and national data collection programs excluded a large portion of the student population with disabilities. At the national level, 40% to 50% of school-age students with disabilities were estimated to be excluded from the most prominent national education data collection programs (e.g., National Assessment of Educational Progress; NAEP). Conversely, exclusion was found to be minimal in a number of noneducational national data collection programs (e.g., National Health Interview Survey). Indicative of the low status placed on the inclusion of students with disabilities in state assessment programs was the finding that data documenting the extent of exclusion was, with few exceptions, largely unavailable (McGrew et al., 1992).

Although available assessment technology precludes the inclusion of all students with disabilities in large scale assessment programs, NCEO concluded that a "sizable portion of excluded students should not have been excluded and could readily participate (some with testing accommodations; others without)" (McGrew, Thurlow, & Spiegel, 1993, p. 347).

Table 1.  NCEO Targeted National Data Collection Programs

Many factors have been found to contribute to the exclusion of students with disabilities.  Among these factors are:

• Use of vague assessment inclusion/exclusion guidelines
• Differential and inconsistent implementation of inclusion/exclusion guidelines
• Incomplete or unsuccessful monitoring of the extent to which the intent of inclusion/exclusion guidelines is followed
• Sampling plans that systematically exclude students who are in separate schools and students who are not in graded programs
• Altruistic motivations, such as lessening the emotional distress to students who are not expected to do well
• Nonavailability of accommodations in assessment materials and procedures
• Incentives created by the desire to have a school or state look good in comparison to others in the state or nation

Identification of students with disabilities in national and state data collection programs. Although large numbers of students with disabilities are excluded from many state and national assessment programs, some students with disabilities (almost all in the case of many noneducational national data collection programs such as those of the National Center on Health Statistics - NCHS) are included in these programs. Thus, NCEO considered it important to investigate the extent to which subpopulations of students with disabilities were identified in the final database, when and if they were included in data collection programs,. Accurate and consistent identification of students with disabilities in state and national databases is a prerequisite for conducting useful secondary data analysis.

Nineteen of the twenty-eight targeted national data collection programs were reviewed to evaluate the correspondence between the disability categories or terms used in each data collection program and those used by the federal government in its Annual Report to Congress on the Implementation of the Individuals with Disabilities Act (McGrew, Algozzine, Spiegel, Thurlow, & Ysseldyke, 1993). The results indicated that there is significant variability between different national data collection programs in the manner in which individuals with disabilities are identified. Notable differences were observed between data collection programs sponsored by educational and noneducational agencies. As would be expected when using the federal special education categorical system as the basis for the analytical framework, those data collection programs sponsored by the U. S. Department of Education contained the largest number of data elements that displayed some correspondence to the federal special education categories. However, even within data collection programs sponsored by the Department of Education there was variability.

Inconsistency in the identification of students with disabilities in state databases was also found to be a problem. Of the initial 27 states contacted, 6 state databases were characterized by unreliable or no coding of students with disabilities. In fact, few states provided for the identification of students with each type of disability at the grade levels assessed. Further complicating the ability to aggregate state information was incompatibility across states in the differentiation within categories (e.g., mental retardation) by level of disability (e.g., educable, trainable, or severe versus a single global category). In addition to problems in the exclusion of many students with disabilities, and inconsistent or no coding of students with disabilities in state data collection programs, aggregation of state databases was also problematic due to the sparseness of data at individual grade levels (even after aggregation) and noncomparability of types of data (national vs. local norm-referenced scores; state-specific minimum competency scales).

Based on the extent of exclusion and variable or no identification of students with disabilities in most state and national data collection programs, we concluded that it is currently not possible to extract, on an regular basis, nationally representative policy-relevant information on the educational and quality-of-life outcomes for students with disabilities. This conclusion echoes the conclusion reached by the Committee on a National Agenda for the Prevention of Disabilities (Pope & Tarlov, 1991) that epidemiologists and statisticians have generally ignored the issue of disability and that the information that is available has been collected in a piecemeal fashion.

Analysis of Targeted Databases According to NCEO's Comprehensive System of Outcomes and Indicators

A goal of NCEO is to evaluate the extent to which the indicators included in the NCEO comprehensive system of outcomes and indicators (see Figure 2) are present in existing and emerging state and national data collection systems. The value of this activity is to: (a) highlight potential gaps in current state and national education data collection systems, systems that have not had the benefit of evolving from an a priori conceptual model, and (b) provide an organizational framework from which to conduct secondary analyses. The identification of information gaps in current data collection systems may serve to stimulate the modification of current data collection programs, or, the development of new data collection programs that produce more comprehensive conceptually-based data that provide useful, policy-relevant information for all children.

Analysis of national data collection programs. The analysis of targeted national data sets followed a sequence that included the review and classification of individual test or survey items according to the NCEO conceptual model of outcomes and indicators (see details in McGrew, Spiegel, Thurlow, & Kim, 1994). To date these analyses have been completed for the NCEO school completion model, which includes 77 indicators. Thirteen national data collection programs that measure relevant outcomes at the completion of school were reviewed.

At the national level, indicators of important school completion outcomes for all students were found to be substantially represented in the existing national data collection system. Across 13 national data collection programs that assess individuals at the completion of their secondary schooling, 91% (70 of 77) of the NCEO outcome indicators were found to be represented by at least one measure. Approximately half of all the NCEO indicators were found to be present in two or more of the thirteen data collection programs reviewed. Although the final number of usable indicators would most likely be less, due to the technical and methodological problems typically encountered in secondary data analysis (McGrew et al., 1991), McGrew, Spiegel, Thurlow and Kim (1994) concluded that the potential exists for producing comprehensive and conceptually organized policy reports regarding the status of all students as they complete school on a number of important NCEO outcomes and indicators.

The NCEO review indicated that the outcome domains of Satisfaction and Accommodation and Adaptation, and to a smaller extent, Personal and Social Adjustment and Responsibility and Independence, are the areas where usable national indicators may be problematic. Outcome domains for which usable national indicators are most likely to be found are Participation, Academic and Functional Literacy, Contribution and Citizenship, and Physical Health.

State data collection programs. Despite a high degree of overlap between many state education goals and the outcome domains in the NCEO conceptual model (Spande & Thurlow, 1994), Ysseldyke, Vanderwood, and Reschly (1994) found that there are few NCEO school completion indicators for which data are currently reported by seven or more states. For state data collection programs, data managers from 36 states and territories were surveyed on the availability of data for the 77 NCEO school completion indicators. The state data managers were asked to indicate on a 5-point Likert scale the extent to which the data for each indicator are available or might be made available. In general, state data are currently available for indicators in the Presence and Participation and Contribution and Citizenship domains. Data on Physical Health, Independence and Responsibility, Satisfaction, Accommodation and Adaptation, and Personal and Social Adjustment are relatively unavailable across states. These results suggest a significant disparity between the kinds of data stakeholders say are needed for purposes of improving instruction and educational outcomes and the data that are currently available or could be made available.

Development of a List of Prioritized Secondary Data Analyses

The final step in the NCEO process is the generation of key informational needs and research questions, the placement of these questions within a conceptual framework, and the specification of the types of comparisons needed to answer the key questions (McGrew et al., 1992). This step focuses on answering the question of "what do we want to know about students with disabilities?" Unfortunately, given the nature of the problems encountered when attempting to locate usable state and national level data on the status of students with disabilities, NCEO has as yet been unable to reach this step. Instead, NCEO has focused its energies on helping state and national agencies find solutions to the challenges of increased inclusion, better sample description, and increased participation through the development and use of assessment accommodations and adaptations for students with disabilities. It is only after progress is made in these areas, and "good" data become available in state and national data sets, that serious attention can be directed to the development and completion of lists of secondary analyses.

Summary and Conclusions

Calls for reform in American education during the past decade have resulted in raised expectations, attempts to develop "world class" standards, and increased interest in the measurement of school outcomes. There is no doubt that the current wave of education reform places significant attention on measurement-driven accountability and evaluation. Results from state and national assessments and national reports (e.g., annual National Education Goals Report) often appear in local and national newspapers in the form of eye-catching headlines such as "eighth grade students in state X rank 20th in the nation in mathematics" or "X percent of high school students experience violence during school."

A look behind the headlines reveals that the data such reports are based on are derived primarily from large-scale state and national data collection programs. Further examination indicates that an answer to the question of "How are students with disabilities doing in the same areas?" is currently impossible to answer. Given that the current education reform activities use measurable indicators from large-scale assessments as the index of progress, the evaluation of the education of most students with disabilities is being short-changed.

The potential exists for the production of useful policy-relevant information about the school completion outcomes of students with disabilities. A significant number of important school completion outcome indicators identified by key stakeholders (the consensus-based NCEO model of outcomes) is present in national data collection programs, and many are mentioned as important outcomes in state goals reports. The bad news is that this potential cannot currently be tapped for the recurring analysis and generation of reports about students with disabilities. The exclusion of significant numbers of students with disabilities from both state and national data collection programs and/or the inconsistent or nonexistent disability-specific variables used to identify students in these data collection programs makes it all but impossible to use all the outcome information that is currently available. In addition, although state goals encompass many important school completion outcome indicators, available state data collection programs only assess a limited number of these indicators.

The conclusions reached in this report should not be construed as a general indictment of most state and national assessment activities. It is important to recognize that the problems encountered in NCEO's secondary data analysis activities surfaced when attempting to use databases originally developed for different purposes. Most large-scale state and national data collection programs provide extremely important, reliable, and valid information for general education state and national level analyses and decision making. Many of the national programs were not originally designed to provide answers to educationally related questions (e.g., data collection programs of the National Center for Health Statistics) and are burdened with many competing goals and objectives. Large-scale state and national data collection programs are designed and operated to meet the unique needs of each state or congressional mandates. They typically are not designed to meet the needs of independent researchers who seek to conduct secondary data analyses, especially aggregated analyses across a number of data programs.

However, in an environment of limited resources for new and expensive large-scale nationally representative data collection programs, we believe it would be most practical and cost-effective to modify and use existing data collection programs to answer new and emerging questions. How individuals with disabilities are performing during and after their educational careers is an important public policy and research question that needs to be addressed now and in the future (Ysseldyke, Thurlow, Bruininks, Gilman, Deno, McGrew, & Shriner, 1992). Although specially designed national studies focused exclusively on the population of students with disabilities (i.e., National Longitudinal Transition Study of Special Education Students) provide valuable information, such special surveys are expensive, limited in coverage of outcome domains, provide no comparable data on students without disabilities, and are typically fixed duration studies that do not provide information as part of a recurring national information system.

Finally, "given the magnitude of federal and state support for educational programs for students with disabilities, support that reflects the valuing of this population in our society, it is time that this implied value be matched by the commitment of resources to address the numerous political and technical hurdles that must be overcome in order to be able to extract useful and routine information on the educational and quality of life outcomes for individuals with disabilities" (McGrew, Algozzine, Spiegel, Thurlow, & Ysseldyke, 1993, p.11). Although currently it is not possible to produce routine, quality information regarding the educational outcomes of students with disabilities through the secondary analysis of data gathered through large-scale state and national data collection programs, this does not mean this approach should be discarded. Toward the goal of improving the collection and reporting of information from the analysis of large-scale state and national data collection programs, NCEO has made a number of suggestions and recommendations.

Recommendations

1. The most important recommendation is to increase the inclusion and participation of students with disabilities in state and national data collection programs.  This can be done by adopting a number of strategies that are described in greater detail by McGrew, Thurlow, and Spiegel (1993), Ysseldyke, Thurlow, McGrew, and Vanderwood (1994) and Ysseldyke, Thurlow, McGrew, and Shriner (1994).

1. Develop broader and more uniform definitions of sample eligibility across state and/or national data collection programs.
   
   
2. Increase adherence to inclusion guidelines.
   
   
3. Develop state and national assessment sampling frames that are more inclusive (e.g., do not exclude students in ungraded programs or those residing in separate facilities).
   
   
4. Include follow-up studies and special analyses of ineligible students as a standard component of data collection programs in order to accurately estimate the effect of exclusion on important statistical estimates and to allow reentry into the sample when a data collection program is longitudinal.
   
   
5. Increase partial participation in data collection programs, such as data collection components that do not require direct student responding (e.g., record reviews, third-party informant ratings, etc.).
   
   
6. Include students with disabilities during instrument development to identify items, questions, tasks, or procedures that may need to be eliminated or modified in order to allow more students with disabilities to participate.
   
   
7. Develop appropriate assessment modifications, accommodations and alternatives for students with disabilities.  Also, initiate research and development activities that focus on investigating the extent to which modification and/or accommodations can be made to existing instruments used in large-scale data collection programs without destroying the basic psychometric properties of the instruments.

2. Expand recurring state and national educational data collection programs to include other outcome domains besides academic achievement.  Important outcome information in the NCEO conceptual model domains of Personal and Social Adjustment, Responsibility and Independence, Physical Health, Contribution and Citizenship, and Satisfaction would provide a more comprehensive picture of the status of all children.  More importantly, assessments in many of these non-achievement domains would not have to be paper-and-pencil "tests," but could be gathered through other methods such as administrative record reviews and third-party informants (e.g., parent and teacher surveys).
   
   
3. Initiate a dialogue among appropriate state assessment personnel (e.g., state data managers) on the feasibility of using a common set of data gathering and reporting strategies, guidelines, and/or standards that might produce more common or related data elements specific to students with disabilities across state assessment programs.  Cooperative efforts similar to those that produced the Standards for Education Data Collection and Reporting (SEDCAR) (CEDCAR, 1991) might be particularly worthwhile.
   
   
4. Individuals and organizations developing instrumentation for state and national data collection programs should review the instruments and methodology reports of the National Longitudinal Transition Study of Special Education Students (NLTS) for ideas on how to develop new indicators or strategies for measuring important outcomes for all students.
   
   
5. A more uniform and standard disability variable system that parallels the federal special education categories should be used for educationally oriented state and national data collection programs, particularly those sponsored by the U.S. Department of Education.  Such a system should be used not only to identify those individuals with disabilities who are excluded from these data collection programs, but also to identify those individuals who do participate.
   
   
6. A dialogue should be initiated between representatives from the appropriate educational (e.g., NCES, NCEO, OSEP) and noneducational (e.g., NCHS, Census Bureau, etc.) federal groups and agencies to identify means by which uniform disability-related variables could be collected across agencies, particularly for the school-age portion of each data collection program.  The feasibility of using the same special education categorical variable system recommended for the U.S. Department of Education should be examined.  Alternatively, the feasibility of developing "cross-walk" procedures that would allow the different disability information collected by different agencies to be converted to the federal special education categories should be explored.
   
   
7. Much has been written about the significant variability between states in the operationalization of the same federal disability categories, and the variability between professionals when implementing the same operation criteria (Ysseldyke, 1987).  These sources of variability can introduce unknown sources of error into any national statistical estimates that might be reported by different disability categories.  Although accurately describing the disability population as identified by current practice, research is needed to determine whether the classification of individuals included in the samples of state and national data collection programs can be made more uniform.  Research and development activities are needed to explore the advantages and disadvantages of developing a small set of standard personal competency variables (e.g., academic, cognitive, adaptive, social, emotional, physical) that could be used in all data collection programs, the results of which then could be used to describe and operationally classify the sample respondents according to the federal special education categories.
   
   
8. Individuals charged with the design of instruments used in state and national data collection programs should consider including additional variables that would better describe those individuals who are included or excluded.  Possible new variables include a better description of a student's (1) primary means of communication, (2) physical health, (3) mobility, and (4) social behavior.

References

CEDCAR. (1991). Standards for education data collection and reporting. Washington, DC: National Center for Education Statistics, U. S. Department of Education.

DeStefano, L., & Wagner, M. (1991). Outcome assessment in special education: Lessons learned. Champaign, IL: University of Illinois at Urbana-Champaign, SRI International.

Hanford, T., & White, K. (1991, June). Reform by the numbers. The American School Board, 178(6), 14-16.

McGrew, K. S., Algozzine, B., Spiegel, A. N., Thurlow, M. L., & Ysseldyke, J. E. (1993). The identification of people with disabilities in national databases: A failure to communicate. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Spiegel, A. N., Thurlow, M. L., & Kim, D. (1994). Matching information in national data collection programs to a model of school completion outcomes and indicators. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Spiegel, A. N., Thurlow, M. L., Shriner, J. G., & Ysseldyke, J. E. (1994). Secondary analysis of state assessment data: Why we can't say much about students with disabilities. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Spiegel, A. N., Thurlow, M. L., Ysseldyke, J. E., Bruininks, R. H., & Shriner, J. G. (1992). Outcomes for children and youth with disabilities: Secondary analysis of national data collection programs. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Spiegel, A. N., Thurlow, M. L., Ysseldyke, J. E., Bruininks, R. H., Deno, S., & Shriner, J. G. (1991). Secondary data analysis: A review of major conceptual, measurement and technical issues. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

McGrew, K. S., Thurlow, M. L., & Spiegel, A. N. (1993). The exclusion of students with disabilities in national and state data collection programs. Educational Evaluation and Policy Analysis, 15 (3), 339-352.

National Education Goals Panel (1991). Measuring progress toward the national education goals: Potential indicators and measurement strategies. Washington, DC: Author.

National Education Statistics Agenda Committee of the National Forum on Education Statistics (1990). Improving our national education data system: An agenda for action. Washington, DC: Author.

NCEO (1992). State special education outcomes 1991. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Pope, A. M., & Tarlov, A. R. (Eds.) (1991). Disability in America: Toward a national agenda for prevention. Washington, DC: National Academy Press.

Skrtic, T. M. (1991). Behind special education: A critical analysis of professional culture and school organization. Denver: Love Publishing.

Spande, G. E., & Thurlow, M. L. (1994). Matching state goals to a model of school completion outcomes and indicators. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Special Study Panel on Educational Indicators (1991). Education counts: An indicator system to monitor the nation's educational health. Washington, DC: National Center for Education Statistics.

Taeuber, R. C., & Rockwell, R. C. (1982). National social data series: A compendium of brief descriptions. Review of Public Data Use, 10, 23-111.

Vanderwood, M. L., Ysseldyke, J. E., & Thurlow, M. L. (1993). Consensus building: A process for selecting educational outcomes and indicators. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes..

Ysseldyke, J. E. (1987). Do tests help in teaching? The Journal of Child Psychology and Psychiatry, 28, 21-24.

Ysseldyke, J. E., & Thurlow, M. L. (1993). Development of a comprehensive model of educational outcomes and indicators for students with disabilities. Minneapolis: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., Bruininks, R. H., Gilman, C. J., Deno, S. L., McGrew, K. S., & Shriner, J. G. (1992). A conceptual model of educational outcomes for children and youth with disabilities. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Erickson, R. N. (1994a). Educational outcomes and indicators for grade 4. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Erickson, R. N. (1994b). Educational outcomes and indicators for grade 8. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993a). Educational outcomes and indicators for early childhood (Age 3). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993b). Educational outcomes and indicators for early childhood (Age 6). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993c). Educational outcomes and indicators for individuals at the post-school level. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993d). Educational outcomes and indicators for students completing school. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., McGrew, K. S., & Shriner, J. G. (1994). Recommendations for making decisions about the participation of students with disabilities in statewide assessment programs: A report on a working conference to develop guidelines for statewide assessments and students with disabilities. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Thurlow, M. L., McGrew, K. S., & Vanderwood, M. L. (1994). Making decisions about the inclusion of students with disabilities in large scale assessments. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., Vanderwood, M. L., & Reschly, B. (1994). Availability of data on school completion outcomes and indicators. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Top of page