Examining Differential Distractor Functioning in Reading Assessments for Students with DisabilitiesJamal Abedi Seth Leon & Jenny C. Kao January 2007 All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as: Abedi, J., Leon, S., & Kao, J. (2007). Examining differential distractor functioning in reading assessments for students with disabilities. Minneapolis, MN: University of Minnesota, Partnership for Accessible Reading Assessment. AbstractThis study examines the incorrect response choices, or distractors, of students with disabilities in standardized reading assessments. Differential distractor functioning (DDF) analysis differs from differential item functioning (DIF) analysis, which treats all answers alike and examines all wrong answers against the correct answer. DDF analysis in contrast examines only the incorrect answers. If different groups, such as students with disabilities and students without disabilities, preferred different incorrect responses to an item, then the item could mean something different to the different groups. Our study results found items showing DDF for students with disabilities in grade 9, but not for grade 3. Results also suggest that items showing DDF were more likely to be located in the second half of the assessments rather than the first half of the assessments. Additionally, results suggest that in items showing DDF, students with disabilities were less likely to choose the most common distractor than their non-disabled peers. Results of this study can shed light on potential factors affecting the accessibility of reading assessments for students with disabilities, in order to provide assessment tools that are conceptually and psychometrically sound for all students. A companion report is available examining differential item functioning for students with disabilities. IntroductionThe reauthorization of the Individuals with Disabilities Education Act (IDEA) has heightened demands for equity and accountability in education for the approximately 6.5 million children and youth with disabilities in the United States (U.S. Department of Education, 2004). In the 2003-2004 school year, almost half of all students with disabilities were in regular classrooms for 80 percent or more of the school day (U.S. Department of Education, 2005). Furthermore, since the inception of the No Child Left Behind Act of 2001 (NCLB, 2002), more students with disabilities participate in assessments than in the past, and states are required to report the achievement of students with disabilities as a separate subgroup. In a review of state practices, Klein, Wiley, and Thurlow (2006) found that 44 states reported participation and performance for students with disabilities on all of their NCLB assessments. Nearly 84% of middle school students with Individualized Education Programs (IEPs) participated in general reading assessments, as reported by states in 2002-2003 Annual Performance Reports (Thurlow, Moen, & Wiley, 2005). Students with disabilities traditionally perform at substantially lower levels than students with no apparent disabilities (Abedi, Leon, & Mirocha, 2003; Klein, et al., 2006; Ysseldyke et al., 1998). While their lower performance may be partially attributed to their specific disability, other factors may potentially interfere with their performance, such as the lack of opportunity to learn or lack of appropriate testing accommodations. Specific characteristics of the test itself may also be a reason. Variables unrelated to the construct of an assessment may affect its reliability and validity for students with disabilities. Haladyna and Downing (2004) created a taxonomy of what they found to be construct-irrelevant variance in high-stakes testing. The taxonomy included 21 potential sources of systematic errors associated with construct-irrelevant variance, including factors relating to test development, such as item quality and test item format. Given the high rates of participation of students with disabilities in standardized assessments, it is necessary to have valid and reliable measures of their knowledge and skills with minimal construct-irrelevant variance. Classical theory of measurement is based on the assumption that measurement error has a similar random distribution for all students and no differential subgroup trend is assumed (see Allen & Yen, 1979). However, test bias can occur when performance on a test requires sources of knowledge different from those intended to be measured, causing the test scores to be less valid for a particular group (Penfield & Lam, 2000). Our current study seeks to identify a potential source of construct-irrelevant variance by examining item bias with the eventual hope of creating tests that more accurately reflect the knowledge of students with disabilities. Educational measurement researchers and theorists have examined many different forms of detecting item bias (Matlock-Hetzel, 1997; ONeal, 1991), including different forms of analyses for examining distractors, such as linear polytomous scoring (Crehan & Haladyna, 1994), point-biserial discrimination index (Attali & Fraenkel, 2000), factorial modeling (Wang, 2000), standardization approach (Dorans & Holland, 1992; Dorans, Schmitt, & Bleistein, 1992), and log-linear modeling (Green, Crone, & Folk, 1989; Marshall, 1983). More recent methods of detecting item bias use the framework of differential item functioning (DIF; Penfield & Lam, 2000). DIF analyses have traditionally been used to examine item bias for members of different demographic groups, such as for determining cultural bias in a test item. For example, if a certain group performs lower on average on a specific item, then one could say that the item is biased against that particular group. DIF analyses compare the performance of two groups of the same level of ability in order to disentangle the effects of unfairness and ability level. Consistent differences between the two groups would suggest that DIF is present. However, it must be noted that considering an item as biased would also require determining the non-target constructs that lead to the between-group differences in performance (Penfield & Lam, 2000). Green et al. (1989) extended the concept of DIF to what they termed differential distractor functioning (DDF). DIF analysis treats all answers alike and examines all wrong answers against the correct answer and DDF analysis, in contrast, examines only the incorrect answers. Green et al. argued that if different groups preferred different incorrect responses to an item, often called foils or distractors, then the item could mean something different to the different groups. Although group differences in distractor choices do not affect test scores (because all distractors are wrong), group differences might suggest differential functioning for different subgroups. In their DDF study, Green et al. used log-linear models to examine subgroups while holding ability constant, to ensure that any group differences detected were not due to differences in ability. DDF analysis acknowledges that people of different abilities are expected to pick different wrong answers, but people of different backgrounds may prefer different distractors. Green et al. argued that when a test shows substantial DDF, it is not blind to a particular group, and therefore test scores cannot be interpreted in the same way for the different groups. While there is research examining measurement issues related to distractors in multiple-choice items, research on the role of distractors specifically on the assessment outcomes of students with disabilities is scarce. Assessment outcome studies that examine distractors tend to focus on other subgroups, such as gender. For instance, Marshall (1983) found a significant interaction between gender and choice of distractor in a large majority of items in a grade 6 assessment. Other past research explored differences between students ability and incorrect option choices (Huntley & Welch, 1993; Levine & Drasgow, 1983). However, as aforementioned, DDF analysis already acknowledges that different ability groups will likely pick different incorrect answers. Given the paucity of studies examining distractor choices for students with disabilities, we were interested in exploring whether there is a differential trend of selecting distractors among students with disabilities and students without disabilities, while controlling for ability, using existing data. It is imperative to provide assessment tools that are conceptually and psychometrically sound for all students, particularly those with special needs. Results of this study can provide insight on factors that may affect the performance of students with disabilities, and may open avenues for future studies as part of an effort to improve assessments for all students. Research QuestionsThe following research questions guided the analyses and reporting of this study:
MethodologyData SourceData from a single state provided the impetus for answering the above research questions. The data were obtained for the 1997-1998 academic year from a small state with an average number of students with disabilities, and included item-level information on students responses in the Stanford Achievement Test, Ninth Edition (Stanford 9). Published by Harcourt Brace Educational Measurement in 1996, the Stanford 9 is a standardized, norm-referenced test in several subject areas, including reading. According to the Harcourt Assessment website, the Stanford 9 uses an easy-hard-easy format in which difficult questions are surrounded by easy questions to encourage students to complete the test. The reading portion of the test is characterized by three different types of reading selections: recreational, textual, and functional, and includes items that assess initial understanding, interpretation, critical analysis, and reading strategy (HarcourtAssessment.com). The present study examines two subscales of the Stanford 9, Reading Comprehension (RC) and Word Analysis (WA) (more commonly known as phonics or decoding), from the state (the state is not named to preserve anonymity). The test results from public school students in grades 3 and 9 were analyzed to present data over a wider age range. Students with valid scores were included in our analyses. Students with LEP (limited English proficient) classifications (including LEP students with disabilities) were excluded from the analyses to reduce the possible confounding of language proficiency issues. Of the 6,611 third-grade students included in the present analyses, 448 (6.8%) were considered to be students with disabilities. Of the 5,287 ninth-grade students, 522 (9.9%) were considered to be students with disabilities. Procedure & Statistical DesignMultiple-choice items in two reading subscales (RC & WA) were selected for this study. Each multiple-choice item had four response options consisting of a correct response and three distractors. Analyses were conducted for grade 3 and grade 9 students. We selected two grades apart enough from each other to examine the possible differences between the grade/age of students. To examine the possibility of differential distractor functioning (DDF) across the categories of students with disabilities and students without disabilities, we used a multi-step logistic regression procedure. Item distractors are often designed to draw the attention of students with partial knowledge of the question. Therefore it is important to control for student ability in the construct when attempting to test for DDF. This is especially true in subgroups such as students with disabilities, who often have performance gaps relative to students without disabilities. Only incorrect responses were considered in this analysis, and responses were grouped into two categories. One category represented students who selected the most common distractor, and the other category consisted of students who selected one of the two less common distractors. This indicator of distractor selection was used as the criterion variable. A total score on the applicable subscale (RC or WA) was computed as a proxy for ability on the construct. This score was standardized to provide easier interpretation of odds ratios. In step 1, the ability proxy was entered into the model and a measure of the explained variance (Naeglekerke R-square) was obtained. In step 2, the students with disabilities grouping variable and an interaction between the grouping variable and the ability proxy were entered into the model. Again the R-square estimate was obtained. The change in R-square between step 1 and step 2 was calculated and tested for significance. Analyses were performed for each item separately. Since there were large numbers of items in each content area, the adjustment of type I error rate (a) for multiple analyses was not practical. Therefore results emphasize trends in the significance levels rather than focusing on findings for any particular item. Items were identified for closer inspection as having differential distractor functioning if the R-square change was at least 0.003 and was significant at p<0.01. A similar approach was used to determine if item location influences DDF for students with disabilities. Rather than using the total score as a proxy for ability, only the combined score on items from the first half of the assessment was used as an ability proxy (first 27 out of 54 items for RC; first 15 out of 30 items for WA). Using this second type of proxy enables us to examine potential influences of item location. Items that exhibited DDF were examined more closely by looking at the odds ratios of the variables in the final model. If systemic differences in the DDF findings arose between the two approaches they could then be compared. For example, if items showed larger DDF effects on the items from the latter portion of the assessment when the second proxy was used, and if the odds ratios on those items were in a consistent direction, then it would be apparent that item location was influencing DDF. There could be multiple reasons why item location might influence DDF for students with disabilities, including but not limited to, time pressures, fatigue, frustration and motivation. However, determining specific reasons was beyond the scope of this study. ResultsThe analyses examine the following research questions:
Separate model approaches were used to answer research questions 1 and 2 respectively. To answer question 1, student ability was measured as the total score on the applicable subscale (RC or WA). To answer question 2, student ability was measured as the score on items from the first half of the applicable assessment. In each model approach the additional variance explained by the introduction of disability status and the interaction between disability status and student ability was examined to determine if the item in question met the specified thresholds to be considered as indicating DDF. Results are described in the following pages by subscale and by grade. Detailed results of the analyses are provided in the Appendix. Reading ComprehensionGrade 3. Table 1 presents a summary of the results of the grade 3 reading comprehension items. With Model 1, in which the total score on the 54-item RC assessment was used as an ability proxy, four items indicated DDF. Three of the four items that showed DDF were from the second half of the assessment. In Model 2 only the score on the first 27 RC items were used to measure student reading ability. If systemic DDF is present for items on the second half of the assessment, we would expect to see more items show DDF using this approach. However, only three items indicated DDF using Model 2. These results suggest that DDF is minimal on the RC assessment in grade 3, and there is little evidence of item location influencing DDF. Table 1. Grade 3 Item-level Reading Comprehension
Note: In Model 1, the total score was used as an ability proxy. In Model 2, the score on the first 27 items was used as an ability proxy. Grade 9. Table 2 presents results for the grade 9 RC subscale that are very different from what was seen with grade 3. With Model 1, in which the total score on the 54-item RC assessment was used as an ability proxy, 10 items indicated DDF. Six of the ten items that showed DDF were from the second half of the assessment. With Model 2, in which the first half of the RC assessment was used as the measure of student reading ability, 13 items exhibited DDF. Using the Model 2 approach, 10 of the 13 items that indicated DDF originated from the second half of the RC assessment. There was substantially more DDF present in grade 9 than in grade 3. Table 2. Grade 9 Item-level Reading Comprehension
Note: In Model 1, the total score was used as an ability proxy. In Model 2, the score on the first 27 items was used as an ability proxy. Table 3 presents the results for items in Model 2 that showed DDF. Results are presented for each step in the logistic regression. Odds ratios are also reported for each variable to determine if the DDF is operating similarly for items from the second half of the RC assessment. Of the 10 items that indicated DDF from the second half of the assessment, seven had a significant main effect for the disability status variable. For each of these seven items the odds ratio for students with disabilities was less than 1.0, indicating that students with disabilities were less likely to choose the most commonly chosen distractor when compared to students without disabilities. For example, on Item 33, while controlling for reading ability on the first 27 items, students with disabilities were about one third as likely (0.35) to select the most commonly selected distractor when compared to students without disabilities. Table 3. Grade 9 Item-level Reading Comprehension Logistic Regression Results for Items Showing DDF with Ability Proxy Based On First 27 Items Score
Note: * denotes significance at p<.05. ** denotes significance at p<.01 Word AnalysisGrade 3. Table 4 presents a summary of the results of the grade 3 Word Analysis items. With Model 1, in which the total score on the 30-item WA assessment was used as an ability proxy, three items indicated DDF. All three items that showed DDF were from the second half of the assessment. In Model 2 only the score on the first 15 WC items were used to measure student reading ability. If systemic DDF is present for items on the second half of the assessment we would expect to see more items showing DDF. Only one item showed DDF using the Model 2 approach. These results are similar to the results on the RC assessment in grade 3 and suggest that DDF is minimal on the WC assessment with little evidence of item location influencing DDF. Table 4. Grade 3 Item-level Word Analysis
Note: In Model 1, the total score was used as an ability proxy. In Model 2, the score on the first 15 items was used as an ability proxy. Grade 9. Table 5 presents results for the grade 9 WA subscale which are again quite different than what was seen with grade 3. With Model 1, in which the total score on the 30-item WA assessment was used as an ability proxy, 12 items indicated DDF. Eight of the twelve items that showed DDF were from the second half of the assessment. With Model 2, in which the first half of the WC assessment was used as a measure of student reading ability, eleven items indicated DDF. Seven of the eleven items that showed DDF were from the second half of the RC assessment. There was substantially more DDF present in grade 9 than in grade 3. Table 5. Grade 9 Item-level Word Analysis
Note: In Model 1, the total score was used as an ability proxy. In Model 2, the score on the first 15 items was used as an ability proxy. Table 6 presents the results for items in Model 2 that showed DDF. Results are presented for each step in the logistic regression. Odds ratios are also reported for each variable to determine if DDF is operating similarly for items from the second half of the RC assessment. Of the seven items that showed DDF from the second half of the assessment, six had a significant main effect for the disability status variable. For each of these six items the odds ratio for students with disabilities was less than 1.0, indicating that students with disabilities were less likely to choose the most commonly chosen distractor when compared to students without disabilities. For example, on Item 20, while controlling for reading ability on the first 15 items, students with disabilities were less than one third as likely (0.31) to select the most commonly selected distractor when compared to students without disabilities. Table 6. Grade 9 Item-level Word Analysis Logistic Regression Results for Items Showing DDF with Ability Proxy Based On First 15 Items Score
Note: * denotes significance at p<.05. ** denotes significance at p<.01 DiscussionThe national achievement trend shows that students with disabilities perform considerably lower than students with no apparent disabilities. While these achievement gaps can be partly explained by the interference of students specific disabilities, other factors may also contribute to the performance gaps. Factors related to both the instruction and assessment of these students play a large role in their achievement. While we believe factors related to instruction and assessment are intricately intertwined, this study focuses on the factors that influence the assessment of students with disabilities. Specifically, this study focuses on the test items themselves, and whether the items have a potential bias against students with disabilities. The present study therefore explored distractor choices amongst students with disabilities using an existing data set. Results of this study can shed light on potential factors affecting the accessibility of reading assessments for students with disabilities, in an effort to provide assessment tools that are conceptually and psychometrically sound for all students. The following research questions guided this study:
To answer these research questions, student responses on multiple-choice items were compared across the disability status categories in two reading subscales of the Stanford 9, Reading Comprehension and Word Analysis, in two grade levels (3 and 9) from public schools in an entire state. Each multiple-choice item consisted of four response options (one correct and three distractors). Item distractors are often designed to draw the attention of students with partial knowledge of the question. Therefore it is important to control for student ability in the construct when attempting to examine differential distractor functioning (DDF). DDF assumes that people of different ability already naturally choose different wrong answers (Green et al., 1989). It is when substantial DDF is shown for a particular group that there is cause for concern. Our present study results suggest that a substantial number of items exhibit DDF for students with disabilities in grade 9. Results also suggest that items showing DDF were more likely to be located in the second half of the assessments rather than the first half of the assessments. Results also suggest that DDF was present for grade 9 test items, but not for grade 3 test items. Even when controlling for ability using only the items in the first half of the assessments, more grade 9 items exhibited DDF than grade 3 items. For items showing DDF, odds ratios for students with disabilities were less than 1.0, suggesting that students with disabilities were less likely to choose the most common distractor as compared to their peers without disabilities. Students with disabilities might be randomly selecting one of the four response options rather than making an educated guess. Our concurrent study that employs DIF analyses sheds additional light on differential response patterns for students with disabilities, and is available in a companion report. Findings from our concurrent DIF study are consistent with the DDF analyses, in that students with disabilities were shown to perform more poorly on items located in the second half of the assessment, even while controlling for their performance on the first half of the assessment (see the companion report for more details, Abedi, Leon, & Kao [2006]). The findings of this study have multiple implications. First we might speculate why items located in the second half of assessments showed more DDF than items located in the first half. It could be that students with disabilities require more time than is allowed to complete the tests, or that they became fatigued or frustrated at a certain point in the test. Since responses to these items appear to have been selected more randomly, it could be that many students with disabilities did not have the time or energy to thoroughly read or give the items much thought. Or it could be that they reached a certain cognitive overload. More research is necessary to determine the actual cause or causes, possibly with qualitative research. We might also speculate why DDF was present for grade 9 items but not for grade 3 items. This might be attributed to the content and construct of the test, or to the students themselves. Again, more research is needed. This study has several major limitations. For example, it does not differentiate between different categories of disabilities. Student performance across different categories of disabilities may be quite different and these factors may affect their performance quite differently. Given the heterogeneity of students with disabilities, it is not ideal to group them together into one category, and additional insight could be gathered from analyzing data by specific disability groups. This study was also limited in terms of scope. We did not have access to the specific types of testing accommodations that these students may have received, such as extended time. Also, without access to the actual test items, we were unable to make any conclusive statements regarding the content of the tests, especially with respect to the differences across the two grade levels. Further investigation is required for future studies. Nevertheless, findings of this study provide evidence that other factors related to the assessments may contribute to the performance gap between students with disabilities and their peers without disabilities. Controlling for factors that are not related to the content being assessed may help test developers provide more accessible and more valid assessments for students with disabilities. ReferencesAbedi, J., Leon, S., & Kao, J. C. (2006). Examining differential item functioning in reading assessments for students with disabilities (PARA report). Minneapolis, MN: University of Minnesota, Partnership for Accessible Reading Assessment. Abedi, J., Leon, S., & Mirocha, J. (2003). Impact of student language background on content-based performance: Analyses of extant data (CSE Tech. Rep. No. 603). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole. Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: Deficiencies in usage and an alternative. Journal of Educational Measurement, 37(1), 77-86. Crehan, K. D., & Haladyna, T. M. (1994). A comparison of three linear polytomous scoring methods. (ERIC Document Reproduction Services No. ED 377246). Dorans, N. J., & Holland, P. W. (1992). DIF detection and description: Mantel-Haenszel and standardization. (Educational Testing Service Report ETS-RR-92-10). Dorans, N. J., Schmitt, A. P., & Bleistein, C. A. (1992). The standardization approach to assessing comprehensive differential item functioning. Journal of Educational Measurement, 29(4), 309-319. Green, B. F., Crone, C. R., & Folk, V. G. (1989). A method for studying differential distractor functioning. Journal of Educational Measurement, 26(2), 147-160. Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17-27. HarcourtAssessment.com. (n.d.). Stanford Achievement Test series, Ninth edition - Complete battery. Retrieved May 18, 2006, from http://harcourtassessment.com/hai/ProductLongDesc.aspx?ISBN=E132C&Catalog=TPC-USCatalog&Category=AchievementAccountability Huntley, R. M., & Welch, C. (1993, April). Numerical answer options: Logical or random order? Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA. Klein, J. A., Wiley, H. I., & Thurlow, M. L. (2006). Uneven transparency: NCLB tests take precedence in public assessment reporting for students with disabilities (Technical Report 43). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Levine, M. V., & Drasgow, F. (1983). The relationship between incorrect option choice and estimated ability. Educational and Psychological Measurement, 43, 675-685. Marshall, S. P. (1983). Sex differences in mathematical errors: An analysis of distracter choices. Journal for Research in Mathematics Education, 14(5), 325-336. Matlock-Hetzel, S. (1997, January). Basic concepts in item and test analysis. Paper presented at the annual meeting of the Southwest Educational Research Association, Austin, TX. No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002). ONeal, M. R. (1991). A comparison of methods for detecting item bias (Doctoral dissertation, The University of Alabama, 1991). Dissertation Abstracts International, 52(5), 1723A. Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment. Educational Measurement: Issues and Practice, 19(3), 5-15. Thurlow, M. L., Moen, R. E., & Wiley, H. I. (2005). Annual performance reports: 2002-2003 state assessment data. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved May 2, 2006, from http://cehd.umn.edu/NCEO/OnlinePubs/APRsummary2005.pdf U.S. Department of Education. (2004). Digest of education statistics, 2003 (NCES 2005-025). Author: National Center for Education Statistics. U.S. Department of Education. (2005). The condition of education 2005 (NCES 2005-094). Author: National Center for Education Statistics. Ysseldyke, J., Thurlow, M., Langenfeld, K., Nelson, J. R., Teelucksingh, E., & Seyfarth, A. (1998). Educational results for students with disabilities: What do the data tell us? (Technical Report 23). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Wang, W. (2000). Factorial modeling of differential distractor functioning in multiple-choice items. Journal of Applied Measurement, 1(3), 238-256. AcknowledgmentsThe work reported here was supported under a grant from the U.S. Department of Education, Office of Special Education Programs. The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of the Office of Special Education Programs or the U.S. Department of Education. The authors acknowledge the valuable contribution of colleagues in this study. The authors are thankful to Martha Thurlow, Ross Moen, Christopher Johnstone, and other staff at the National Center on Educational Outcomes, and other members of the Partnership for Accessible Reading Assessment for their helpful comments and suggestions. Danna Schacter at CRESST also helped us substantially with the preparation of this manuscript. The authors are also grateful to Eva Baker for her support of this work, and to Joan Herman for her extensive involvement, advice, and support of this work. AppendixDetailed DDF ResultsTable A1. Grade 3 Item-level Reading Comprehension Ability Proxy Based On All 54 Items
Table A2. Grade 3 Item-level Reading Comprehension Ability Proxy Based On First 27 Items
Table A3. Grade 9 Item level Reading Comprehension Ability Proxy Based on All 54 Items
Table A4. Grade 9 Item-Level Reading Comprehension Ability Proxy Based On First 27 Items
Table A5. Grade 3 Item-Level Word Analysis Ability Proxy All 30 Items
Table A6. Grade 3 Item-Level Word Analysis Ability Proxy First 15 Items
Table A7. Grade 9 Item-Level Word Analysis Ability Proxy All 30 Items
Table A8. Grade 9 Item-Level Word Analysis Ability Proxy Based on First 15 Items
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||