The Effect of a Simplified English Language Dictionary on a Reading Test

LEP Projects Report 1

Published by the National Center on Educational Outcomes

Prepared by Deb Albus, John Bielinski, Martha Thurlow, and Kristin Liu

March 2001

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Albus, D., Bielinski, J., Thurlow, M., & Liu, K. (2001). The effect of a simplified English language dictionary on a reading test (LEP Projects Report 1). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/LEP1.html

Executive Summary

This study was conducted to examine whether using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient (LEP) Hmong students improved test performance. Hmong students were chosen because they are often not literate in their first language due to a lack of educational experiences in Hmong, which was first put into written form in the 1970s.  For these students, bilingual dictionaries are unlikely to be useful. Thus, we studied the possible usefulness of a monolingual English dictionary for these students.  Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 students in the non-LEP group, and 133 students in the Hmong LEP group. The study was conducted using a randomized counter-balanced design, with a control group of non-LEP students and an experimental group of Hmong LEP students.   All students were administered two reading passages with an English dictionary available, and two passages without the dictionary, varying passage order and order of accommodation in both study groups. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure. Results showed that there was not a significant difference in reading comprehension scores for students in either the LEP or non-LEP group under accommodated conditions. However, it was found that intermediate level English proficiency students in the Hmong LEP group who reported using the dictionary in the accommodated condition showed a moderately significant gain. Issues discussed include student dictionary ability, dictionary interactions with test items, test development considerations, and current beliefs about dictionary accommodations and reading assessment.

Overview

A challenge facing states and districts is determining best practices for including Limited English Proficient (LEP) students in their testing systems. One recommended approach is to provide testing accommodations that are designed to reduce language barriers while not changing what a test is supposed to measure. One accommodation that is available in several states is allowing LEP students to use a dictionary during a test. However, there are few studies on the effects of dictionary use on test performance and score validity.

The way in which dictionaries are used as an accommodation on reading tests varies across the country. In a survey of state assessment directors for 1998-99, Rivera, Stansfield, Scialdone, and Sharkey (2000) identified 21 states that allowed the use of bilingual dictionary accommodations and only 3 states that specifically prohibited them. Among the states that allowed bilingual dictionaries, 11 states allowed them on all assessment components and 10 states allowed them on some of the assessment components (Rivera, et al., 2000). According to a survey conducted by the Council of Chief State School Officers (Olson, Bond, & Andrews, 1999), which covered 1997-98 assessments, only two of these states allowed bilingual dictionaries and the third allowed an English language dictionary.

Current views about the appropriateness of dictionary accommodations in testing differ among researchers, educators, and LEP students. For example, there are both research and opinions that support using dictionary accommodations. Researchers have shown that unfamiliar vocabulary may cause difficulty in understanding items on tests (Garcia, 1991) and that using dictionaries can help students’ reading comprehension (Goyette, 1996; Laufer & Hadar, 1997) and equalize skill assessments for LEP students (Rivera & Vincent, 1997). Further, LEP students and English as a Second Language (ESL) and Bilingual Education teachers have requested them in testing situations because dictionaries are an accommodation that students use in classes every day and will most likely use throughout their lives (Bensoussan, 1983; Liu, Spicuzza, Erickson, Thurlow & Ruhland, 1997; Quest, Liu, & Thurlow, 1997).

There is also research and opinion that opposes using dictionary accommodations. It is a common belief among some researchers and test developers that any alteration to the standard administration necessarily alters the validity of the test score. Others argue that dictionaries should not be allowed because they may negatively affect the validity of a test (Rivera & Stansfield, 1998; Spolsky, 1997). Other reasons given against using dictionaries are that students will need more time with dictionaries, that students may over rely on them (Roizen, 1984), and that they have not been found to significantly affect reading comprehension test scores of individuals learning English as a foreign language (Bensoussan, 1983; Nesi & Meara, 1991).

One argument against dictionary use is that it negates the role of specific vocabulary knowledge as an essential component of reading ability (Bensoussan, 1983). However, Bensoussan argues that a student using a dictionary still needs to be able to successfully choose the right meaning of a word based on the context of a passage in order to correctly answer a test question. Contextual clues may not always be readily available in a passage to infer meanings of unknown words (in tests or in everyday reading). Therefore, the availability of a dictionary does not guarantee understanding of unknown words with or without sufficient contextual clues.

According to some researchers, readers need to comprehend a certain percentage of a text to be able to infer meaning of unknown words. Laufer (1997) suggested 95% text comprehension (understanding of 3,000 word families) before reading skills in a reader’s first language will aid reading in the second language, including inferring meanings of words from context. Hirsh and Nation (1992) suggested 98% text comprehension for pleasure reading, requiring readers to have knowledge of approximately 5,000 word families. Although reading language skills may be more developed in the second language of some students than in their first language, adequate text comprehension still requires understanding of a high percentage of the words. Also, the extent of word family knowledge needed in a reader’s “sight” or “automatic” vocabulary suggests that dictionary use during a reading comprehension test may not greatly enhance performance if students’ overall threshold vocabulary is too low. Observed problems with vocabulary thresholds have led some researchers to conclude that students’ problems with comprehension are basically lexical rather than due to lack of reading strategies (Haynes & Baker, 1993). However, a dictionary would not be able to compensate a student with great gaps in vocabulary knowledge, therefore its use may be more beneficial for students whose proficiency is near the level required for comprehending a text.

In some studies conducted with students learning English as a Foreign Language (EFL) in Israel, it was found that dictionaries were more useful for students with a moderate level of English proficiency and some dictionary skills (Bensoussan, 1983; Laufer & Hadar, 1997), and that they did not benefit students with very low or very high proficiency. Therefore, the accommodation may not benefit the lower proficiency students who would need the most help (Shepard, Taylor, & Betebenner, 1998).

The majority of dictionary accommodation reading studies have been conducted in EFL settings, thus it is important to study dictionary use in an English as a Second Language setting, especially where most students are not literate in their first spoken language. Further, because most dictionary studies have used EFL tests, our goal was to seek evidence to either support or refute the use of dictionary accommodations in large-scale tests in the United States.

Researchers who support dictionary accommodations favor the use of bilingual or bilingualized dictionaries (English dictionaries with a native language translation) for both LEP students in the United States and LEP students overseas (Laufer & Hadar, 1997; Rivera & Stansfield, 1998). Laufer and Hadar (1997) found that, in general, monolingual dictionaries were the least useful accommodation on tests of English as a foreign language. However, in the United States where students are in an English as a second language setting, bilingual dictionaries may not always be the best choice, particularly for individual students who may not be literate or may not have received any education in their first language. Despite being classified as LEP, the language that these students read and write the most fluently may, in fact, be English.

The quality and appropriateness of dictionaries (including the quality of their translations), whether bilingual or monolingual, vary greatly. Some bilingual dictionaries only give word for word translations or incomplete meanings, while others give definitions. English monolingual dictionaries, on the other hand, sometimes provide only the most basic definition of a word. If students are allowed to bring their own dictionaries to a test, some may be at a disadvantage because of the specific dictionary they choose. English language dictionaries tend to give more detailed explanation about the function of words in context; however, the number and depth of entries and illustrations are not equal across dictionary versions. There are also differences between regular and simplified English dictionaries. Abedi, Lord and Plummer (1997) studied the impact of simplified English in the test items themselves on the National Assessment of Educational Progress (NAEP), and found that simplified English on tests had a positive effect for all students, not just for those who are LEP. If simplified English has been found to affect students’ performance on tests, it is likely that simplified English in dictionaries could also have an effect.

Goal of the Study

This study was conducted to examine the possible effects of using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient Hmong students.

Research Questions

Four primary research questions were posed for this study.

1.  Do Hmong LEP students provided with a simplified English dictionary perform better than when the dictionary is not provided?

2.  How does use of a simplified English dictionary by Hmong LEP students affect the reliability of test scores?

3.  What are the characteristics of Hmong LEP students whose scores are most affected by the use of a simplified English dictionary accommodation?

4.  Do students (LEP and non-LEP) want to use a simplified English dictionary as a test accommodation?

Method

Participants

Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 regular education students in the non-LEP group and 133 students in the Hmong LEP group. No restrictions were placed on the backgrounds of students in the non-LEP groups other than they not be LEP students or students receiving special education services. Some students in the non-LEP group were from Hmong language backgrounds, but none of them was receiving services for limited English proficiency. Students from both groups were recruited at all three schools; however, only two of the schools provided students from both groups. The third school chose only to allow participation for its Hmong LEP student population. Data on an economic status indicator (receiving free or reduced lunch) were collected for both the Hmong LEP group and the non-LEP group. The students in the two testing groups were comparable.

Schools used similar ESL level designations (1-5), but the specific description of each level varied across sites. The levels indicate the range of students within ESL classes, from beginning to high levels of English proficiency. The number and percent of LEP students by level are presented in Table 1 (1=lowest proficiency level).

Table 1. Number and Percent of LEP Students by ESL Level

Note: Level of English proficiency as assigned by school personnel.

Design

The study was conducted using a randomized counter-balanced design, with a control group of regular education students and an experimental group of Hmong LEP students. Students were administered two reading passages with the English dictionary available, and two passages without the dictionary. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure.

The passages were designed to parallel Minnesota’s Basic Standards Reading Test, which is part of the state graduation exam. However, the items used for this study had not been used or equated with officially administered tests. The passages represent sample passages that had been reworked by an assessment specialist for use in the Minnesota Assessment Project study of bilingual reading test items (Anderson, Liu, Swierzbin, Thurlow, & Bielinski, 2000). In addition, an LEP graduation standards specialist at the Minnesota Department of Children, Families and Learning and a bilingual adult member of the ESL community provided advice on cultural background in the development of the test passages. The two halves of the test were divided into Form A and Form B. The passages were assigned to forms so that Form A and Form B had the same overall difficulty. Table 2 shows the study design, which is a modification of one design presented by Thurlow, McGrew, Tindal, Thompson, Ysseldyke, and Elliott (2000).

Table 2. Study Design

Test Instrument

Form A and Form B each contained two passages. Each passage was 900-1,040 words in length. For each reading passage, the examinee was asked 10 multiple-choice questions that addressed both literal and inferential comprehension. The test used in this study was previously used in a study on bilingual translation accommodations by researchers of the Minnesota Assessment Project. Evaluation of test score reliability indicated that it had the same or higher internal consistency as the actual Minnesota Basic Standards reading test for most of the test groups (Anderson et al., 2000).

A dictionary exercise, developed by research staff with ESL teaching backgrounds, was also administered to determine each student’s proficiency with the simplified English dictionary. The exercise was composed of four questions. Two questions asked students to provide written definitions of words appropriate to context sentences that were provided to them. A third question was aimed specifically at alphabetizing skill. The fourth question was designed to determine students’ ability to use the dictionary for other information about parts of speech. A four-point scoring rubric was developed for raters to evaluate and score the exercise.

Dictionary Accommodation

A simplified English dictionary was used, as opposed to a Hmong Bilingual dictionary because most Hmong background LEP students have limited literacy in their first language. The dictionary chosen for the study was The American Heritage English as a Second Language Dictionary (Houghton Mifflin Company, 1998). This simplified English dictionary was chosen for several reasons. First, the dictionary for the study had to include more difficult academic words specific to the test passages (e.g., cum laude, precocious), and had to provide a range of definitions for words with multiple meanings (e.g., stand, produce). It also had to present the definitions at a simplified level of English for clarity, and yet fit the needs of a range of proficiency levels among LEP students.

Some of the other available ESL dictionaries had more pictorial content, but had fewer of the words that had been identified as potentially problematic in the study passages. Other ESL dictionaries were too basic, providing only one meaning for words with multiple meanings, and often giving meanings that did not match the meaning of words in the test passages. Also, for one of the vocabulary items on the test, another dictionary provided a definition with an example sentence that could have misled students to choose an incorrect answer. The example sentence included extra descriptive information that was contradictory to the correct test response.

Study Procedure and Timing

Students were assembled into either an auditorium or classroom to take the test. The size of the groups ranged from 11 to 58 students per room. Students were first asked to fill out a brief pre-test questionnaire about language background to provide self-ratings of their English and Hmong proficiency in several modalities: speaking, listening, and reading.

Next, students were administered one half of the test; some of the students getting the dictionary accommodation and some not. Students were allowed as much time as they needed to complete each half of the test, having been given a general time limit of two hours. After completing the first half of the test, the student raised his or her hand and a test administrator started the student on the second half. If the student had the dictionary for the first half, dictated by the color of the test cover, the administrator removed it; if the student took the first half without the dictionary, then it was provided to the student on the second half. During test administration, staff recorded start and finish times on the test covers at the end of each half of the test while providing and removing dictionaries for students’ use so that the time taken by students with and without dictionaries could be tracked. Immediately after completion of the whole test, students were given a post-test questionnaire about dictionary use during the test, their opinions on possible usefulness of an English dictionary on a reading test, and other background information on dictionary use and instruction in the classroom. A short dictionary exercise also was given after the post-test survey to determine levels of student ability in using a dictionary.

Students were allowed approximately two-and-one-half hours to complete all materials. This time allotment was determined on the basis of schedule limitations in the schools. Following completion of the test, students either stayed in the same room or simply returned to class, depending on school requests. For example, at one site, the students who completed the test were given other activity sheets to work on while waiting for other students to finish before being released back to class. At other sites, students were allowed to leave after they completed the study materials. It is uncertain whether these varying procedures had any effect on test results.

Results

Proficiency in English and Hmong

On student pre-test questionnaires, the control group and the Hmong LEP group answered a series of questions on language proficiency for speaking, listening, and reading in English and Hmong, and the length of time they had spent in U.S. schools. Most students in the Hmong LEP group reported higher reading ability in English than in Hmong (see Table 3), with most students rating their English reading ability as either “well” or “pretty well.” For speaking and understanding spoken English/ Hmong, the majority of the group described themselves between “Well” and “Pretty well” for English, and between “Very well” and “Pretty well” for Hmong.

Table 3.   Hmong LEP Group Self-Report for English and Hmong Reading and Speaking Ability

In the non-LEP group (see Table 4), as might be expected, the majority reported their level of English proficiency as “Very well” in both modalities. This group did include some students with Hmong language background who were not limited English proficient. For these students, some reported a range of “Well” to “Very well” in understanding and speaking Hmong. However, very few reported reading Hmong above “Not very well.”

Table 4   Non-LEP Group Self-Report for English and Hmong Reading and Speaking Ability

Time in U.S. Schools

Table 5 shows the length of time in U.S. schools for each group. The percentage of Hmong LEP students in U.S. schools 9 years or less (45.4%) was greater than the non-LEP group (11.5%) (X2(3)=24.9, p<.01).

Table 5.   Student Self-Report on Time in U.S.

Reported Dictionary Proficiency

Table 6 shows the result of students’ self report of monolingual English dictionary skills. All but eight students in the study (6.3%) described their dictionary skills as “Good” or above. However, a greater percent of non-LEP students described their skills as “Very Good,” while more LEP students described their skills as “Good.” (X2(3) = 29.7; p<.01)

Table 6.   Self-Report on Using Dictionary for Non-LEP and LEP students

Dictionary Exercise Results

For the dictionary exercise, raters independently rated each item using a four-point rubric, then scores were compared. The agreement between raters for scoring the exercise was 94%. Disagreements were resolved by rater consensus after discussion.

The results of the dictionary exercise were somewhat problematic for identifying specific levels of skill. For example, in choosing a correct definition from words with multiple entries, there was an observed tendency for students to choose the first definition in the dictionary regardless of whether it fit the context of the example sentence. We believe that students’ scores do provide a good estimate of specific dictionary skills, such as alphabetizing, locating a word entry, and choosing the correct definition. Given that the highest possible score on this exercise was 4.0, the mean scores for each group demonstrate at least basic dictionary skills (see Table 7).

Table 7. Dictionary Exercise Mean Scores for Non-LEP and LEP Students

Note: One point was given for each correct answer (total possible 4). Half points were allowed on some items.

We also compared dictionary exercise results to students’ self-ratings of dictionary ability. Table 8 shows the average score on the dictionary exercise with each self-reported dictionary proficiency rating. Generally, the Hmong LEP students’ performance increased with their self-ratings of dictionary proficiency, but the non-LEP students’ performance was nearly the same across each self-rated proficiency level.

Table 8.   Students’ Self-Report on Dictionary Ability by Dictionary Exercise Means

Test Results for Accommodated vs. Non-Accommodated Performance

Initial analyses were conducted to determine whether there were possible effects for form or the order the accommodation was given. This involved looking at the mean performance on each half of the test (20 points possible on each half of the test) and the mean for the entire test for LEP and non-LEP groups by the order of form (i.e., Form A and Form B) and order of administration (accommodation given on first half vs. accommodation given on second half). Table 9 shows that there was no effect for the Form that students received with the accommodation (F(1,198) =.12; p =.73).

Table 9.   Effect of Accommodation by Form

To examine the possibility that the order in which the accommodation was administered made a difference on test performance, we compared test scores for those receiving the accommodation on the first half with the scores of those receiving the accommodation on the second half. The mean score for the LEP group with the accommodation on the second half of the test was 19.4, whereas the mean score with the accommodation on the first half was 18.3. A 2x2 ANOVA was run with Group and Order treated as fixed effects (see Table 10). The effect for the order of accommodation was not significant (F (1,198)=.043; p=.73).

Table 10.   Effect of Accommodation by Order

The means shown in Table 11 indicate that regardless of the order the accommodation was given, students performed better on the first half of the test than on the second half (F(1,200) = 22.1;p<.01).

Table 11.   Student Performance on Both Halves of Test

The main research question in the study was whether the use of a simplified English dictionary would improve test performance, and whether improvement would be greater for the LEP group than for the non-LEP group. A repeated measures ANOVA was run wherein the within group variable had two levels: (1) test score without accommodation, and (2) test score with accommodation. The between group variable was LEP status (LEP vs. non-LEP). The difference between these two groups was significant (F 1,200=49.7; p<.01). The effect for the accommodation was not significant (F 1,200=.15; p=.70), and the interaction also was not significant (F 1,200=.39; p=.54). Table 12 shows the cell means and 95% confidence interval for each condition by group. The non-LEP group performed the same under both conditions whereas the LEP group performed slightly better with the dictionary accommodation.

Table 12. Student Performance With and Without Dictionary Accommodation

Because not every student in either the control or Hmong LEP group self-reported using the dictionary accommodation when it was provided, a second analysis was conducted for only those students who reported that they had used the accommodation. In this analysis we compared the performance with and without the accommodation by self-reported English language proficiency. The analysis indicated that LEP students who self-reported lower English proficiency (“well” to “not well at all”) did not benefit as much from using the dictionary accommodation as LEP students who self-reported higher English proficiency (“pretty well” to “very well”). This high English proficiency group scored an average 1.2 points higher using the accommodation compared to performance without the accommodation; the lower proficiency group performed nearly the same under both conditions (see Table 13). This interaction between proficiency level and condition was significant ( F(1,97)=4.78; p=.03).

Table 13.   Student Performance by Accommodation Conditions and Self-Reported English Reading Ability

A similar analysis was conducted for the non-LEP group using self-reports of dictionary use during the test. It showed that reading score means in this non-LEP group were not affected by the dictionary accommodation (F(1,67) = .61; p=.44).

Affect of Accommodation on Test Reliability

Test score reliability was estimated using Chronbach’s alpha, a measure of internal consistency. This statistic can range from 0.0 (completely unreliable) to 1.0 (perfectly reliable). Achievement tests are designed to obtain a reliability of about .85. We computed the reliability for each half of the test (20 questions each) taken under both accommodated and non-accommodated conditions. Because reliability is affected by test length, we anticipated that the reliability for each half of the test would be slightly lower than .85. Table 14 below shows Chronbach’s reliability coefficient for each half of the test by condition, accommodated and non-accommodated.

Table 14  Chronbach’s Alpha for Test Reliability

Performance on Test’s Vocabulary Items

A closer analysis was conducted on the test results of specific vocabulary items to determine whether performance gains with the dictionary accommodation could be attributed to performance on two questions that required only the definition of a term. There was one vocabulary item on each half of the test so that one half of the students would have encountered each word under accommodated conditions. The item questions were as follows:

Items 8 and 28              The term precocious used in the title means:
Items 5 and 25             As used in this article, the word produce is best defined as:

The dictionary entries available to students (if in the dictionary accommodation condition) are reproduced below (without pronunciation information):

Precocious. adj. Showing mental skills or abilities at an earlier age than is normal: He was a precocious child who learned to read at three. …

Produce. Tr.v. 1. To bring forth (sthg.): Seeds grow up to produce plants. 2. To create (sthg.) by mental or physical effort: produce a painting. 3. To manufacture (sthg.): produce parts for machines. 4. To cause (sthg.) to exist: Industrial growth produced a new kind of business organization. 5. To supervise and finance the public presentation of (a movie, for example): produce a play.-n. (U) Farm products, especially fruits or vegetables: Excuse me, where can I find the produce?

It should be noted that “precocious” was not embedded in any of the text of the passage but was presented somewhat independent of the context in the title, “Precocious preteen is youngest college graduate.” Students would need to have understood its part of speech and rhetorical function in the title and connected that to the overall description of the boy’s qualities in the passage to infer the meaning of the word. The “produce” item, in contrast, occurred with relatively high frequency in the passage, embedded in sentences with numerous descriptors of fruit and vegetables.

To investigate whether the performance on these vocabulary items was enhanced by dictionary use, thus potentially responsible for part of the average gain, we calculated chi-square results of number correct by school-assigned ESL level and accommodation condition with these vocabulary items (see Table 15). Only LEP students who reported using the dictionary for at least a few words were included.

Table 15.    Percentage Passing ‘Precocious’ Item.

 Statistics for table: (X²₍₁₎=2.10; p=.15)

Table 15 shows that for the item asking students to select the correct definition for “precocious,” lower ESL level students did slightly less well on this item with the accommodation available than did those without it. For higher ESL level students, 50% more students got the correct answer with the accommodation (N=16) than without it (N=8). It is uncertain why the lower level group without the accommodation was able to do better than the higher level group (52.2% vs. 43.5%). Even when the correct single meaning dictionary entry was available, some students did not choose the correct answer, regardless of school assigned ESL level.

Table 16 shows that lower ESL level students did slightly better with the dictionary accommodation on the “produce” item than without the accommodation. The higher level students did slightly better without the accommodation than with the accommodation.

Table 16.   Percent Passing ‘Produce’ Item

 Statistics for table: (X²₍₁₎=1.38; p=0.24)

Reported Dictionary Use

Table 17 shows results of post-test self reports from students about their overall dictionary use during the test. Students in the Hmong LEP group reported using the dictionary more than the non-LEP group. For new words, the Hmong LEP group reported more use (73%) than the non-LEP group (46%). However, the percentage of students who used the dictionary to check words they already knew was similar for the Hmong LEP group (12%) and the non-LEP group (7%). The third question about dictionary use was included in case students used the dictionary in a way that was different from the ways we thought it would be used during the test (e.g., looking up part of speech information, pronunciation, etc.). We are not able to say how the students defined “other information” in this question, but this may be a useful area for further study.

Table 17.   Student Self-Report on Dictionary Use During Test

We analyzed self-reports according to which passage(s) students said they used the dictionary. Although the Hmong LEP group reported a larger overall percentage of dictionary use for each passage than the non-LEP group, the two groups showed the same pattern of use for the four passages (see Table 18).

Table 18. Student Self-Report of Using Dictionary by Reading Passages in Test

Most students reported using the dictionary for Passage 2, which contained the question on defining “precocious.” Passage 3 contained the item on defining “produce.” It may be that Passage 2 had more reported dictionary users because the word “precocious” is probably more difficult for students in both groups than is the word “produce” (fruits and vegetables) found in Passage 3. Both groups also reported using the dictionary less for Passage 1 and about equally for Passages 3 and 4.

Use of Dictionary Accommodation During Test as a Function of Reported English Proficiency

Table 19 shows whether LEP students who reported lower English proficiency used the dictionary more or less than students reporting higher English proficiency. Findings showed, as may be expected, that more students who reported higher language proficiency also reported that they did not use the dictionary accommodation. Still, 14.6% of the students in the lower proficiency group reported not using the dictionary at all.

Table 19. Reported English Ability by Reported Dictionary Use During Test

 *No student self-rated “not well at all.”

Which Characteristics Predicted Overall Test Performance?

A multiple regression analysis was conducted to assess which factors may account for differences in the performance of LEP students. Factors included in this analysis were gender, English reading ability self report, years in U.S., ESL level, difference in time on the test half with accommodation and the test half without accommodation, and dictionary exercise scores. After accounting for these variables, only school-assigned ESL level and dictionary exercise score accounted for a significant proportion of the variance in test scores.

We also wanted to determine which characteristics, if any, were associated with the test score gains on accommodated items. A multiple regression analysis was run with the following predictors: gender, reported English reading ability, time in U.S., ESL level, time differences on the test half with accommodation and the test half without accommodation, and dictionary exercise scores. For the LEP students who showed the most gain on accommodated items, the only significant correlation was for time difference between the halves of the test with and without accommodation. LEP students who spent more time on the half of the test with the dictionary accommodation than on the half without the dictionary accommodation performed better under the accommodated condition (r=.257, p=.006). To test whether these students might have “rushed” the half without the dictionary, we compared the mean number of minutes spent on the half without the dictionary for the LEP students showing a gain compared to those who did not show a gain. For greater specificity, we looked at three groups: (1) students who lost at least 3 points on the accommodated half, (2) students whose difference between halves was 2 points or less, and (3) students who gained at least 3 points on the accommodated half (see Table 20). We found no significant difference in the amount of time spent by each group (F 2,124=1.28; p=.28).

Table 20. Test Time Means for Three Groups (Non-Accommodation Condition) in Minutes

We also found that the correlation between testing time and performance, regardless of whether the student was in the dictionary accommodation condition, was not significant. These results suggest that students taking full advantage of the accommodation showed statistically significant performance gains even after controlling for other variables.

Even though the dictionary exercise scores did not correlate significantly with LEP student performance, the gain that correlated with more time spent during the accommodation condition may actually have been due to increased time with dictionary use.

Analysis of Testing Time

We found that both groups (LEP and non-LEP) spent more time on the first half of the test overall, regardless of the order in which they received the accommodation. Also, the non-LEP group spent an average of 4 minutes more time on the first half of the test than did the LEP group, whereas the LEP group spent an average of 6 minutes more time on the second half than did the non-LEP group.

Opinion on Dictionary Usefulness

Even though 34.5% of students reporting high English proficiency did not report using the dictionary in the two test groups, most students in the non-LEP group (84.1%) and the Hmong LEP group (95.8%) reported that access to a dictionary would be helpful on a reading test.

Dictionary Type, Usage, and Instruction

The two groups also responded similarly to questions about instruction in dictionary use and classroom usage (see Table 21). A majority in both groups had been instructed in the last year on how to use a dictionary (non-LEP, 66.7%; Hmong LEP, 65.3%), and had used them in classes (non-LEP, 79.7%; Hmong LEP, 87.3%). The type of dictionary used in classes for both groups was a monolingual English dictionary (non-LEP, 92.8%; Hmong LEP, 90.7%). The fact that there was a lower reported ability level in reading Hmong language than in reading in English helps explain why bilingual dictionaries are not used as often (4.2%) in classes for these students.

Table 21. Type of Dictionary Used in Classes

Discussion

The primary goal of this study was to determine whether providing a monolingual English dictionary would improve the test performance of LEP students. A repeated measures ANOVA indicated that, overall, the effect of the monolingual English dictionary accommodation was not significant. In other words, test performance for LEP students was about the same under both standard and accommodated conditions. Not all students reported using the accommodation when it was provided. For those students who reported using the accommodation, LEP students with intermediate self-reported English reading proficiency had a statistically significant test score gain due to the presence of the dictionary accommodation. Hmong students with poor English-reading proficiency did not appear to benefit from the accommodation even when they made use of it. This result is consistent with findings that lower and higher level proficiency students in EFL settings do not benefit from dictionary accommodations on reading tests as much as intermediate level students (Bensoussan, 1983; Laufer & Hadar 1997). It is interesting to note that even though the accommodation did not result in a significant overall gain for LEP students, 95.8% of the LEP group believed that providing an English dictionary would be helpful on a reading test. It is unknown whether this discrepancy is related to the fact that students knew the test used in the study did not actually count. They may have believed that dictionaries would be helpful on a test that actually counted.

Anytime a test is administered under non-standard conditions, there is concern that test score validity may be compromised. We assessed this possibility by examining the correlation between item difficulty for LEP students under standard and accommodated conditions, and item difficulty estimates for the non-LEP group without the accommodation. The correlations were similar. In this study, it would appear that the presence of the accommodation neither enhanced nor compromised the validity of the test scores.

The question of how the added lexical resource of a dictionary may have interacted with test items should also be addressed. For example, the results of the two vocabulary items on the test appeared similar to the dictionary exercise results where an observed tendency was for students to choose the first definition in the dictionary to answer a question. It is possible that if students with this tendency used the dictionary for these items “precocious” would have more correct answers because it had only one entry and the correct answer was therefore listed first, whereas “produce” had several definitions with the correct one listed last. However, there may be more complex reasons for the difference in performance for these two items such as the availability of contextual cues, ability to infer meaning, dictionary skill, item distractors, or too low a vocabulary threshold for efficient inferencing or dictionary use.

Although we do not know whether students used the dictionary for these specific items, the results do show that the availability of a dictionary did not guarantee that students would be able to answer these vocabulary items correctly. Further, we do not know whether the dictionary accommodation was possibly disadvantageous for students on at least one of the items.

While the results of this study are consistent with Bensoussan’s point that using a dictionary on a reading comprehension test does not give away answers (even on some vocabulary items), the validity of items still depends on how the item choices interact with dictionary entries and the context of the test passage. For vocabulary items, if distractors include options that are not other meanings or plausible meanings of words, or if one of the options closely matches a dictionary’s entry (as was the case with “precocious” in our study), an item is potentially testing only basic dictionary skill and visual word matching. In contrast, if vocabulary item choices included multiple meanings of a word (i.e., our “produce” item), then the student would need to use more sophisticated reading and dictionary skills to determine the correct answer. It is important that test items are written to appropriately narrow the reading abilities being tested, and that where a dictionary accommodation is being considered, this be taken into account in the development of the reading test items. For reading comprehension items, it may be similarly argued that dictionaries do not “give away” answers, because students still need to make use of vocabulary knowledge using their reading strategies or strategic competence. For example, looking up the word “fact” or “opinion” in the wording of a comprehension question is not going to give students the correct answer requiring them to identify an opinion from a passage (i.e., Which of the following is an opinion, not fact, expressed in the article?)

The results of this study are, to some extent, confined to tests similar to the one used in our study. It was a basic skills test that used newspaper-like passages and asked questions about them. Context was important in providing cues for students to answer questions. Dependence on context reflects an overall trend in performance–based testing, as well as in second language reading and vocabulary testing. When a test is designed to test specific vocabulary where context is not needed to choose the correct definition, then access to a dictionary could very well invalidate students’ test scores. Despite criticism that discrete point vocabulary items stress knowledge of words without having to relate them to the context in which they are used (Read, 2000), these types of items do still appear in tests. When that is the case, allowing the use of a dictionary accommodation probably is not appropriate.

An accommodation option that is similar to dictionaries is word glossaries, which may provide a way to control for the varied definitions across different types and skill levels of dictionaries. Choosing items for glossaries may prove difficult for students with certain backgrounds. For example, Hmong students may need some Latin based words glossed that a Spanish language background student may not, and some Latin based words in English may be false cognates for the Spanish student. Also, there may be conceptual differences between language groups in general background knowledge and experiences that would need to be considered in developing glossaries. Earlier research (Brutten, 1981) showed that experienced teachers could predict to some extent words that would be problematic for students in a reading text. This suggests that glossaries might need to be individualized. There would still need to be consensus on how glossaries or dictionaries fit into the definition and demonstration of vocabulary knowledge and reading abilities for assessment purposes.

An issue raised by some researchers is that students may over-rely on dictionaries in testing situations. In this study, we did not collect information on how students used the dictionary on specific items or their attitudes toward taking the test with the dictionary accommodation related to test preparation, so it is unclear whether some students may have over-relied on the dictionary. Generally, over-reliance on the dictionary during the test was not observed by staff administering the test or in the students’ own self-reports, which showed that most LEP students used the dictionary for “few” (37.8%) or “some” of the words (32.3%). Only two (1.6%) of the LEP students reported using the dictionary for all the words, and 22% of the LEP students reported not using the dictionary at all. Of course, students did realize that the test used in the study did not actually count; this may have affected the extent to which they used the dictionaries.

Although most studies on dictionary accommodations have recommended bilingual or bilingualized dictionaries, this option was not appropriate for this student population because even though they spoke Hmong as their first language, most of them read and wrote more fluently in English. There are other populations with similar characteristics. The choice of an English only dictionary fits these populations as suggested by the survey results of the LEP group, which showed that most used English dictionaries in classes (90.7%). Of course, were students literate in their first language they would possibly benefit more from a high quality bilingual Hmong/English dictionary. However, there are many languages for which bilingual dictionaries are not available. Furthermore, the cost effectiveness of offering bilingual dictionaries for every language group is a problematic ideal not uncommon to similar accommodations such as translated tests or test instructions.

A factor in this study that may have contributed to the general outcome was test time constraints. To work around regular schedules across sites, two methods for managing the return of students to classrooms were used. These could have affected the length of time taken to finish the tests or influenced the outcome of spending significantly less time on the second half of the test. At one school, the policy was that students would remain in the testing room until all students were finished. At the two other schools, each student was allowed to leave after he or she completed the testing materials. Further, other student factors such as time management skills during test-taking, peer pressure to finish, the test not having as much real world significance as the real Basic Standards Test administration, or students wanting to return to friends in other classes could have influenced the overall outcome. It is difficult to determine what possible effect, if any, these student release policies may have had on test results given.

It may also be a consideration that in studies of this type dictionaries and passages are inadvertently chosen that are more appropriate for the middle to upper intermediate level students, which is consistent with the nature of the tests used in standards-based assessments. If the level of reading text and dictionary are set at an easier ability level, the lower intermediate level students may benefit more, and the higher intermediate level would benefit less. This is similar to the advanced student not benefiting from the dictionary in this study. It is also likely that a general vocabulary threshold prevents lower level ability students from using a basic level dictionary with meaningful benefit.

These challenges of choosing an appropriate dictionary and writing items that take into consideration the possibility that a dictionary accommodation may be made available help to highlight some of the issues involved in allowing dictionaries on reading tests. Some researchers and test developers may still argue that dictionaries give away answers, or that reading comprehension includes vocabulary knowledge, making the difficulty added by the vocabulary load of a passage appropriate. Others may argue that specific vocabulary knowledge is less critical to demonstrating reading for comprehension, and that strategic competence (how students use their knowledge) should also be assessed in reading tests (Bachman, 1996). Of these, some may argue that reading skill should include the abilities needed to effectively use a dictionary. Those choosing this option should be prepared to make decisions about different kinds of dictionaries, varying levels of student language and dictionary proficiency, and the specific interactions between test items, test passages, and dictionary entries.

Recommendations

While the simplified English dictionary accommodation may be helpful for intermediate level LEP students who have some dictionary skills, as shown in this study, students whose English proficiency falls within lower and higher ability levels would not be expected to benefit as much from the accommodation. This finding, perhaps more than any other, highlights the need for decisions about accommodations to be made on an individual basis- not for students as a group, but for students who need a specific accommodation. The current research also leads to several recommendations.

•    It is important for test administrators and developers to consider whether dictionaries will be made available, how the availability of the dictionary may interact with the test, and what specific language skills are to be tested when writing test items. It is unlikely that new dictionaries would be chosen each time a test is administered, or that tests would be developed to align with dictionary definitions. On the other hand, one of the dictionary skills that students eventually gain is determining whether the information matches their needs. Students experiencing exposure to different dictionaries not only choose the information they use, but also acquire the added skills needed to take the test. It is questionable whether these added skills increase the difficulty more than the dictionaries reduce barriers created by limited English proficiency.

•   Questions about what specific skills are included within the term “reading skill” need to be resolved in developing assessments with a dictionary or glossary accommodation.

•    If dictionaries are used, it is important for administrators to consider the quality and appropriateness of a dictionary chosen for varying student populations. It is most likely not feasible to provide all biliterate students with dictionaries for each language represented in an LEP population. English glossaries may appear to be more practical from a test development viewpoint and cost effectiveness. This option may still raise questions about how one defines reading skills and insuring test validity.

•    The process of selecting a dictionary for this study revealed that test administrators should ensure that the level of vocabulary in test passages and items is compatible with dictionary entries, assuming test administrators are controlling for dictionary quality. The process also revealed that certain dictionaries could actually put students at a disadvantage unless there was careful consideration of the match between test and dictionary.

 References

Abedi, J., Lord, C., & Plummer, J. (1997, August). Final report of language background as a variable in NAEP mathematics performance (CSE Technical Report 429). Los Angeles, CA: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation. Retrieved from the World Wide Web: http://www.cse.ucla.edu/CRESST/Reports/RECH429.pdf

Anderson, M., Liu, K., Swierzbin, B., Thurlow, M., & Bielinski, J. (2000). Bilingual accommodations for limited English proficient students on statewide reading tests: Phase 2 (Minnesota Assessment Report 31). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Bachman, L., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.

Bensoussan, M. (1983). Dictionaries and tests of EFL reading comprehension. English Language Teaching Journal, 37 (4), 341-396.

Brutten, S. (1981). An analysis of student and teacher indications of vocabulary difficulty. RELC Journal, 12 (1), 66-71. (Referred to in I.S.P. Nation, 1990, Teaching and learning; New York: Newbury House Publishers.)

Garcia, G. E. (1991). Factors influencing the English reading test performance of Spanish-speaking Hispanic children. Reading Research Quarterly, 26 (4), 371-392.

Goyette, S. (1996). The effects of dictionary usage on text comprehension. Dissertation Abstracts International Section A: Humanities and Social Sciences, 1997 April, Vol. 57 (10-A), 4264.

Haynes, M., & Baker, I. (1993). American and Chinese readers learning from lexical familiarization in English text. In T. Huckin, M. Haynes, & J. Coady (Eds.), Second language reading and vocabulary learning (pp. 130-152). Norwood, NJ: Ablex.

Hirsch, D., & Nation, P. (1992). What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a foreign language, 8 (2), 689-696.

Laufer, B. (1997). The lexical plight in second language reading. In J. Coady, & T. Huckin (Eds.), Vocabulary acquisition. Cambridge: Cambridge University Press.

Laufer, B., & Hadar, L. (1997). Assessing the effectiveness of monolingual, bilingual, and “bilingualized” dictionaries in the comprehension and production of new words. The Modern Language Journal, 81, 189-196.

Liu, K., Spicuzza, R., & Erickson, R. (1996). Focus group input on students with limited English proficiency and Minnesota’s Basic Standards Tests (Minnesota Report 4). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Nesi, H., & Meara, P. (1991). How using dictionaries affects performance in multiple-choice EFL tests. Reading in a Foreign Language, 8 (1), 631-643.

Olson, J., Bond, L., & Andrews, C. (1999). Data from the annual survey: State student assessment programs. Washington, DC: Council of Chief State School Officers.

Quest, C., Liu, K., & Thurlow, M. (1997). Cambodian, Hmong, Lao, Spanish-speaking, and Vietnamese parents and students speak out on Minnesota’s Basic Standards Tests (Minnesota Report 12). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Read, J. (2000). Assessing vocabulary. Cambridge: Cambridge University Press.

Rivera, C., & Stansfield, C. (1998). Leveling the playing field of English language learners: Increasing participation in state and local assessments through accommodations. In R. Brandt, (Ed.), Assessing student’s learning: New rules, new realities (pp. 65-92). Arlington, VA: Educational Research Service.

Rivera, C., Stansfield, C. W., Scialdone, L., & Sharkey, M. (2000). An analysis of state policies for the inclusion and accommodation of English language learners in state assessment programs, 1998-1999. Arlington, VA: George Washington University, Center on Equity and Excellence in Education.

Rivera, C., Vincent, C., Hafner, A., & LaCelle-Peterson, M. (1997). Statewide assessment practices for the inclusion of limited English proficient students. (ERIC Document Reproduction Service Number 421 484).

Roizen, M.A. Test performance vis a vis test-taking strategies in reading comprehension of English as a second language. (ERIC Document Reproduction Service Number 252 053)

Scialdone, L. (2000). State policies for the inclusion and accommodation of ELLs 1998-1999. Arlington, VA: George Washington University, Center on Equity and Excellence in Education.

Shepard, L., Taylor, G., & Betebenner, D. (1998). Inclusion of limited English-proficient students in Rhode Island’s grade 4 mathematics performance assessment (CSE Technical Report 486). Los Angeles, CA: University of California, Los Angeles, Center for the Study of Evaluation. Retrieved from the World Wide Web: http://www.cse.ucla.edu/CRESST/Reports/TECH486.pdf

Spolsky, B. (1997). Foreign language dictionaries and examinations. Unpublished manuscript.

The American Heritage English as a Second Language Dictionary (1998). Boston: Houghton Mifflin Company.

Thurlow, M. L., McGrew, K. S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., & Elliott, J. L. (2000). Assessment accommodations research: Considerations for design and analysis (Technical Report 26). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Appendix A

Pre-test Questions from Questionnaire (bubble) Form

1. How well do you understand spoken Hmong?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

2. How well do you speak Hmong?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

3. How well do you read Hmong?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

4. How well do you understand spoken English?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

5. How well do you speak English?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

6. How well do you read English?

A. Very well           B. Pretty well      C. Well             D. Not very well             E. Not well at all

7. Do you read or write a language other than Hmong or English?

A. Yes              B. No
If yes, what languages?_______________________

8. How many years have you lived in the United States? (choose one.)

A. Less than 1               B. 1-3 years                C. 4-6 years
D. 7-9 years                 E. More than 9

Appendix B

Post-test Questions from Questionnaire Form

1. How good are you at using an English-only dictionary?

A. Very good       B. Pretty good    C. Good               D. Not very good    E. Not good at all

2. How often did you use the dictionary to look up words you didn’t know during the tests? (choose one.)

A. I didn’t use it.                      B. For very few words             C. For some of the words
D. For most of the words             E. For all the words

3. Did you use the dictionary for “the first champions of the little league?”

A. Yes              B. No

4. Did you use the dictionary for “precocious preteen is youngest college graduate”?

A. Yes                         B. No

5. Did you use the dictionary for “stand by your stand”?

A. Yes              B. No

6. Did you use the dictionary for “young historians take projects to granddaddy of museums”?

A. Yes              B. No

7. Did you open the dictionary and look inside?

A. Yes              B. No

8. Did you look up the meanings of new words using the dictionary?

A. Yes              B. No

9. Did you look at other information about words using the dictionary?

A. Yes              B. No

10. Did you check the meanings of words you knew before using the dictionary?

A. Yes              B. No

11. Would it help you to use an English-only dictionary when you take a reading test?

A. Yes              B. No

12. Do you use a dictionary in any of your school classes?

A. Yes              B. No

13. In the last year, has your teacher shown you how to use an English-only dictionary?

A. Yes              B. No

14. If you do use a dictionary in any of your classes, what kind of dictionary do you use the most? (Choose one.)

A. English-only             B. Bilingual                         C. Hmong-only             D. Other

Appendix C

Post-test Dictionary Exercise

Dictionary Use Check

Use your dictionary to find the answers to these questions.

The underlined words in the sentences below have more than one meaning. Use your dictionary to find the correct meaning for each sentence.

1. My mother won’t let me watch television on Saturday mornings because she says I get too immersed in the cartoons.

Write the meaning of immersed:

2. The bank gives a premium to customers who open a savings account.

Write the meaning of premium:

3. What word comes after “stale” in the dictionary?

4. What part of speech is the word “haphazard?” (circle one answer)

a. noun             b. verb                 c. adjective             d. adverb  e. other

  Top of page