The Nation’s
Report Card
U.S. History 2001
NCES 2002–483 May02
[This file contains excerpts of the report. For a complete version see information immediately below]
U.S. Department of Education.
Office of Educational Research and Improvement.
National Center for Education Statistics (NCES).Michael S. Lapp, Wendy S. Grigg, Brenda S.-H. Tay-Lim, in collaboration with Tatyana Petrovicheva, and Satwinder Thind
THE NATION’S REPORT CARD, the National Assessment of Educational Progress (NAEP), is the only nationally representative and continuing assessment of what America’s students know and can do in various subject areas. Since 1969, assessments have been conducted periodically in reading, mathematics, science, writing, history, geography, and other fields. By making objective information on student performance available to policymakers at the national, state, and local levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. Only information related to academic achievement is collected under this program. NAEP guarantees the privacy of individual students and their families.
NAEP is a congressionally mandated project of the National Center for Education Statistics, the U.S. Department of Education. The Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project through competitive awards to qualified organizations. NAEP reports directly to the Commissioner, who is also responsible for providing continuing reviews, including validation studies and solicitation of public comment, on NAEP’s conduct and usefulness.
In 1988, Congress established the National Assessment Governing Board (NAGB) to formulate policy guidelines for NAEP. The Board is responsible for selecting the subject areas to be assessed from among those included in the National Education Goals; for setting appropriate student performance levels; for developing assessment objectives and test specifications through a national consensus approach; for designing the assessment methodology; for developing guidelines for reporting and disseminating NAEP results; for developing standards and procedures for interstate, regional, and national comparisons; for determining the appropriateness of test items and ensuring they are free from bias; and for taking actions to improve the form and use of the National Assessment.
To obtain single copies of this report, limited number of copies available, or ordering information on other U.S. Department of Education products, call toll free 1–877–4ED-PUBS (877–433–7827), or write: Education Publications Center (ED Pubs), U.S. Department of Education, P.O. Box 1398, Jessup, MD 20794–1398, TTY/TDD 1–877–576–7734, FAX 301–470–1244. Online ordering via the Internet: http://www.ed.gov/pubs/edpubs.html Copies also are available in alternate formats upon request. This report also is available on the World Wide Web: http://nces.ed.gov/nationsreportcard
Figure 2.3 Percentage of students within and at or above U.S. history achievement levels, grades 4, 8, and 12: 1994 and 2001
National Achievement-Level Results
 |
 |
 |
* Significantly different from 1994. NOTE: Percentages within each U.S. history achievement level may not add to 100, or to the exact percentages at or above achievement levels, due to rounding. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1994 and 2001 U.S. History Assessments.
Chapter 4 Becoming a More Inclusive
National Assessment
In its efforts to assess a representative sample of all students in the nation, NAEP consistently has striven to include special-needs students—those with disabilities (SD) or limited English proficient students (LEP). A certain percentage of such students, however, has always been excluded because they could not be assessed meaningfully without accommodations. Schools that participate in NAEP have been asked to use specific criteria in making decisions to exclude certain students who have been classified as having a disability under the Individuals with Disabilities Education Act (IDEA), based upon their Individualized Education Programs (IEP) and Section 504 of the Rehabilitation Act of 1973. Similarly, schools have been permitted to exclude some students they identify as being limited English proficient.1
In order to increase the inclusiveness of NAEP’s samples, and in an attempt to remain consistent with state- and district-level testing policies that increasingly offer accommodations to special-needs students, NAEP began to explore the use of accommodations in the 1996 and 1998 assessments. A split-sample design was used to identify a portion of schools that were permitted to provide accommodations to their special-needs students who required them, and a portion of schools in which accommodations were not offered (the standard administration procedure prior to 1996). The split-sample design made it possible to study the effects on NAEP results of including special-needs students who required and were provided accommodations, while at the same time, obtaining results that were comparable to those from previous assessments. Based on research conducted and published since that time, it was determined that NAEP could begin a transition to reporting results that included the performance of accommodated special-needs students.2 It is anticipated that in the near future, NAEP will only report results based on this more inclusive sample.
1 See appendix A for a description of specific criteria provided to assist them in making exclusion decisions.
2 Olson, J. F. & Goldstein, A. A. (1997). The inclusion of students with disabilities and limited-English-proficient students in large-scale assessments: A summary of recent progress. (NCES Publication No. 97–482). Washington, DC: National Center for Education Statistics.
Mazzeo, J., Carlson, J. E., Voelkl, K. E., & Lutkus, A. D. (1999). Increasing the participation of special needs students in NAEP: A report on 1996 research activities. (NCES Publication No. 2000–473). Washington, DC: National Center for Education Statistics.
Two Sets of 2001 NAEP U.S. History Results
This report is the first to display two different sets of NAEP U.S. history results based on the split-sample design: 1) those that reflect the performance of regular and special-needs students when accommodations were not permitted, and 2) those that reflect the performance of regular and special-needs students—both those who were accommodated and those who could be tested without accommodations—when accommodations were permitted. It should be noted that accommodated students make up a small proportion of the total weighted number of students assessed (see table A.6, page 115 in appendix A for details). Making accommodations available may change the overall assessment results in subtle and different ways. For example, when accommodations are permitted, there may be some occurrences of students being accommodated who might have taken the test under standard conditions if accommodations were not permitted. This could lead to an overall increase in the average assessment results if accommodations were to increase special-needs students’ performance. Conversely, when accommodations are permitted, special-needs students who could not have been tested without accommodations could be included in the sample. Assuming that these are generally lower-performing students, their inclusion in the sample—even with accommodations —could result in an overall lower average score.
The two sets of results presented in this chapter were obtained by administering the assessment to a nationally representative sample of students and schools. In one sample, no accommodations were permitted; all students were assessed under the same conditions that were the basis for reporting results from the 1994 NAEP U.S. history assessment. In another part of the schools sampled, accommodations were permitted for students with disabilities and limited English proficient students who normally receive accommodations in their district or state assessment programs. Most accommodations that schools routinely provide for their own testing programs were permitted. The permitted accommodations included, but were not limited to the following:
- one-on-one testing,
- bilingual dictionary,
- large print book,
- small-group testing,
- extended time,
- oral reading of questions, and
- use of an aide for transcribing responses.
(See appendix A, table A.7, page 117, for greater detail on the numbers and percentages of students accommodated by accommodation type in the 2001 assessment.)
Figure 4.1 provides a visual representation of how the two sets of results were based on the two samples in 2001. Included in both sets of results (accommodations not permitted and accommodations permitted) are those students from both samples of schools who were not identified as either SD or LEP. In addition, the first set of results (accommodations not permitted) includes SD and LEP students from the sample of schools where accommodations were not permitted (see middle portion of figure 4.1). This is the set of results that allows for trend comparisons back to 1994 and are presented in the other chapters of this report.
The second set of results, accommodations permitted (see bottom portion of figure 4.1), includes SD and LEP students from the sample of schools where accommodations were permitted. This is the set of results that form the new, more inclusive baseline for future reporting of trend comparisons for the NAEP U.S. history assessment.
In the NAEP 2001 sample where accommodations were not permitted, 16 percent of fourth-graders, 16 percent of eighth-graders, and 11 percent of twelfthgraders, were identified by their schools as having special needs (i.e., either as students with disabilities or limited English proficient students). In the other sample where accommodations were offered, 18 percent of fourth-graders, 17 percent of eighthgraders, and 10 percent of twelfth-graders were identified as having special needs. In the sample where accommodations were not permitted, between 45 and 51 percent of the special-needs students at each of the three grade levels (between 4 and 8 percent of all students—see appendix A, table A.5, page 114) were excluded from NAEP testing by their schools. In the sample where accommodations were offered, between 23 and 33 percent of the specialneeds students were excluded from the assessment (between 2 and 3 percent of the total sample). Thus, offering accommodations would appear to lead to greater inclusion of special-needs students.
Figure 4.1 Split-Sample Design
The two sets of NAEP results based on a split-sample design
Split-sample design The national sample was split. In part of the schools, accommodations were not permitted for students with disabilities (SD) and limited English proficient (LEP) students. In the other schools, accommodations were permitted for SD and LEP students who routinely received them in their school assessments. |
Accommodations-not-permitted results The accommodations-not-permitted results include the performance of students from both samples who were not classified as SD or LEP and the performance of SD and LEP students from the sample in which no accommodations were permitted. |
Accommodations-permitted results The accommodations-permitted results also include the performance of students from both samples who were not classified as SD or LEP; however, the SD and LEP students whose performance is included in this set of results were from the sample in which accommodations were permitted. Since students who required testing accommodations could be assessed and represented in the overall results, it was anticipated that these results would include more special-needs students and reflect a more inclusive sample. |
Chapters 2, 3, 5, and 6 of this report are based on the first set of results (no accommodations permitted). This chapter presents an overview of the second set of results—results that include students who were provided accommodations during the assessment administration. Overall results are provided for the nation and for student subgroups by gender and by race/ethnicity. These results are discussed in terms of statistically significant differences between the two sets of results and differences between subgroups of students within each set of results. Throughout this chapter, the assessment results that include SD and LEP students for whom accommodations were not permitted will be referred to as the “accommodations-not-permitted” results. The set of results that includes SD and LEP students for whom accommodations were permitted will be referred to as the “accommodations- permitted” results.
Results for the Nation
Accommodations Not Permitted and Accommodations Permitted
Table 4.1 displays the average U.S. history scale scores for the nation in 2001 for two sets of results: 1) accommodations not permitted, and 2) accommodations permitted. There were no significant differences in the average scores between the two sets of results at grades 4 and 12. At grade 8, however, the average score when accommodations were permitted was lower than the average score when accommodations were not permitted.
As noted in the introduction to this chapter, NAEP has always sought to include special-needs students proportional to their representation in the U.S. population. Offering accommodations tends to reduce exclusion rates for special-needs students and therefore allows NAEP to offer a fairer and more accurate picture of the status of American education. Because special-needs students are typically classified as eligible for special educational services after having shown some difficulty in the regular learning environment, some may assume that including the performance of these students would tend to lower the overall results. This assumption appears to have been justified only in the observed difference between the two sets of grade 8 U.S. history results in 2001, where the accommodations-permitted results, which included slightly more special-needs students because of the availability of accommodations, were lower than the accommodations-not-permitted results. It is important to examine the percentages of students attaining the NAEP achievement levels, however, to see if there were higher percentages at the lower achievement level (i.e., Basic), when students were assessed with accommodations.
Table 4.1 Comparison of Two Sets of National Scale Score Results
National average U.S. history scale scores by type of results, grades 4, 8, and 12: 2001
Accommodations Accommodations
not permitted permitted
Grade 4 209 208
Grade 8 262 260 *
Grade 12 287 287
* Significantly different from the sample where accommodations were not permitted.
SOURCE: U.S. Department of Education, Office of Educational Research and Improvement,
National Center for Education Statistics, National Assessment of Educational Progress (NAEP),
2001 U.S. History Assessment.
Table 4.2 shows the percentages of students attaining each of the achievement levels. The percentages are similar across the two sets of results for grades 4 and 12; apparent differences between the accommodations-not-permitted and the accommodations-permitted results were not significantly different. At grade 8, however, the percentage of students below Basic was higher when accommodations were permitted than when they were not permitted.
Table 4.2 Comparison of Two Sets of National Achievement-Level Results
Percentage of students within and at or above U.S. history achievement levels by type of results, grades 4, 8, and 12: 2001
BB AB AP AA AAB AAP Grade 4 ANP 33 49 16 2 67 18 AP 34 48 16 2 66 18 Grade 8 ANP 36 48 15 2 64 17 AP 38* 46* 14 1 62* 16 Grade 12 ANP 57 32 10 1 43 11 AP 57 32 10 1 43 11 Key: Below Basic BB At Basic AB At Proficient AP At Advanced AA At or above Basic AAB At or above Proficient AAP Accommodations were not permitted ANP Accommodations were permitted AP * Significantly different from the sample where accommodations were not permitted. NOTE: Percentages within each U.S. history achievement-level range may not add to 100, or to the exact percentages at or above achievement levels, due to rounding. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.
National Results by Gender
Accommodations Not Permitted and Accommodations Permitted
The average U.S. history scale scores by gender for both sets of results in 2001 are provided in table 4.3. Both male and female students at grade 8 had higher U.S. history scores when accommodations were not permitted than when accommodations were permitted.
At all three grades, the average scores for male students were not significantly different from that of female students regardless of whether or not accommodations were permitted.
Table 4.3 Comparison of Two Sets of National Scale Score Results by Gender
National average U.S. history scale scores by gender and type of results, grades 4, 8, and 12: 2001
Male Female Grade 4 ANP 209 209 AP 207 209 Grade 8 ANP 264 261 AP 261* 260* Grade 12 ANP 288 286 AP 288 286 Key: Accommodations were not permitted ANP Accommodations were permitted AP * Significantly different from the sample where accommodations were not permitted. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.
The percentages of male and female students attaining the Basic, Proficient, and Advanced levels are provided in table 4.4. Comparing the two sets of results in 2001, there were no statistically significant differences in the percentages of male or female students attaining each of the achievement levels at grades 4 or 12. At grade 8, however, a higher percentage of male students were below Basic when accommodations were permitted than when they were not.
Table 4.4 Comparison of Two Sets of National Achievement-Level Results by Gender
Percentage of students within and at or above U.S. history achievement levels by gender and type of results, grades 4, 8, and 12: 2001
BB AB AP AA AAB AAP Grade 4 Male ANP 34 47 17 2 66 19 AP 35 46 16 2 65 19 Female ANP 32 51 15 2 68 17 AP 33 50 15 2 67 17 Grade 8 Male ANP 35 47 17 2 65 18 AP 38 45 16 2 62 17 Female ANP 37 48 14 1 63 15 AP 39 47 13 1 61 14 Grade 12 Male ANP 55 33 11 1 45 12 AP 55 32 11 2 45 12 Female ANP 59 31 9 1 41 10 AP 60 31 9 1 40 10 * Significantly different from the sample where accommodations were not permitted. NOTE: Percentages within each U.S. history achievement-level range may not add to 100, or to the exact percentages at or above achievement levels, due to rounding. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.
National Results by Race/Ethnicity
Accommodations Not Permitted and Accommodations Permitted
NAEP assessments across academic subjects have typically reported large score differences according to race and ethnic group membership. If students with disabilities or limited English proficient students are over-represented in a particular racial or ethnic group, that group’s assessment scores may decrease. Table 4.5 provides the average U.S. history scale scores for each of the race/ethnicity categories for the two sets of results in 2001. At grade 8, both White students and Black students had higher average scores when accommodations were not permitted than when accommodations were permitted. There were no statistically significant differences observed between the average scores when accommodations were not permitted and when accommodations were permitted for any of the race/ethnicity categories at grades 4 and 12.
As noted in chapter 3, a pattern of performance differences by race/ethnicity can be seen in the accommodations-notpermitted results in 2001. Both White and Asian/Pacific Islander students at all three grades scored higher than Black and Hispanic students. The same pattern can be observed in the accommodations-permitted results. However, while White students outperformed their Asian/Pacific Islander peers at grade 4 when accommodations were not permitted, the difference was not statistically significant when accommodations were permitted.
Table 4.5 Comparison of Two Sets of National Scale Score Results by Race/Ethnicity
National average U.S. history scale scores by race/ethnicity and type of results, grades 4, 8, and 12: 2001
Asian/Pacific American White Black Hispanic Islander Indian Grade 4 ANP 220 188 186 213 197 AP 218 186 187 214 197 Grade 8 ANP 271 243 243 267 249 AP 269* 240* 240 265 248 Grade 12 ANP 292 269 274 295 277 AP 292 268 271 294 274 * Significantly different from the sample where accommodations were not permitted. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.
The percentages of students in each race/ethnicity category who attained the Basic, Proficient, and Advanced levels are provided in table 4.6. No significant differences were found at any of the three grades between the accommodations-not-permitted results and the accommodationspermitted results for the percentages of students attaining each of the achievement levels in 2001.
Table 4.6 Comparison of Two Sets of National Achievement-Level Results by Race/Ethnicity
Percentage of students within and at or above U.S. history achievement levels by race/ethnicity and type of results, grades 4, 8, and 12: 2001 At or above At or above
Below Basic At Basic At Proficient At Advanced Basic Proficient Grade 4 White Accommodations were not permitted 21 55 21 3 79 24
BB AB AP AA AAB AAP Grade 4 White ANP 21 55 21 3 79 24 AP 23 53 21 3 77 24 Black ANP 56 38 5 # 44 6 AP 58 36 5 # 42 5 Hispanic ANP 58 35 6 1 42 7 AP 58 36 6 # 42 6 Asian/Pacific Islander ANP 29 53 16 3 71 19 AP 26 54 17 3 74 20 American Indian ANP 47 41 8 4 53 12 AP 44 44 9 3 56 12 Grade 8 White ANP 25 53 19 2 75 21 AP 27 52 19 2 73 20 Black ANP 62 34 4 # 38 4 AP 65 31 4 # 35 4 Hispanic ANP 60 34 5 # 40 5 AP 63 32 4 # 37 4 Asian/Pacific Islander ANP 32 48 18 2 68 20 AP 34 47 17 2 66 19 American Indian ANP 50 42 7 1 50 8 AP 54 38 7 1 46 8 Grade 12 White ANP 51 36 12 1 49 13 AP 51 36 12 1 49 13 Black ANP 80 18 3 # 20 3 AP 80 17 3 # 20 3 Hispanic ANP 74 21 5 # 26 5 AP 74 21 5 # 26 5 Asian/Pacific Islander ANP 47 31 17 5 53 21 AP 48 31 16 5 52 21 American Indian ANP 66 33 1 0 34 1 AP 68 31 1 0 32 1 # Percentage is between 0.0 and 0.5. NOTE: Percentages within each U.S. history achievement-level range may not add to 100, or to the exact percentages at or above achievement levels, due to rounding. SOURCE: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.
From the Executive Summary
The Setting of Achievement Levels
The 1988 NAEP legislation that created the National Assessment Governing Board directed the Board to identify “appropriate achievement goals…for each subject area” that NAEP measures.4 The 2001 NAEP reauthorization reaffirmed many of the Board’s statutory responsibilities, including developing “appropriate student achievement levels for each grade or age in each subject area to be tested . . . ”5 In order to follow this directive and achieve the mandate of the 1988 statute to “improve the form and use of NAEP results,” NAGB undertook the development of student performance standards called “achievement levels.” Since 1990 the Board has adopted achievement levels in mathematics, reading, U.S. history, geography, science, writing, and civics.
The Board defined three levels for each grade: Basic, Proficient, and Advanced. The Basic level denotes partial mastery of the knowledge and skills that are fundamental for proficient work at a given grade. The Proficient level represents solid academic performance. Students reaching this level demonstrate competency over challenging subject matter. The Advanced level presumes mastery of both the Basic and Proficient levels. Figure 1.2 presents the policy definitions of the achievement levels that apply across all grades and subject areas. The policy definitions guided the development of the U.S. history achievement levels, as well as the achievement levels established in all other subject areas. Adopting three levels of achievement for each grade signals the importance of looking at more than one standard of performance. The Board believes, however, that all students should reach the Proficient level: the Basic level is not the desired goal, but rather represents partial mastery that is a step toward Proficient.
4 National Assessment of Educational Progress Improvement Act of 1988. Pub. L. No. 100-297, 20, U.S.C. 1211.
5 No Child Left Behind Act of 2001: Reauthorization of the Elementary and Secondary Education Act. Pub. L. No. 107-110 (H.R. 1).
Figure 1.2 Policy definitions of the three NAEP achievement levels
Achievement Levels
Basic
This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.Proficient
This level represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.Advanced
This level signifies superior performance.
SOURCE: National Assessment Governing Board. U.S. History Framework for the 1994 and 2001 National Assessment of Educational Progress.
The achievement levels in this report were adopted by the Board based on a standard-setting process designed and conducted under a contract with ACT, Inc. To develop these levels, ACT convened a cross section of educators and interested citizens from across the nation and asked them to judge what students should know and be able to do relative to a body of content reflected in the NAEP framework for U.S. history. This achievement-levelsetting process was reviewed by a variety of individuals including policymakers, representatives of professional organizations, teachers, parents, and other members of the general public. Prior to adopting these levels of student achievement, NAGB engaged a large number of persons to comment on the recommended levels and to review the results.
The results of the achievement-levelsetting process, after NAGB’s approval, became a set of achievement-level descriptions and a set of achievement-level cut points on the 0-500 NAEP U.S. history scale. The cut points are the scores that define the boundaries between below Basic, Basic, Proficient, and Advanced performance at grades 4, 8, and 12. The Board established these U.S. history achievement levels based upon the U.S. history content framework.
Achievement-Level Descriptions for Each Grade
Specific definitions of the Basic, Proficient, and Advanced U.S. history achievement levels for grades 4, 8, and 12 are presented in figures 1.3 through 1.5. As noted previously, the achievement levels are cumulative. Therefore, students performing at the Proficient level also display the competencies associated with the Basic level, and students at the Advanced level also demonstrate the skills and knowledge associated with both the Basic and the Proficient levels. For each achievement level listed in figures 1.3 through 1.5, the scale score that corresponds to the beginning of that level is shown in parentheses. For example, in figure 1.3 the scale score of 243 corresponds to the beginning of the grade 4 Proficient level of achievement.
Figure 1.3 Descriptions of NAEP U.S. history achievement levels for grade 4
Achievement Levels
Basic (195)
Fourth-grade students performing at the Basic level should be able to identify and describe a few of the most familiar people, places, events, ideas, and documents in American history. They should be able to explain the reasons for celebrating most national holidays, have some familiarity with the geography of their own state and the United States, and be able to express in writing a few ideas about a familiar theme in American history.Proficient (243)
Fourth-grade students performing at the Proficient level should be able to identify, describe and comment on the significance of many historical people, places, ideas, events, and documents. They should interpret information from a variety of sources, including texts, maps, pictures, and timelines. They should be able to construct a simple timeline from data. These students should recognize the role of invention and technological change in history. They should also recognize the ways in which geographic and environmental factors have influenced life and work.Advanced ( 276)
Fourth-grade students performing at the Advanced level should have a beginning understanding of the relationship between people, places, ideas, events, and documents. They should know where to look for information, including reference books, maps, local museums, interviews with family and neighbors, and other sources. They should be able to use historical themes to organize and interpret historical topics, and to incorporate insights from beyond the classroom into their understanding of history. These students should understand and explain the role of invention and technological change in history. They should also understand and explain the ways in which geographic and environmental factors have influenced life and work.
SOURCE: National Assessment Governing Board. U.S. History Framework for the 1994 and 2001 National Assessment of Educational Progress.
Figure 1.4 Descriptions of NAEP U.S. history achievement levels for grade 8
Achievement Levels
Basic (252)
Eighth-grade students performing at the Basic level should be able to identify and place in context a range of historical people, places, events, ideas, and documents. They should be able to distinguish between primary and secondary sources. They should have a beginning understanding of the diversity of the American people and the ways in which people from a wide variety of national and cultural heritages have become part of a single nation. Eighthgrade students at the Basic level should also have a beginning understanding of the fundamental political ideas and institutions of American life and their historical origins. They should be able to explain the significance of some major historical events.Proficient (294)
Eighth-grade students performing at the Proficient level should be able to explain the significance of people, places, events, ideas, and documents, and to recognize the connection between people and events within historical contexts. They should understand and be able to explain the opportunities, perspectives and challenges associated with a diverse cultural population. They should incorporate geographic, technological, and other considerations in their understanding of events and should have knowledge of significant political ideas and institutions. They should be able to communicate ideas about historical themes while citing evidence from primary and secondary sources to support their conclusions.Advanced (327)
Eighth-grade students performing at the Advanced level should recognize significant themes and movements in history and begin to understand particular events in light of these themes and movements. They should have an awareness of continuity and change over time and be able to draw relevant analogies between past events and present-day situations. They should be able to frame questions about historical topics and use multiple sources to develop historical generalizations and interpretations. They should be able to explain the importance of historical themes, including some awareness of their political, social, and economic dimensions.
SOURCE: National Assessment Governing Board. U.S. History Framework for the 1994 and 2001 National Assessment of Educational Progress.
Figure 1.5 Descriptions of NAEP U.S. history achievement levels for grade 12
Achievement Levels
Basic (294)
Twelfth-grade students performing at the Basic level should be able to identify the significance of many people, places, events, dates, ideas, and documents in U.S. history. They should also recognize the importance of unity and diversity in the social and cultural history of the United States, and an awareness of American’s changing relationships with the rest of the world. They should have a sense of continuity and change in history and be able to relate relevant experience from the past to their understanding of contemporary issues. They should recognize that history is subject to interpretation and should understand the role of evidence in making an historical argument.Proficient (325)
Twelfth-grade students performing at the Proficient level should understand particular people, places, events, ideas, and documents in historical context, with some awareness of the political, economic, geographic, social, religious, technological, and ideological factors that shape historical settings. They should be able to communicate reasoned interpretations of past events, using historical evidence effectively to support their positions. Their written arguments should reflect some in-depth grasp of issues and refer to both primary and secondary sources.Advanced (355)
Twelfth-grade students achieving at the Advanced level should demonstrate a comprehensive understanding of events and sources of U.S. history. Recognizing that history is subject to interpretation, they should be able to evaluate historical claims critically in light of the evidence. They should understand that important issues and themes have been addressed differently at different times and that America’s political, social, and cultural traditions have changed over time. They should be able to write well-reasoned arguments on complex historical topics and draw upon a wide range of sources to inform their conclusions.
SOURCE: National Assessment Governing Board. U.S. History Framework for the 1994 and 2001 National Assessment of Educational Progress.
The Trial Status of Achievement Levels
The 2001 NAEP reauthorization law requires that the achievement levels be used on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are “reasonable, valid, and informative to the public.”6 Until that determination is made, the law requires the Commissioner and the Board to state clearly the trial status of the achievement levels in all NAEP reports.
In 1993, the first of several congressionally mandated evaluations of the achievement level setting process concluded that the procedures used to set the achievement levels were flawed and that the percentage of students at or above any particular achievement level cutpoint may be underestimated. 7 Others have critiqued these evaluations, asserting that the weight of the empirical evidence does not support such conclusions.8
In response to the evaluations and critiques, NAGB conducted an additional study of the 1992 reading achievement levels before deciding to use those reading achievement levels for reporting 1994 NAEP results.9 When reviewing the findings of this study, the National Academy of Education (NAE) Panel expressed concern about what it saw as a “confirmatory bias” in the study and about the inability of this study to “address the panel’s perception that the levels had been set too high.”10 In 1997, the NAE Panel summarized its concerns with interpreting NAEP results based on the achievement levels as follows:
First, the potential instability of the levels may interfere with the accurate portrayal of trends. Second, the perception that few American students are attaining the higher standards we have set for them may deflect attention to the wrong aspects of education reform. The public has indicated its interest in benchmarking against international standards, yet it is noteworthy that when American students performed very well on a 1991 international reading assessment, these results were discounted because they were contradicted by poor performance against the possibly flawed NAEP reading achievement levels in the following year.11
6 No Child Left Behind Act of 2001: Reauthorization of the Elementary and Secondary Education Act. Pub. L. No. 107-110 (H.R. 1).
7 United States General Accounting Office. (1993). Education achievement standards: NAGB’s approach yields misleading interpretations. U.S. General Accounting Office Report to Congressional Requestors. Washington, DC: Author. National Academy of Education. (1993). Setting performance standards for achievement: A report of the National Academy of Education Panel on the evaluations of the NAEP Trial State Assessment: An evaluation of the 1992 achievement levels. Stanford, CA: Author.
8 Cizek, G. (1993). Reactions to National Academy of Education report. Washington, DC: National Assessment Governing Board. Kane, M. (1993). Comments on the NAE evaluation of the NAGB achievement levels. Washington, DC: National Assessment Governing Board.
9 American College Testing. (1995). NAEP reading revisited: An evaluation of the 1992 achievement level descriptions. Washington, DC: National Assessment Governing Board.
10 National Academy of Education. (1996). Reading achievement levels. In Quality and utility: The 1994 Trial State Assessment in reading. The fourth report of the National Academy of Education Panel on the evaluation of the NAEP Trial State Assessment. Stanford, CA: Author.
11 National Academy of Education. (1997). Assessment in transition: Monitoring the nation’s educational progress (p. 99). Mountain View, CA: Author.
The National Center for Education Statistics and the National Assessment Governing Board have sought and continue to seek new and better ways to set performance standards on NAEP.12 For example, NCES and NAGB jointly sponsored a national conference on standard setting in large-scale assessments, which explored many issues related to standard setting.13 Although new directions were presented and discussed, a proven alternative to the current process has not yet been identified. The Deputy Commissioner of Education Statistics and the Board continue to call on the research community to assist in finding ways to improve standard setting for reporting NAEP results.
The most recent congressionally mandated evaluation conducted by the National Academy of Sciences (NAS) relied on prior studies of achievement levels, rather than carrying out new evaluations, on the grounds that the process has not changed substantially since the initial problems were identified. Instead, the NAS Panel studied the development of the 1996 science achievement levels. The NAS Panel basically concurred with earlier congressionally mandated studies. The Panel concluded that “NAEP’s current achievement level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results.”14
The NAS Panel accepted the continuing use of achievement levels in reporting NAEP results on a developmental basis, until such time as better procedures can be developed. Specifically, the NAS Panel concluded that “....tracking changes in the percentages of students performing at or above those cut scores (or, in fact, any selected cut scores) can be of use in describing changes in student performance over time.”15
The National Assessment Governing Board urges all who are concerned about student performance levels to recognize that the use of these achievement levels is a developing process and is subject to various interpretations. The Board and the Deputy Commissioner believe that the achievement levels are useful for reporting trends in the educational achievement of students in the United States.16 In fact, achievement level results have been used in reports by the President of the United States, the Secretary of Education, state governors, legislators, and members of Congress. Government leaders in the nation and in more than 40 states use these results in their annual reports.
12 Reckase, Mark, D. (2000). The evolution of the NAEP achievement levels setting process: A summary of the research and development efforts conducted by ACT. Iowa City, IA: ACT, Inc.
13 National Assessment Governing Board and National Center for Education Statistics. (1995). Proceedings of the joint conference on standard setting for large-scale assessments of the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES). Washington, DC: Government Printing Office.
14 Pellegrino, J.W., Jones, L.R., & Mitchell, K.J. (Eds.). (1998). Grading the nation’s report card: evaluating NAEP and transforming the assessment of educational progress. Committee on the Evaluation of National Assessments of Educational Progress, National Research Council. (p.182). Washington, DC: National Academy Press.
15 Ibid., page 176.
16 Forsyth, Robert A. (2000). A description of the standard-setting procedures used by three standardized test publishers. In Student performance standards on the National Assessment of Educational Progress: Affirmations and improvements. Washington, DC: National Assessment Governing Board. Nellhaus, Jeffrey M. (2000). States with NAEP-like performance standards. In Student performance standards on the National Assessment of Educational Progress: Affirmations and improvements. Washington, DC: National Assessment Governing Board.
However, based on the congressionally mandated evaluations so far, the Deputy Commissioner agrees with the National Academy’s recommendation that caution needs to be exercised in the use of the current achievement levels. Therefore, the Deputy Commissioner concludes that these achievement levels should continue to be used on a trial basis and should continue to be interpreted with caution.
Interpreting NAEP Results
The average scores and percentages presented in this report are estimates because they are based on samples of students rather than on entire populations. Moreover, the collection of questions used at each grade level is but a sample of the many questions that could have been asked to assess student knowledge of the framework content. As such, the results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in this report are provided in appendix B.
The differences between scale scores and between percentages discussed in the following chapters take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics. Throughout this report, differences between scores or between percentages are pointed out only when they are significant from a statistical perspective. All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term significant is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policymakers, educators, and the public.
Readers are cautioned against interpreting NAEP results in a causal sense. Inferences related to student subgroup performance or to the effectiveness of public and nonpublic schools, for example, should take into consideration the many socioeconomic and educational factors that may also impact on performance in U.S. history.
Overview of the Remaining Report
The results in chapters 2 and 3 of this report are based on the set of data with no accommodations offered to students. Findings are presented for the nation and for all the major reporting subgroups included in all NAEP report cards. Comparisons with results from the 1994 assessment are noted where the data permit.
NAEP has sought to assess samples that are as inclusive as possible. Nevertheless, there has always been some exclusion of students with disabilities (SD) and limited English proficient (LEP) students who could not be assessed meaningfully without accommodations. Local school officials have made decisions about exclusion in accordance with explicit criteria provided by the NAEP program. In order to expand the proportion of students who can be assessed meaningfully, the NAEP program began in recent assessments to explore the use of accommodations with special-needs students. Chapter 4 presents an overview of a second set of results—those that include students who were provided accommodations during the test administration. By including these results in the nation’s U.S. history report card, the NAEP program continues a phased transition toward a more inclusive reporting sample. Future assessment results will be based solely on a student and school sample in which accommodations are permitted.
Chapter 5 provides sample assessment questions and student responses from the 2001 assessment. Also presented in chapter 5 are item maps that position selected question descriptions along the NAEP U.S. history scale where they are likely to be answered successfully by students. The descriptions used on these item maps focus on the U.S. history skill or knowledge needed to answer the question. Chapter 6 examines contexts for learning U.S. history in terms of classroom practices and student variables. The data presented in both chapters 5 and 6 are based on the set of results that did not include accommodated special-needs students.
This report also contains appendices that support or augment the results presented. Appendix A contains an overview of the NAEP U.S. history framework and specifications, information on the national sample, and a more detailed description of the major reporting subgroups featured in chapters 2 and 3. Appendix B contains the full data with standard errors for all tables and figures in this report. Appendix C contains a list of the NAEP U.S. history committee members.
|
If you have come to this page from an outside location click here to get back to mindfully.org |
