Chapter Two

How Is the NAEP Reading Assessment Designed?

Purposes of the Assessment

With the reauthorization of the NAEP program in 1988, 1994, and 2002, Congress mandated that NAEP provide a fair and accurate measurement of academic achievement in reading and other subjects. In this mission, NAEP:

  • informs the public about trends in the reading proficiency of students at state and national levels;

  • provides data for comparing student achievement across states;

  • informs policymakers about relationships between student achievement and key background variables;

  • provides comparative data on the performance of states, regions, and groups.
As the "Nation's Report Card," NAEP provides descriptive information about student reading achievement to the general public, policymakers, and educators. The NAEP reading assessment is not intended to be a model for teaching reading. It measures the outcomes of instruction as reflected in the behaviors of readers.

Methodology

NAEP measures student achievement by taking a stratified random sample of schools throughout the nation and in participating states. NAEP also uses a matrix-sampling design of test items so that no one student takes the entire test.1 The design distributes the large numbers of items across school buildings, districts, and states but limits the number of items that an individual student takes. The assessment is designed to glean information from hundreds of items but restricts the amount of time that any student has to spend responding to the assessment to approximately 50 minutes. Consequently, students taking the assessment will have one of eight or more possible booklets of passages and questions.

Format of the Assessment

Given the broad goals of the NAEP Reading Framework and its emphasis on the use of authentic texts, the assessment is given either in two 25-minute blocks or one 50-minute block. In the fourth grade, only 25-minute blocks are used. Students receive a booklet containing the reading materials and questions. Questions are presented in multiple-choice or constructed-response formats. At least half are constructed-response questions, which allows students to write their answers and explain and support their ideas. There are two types of constructed-response questions: short, requiring a one- or two-sentence answer, and extended, requiring a paragraph or full-page response.

All NAEP questions emphasize critical thinking and reasoning rather than factual recall. Multiple-choice questions require students to choose a single, clear answer. In contrast, constructed-response questions require students to integrate information from the text with their background knowledge, reorganize ideas, and critically consider the text. In an assessment of reading, it is important to have items that can directly and accurately reflect how readers use multiple strategies to build understanding.

Reading Passages

To reach the goal of approximating actual reading experiences, NAEP reading passages are typical of those read by students every day. The passages are taken from authentic texts found in the environments of students in grades 4, 8, and 12. Selected passages are original sources, not simplifications, paraphrases, abridgments, or modernizations. Only minimal changes are permitted in reading passages used in NAEP (for example, substituting an obscure word, revising a confusing phrase, or replacing an unfamiliar popular name). Such changes do not affect overall passage difficulty level, purpose, or meaning. Complete stories, articles, or chapters of textbooks are used to engage students' interest. These passages make it possible to ask questions that elicit various reading strategies.

Passages vary in length. In 4th grade, students read passages of 250 to 800 words; in 8th grade, 400 to 1,000 words; and in 12th grade, 500 to 1,500 words. In grades 8 and 12, students may be given two related reading passages in the 50-minute block.

The selected passages for each grade are developmentally and topically appropriate. In addition, passages are intact and meet criteria for genre, language, interest, and organization. These criteria include curricular considerations, appropriateness of the topic, style, perspective, and fairness. Documents are genuine and relate to tasks that are appropriate for the grade level and experiences of the students being assessed. Exhibit 6 outlines the major criteria for passage selection.

Exhibit 6. Criteria for Passage Selection
  • Developmental appropriateness

  • Topic appropriateness

  • Language appropriateness

  • Fairness

  • Interest level

  • Reproducibility (copyright, format, Internet use)

  • Diversity among authors

    • Item difficulty is a function of the difficulty of the passage, the complexity of the text, and the amount of background knowledge required to respond correctly. Because of their limitations, conventional readability estimates are not the main criteria for determining the difficulty of a passage, which is a function of the complexity of its arguments, the abstractness of its concepts, and the inclusion of unusual points of view and shifting time frames—factors not addressed by traditional readability measures. As the difficulty of the passages increases, so does the difficulty of the questions because the questions focus on important points in the text.

    Passages range in difficulty from those that could be read by the least proficient readers (for example, about second-grade level in a fourth-grade class) to those that could be read by only the most proficient readers (for example, possibly eighth-grade level in a fourth-grade class), as determined by teachers in specific grades.

    Item Development

    Items are based on the aspects of reading and the related strategies readers use to build understanding. In developing the items, NAEP focuses on the range of questions that tap into each aspect of reading as well as the important elements of text. Exhibits 7 through 9 list sample questions that address each aspect of reading and element of text in a given context: reading for literary experience (exhibit 7), reading for information (exhibit 8), and reading to perform a task (exhibit 9). Note that the elements of text are related to the organization and type of text and therefore vary with the context for reading.

    Exhibit 7. Sample NAEP Items, by Element of Literary Text and Aspect of Reading
    Element of Literary Text Aspect of Reading
    Forming a General Understanding Developing Interpretation Making Reader/Text Connections Examining Content and Structure
    Theme What is the moral in the story? Use evidence from the story in your response. How does the setting help to illustrate the theme of the story? Do you think the lesson in this story is true today? Why or why not? Explain what makes this story a fable.
    Major characters What was the major character’s opinion of _____? What causes the main character to do _____? Use evidence from the story in your response. How do you think the character’s actions might be different today? Support your response with evidence from the story. How does the author’s description of _____ help explain the character’s actions?
    Major events Write a short summary of the major events in the story. What happens after _____? How do you think the story would have ended if _____ had not happened? How do the first events help you predict the ending?
    Problem How does _____ make the problem worse? Use evidence from the text to support your response How did _____ help solve the problem? How does the problem in the story compare with another story you have read? Include evidence from the text and another story. Why does the author explain the problem in the first part of the story? Explain with evidence from the story.
    Vocabulary Which words describe what the story is mostly about? Use evidence from the text to support your response. Which words let you know that time has gone by? Explain with evidence from the story. Explain the double meaning of _____. Tell which meaning better explains the major ideas in the passage. Why does the author use the words _____ to describe how _____ feels?

    Exhibit 8. Sample NAEP Items, by Element of Informational Text and Aspect of Reading
    Element of Informational Text Aspect of Reading
    Forming a General Understanding Developing Interpretation Making Reader/Text Connections Examining Content and Structure
    Central purpose What might be the author’s message in this article? How does the author support the message? Do you agree with the author’s message? Give evidence from the text. Based on what you read, what might be the reason the author wrote this?
    Major ideas Give a summary of the major ideas. How does the big idea in the first section relate to the big idea in the last section? Who might need or want this information? Use details from the text in your answer. What did the author do to present information clearly?
    Supporting ideas Identify ideas that most closely relate to the topic. Give evidence from the text to support your choice. How does the author show you that the main idea is important? Which details about the _____ help you to have a clear image of the topic? Explain why you chose them. What information did the author have to know before writing the article?
    Adjunct aids The chart in this article is mostly used to _____? How does the information in the chart support the information in the article? Why did the author include the picture with the chart? Explain using what you know and information from the text. What is the significance of the map to the article? Explain.
    Vocabulary Which words describe what the passage is mostly about? Use the evidence from the text to support your choice. Which words do you think mean the same as the title? Tell why you think so. Explain the double meaning of _____. Tell which meaning better explains the major ideas in the passage. Why did the author give a definition of _____ in paragraph 2?

    Exhibit 9. Sample NAEP Items, by Element of Practical Text and Aspect of Readi
    Element of Practical Text Aspect of Reading
    Forming a General Understanding Developing Interpretation Making Reader/Text Connections Examining Content and Structure
    Central purpose What is the purpose of the document? Use evidence from the document in your response. Do these directions list all the materials you need? Give a reason for your answer. Do you think the directions would be the same for _____? Use evidence from the text. Is this mainly for readers familiar with the activity? Explain why or why not.
    Key information Tell how this information would be useful. Use evidence from the document. Why is it important to do _____ before _____? Use evidence from the document in your response. Compare the dir- ections to a set of directions you have used in the past. Which set is easier to follow? Tell why. Name one step that is important in order to follow the directions. Explain why this step is important.
    Key organizing features Tell what you need to complete the steps. Which features indicate this? What happens after _____? Explain why this is an important step. When might you use these steps? Support your answer with evidence from the text. Explain how the author organized this document. Use support from the text.
    Key graphics What is the signifi- cance of the graphics to the total set of directions? Use support from the text in your response. Why does the second picture show _____? Which additional graphics would you add to make the directions clear? Use support from the text. Why is there boldface print at the front of each section?
    Vocabulary Which words tell what the document is mostly about? Use evidence from the text to support your response. Use the context to tell the meaning of _____. Explain why the phrase _____ is useful in these directions. Why does the author use the words _____ to present the last step of the directions?

    Review Process

    An extensive review process ensures that the assessments are consistent in meeting the criteria outlined in the framework. External reading committees and state-level testing and reading experts check the content of the assessments. Items are reviewed for bias and sensitivity, and test specialists review the items for technical qualities. An extensive editorial review provides quality assurance. Finally, as mandated by NAEP law, the Governing Board reviews the items both before and after field testing for appropriateness and bias.

    Accommodations

    NAEP reading is administered to English-language learners and students with disabilities who, by judgment of the school staff, are capable of taking the assessment. NAEP's intent is to assess all selected students from the target population. Therefore, every effort is made to ensure that all selected students who are capable of participating in the assessment are assessed. Some students sampled for NAEP participation are excluded according to carefully defined criteria. These criteria were revised in 1996 to more clearly communicate a presumption of inclusion except under special circumstances. Students are excluded only if the school's Individual Education Plan (IEP) team determines that the student cannot participate or the student's cognitive functioning is so severely impaired that she or he cannot participate. According to these criteria, students who are English-language learners, as well as those with disabilities who have IEPs that indicate accommodation strategies and procedures, participate in the assessment. Accommodations include (but are not limited to):

    • extra testing time

    • individual or small-group administrations

    • large-print booklets

    • multiple testing sessions

    Accommodations may be provided in combination—for example, extra testing time and individual administration. Because NAEP considers the domain of its reading assessment to be reading, the assessment cannot be read aloud. Moreover, NAEP does not provide an alternate language version, since the assessment measures reading in English.

    1 Passages and accompanying items are divided into blocks and administered to representative samples of students. Results are pooled, profiling the performance of groups of students on the entire assessment.

    Previous Contents Next
    Reading Framework for the 2005 National Assessment of Educational Progress