Selected Response

As the name implies, selected-response items are those in which students read a question and are presented with a set of responses from which they choose the best answer. In other NAEP assessments, selected-response items most often take the form of multiple-choice items, in which students select an answer from, say, four options provided. The choices include the most applicable response—the "answer"—as well as three "distractors." The distractors should appear plausible to students but should not be justifiable as a correct response, and, when feasible, the distractors should also be designed to reflect current understanding about students' mental models in the content area. The NAEP Technology and Engineering Literacy Assessment will include such multiple-choice items within both the scenario-based and the discrete-item assessment sets as one type of selected response.

In addition to the conventional multiple-choice selected-response items, the scenario-based sets in the NAEP Technology and Engineering Literacy Assessment will include other types of selected-response formats. The computer-based nature of the scenarios will allow a variety of types of student selections to be measured. For example, a student might be given a task to perform and asked to select an individual tool from a set of virtual tools. When a student selects a tool by clicking on it, it provides a measurable response that is, in essence, a selected response.

A selected-response item in such a scenario might have fewer choices than in a conventional multiple-choice item. For example, a student might select between two alternatives, such as deciding whether a switch in a circuit should be open or closed in order to produce a particular outcome, but the student might also have to justify or explain the selection. In this case, the first part of the answer is a selected-response, but it might be necessary to score the two parts of the item together so that the selection and justification together determine the score. In complex, real-world scenarios, it might be the case that there is not a "correct" selection, and in such a situation what matters is that the selection is justified adequately.

An example of a selected-response item that might be part of a series of items embedded in a scenario-based assessment set is shown on the next page. The question was part of an ICT assessment about reintroducing lynx into a Canadian park overrun by hares. The assessment was designed for 13 year-olds.

Sample item. Text reads, "Park rangers noticed the problem because they've been estimating the number of hares in the area for the last four years. Here's what they found. Last year, 2002, there were about 95,000 hares. The year before that, 2001, there were about 80,000. In 2000, there were 25,000. And in 1999, there were only about 1,000 hares. Your task is to organize the data to see if there is a trend. Pick a tool to use:" Options: "Word processor, Spreadsheet, Presentation."
[Source: Quellmalz & Kozma, 2004]

Other selected-response types within a scenario-based assessment set might include a task in which a student selects all of the options that apply from a given set of choices. Again, in a real-world situation there might be one "best" combination of choices but also one or more other combinations that are partially correct. In such a situation, it makes sense to use a scoring rubric that rewards different combinations of selected response items with different scores.

In the following item, students observe organisms interacting in an ecosystem before choosing all of the organisms that are consumers. The assessment was designed for grade 8. In the NAEP Technology and Engineering Literacy Assessment, students might choose all of the organisms in an ecosystem most immediately affected by a pollutant.

Screenshot of a sample item. Screen shows fish in the ocean. Text reads, "Select all the organisms that are consumers." Options: "Algae, Shrimp, Trofa, Alewife, Trout."
[Source: Quellmalz, Timms, & Buckley, 2009]

Another form of selected response is a "hot spot" in which a student answers by clicking on a spot on an image such as a map, picture, or diagram. The example below was designed for grade 8 students as part of a set of items on a state science test. For the NAEP Technology and Engineering Literacy Assessment, a task might be designed that asks students to click on the places in the image where pollution originated in the water system.

Screenshot of a program that features a sample item, with a diagram depicting the water cycle. Diagram includes land, a lake, clouds, and rain, with arrows between them showing the cycle. Text reads, "Choose the part of the water cycle that takes minerals into a lake. Click on the diagram to put a '+' on the part of the water cycle that takes minerals into a lake."
[Source: Minnesota Department of Education, 2009]