Chapter Four: Mathematical Complexity of Items

Each NAEP item assesses an objective that can be associated with a single content area of mathematics, such as number or geometry. The item also makes certain demands on students’ thinking. These demands constitute the mathematical complexity of the item, which is the second dimension of the mathematics framework. The demands on thinking that an item makes (what it asks the student to recall, understand, reason about, and do) are determined on the assumption that the student is familiar with the mathematics of the task. If a student has not studied the mathematics, the task is likely to make different and heavier demands, and the student may well not be successful. Items are chosen for administration at a given grade level in part on the basis of their appropriateness for typical curricula, but the complexity of those items is always independent of the particular curriculum a student has experienced.

The categories (low complexity, moderate complexity, and high complexity) form an ordered description of the demands an item may make on a student. Items at the low level of complexity, for example, may ask a student to recall a property. At the moderate level, an item may ask the student to make a connection between two properties; at the high level, an item may ask a student to analyze the assumptions made in a mathematical model. This is an example of the distinctions made in item complexity to provide balance in the item pool. The ordering is not intended to imply that mathematics is learned or should be taught in such an ordered way.

The complexity dimension is both similar to and different from the levels of mathematical ability (conceptual understanding, procedural knowledge, and problemsolving) that were used in the NAEP Mathematics Framework for the 1996 and 2000 assessments. The dimensions are similar in that both attempt to address the kind of thinking that the student is doing when working on an item. They are also similar in that although neither dimension is used to define specific percentages of items in each content area, both are used to help define item descriptors and to achieve a balance across the tasks administered at each grade level. Level of complexity is different from level of mathematical ability, however, in that complexity describes the mathematical expectations of an item, whereas mathematical ability (along with the associated construct of mathematical power) requires an inference about the skill, knowledge, and background of the students taking the item.

The mathematical complexity of an item is not directly related to its format (multiple choice, short constructed response, or extended constructed response). Items requiring that the student generate a response tend to make somewhat heavier demands on students than items requiring a choice among alternatives, but that is not always the case. Any type of item can deal with mathematics of greater or lesser depth and sophistication. There are multiple-choice items that assess complex mathematics, and constructed-response items can be crafted to assess routine mathematical ideas. Moreover, the mathematical complexity of an item is constant; it does not vary depending on the score given for a certain kind or level of response.

Low Complexity

This category relies heavily on the recall and recognition of previously learned concepts and principles. Items typically specify what the student is to do, which is often to carry out some procedure that can be performed mechanically. It is not left to the student to come up with an original method or solution. The following are some, but not all, of the demands that items in the low-complexity category might make:

• Recall or recognize a fact, term, or property.

• Recognize an example of a concept.

• Compute a sum, difference, product, or quotient.

• Recognize an equivalent representation.

• Perform a specified procedure.

• Evaluate an expression in an equation or formula for a given variable.

• Solve a one-step word problem.

• Draw or measure simple geometric figures.

• Retrieve information from a graph, table, or figure.

Examples

See appendix B for solutions and scoring guides

• Recall or recognize a fact, term, or property.

  How many fourths make a whole?

  Answer: ___________________

  Source: 1996 NAEP (grade 4)      Percent correct: 50

• Recall or recognize a fact, term, or property.

  Which of the following is NOT a property of every rectangle?

  1. The opposite sides are equal in length.

  2. The opposite sides are parallel.

  3. All angles are equal in measure.

  4. All sides are equal in length.

  5. The diagonals are equal in length.

  Source: 1992 NAEP (grade 12)       Percent correct: 71

• Compute a sum, difference, product, or quotient.

  What is 2/3 of 15 marbles?

  1. 10 marbles

  2. 9 marbles

  3. 8 marbles

  4. 5 marbles

  Source: 1990 NAEP (grade 12)       Percent correct: 74

• Compute a sum, difference, product, or quotient.

  +6 + -12 =

  • -6

  • +6

  • -18

  • +18

  Source: 1990 NAEP (grade 8)       Percent correct: 68

• Provide or recognize equivalent representations.

  N stands for the number of stamps John had. He gave 12 stamps to his sister.

  Which expression tells how many stamps John has now?

  1. N + 12

  2. N – 12

  3. 12 – N

  4. 12 x N

  Source: 1996 NAEP (grade 4)       Percent correct: 67

• Perform a specified procedure.

  How many hours are equal to 150 minutes?

  1. 1¹/2

  2. 2¹/4

  3. 2¹/3

  4. 2¹/2

  5. 2⁵/6

  Source: 1990 NAEP (grade 12)       Percent correct: 74

• Retrieve information from a graph, table, or figure.

  

  What is the weight shown on the scale?

  1. 6 pounds

  2. 7 pounds

  3. 51 pounds

  4. 60 pounds

  Source: 1992 NAEP (grade 4)       Percent correct: 44

Moderate Complexity

Items in the moderate-complexity category involve more flexibility of thinking and choice among alternatives than do those in the low-complexity category. They require a response that goes beyond the habitual, is not specified, and ordinarily has more than a single step. The student is expected to decide what to do, using informal methods of reasoning and problemsolving strategies, and to bring together skill and knowledge from various domains. The following illustrate some of the demands that items of moderate complexity might make:

• Represent a situation mathematically in more than one way.

• Select and use different representations, depending on situation and purpose.

• Solve a word problem requiring multiple steps.

• Compare figures or statements.

• Provide a justification for steps in a solution process.

• Interpret a visual representation.

• Extend a pattern.

• Retrieve information from a graph, table, or figure and use it to solve a problem requiring multiple steps.

• Formulate a routine problem, given data and conditions.

• Interpret a simple argument.

Examples

See appendix B for solutions and scoring guides

The following shapes were provided to students. (Shapes were larger than shown.)

(Grade 8 version)

Bob, Carmen, and Tyler were comparing the areas of N and P. Bob said that N and P have the same area. Carmen said that the area of N is larger. Tyler said that the area of P is larger.

Who was correct? _______________________________

Use words or pictures (or both) to explain why.

Source: 1996 NAEP (grades 4 and 8)      Percent correct: 27

• Interpret a visual representation.

  

  In this figure, how many small cubes were put together to form the large cube?

  1. 7

  2. 8

  3. 12

  4. 24

  Source: 1996 NAEP (grade 4)       Percent correct: 33

• Extend a pattern.

  From any vertex of a 4-sided polygon, 1 diagonal can be drawn.

  From any vertex of a 5-sided polygon, 2 diagonals can be drawn.

  From any vertex of a 6-sided polygon, 3 diagonals can be drawn.

  From any vertex of a 7-sided polygon, 4 diagonals can be drawn.

  How many diagonals can be drawn from any vertex of a 20-sided polygon?

  Answer: ___________________________

  Source: 1996 NAEP (grade 8)      Percent correct: 54

• Interpret a simple argument.

  Tracy said, “I can multiply 6 by another number and get an answer that is smaller than 6.”

  Pat said, “No, you can’t. Multiplying 6 by another number always makes the answer 6 or larger.”

  Who is correct? Give a reason for your answer.

  Source: 1992 NAEP (grades 8 and 12)
  
  Percent correct: 49 (grade 8); 63 (grade 12)

High Complexity

High-complexity items make heavy demands on students, who must engage in more abstract reasoning, planning, analysis, judgment, and creative thought. A satisfactory response to the item requires that the student think in abstract and sophisticated ways. Items at the level of high complexity may ask the student to do any of the following:

• Describe how different representations can be used for different purposes.

• Perform a procedure having multiple steps and multiple decision points.

• Analyze similarities and differences between procedures and concepts.

• Generalize a pattern.

• Formulate an original problem, given a situation.

• Solve a novel problem.

• Solve a problem in more than one way.

• Explain and justify a solution to a problem.

• Describe, compare, and contrast solution methods.

• Formulate a mathematical model for a complex situation.

• Analyze the assumptions made in a mathematical model.

• Analyze or produce a deductive argument.

• Provide a mathematical justification.

Examples

See appendix B for solutions and scoring guides

• Generalize a pattern.

  A pattern of dots is shown below. At each step, more dots are added to the pattern. The number of dots added at each step is more than the number added in the previous step. The pattern continues infinitely.

  Marcy has to determine the number of dots in the 20th step, but she does not want to draw all 20 pictures and then count the dots. Explain or show how she could do this and give the answer that Marcy should get for the number of dots.

  Source: 1992 NAEP (grade 8)       Percent correct: 6

• Analyze or produce a deductive argument.

  Jaime knows the following facts about points A, B, and C.

  • Points A, B, and C are on the same line, but might not be in that order.

  • Point C is twice as far from point A as it is from point B.

  Jaime concluded that point C is always between points A and B.

  Is Jaime’s conclusion correct?

  Yes     No

  In the space provided, use a diagram to explain your answer.

  Source: 1996 NAEP (grade 8)      Percent correct: 23

• Provide a mathematical justification

  

  In Mr. Bell’s classes, the students voted for their favorite shape for a symbol.

  Here are the results.

  	Class 1	Class 2	Class 3
  Shape N	9	14	11
  Shape P	1	9	17
  Shape Q	22	7	2

  Using the information in the chart, Mr. Bell must select one of the shapes to be the symbol. Which one should he select and why?

  The shape Mr. Bell should select: _________________

  Explain.

  Source: 1996 NAEP (grades 4 and 8)
  
  Percent correct: 31 (grade 4); 58 (grade 8)

Balance of Mathematical Complexity

The ideal balance sought for the 2005 NAEP is not necessarily the balance one would wish for curriculum or instruction in mathematics education. Balance here must be considered in the context of the constraints of an assessment such as NAEP. These constraints include the timed nature of the test and its paper-and-pencil format. Items of high complexity, for example, often take more time to complete. At the same time, some items of all three types are essential to assess the full range of students’ mathematical achievement. Within that context, the ideal balance would be that half of the score on the assessment is based on items of moderate complexity, with the remainder of the score based equally on items of low and high complexity. This balance would apply for all three grade levels.