|
Chapter One Overview Since 1973, the National Assessment of Educational Progress (NAEP) has gathered information about levels of student proficiency in mathematics and the related practices of teachers in our nation’s schools. The results of these periodic assessments are published in The Nation’s Report Card to inform citizens about the nature of students’ comprehension of the subject, curriculum specialists about the level and nature of student understanding, and policymakers about factors related to schooling and its relationship to student proficiency in mathematics. Based on these surveys of students at the end of elementary, middle, and high school, The Nation’s Report Card has provided comprehensive information about what students in the United States know and can do in the area of mathematics and in several other subjects. These reports present information about strengths and weaknesses in students’ understanding and their ability to apply that understanding in problem-solving situations; provide comparative student data according to race/ethnicity, type of community, and geographic region; describe trends in student performance over time; and report relationships between student proficiency and certain background variables. This framework document describes the content and format of the NAEP mathematics assessments in 1996, 2000, and 2003. Although there have been revisions to the framework, NAEP has maintained the mathematics trend begun in 1990. Context for Planning the Mathematics Assessment The National Assessment Governing Board (NAGB), created by Congress in 1988, is responsible for formulating policy for NAEP. NAGB is specifically charged with developing assessment objectives and test specifications, identifying appropriate achievement levels, and carrying out other NAEP policy responsibilities. In 1990, the U.S. Department of Education conducted the first voluntary state-by-state assessment of mathematics as an adjunct to its periodic NAEP national assessments of mathematics. The 1990 state-level trial was limited to the 8th grade. In 1992, the second voluntary state-level assessments associated with NAEP were carried out at the fourth- and eighth-grade levels in mathematics and at the fourth-grade level in reading. Current NAEP legislation in the No Child Left Behind Act of 2001 requires that NAEP assess mathematics and reading every two years at the national and state levels in grades 4 and 8. This schedule begins with the 2003 assessment. To prepare for the 1990 trial state assessment, the National Center for Education Statistics awarded a contract in 1987 to the Council of Chief State School Officers (CCSSO) to design a framework for the assessment. The Mathematics Framework Project gave special attention to the nature of formal state objectives and frameworks for mathematics instruction. In doing so, the Framework panels sampled state-, district-, and school-level objectives; examined the curricular frameworks on which previous NAEP assessments were based; consulted with leaders in mathematics education; and reviewed a draft version of the National Council of Teachers of Mathematics (NCTM) Curriculum and Evaluation Standards for School Mathematics . This project resulted in the “content-by-mathematical-ability” matrix design used to guide both the 1990 and 1992 NAEP mathematics assessments. To prepare for the next NAEP mathematics assessment, NAGB awarded a contract in fall 1991 to The College Board to develop assessment and item specifications for the 1994 mathematics assessment. The process of developing the recommendations for the planned 1994 NAEP mathematics assessment occurred between September 1991 and March 1992. Because of a budget shortfall, however, both the new NAEP mathematics and science assessments were rescheduled from 1994 to 1996. The NAEP mathematics project conducted by The College Board had two primary purposes. The first was to recommend a framework for the overall design of the mathematics assessment—that is, a structure for describing what students should know and be able to do in mathematics. The second was to develop specifications for the assessment items, with particular attention to a mix of formats, the item distribution for content areas within mathematics, and the conditions under which items are presented to students (e.g., use of manipulatives, use of calculators, and other factors). The new NAEP Mathematics Framework was considered in light of the three NAEP achievement levels—Basic, Proficient, and Advanced. Basic denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. Proficient represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter. Advanced represents superior performance. These levels are intended to provide descriptions of what students should know and be able to do in mathematics. Established for the 1992 mathematics scale through a broadly inclusive process and adopted by NAGB, the three levels per grade are the primary means of reporting NAEP data. The new mathematics assessment was constructed with these levels in mind to ensure congruence between the levels and the test content. See appendix A for the NAEP Mathematics Achievement Level Descriptions. Framework and Specifications Development Process The College Board convened a steering committee representing national education organizations, policymakers, and business to review the direction and scope of the project. A planning committee of mathematics educators met to draft the assessment framework. Both committees considered (1) the status of national reform efforts in mathematics education and assessment evaluations of the NAEP trial state assessment in mathematics (Silver, Kenney, and Salmon-Cox, 1991) and (2) the fit between NAEP assessments and the teaching of mathematics at grades 4, 8, and 12 in the nation’s schools (Romberg, Wilson, Smith, and Smith, 1991). Committee members are listed in appendix B. The suggested revisions in the framework are intended to reflect the 1990 and 1992 NAEP assessments. Committee members also made use of the findings of evaluation studies concerning the NAEP assessments. The findings of these studies were merged with research and state standards for the assessment of student proficiency in mathematics. Finally, the committee reviewed information provided by the 1990 assessment, noting features of the framework and how those features assisted or hindered the clear understanding of what students knew and were able to do in mathematics appropriate to their ages and levels of education. Another important phase in the process involved conducting a national mail review and convening focus groups in six states to gather input on the committee’s recommendations. The suggested revisions in the framework for the new NAEP assessment in mathematics are intended to reflect curricular emphases and objectives; include what various scholars, practitioners, and interested citizens believe should be in the assessment; and maintain ties to previous assessments to permit the reporting of trends in student achievement across time. Recommendations for the 1996 and Future NAEP Mathematics Assessments As a result of analysis and review, the steering committee and planning committee endorsed the following recommendations for the 1996 and future NAEP mathematics assessments:
The matrix framework employed in the 1990 and 1992 NAEP assessments should be discontinued in favor of a model consisting primarily of the five major content strands used in that matrix model. Evaluation studies of the NAEP trial state assessment and other cognitive science recommendations dealing with assessment suggest that forcing content into a rigidly structured, content-by-ability-level matrix distorts the nature of the discipline. A model that calls for the assessment of knowledge in discrete, content-by-ability-level categories is inappropriate in an era in which more progressive recommendations call for attention to a student’s ability to connect knowledge in one area of mathematics with knowledge and abilities in other areas of mathematics. Therefore, the recommendation was to use the five major content strands: (1) Number Sense, Properties, and Operations; (2) Measurement; (3) Geometry and Spatial Sense; (4) Data Analysis, Statistics, and Probability; and (5) Algebra and Functions. These strands have their foundation in NAEP mathematics assessments beginning in the 1970s. The nature of the strands is further discussed in chapters two and three. The levels of mathematical ability (conceptual understanding, procedural knowledge, and problem solving) should not be used to define specific percentages of items in each of the five content strands, as was done in the 1990 and 1992 assessments. However, these descriptors, along with the more encompassing process goals of reasoning, connections, and communication, should play a central role in defining item descriptors and achieving balance across the tasks for each grade level in the NAEP mathematics assessment. This recommendation is discussed further in chapters two and four. The percentage of items allotted to each of the five strands should continue the move begun with the 1990 assessment toward a balance among the five strands and away from an assessment dominated by number and operations. The recommendations, although retaining a core of items that reflect traditional goals in the Basic skills, represent continued movement toward a broad algebra- and geometry-oriented program at the eighth- and twelfth-grade levels. The specific percentage of items recommended is further discussed in chapter two. To measure the breadth and depth of student knowledge in mathematics, “families” of tasks/items should be created for each grade level of the assessment. A family of tasks/items is a related set of assessment tasks that can probe the vertical or horizontal nature of a student’s understanding. A vertical family might include items that measure students’ abilities to define a concept, apply the concept in a familiar setting, use the concept or related principles to solve a new problem, and ultimately generalize knowledge about the concept or related principles to represent a new level of understanding. A vertical family might lie within a single grade level or extend across grade levels. Another family of items might measure students’ horizontal understanding of a concept or principle across content strands. For example, students’ proficiency in solving the proportion 2/3 = 16/x might be measured in a number context, in a measurement setting, in a geometry setting, in a probability setting, and in an algebraic setting. Students’ ability to work with the proportion in each of these contexts tells a great deal about the richness of their understanding of the concept and the related procedural skills. The number of items requiring students to construct a response should be increased as much as possible within the bounds of the statistical design used to carry out the assessment. Furthermore, these items provide excellent opportunities to measure students’ abilities to reason mathematically as well as connect and communicate their knowledge of mathematics. In particular, the number of extended open-ended items should be increased from the number given in the 1992 assessment. At the twelfth-grade level, a special study should be carried out using graphing calculators to establish baseline data for gradually introducing these calculators, which can assist students in visualizing algebraic relations, into the curriculum. The assessment should continue using reasonable manipulative materials, where possible, to measure students’ knowledge and problem-solving abilities. Such manipulative materials and accompanying tasks should be carefully chosen to minimize disruption of the test administration process. Although bias analysis is consistently conducted on NAEP items and student performance as mandated by law, recommendations for shifting the types of items used on the assessment merit an especially careful look at potential item bias. Data should be gathered during field testing and during the actual assessment and analyzed for any unforeseen item bias that may arise from incorporating less widely used types of assessment items. The 1996 and future NAEP assessments will incorporate awareness of this critical consideration, especially related to students’ previous opportunities to learn and their experience and background both in school and outside of school. Sensitivity and a sound research base will guide not only test construction but also the reporting of student performance. These recommendations were made in an attempt to reflect the increasing realization that student proficiency in mathematics is not the result of the interaction of discrete cells of knowledge with a discrete list of special mathematical abilities. Rather, student proficiency in mathematics results from broad experience in forming networks of connections among mathematical ideas and skills. The current framework and specifications reflect a more integrated view of school mathematics than previous NAEP frameworks. Note: For the 2005 mathematics assessment, the Governing Board conducted a comprehensive Framework Update Project to further enhance the current framework and specifications. Although changes were recommended at grades 4 and 8, these modifications are not expected to disrupt the trends begun in 1990. More substantial changes were recommended for the 12th-grade assessment. Refer to the NAGB Web site at www.nagb.org for further information on the 2005 Mathematics Framework.
|
||||||||