Developing NAEP Frameworks: A Look Inside the Process

Mary Crovo
Deputy Executive Director
National Assessment Governing Board
November 17, 2007

NAEP Frameworks:.Governing Board Responsibilities in Current NAEP Legislation

select the subject areas to be assessed
develop assessment objectives and test specifications that produce an assessment that is valid and reliable, and are based on relevant widely accepted professional standards
develop a process for review of the assessment which includes the active participation of teachers, curriculum specialists, local school administrators, parents, and concerned members of the public
(P.L. 107-279)

Governing Board Policy:.Framework Development
WHAT - comprehensive, inclusive, and deliberative process to determine the content and format of NAEP subject area assessments
WHO - educators, testing experts, state and local policymakers, researchers, business representatives, military representatives, members of the public; key 'users and consumers' in the subject area
HOW - Steering and Planning Committees; broad-based input and review from NAEP stakeholders; input and guidance from ADC and the Board

Overview of the NAEP Framework Process
Board Input
Procurement
Contract Award
Planning Document & Committee Composition
Framework and Specifications Development
Framework Review Process
Board Action
Framework & Specs to NCES

NAEP Framework Components
WHAT
Content areas and objectives for each NAEP grade
Process or cognitive dimension

HOW
Types of test questions
Distributions
by content area
by cognitive dimension
by item type
Administration procedures, delivery mode
Sample test questions

HOW MUCH
Sample scoring rubrics
Preliminary achievement level descriptions

NAEP Specifications
Audience
NCES and NAEP assessment development contractors

Purpose
to provide a detailed blueprint for constructing the assessment

Framework and Specifications are both public documents

Developing the Framework & Specifications
Charge to the project Steering Committee
Issues Paper

Committee meetings
Consideration of research and key documents
Drafting of Framework and Specifications
Ongoing ADC and Board review, input, guidance

Framework Process Balances Consideration of:
Research and Best Practice
Research on student learning, cognitive development and assessment
Current instructional and measurement issues
Curricular reform efforts

Standards and Assessments
State standards and assessments
International standards and assessments
(e.g., PIRLS, PISA, TIMSS)
Professional association standards
(e.g., national content standards)

Framework Review Process
Framework reviews in diverse venues across the U.S.
External review panel
Board action on Framework and Specifications
Framework & Specifications sent to NCES for assessment development
Framework dissemination strategies
Framework Process Balances Consideration of (continued)

Other considerations
Key documents having national or international significance
Preparedness reports and research (grade 12)
Previous NAEP Frameworks
Following vs. leading the field
(given 5-6 year NAEP development cycle)
Assessment content
Assessment format/delivery
Resources

Lessons Learned
Participants
Committee membership - cast a wide net
Framework review - use broad outreach with diverse venues and audiences
Often 1,000+ individuals involved per Framework
Process
Project timeline - 18+ months
Scope of the project, focus on:
What NAEP can do well
What NAEP is not designed to do
Realities, constraints, and timing
Resolving difficult issues - trust the process
iterative, not linear

New Challenges in NAEP Framework Process
Assessing 12th grade preparedness
Considering the role of technology
Maintaining trend lines
Increasing framework dissemination
Funding for new frameworks

NAEP Related Resources

For further information regarding the Framework process, policies, and publications, visit these websites:
http://nces.ed.gov/nationsreportcard/frameworks.asp
http://nces.ed.gov/nationsreportcard/frameworkhow.asph
http://nces.ed.gov/nationsreportcard/tdw/item_development/2000_2001/framework.asp
http://www.nagb.org/policies/pl-index.htm
http://www.nagb.org/publications/frameworks.htm

Developing NAEP Frameworks:.A Look Inside the Process

Mary Crovo
Deputy Executive Director
National Assessment Governing Board
November 17, 2007