Paper - Preparing Job Trainers to Describe Knowledge, Skills, and Abilities Measured in an Academic Assessment

Preparing Job Trainers to Describe Knowledge, Skills, and Abilities Measured in an Academic Assessment: Commentary on Computer-Aided Bookmarking (CAB)

MEASURED PROGRESS, 2012, GRADE 12

This paper describes the Computer-Aided Bookmarking (CAB) method of the Judgmental Standard Setting (JSS) studies and presents findings about the knowledge, skills, and abilities (KSA) statements that JSS panelists deemed necessary to answer NAEP items correctly.

The Judgmental Standard Setting (JSS) studies conducted in the Governing Board's preparedness research program requested that the convened panelists of college and job training program instructors individually note the knowledge, skills, and abilities (KSAs) required to answer each item correctly. As one of the contractors implementing this work, Measured Progress proposed a process that computerized these KSA annotations, in a fully computer-based standard-setting process -- Computer-Aided Bookmarking (CAB). The computerization of the Bookmarking method improved the efficiency of the standard-setting process and enabled analysis for research questions such as whether panelists’ were they able to properly identify KSAs.

This paper provides details of CAB as background information and presents findings about the type of information that JSS panelists regarded as KSA annotations. Process improvements used in the JSS process based on the findings are discussed. Lastly, recommendations for further improvements are presented. Supporting figures are included in the appendix.

Judgmental Standard Setting: JOB TRAINING | COLLEGE | READING | MATHEMATICS