| The purpose of this study was to examine an art portfolio assessment system for measuring student art abilities. The art portfolio assessment system included: (1) faculty that assigned 12 portfolio projects aligned with curriculum objectives, (2) 15 commercial art students that completed these projects at a New England community technical college, (3) a validity committee that monitored rater severity/leniency, (4) 10 professional graphic designers that rated student art projects, (5) and a statistical framework for analysis of project ratings. Professional graphic designers rated student art projects using a scoring rubric that included art project design criteria, four rating dimensions assessing specific aspects of the project, and a 5-point rating scale. Each student portfolio received ratings from three randomly assigned raters. A many-facet Rasch model (MFRM) was used as the statistical framework to analyze student ability, project difficulty, rater severity, and rating dimension difficulty.;Three students' art abilities were over-estimated and three students were under-estimated on the basis of raw score averages. The MFRM adjusted the student ability measure for the rater severity/leniency and changed the ranking of the six students accordingly. Rater severity/leniency was found to be significant with a reliability of separation index of .93, and inter-rater agreement of 43.3 percent. Raters failed to distinguish differences in the rating dimensions with a reliability of separation index of .73. Rating scale categories were consistently selected at the same category 42 percent of the time. Project difficulty and the rating dimension difficulty components had no effect on student ability ranking.;Content validity was addressed as an integral aspect of the art portfolio assessment system; construct validity was examined from the rank ordering of item difficulty; and internal validity was calibrated for each component providing evidence that the data fit the MFRM. The art portfolio assessment system using the MFRM as the statistical framework provided measures of student art ability that were free of subjective judgments. |
