Recently, I facilitated an assessing writing workshop to high school teachers. The supervisor, who coordinated the workshop, shared that the teachers across content areas are not arriving to the same rubric score when assessing the same student writing piece.
What can we infer from this statement? A few questions can be considered.
First, do the teachers scoring the writing understand what the students are expected to know? Secondly, of the standards that are being assessed, have they been identified and understood by the teachers scoring the piece? Thirdly, have the teachers agreed on how the rubric applies to particular examples of student work?
A calibrated protocol for scoring student work not only answers the aforementioned questions, it increases reliability of assessing the same data. In other words, when scoring is calibrated, the student writing piece will receive the same score regardless of who scores it. The rubric is interpreted and applied in the same way by whomever is scoring it. This creates consistency and alignment with the scoring rubric and how it’s being used.
The purpose of the 7-Step Calibration Protocol is to score student work and consider the instructional implications of the task, student work and rubric in a collaboratively with a group of educators.
The protocol takes about 2-3 hours to conduct. This can be coordinated during a Professional Learning Community (PLC) meeting, a grade level meeting, during a professional learning workshop, or other collaborative work sessions.
The group size can range from four to eight grade level and/or content area teachers as well as special education/ English Language Learner teachers, and basic skills teachers.
A facilitator leads the work and explains the task at hand. Teachers have the opportunity to pose clarifying questions about the task at hand. The group of teachers examine the prompt, rubric, and student work.
Individually, each teacher, uses the rubric to read and score each student pieces that have been previously placed in a high, mid, and low piles (See the 3 Stack Protocol from our previous post). Scorers should take notes on the descriptors in the rubric and how evidence in the student work justifies the score rather than the, “relative strength or weakness of the writing piece”. “The student work sample must be truly aligned to the description of the assigned score for the integrity of the exercise to be preserved.”
The next phase, and an important one, is for teachers to share their scores for each of the rubric categories – without providing a justification for the scores.
A discussion ensues led by the facilitator on the differences between scores. This is where the scorers can justify their score by identifying specific language in the student piece that aligns to the descriptors in the rubric. The discussion continues until consensus on each student piece is reached.
Deep analysis of student work coupled with discussion of what constitutes a specific score will yield consistency and alignment to standard based rubrics and reliability from one teacher’s score and another teacher’s score. Ultimately, this will have a powerful impact on creating uniformity and clear expectations for teachers and student learning by grade level and across content areas.
Until next time, happy teaching!