Calibrate screening decisions before inconsistency spreads

Choose a useful sample

Select several consented, anonymised, or fictional profiles that represent direct, adjacent, and less familiar backgrounds. Include at least one ambiguous case. A sample of obvious yes decisions cannot expose judgment differences. Keep the role brief fixed during the exercise so disagreements concern evidence rather than different versions of the job.

Review independently first

Each screener records the evidence, uncertainty, and recommendation before discussion. This prevents a confident colleague from anchoring the room. Use the real checklist and expected time limit. The goal is to reproduce ordinary work closely enough that the team can see where its process, instructions, or assumptions create variance.

Diagnose the disagreement

Ask whether reviewers saw different evidence, weighted a criterion differently, used an unstated proxy, or filled missing information with opposing assumptions. A disputed decision is useful only when the team identifies its source. Avoid settling everything by majority vote; the most common interpretation may still conflict with the role brief.

Write the corrected standard

Clarify criteria, add positive and negative evidence examples, and define when a gap should advance to a screen. Remove proxies the team cannot defend. Record the change and who approved it. If the correction materially affects current applications, review them again rather than applying the new standard only to future candidates.

• Same sample reviewed independently
• Evidence precedes recommendation
• Disagreement source identified
• Correction applied to live work

Uncover a familiar-brand bias

Illustrative example: two screeners advance candidates from well-known companies despite vague ownership, while rejecting a cooperative-bank operations lead with detailed process evidence. Calibration shows that employer recognition replaced the outcome criterion. The team adds an ownership prompt, removes brand-based comments from notes, and rechecks the rejected profile against the actual role.

Make calibration part of operations

Run a short session when a role opens, a new screener joins, criteria change, or decision patterns become inconsistent. Track questions that recur and feed them back into the brief. Do not turn calibration into a meeting for every application. Its purpose is to improve the mechanism so individual reviews become faster, fairer, and easier to explain. Set a trigger for another calibration, such as a material brief change or repeated disagreement on one criterion. Name an owner to collect examples between sessions. Without this feedback loop, the written standard stays static while reviewers quietly develop separate local rules. Date each revision, notify every active screener promptly, and retire obsolete guidance.