Insights
Same answer, different score: when interviewers read the criteria differently
Even with the same criteria, if people read them differently, a score depends on who gave it. Scores shaped by each interviewer’s tendencies cannot simply be pooled and compared, or checked against post-hire outcomes. This article explains how gaps between human and AI scores help you find where criteria are interpreted differently, and why fixing them matters.
The problem
Criteria open to different interpretations cause three problems
People can read the same criteria in different ways. Criteria like that cause three problems.
01
Who interviewed decides who passes
The same answer can gain or lose points depending on the interviewer. Near the pass mark, that difference can change the result.
02
Scores cannot be compared over time
When interviewers change, or one interviewer’s habits shift over time, the same score comes to mean something different. Putting last year’s scores next to this year’s is not a like-for-like comparison.
03
Post-hire outcomes cannot confirm anything
Even if you compare selection scores with later performance reviews, you cannot tell whether the criteria were right or whether interviewers simply interpreted them differently.
For how to check against post-hire outcomes, see “Was your interview evaluation right? You find out after the hire”.
Findings
A rating often says more about the rater than about the person rated
In 2000, Scullen and colleagues published an analysis of on-the-job performance ratings. In two data sets of 2,350 and 2,142 managers, each manager was rated by seven people: two bosses, two peers, two subordinates and themselves. In the two data sets respectively, 62% and 53% of the variation in ratings was explained by each rater’s own tendencies, and only 21% and 25% by how the person being rated actually performed. In other words, more than half the difference in scores came from who did the rating, and only about a quarter from differences in how people actually worked.
Interview ratings also differ between interviewers. In a 2013 analysis by Huffcutt and colleagues, the correlation between interviewers’ ratings averaged 0.74 when they sat in the same interview and 0.44 when each interviewed separately. A correlation runs from −1 to 1; the closer to 1, the more the two ratings agree. So ratings agree more when interviewers sit in the same interview than when each sees the candidate separately. Even so, hearing the same answers side by side does not make two ratings match completely.
The same pattern has been reported in Japanese new-graduate hiring, where companies take on an annual intake of students straight from university. In a 2016 study, Suzuki examined ratings from final-round interviews for new graduates at a Japanese software company. Two interviewers, a board director and a division head, each rated candidates from 0 to 8 on five criteria: attitude as a working adult, interpersonal skills, personality fit, sincerity, and potential to deliver results. The rank correlation between the two was highest for personality fit at 0.41 and between 0.23 and 0.25 for attitude, interpersonal skills and sincerity. For potential to deliver results it was 0.04, showing no relationship. So even when rating the same candidates, the two interviewers agreed only weakly on most criteria, and barely at all on the one that looks ahead to job performance.
In a 2010 conference paper, Imashiro reasoned that Japanese new-graduate hiring, which recruits people into the company rather than into a defined role, leaves more room for interviewers to differ than Western hiring for a specific job. Surveying interviewers, she found that those with long sales experience valued some candidate traits differently from those without.
Things the criteria do not mention also find their way into ratings. A 2009 meta-analysis by Barrick and colleagues examined how candidates’ appearance, impression management, and verbal and non-verbal behaviour relate to interviewer ratings. The effect was larger in unstructured interviews, and the authors concluded that what you see in the interview may not be what you get on the job.
Sources: Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956–970. https://doi.org/10.1037/0021-9010.85.6.956 / Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264–276. https://doi.org/10.1111/ijsa.12036 / Barrick, M. R., Shaffer, J. A., & DeGrassi, S. W. (2009). What you see may not be what you get: Relationships among self-presentation tactics and ratings of interview and job performance. Journal of Applied Psychology, 94(6), 1394–1411. https://doi.org/10.1037/a0016532 / Suzuki, T. (2016). Inter-rater reliability of recruiting interviews: Experimental study on evaluation components [in Japanese]. Japan Journal of Human Resource Management, 17(1), 69–91. https://doi.org/10.24592/jshrm.17.1_69 / Imashiro, S. (2010). Why do interviewers differ in what they look for? Effects of interviewers’ job experience [in Japanese]. Paper presented at the 2010 annual conference of the Japanese Group Dynamics Association. https://www.recruit-ms.co.jp/research/thesis/pdf/2010jgda.pdf
Why it goes unnoticed
Differences in how people interpret criteria are hard to see
Differences between human ratings get mixed up with each interviewer’s leniency or strictness, so the part that comes from interpreting the criteria differently is hard to separate out. Few candidates are rated by several interviewers. And one interviewer’s scores alone do not show how that interviewer reads the criteria.
What AI brings
AI applies the criteria as written, giving you a reference point
AI scores using only the wording of the criteria and what was said in the record. It does not use manner or appearance, and it does not bring in standards the criteria do not state. When people and AI score against the same criteria, the AI score shows the criteria applied literally, so it can serve as a reference point for human scores. Generative AI scores are not always exactly the same each time, but scores for clear answers barely change.
How it happens (an illustrative example)
Takes initiative (L5: action, who was involved and outcome are all described / L3: describes an action, but not the outcome)
At my last job, when enquiries suddenly rose, I put the handling steps together myself and shared them with the team.
AI score
3
Describes an action but not the outcome, so it fits L3.
Human score
4
No figures were given, but from the way they spoke, they seemed to have got results.
The human scores it one level higher, based on an assumption the criterion does not mention.
Interpreting the gaps
Read the gaps by their pattern
When human and AI scores differ, the pattern of the gap tells you what is going on.
01
Humans score higher or lower across the board
If human scores are higher, or lower, on every criterion, the issue is the interviewer’s leniency or strictness, not the criteria.
02
Gaps on particular criteria only
If gaps keep appearing on some criteria only, their wording may allow different interpretations, so they are worth reviewing.
03
Gaps for particular candidates only
If gaps appear for some candidates only, people may be picking up things the record does not capture, such as manner or atmosphere. That is material for checking whether the grounds for the rating are sound.
Why aligning matters
When everyone reads the criteria the same way, scores become a lasting asset
Resolve the gaps starting from the wording of the criteria. If a human judgement has a reason behind it, write that standard into the criteria. If it does not, bring the human scoring in line with the criteria. Once interpretations align, you can do the following.
01
The same yardstick, whoever interviews
The pass mark keeps its meaning when interviewers change. New interviewers can score the same way by reading the same criteria.
02
Scores can be compared across periods
The same score carries the same meaning, so the more scores you gather, the more you have to judge from.
03
Post-hire outcomes can test the criteria
The smaller the differences in how interviewers interpret the criteria, the easier it becomes to check whether the criteria themselves were right.
Rewritten criteria can be tested on past interviews. See “Rewrite your criteria, then test them on past interviews”.
What makes the comparison possible
Because people and AI use the same criteria, the gaps can be compared
Human and AI scores can be compared only when the human scoring sheet and the AI criteria are written in the same words. Where the service provides fixed criteria, the AI criteria and your own scoring sheet use different words, so mapping one onto the other takes a separate step.
In Proomi you write the criteria in your own words. People can score against the same criteria, so human and AI scores can be viewed side by side, criterion by criterion.
What to compare
Three checks for whether human and AI scores can be compared
When comparing AI interview services, these three checks tell you whether you can bring human and AI scoring into line.
01
Can people score against the same criteria as the AI?
If the criteria differ, the gap between human and AI scores cannot be compared.
02
Is the reasoning behind AI scores kept?
With the reasoning, you can see which reading of which remark caused the gap.
03
Can human and AI scores be viewed side by side per criterion?
A total score alone does not show which criterion the gap is on.
Q&A
Frequently asked questions
Which is right, the human score or the AI score?
Neither is right by default. The AI score shows the criteria applied literally. When it differs from the human score, your company decides where the standard should sit, and that decision goes into the wording of the criteria.
Should gaps be eliminated?
There is no need to get them to zero. A gap that appears for particular candidates only may be a sign that people are noticing something the record does not capture, which is worth having someone check. What you want to align is how the criteria are interpreted.
How many interviews before a pattern shows?
One gap at a time cannot tell you whether it is chance. As a guide, look across around 100 interviews per role, criterion by criterion, at whether the gaps keep pointing the same way.
Read next
Once you fix where criteria are read differently, test them on past interviews
Once you rewrite the parts that people interpret differently, you can test them on past interviews. Checking whether scores tell candidates apart at the same time makes it easier to decide which criteria to fix first. How to prepare criteria and refine them by comparing human and AI scores is covered in another article.
Rewrite your criteria, then test them on past interviews →
Re-score past interviews against rewritten criteria and see how the scores move.
More criteria do not mean more evidence →
Why scores bunch up, checked against public civil-service rating data.
Can AI evaluate candidates correctly? HR writes the criteria →
AI scoring removes the differences between interviewers. How to prepare criteria and refine them by comparing human and AI scores.
Related pages
How it applies, by situation
Differences in how criteria are interpreted show up the same way in multi-site hiring and in interviewer training.
Set the criteria first and narrow the variation between interviewers.
AI interview services compared →
The major services, sorted by what each one optimises for.
Show us your current evaluation criteria
We will look with you at which of your criteria are most likely to be read differently by different people. Free, about 30 minutes.