Insights

Can you rely on AI interview scores? Test them against your own interviewers

Once you adopt AI interviews, the scores the AI gives become part of how you decide who moves to the next stage. How closely those scores match your interviewers’ scores varies by company and by criterion, so a vendor’s materials alone cannot tell you whether the scores will work with your criteria. This article shows how to use a trial to have your interviewers and the AI score the same interviews separately and compare the results. Where the scores are close, use the AI score in your decisions. Where they are far apart, review the criteria or check that point in a human interview.

Published

Introduction

What this article covers

This article explains how to run a trial in which two of your interviewers and the AI score the same interviews, so that you can decide how to treat the AI score for each criterion. You need two interviewers because people’s scores differ too. Even if the gap between the AI and an interviewer looks large, when the two interviewers differ by just as much, the gap may come from how the criteria are written or read. You do not have to make a single yes-or-no decision on adoption. You can decide criterion by criterion, using the AI score for some and checking others in a human interview.

For how to put the trial results into an approval request, with samples, see Getting Approval for AI Interviews: Likely Questions, Evidence and Sample Requests.

Summary

Key points

  • In the trial, two interviewers and the AI score the same interviews separately, and you start by comparing the interviewers’ scores with the AI’s
  • Where there is a gap, check whether the two interviewers also differ, and find the cause
  • Depending on the cause, change the criteria or the questions, or check that point in a human interview

Running the trial

In the trial, two interviewers and the AI score the same interviews separately

You do this check in a trial, before going live. For about a month, you run real AI interviews and use the records to compare the interviewers’ scores with the AI’s. The interviews can be with real applicants, or roleplays in which employees act as applicants. When deciding whether to use the AI score, compare scores from interviews with real applicants.

The scorers are two interviewers and the AI. The two interviewers score from the transcript of the interview conducted by the AI interviewer; they do not sit in on it. With real applicants, choose interviewers who will not meet that candidate in person, so that no impression from a face-to-face interview finds its way into the score.

If you cannot find two people, you can start by comparing one interviewer with the AI. But with only one, you cannot tell whether a gap between that interviewer and the AI is one that would also appear between two interviewers. When the time comes to decide which criteria to use the AI score for, ask a second person to score the same interviews without seeing the earlier scores.

Interviewers receive the same transcript the AI scores from. When everyone works from the same material, a gap in scores can be read as a difference in how the answer was evaluated. In Proomi, you can view the transcript in the admin screen right after the interview. If you want to give interviewers a batch of transcripts, we can extract them on request. Handle transcripts as you would CVs and other selection documents. Before you start, explain the purpose to candidates and to employees taking part in roleplays, and complete any agreement with the vendor that is needed before interviewing applicants.

The two interviewers score without talking to each other and without seeing the AI’s score. If they saw it first, it would influence their own scores and the comparison would no longer be fair. Ask them to add a short reason to each score, and to mark any criterion they found hard to judge from the transcript with the note “hard to judge from the transcript alone”.

Counting the gaps

Count the gaps for each criterion and decide how to treat it

Once scoring is done, take each criterion and count the interviews where the gap is two points or more, for each of three pairs: interviewers A and B, A and the AI, and B and the AI. On a five-point scale, one-point gaps are common, so two points is the threshold.

How you treat the criterion depends on where the gaps appear.

Where the gaps appear, and how to treat the criterion

Are the scores of both interviewers and the AI all close?

Close

Use the AI score in your decisions

Some are far apart

Compare the two interviewers’ scores

The two are close

Only the AI differs. Find the cause, then change the criteria or check it in a human interview

The two also differ

Change the criteria and test again

There is no common target percentage. Decide before the trial how close counts as close, judged against the gap between the two interviewers. If you decide after seeing the results, the threshold ends up fitted to the numbers.

For criteria with gaps, put the statements the AI quoted as evidence next to the interviewers’ reasons, and decide with the interviewers what to add to the criteria. When testing again, interviewers read interviews that were not used for the review, because with the same interviews their earlier reading stays with them. The AI can rescore past interviews against the revised criteria.

Review

Rewrite your criteria, then test them on past interviews →

Rescore past interviews against the rewritten criteria.

Because you can bring your own criteria to Proomi, this comparison is also a chance to examine them. Interviewers and the AI score against the same criteria, so a gap lets you look into how the criteria are written and read. If the two interviewers also differ, they may be reading that criterion differently. Where one person handles each first-round interview, there are few chances to see how other interviewers read the criteria. This comparison provides one.

Even when scores are close, there is more to check. If the AI gives 3 points on a criterion in most interviews, it may simply be giving every candidate the same score, even if that score is not far from the interviewers’. Hold off on using the AI score for a criterion like that. And if a criterion collects many “hard to judge from the transcript alone” notes, check it in a human interview even when the scores are close.

A trial covers few interviews, and some gaps will be down to chance, so these decisions are provisional. Keep checking them as interviews build up in live use.

We will explain the details, such as how to build the scoring sheet and set your thresholds, during the trial.

Why

Why there is no common target percentage

Even when interviewers hear the same interview in the same room, their evaluations do not match perfectly. A meta-analysis of studies on interviewer agreement found that the measure of agreement averaged 0.74 for interviewers sitting in the same interview, where 1 would mean perfect agreement.

Source: Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264–276. https://doi.org/10.1111/ijsa.12036

Japanese graduate hiring shows the same tendency. A 2005 study by Imajo, covering graduate interviews at four companies, suggested that interviewers differ in the kind of applicant they rate highly. A 2016 study by Suzuki looked at final-round interviews at a software company, where a director and a division head rated the same candidates on five criteria. Agreement varied widely by criterion: they agreed to some extent on personality fit, but there was almost no relationship on the criterion for expected performance after joining (rank correlations of 0.41 and 0.04; closer to 1 means the same ranking, and 0 means no relationship).

Source: Imajo, S. (2005). Empirical study on employment interviews: Multilevel analyses of applicant’s, interviewer’s, and organization’s effect on interview ratings [in Japanese]. Japanese Association of Industrial/Organizational Psychology Journal, 19(1), 3–16. https://doi.org/10.32222/jaiop.19.1_3

Source: Suzuki, T. (2016). Inter-rater reliability of recruiting interviews: Experimental study on evaluation components [in Japanese]. Japan Journal of Human Resource Management, 17(1), 69–91. https://doi.org/10.24592/jshrm.17.1_69

We believe one reason evaluations differ this much is that part of any set of criteria cannot be fully put into words, and each interviewer interprets that part differently.

You could set one figure for how often the AI and interviewers should agree, but the level of agreement varies by company and by criterion, so that figure would have no basis. That is why the AI score is compared with the scores of two of your own interviewers, who read the same record against the same criteria. If the interviewers agree on a criterion, and the gap between the AI and the interviewers is about the same as the gap between the interviewers, you can use the AI score for that criterion in your decisions.

What this comparison shows is whether the AI reads answers the way your interviewers do. It does not show whether scores predict retention or performance after joining. To see how scores relate to retention, compare them with your post-hire employment data.

After hiring

Was your interview evaluation right? You find out after the hire →

Check whether evaluations were right using post-hire data.

Interviewers

The role of your interviewers, and the time it takes

When scores differ, put the AI’s evidence next to the interviewer’s reasons

Setting scores side by side also shows differences between interviewers. But this comparison is a test of the criteria; it is not used to evaluate the interviewers themselves. Tell your interviewers this before the trial.

When an interviewer’s score differs from the AI’s, put the statements the AI quoted next to the interviewer’s reasons, and check with HR whether the point can be written into the criteria. If it can, add it. If the cause is a misreading by the AI or a transcription error, record it as an error. If a gap appears only for particular candidates, the score may rest on something the interviewer read between the lines of the transcript, so check those candidates in a human interview.

Your interviewers supply what goes into the criteria

Your interviewers decide which criteria need a human interview, and supply the points to add to the criteria.

Interviewers know better than HR how a strong performer answers a given question. When they read the criteria and add real example answers at each level, that knowledge stays in the criteria and becomes available both to the AI’s scoring and to the next interviewer. Have the head of the business unit review and agree to any additions before they go in. Criteria that cannot be agreed are checked in a human interview.

Proomi organises the information needed to make the decision. The decision on who goes through belongs to your interviewers and HR.

Estimating the time

You will ask interviewers to do the scoring (a score, a short reason and any notes) and attend two meetings: one to read through the criteria and one to compare results after scoring. HR compiles the table.

For the transcript of an interview of about 30 minutes, allow about 20 minutes per interview as a rough guide. To estimate the time per interviewer, multiply the number of interviews by 20 minutes and add the two meetings, then time the first few and adjust.

In live use, take a sample of interviews each month for two people to score, and set their scores next to the AI’s. At ten interviews a month, that is a little over three hours a month per interviewer. Interviewers will also spend time checking criteria in second-round interviews and reading AI interview records beforehand. If large gaps stop appearing as interviews build up, you can reduce the sample.

Records

What else to check in the trial

Transcription errors and the AI’s follow-up questions

Proomi scores by reading the interview transcript directly and quoting what the candidate said as evidence for each criterion. So you can check in the record whether a transcription error may have affected a score.

In the trial, HR checks the record of every interview. In live use, whoever decides on a pass looks at the AI score and the statements behind it. For borderline candidates, or those who have asked for an adjustment, read the whole record. As you check, see whether job-specific terms and proper nouns were transcribed correctly, and whether the AI’s follow-up questions fit your hiring policy.

Bias from differences in speaking style and phrasing

Bias can come from the AI’s scoring itself, from how criteria are written, and from how the AI interviewer conducts the interview. On scoring: the AI reads the words in the transcript, so you need to check that dialect, or the phrasing of non-native speakers of the interview language, is not affecting scores. Interviewers read the same transcript, so this effect can stay hidden even when the interviewers’ and the AI’s scores are close.

How criteria are written can also cause bias. With criteria written in terms of speaking style, such as “speaks with confidence” or “actively communicates achievements”, the more faithfully the AI follows them, the more easily differences in speaking style enter the score. In Proomi you can also define each level in your own words, so write them around the content of the answer: what the person did and how they decided.

You can test the effect of phrasing without asking where candidates are from. If you wish, we can take a few transcripts from the trial, rewrite only the phrasing while keeping the content of the answers, score them and compare the results with the original scores.

With real interviews, read low-scoring and high-scoring interviews side by side and see whether any features of speaking style recur. There is no need to guess at or record a candidate’s origin or nationality. Speaking style can also reflect a disability or a health condition, so any adjustment for an individual candidate should be based on that person’s own request for one. If the AI’s follow-up questions often stop after a short answer, we can work with you to adjust how the AI interviewer proceeds (this is custom work and is quoted separately).

Criteria with “no evidence” recorded

When nothing a candidate says supports a criterion, Proomi can record it as “no evidence”. If a criterion often has no evidence, there may be too few questions that explore it. When we simulated interviews with 30 graduate personas, adding three questions aimed at the items that drew little evidence cut “no evidence” records from one to six per persona down to zero to two. The personas’ answers were written by AI, so this is not a result from real applicants. It confirms that adding questions can draw out evidence, but does not show that the scores are right.

Evidence

Use your performance review criteria in interviews, word for word →

Adding questions reduced “no evidence”.

Our regular checks

On every plan, we analyse trends in transcripts and evaluations, and regularly check for bias or problems that appear across our customers. On the Growth plan, the operations review also shows you evaluation bias in your own interviews.

The time spent revising the criteria

Record which criteria you revised, how many times and how long it took, along with the two interviewers’ scoring time and the time spent preparing records for scoring. If a vendor supported the review, note that too, because the effort will differ when you run it alone.

Scoring

Choose an AI interview tool that can explain its scores →

Check what it reads and what it scores against.

Candidates

Question design and AI interviewer behaviour →

Improve how candidates experience the interview, using the records.

Q&A

Frequently asked questions

Can we run the trial with employees?

Yes. You can run it with employee roleplays or with real applicants. Roleplays can start once you have explained the purpose to the employees taking part. For interviews with applicants, we will put the agreement on personal data in place with you. Employees answer differently from real candidates, so decide whether to use the AI score by comparing scores from interviews with applicants.

Can we decide how to treat each criterion within the trial?

A trial of about a month settles which criteria to revise and how to treat them for now. You then keep checking as interviews build up for each role in live use.

Related pages

Read next

Approval

Getting Approval for AI Interviews: Likely Questions, Evidence and Sample Requests →

Put the trial results into the approval request and the trial report.

Criteria

Can AI evaluate candidates correctly? HR writes the criteria →

Prepare the criteria, then develop them by comparing human and AI scores.

Interpretation

Same answer, different score: when interviewers read the criteria differently →

Read gaps between people and AI by where they appear.

Distribution

More criteria do not mean more evidence →

Why scores bunch together, and how to read the distribution.

Evidence

Use your performance review criteria in interviews, word for word →

Adding questions reduced “no evidence”.

Getting started

Onboarding and Support →

The stages from first enquiry to live use, and who does what.

We will build a scoring sheet from your current criteria

Send us the names of your criteria, and we will build a scoring sheet that sets two interviewers’ scores next to the AI’s, and return it within about five working days. You can also talk to us about setting thresholds for the trial.