Can AI evaluate candidates correctly? HR writes the criteria

Introduction

What this article covers

For HR teams considering AI interviews, the first thing to establish is whether AI can evaluate candidates correctly. There are two ways to check whether AI scores can be trusted.

• Agreement between human and AI scores: how closely the AI’s scores agree with your own interviewers’ judgements
• Comparison with performance after hiring: whether selection scores predict how people perform after they join

Agreement depends on how the evaluation criteria are written, and in our view the criteria also affect how well scores predict performance after hiring. In this article we set out how to prepare the criteria and how to keep reviewing them in use, drawing on research, a case study and a worked example.

Key points

Key points

• Whether AI scores can be trusted is checked in two ways: agreement between human and AI scores, and comparison with performance after hiring
• As long as scoring uses the same criteria and model, AI scoring removes the differences in leniency and interpretation that arise when the interviewer changes. Because generative AI output is probabilistic, scores for the same record can still vary, so do not base a decision solely on a gap of a point or so, or on a narrow margin near the pass mark; read the records and the quoted statements as well
• Criteria consist of evaluation axes (what to look at) and the definition of each level (what kind of answer earns which score). You can bring in your own criteria or build them from scratch with AI support
• Criteria improve as you compare human and AI scores and rewrite them. HR decides; much of the insight lies with interviewers from the business
• Who writes the criteria is one of the ways AI interview services differ. Check it when you choose a service

Variation

AI scoring removes the differences between interviewers

In human interviews, ratings of the same interview can differ depending on who does the rating. The difference is not only strictness or leniency; people also read the same answer differently. Candidates in Japan sometimes call this an “interviewer lottery”. In a meta-analysis by Huffcutt and colleagues (2013), the variation between interviewers’ ratings was as follows.

What AI scoring removes is the part of the gap in the first row that comes from who is doing the rating; the AI’s own variation when it re-scores remains, as described below. In the second row, the differences in the questions and in how candidates answer remain even when AI does the scoring. When AI scores interviews, every candidate’s record is read with the same criteria and, as a rule, the same model, so differences in leniency or interpretation that arise when the interviewer changes do not find their way into the scores.

Because generative AI output is probabilistic, however, AI scores for the same record can still vary each time it is scored. In a 2026 study by physicians in Japan, records of physicians and medical students taking a history from an AI-simulated patient were scored by generative AI and by five physicians with the same scoring rubric, and the spread of scores for the same record (the coefficient of variation) was compared. For the AI, this is the spread when the same model re-scores; for the physicians, it is the spread between five different people, so the two are different kinds of variation.

The study did not report the spread for each item (axis), and for some records the AI’s spread was about the same as, or slightly above, the physicians’. It also did not measure how much scores move when the same physician re-scores, and for the same record, scores differed when a different AI model was used. The study was a preliminary one, using seven people’s records of a single history-taking case, so its results should be applied to employment interviews with caution. Separately from interviewer strictness, AI scores can also reflect the AI’s own tendencies in reading answers and biases linked to candidates’ attributes or ways of speaking, and these biases do not necessarily affect everyone equally. The next section explains how to compare AI scores with your own interviewers’ view, and the section “When reviewing, check score distributions and bias in the records” explains how to check for bias.

Sources: Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264-276. https://doi.org/10.1111/ijsa.12036; Takahashi, H., Shikino, K., Kondo, T., et al. (2026). AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI-Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education, 12, e81673. https://doi.org/10.2196/81673

How to check

Check whether AI scores can be trusted from two angles

These two checks can be made at different stages.

When Proomi scores an interview, the AI reads the evaluation criteria your company has written and scores directly from the interview transcript. For each axis it quotes the candidate’s statements that support the score, and an axis with no supporting statement can be recorded as “no evidence”. The AI model and its built-in instructions also shape how the criteria are read.

The Proomi admin console can show the AI’s evaluation and a human reviewer’s evaluation side by side, axis by axis, in a table and a radar chart. Human scores vary too, so ideally have two or more interviewers score each interview. On an axis where scores diverge, read the record, check which reading fits the criteria better, and rewrite the criteria if they are unclear.

Agreement between people and the AI tells you only whether the AI has captured your company’s standards. To check whether the evaluation itself is right, compare scores with performance after hiring.

The purpose of the free trial is to check, on real roles, whether the questions and axes fit your company’s standards. We explain the scope of our hands-on support, including backtesting, when you apply for the trial and when we prepare your quotation.

Research

Interviews with set questions and a set way of rating predict performance better

In 2022, Sackett and colleagues re-examined earlier meta-analyses of how well selection methods, including human interviews, predict job performance. The paper’s figures for structured interviews (where the questions and the way answers are rated are fixed in advance) and for unstructured interviews are as follows.

The values for structured interviews vary widely from study to study, however, and the lower bound of the 80% credibility interval was 0.18. The paper does not identify the cause of this variation; in our view, the content of the evaluation criteria is one of the factors.

The paper also notes that the predictive validity of interviews whose administration or scoring is automated needs to be verified afresh. For a company introducing AI interviews, the next step is to compare selection scores with performance reviews after hiring, axis by axis.

Performance reviews exist only for people who were hired, so the correlation you calculate in-house tends to come out lower than the paper’s corrected values, and with few cases it fluctuates widely. Build the data up over several years; retention data can supplement performance reviews.

Source: Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107, 2040-2068. https://doi.org/10.1037/apl0000994

Case

To pass experts’ judgement on to AI, first capture it in words or data

According to a casebook from the Kansai Economic Federation, at Tsuyakin, a dyeing and finishing company in Gifu Prefecture, whether a fabric’s colour passed used to depend on the intuition of experienced staff. Tsuyakin turned 2,000 past decisions into data and had AI learn the experts’ pass or fail decisions. As a result, for some decisions, junior staff no longer needed to check with experienced colleagues, and decisions that could take five to ten minutes because of that checking could be made in one minute. By our calculation, that saves up to about nine minutes.

The 2022 casebook notes, however, that the AI’s judgement had not reached the level of the experienced staff, and that it served as a reference for junior staff’s decisions. Tsuyakin had AI learn from past decisions; in Proomi, the criteria are written in words. The way judgement is handed over differs, but both start by getting the experts’ judgement out of their heads. An experienced interviewer’s sense of “this is how a strong performer talks” can likewise become the starting point for handing it to other interviewers or to AI interviews, once it is written down as criteria or kept as example ratings.

Source: Kansai Economic Federation, ICT, IoT and AI Case Studies, Tsuyakin: Passing on dyeing judgement (2022, in Japanese; titles translated by us) https://www.kankeiren.or.jp/iot/pdf/iot64.pdf

How to write

Write criteria as evaluation axes and level definitions

An axis sets what the interview looks at; a level definition sets what kind of answer earns which score. Below is an example, made for illustration, of definitions for the axis “Gets results by bringing others along” (three of five levels).

Writing “how they worked things out and reached agreement” into the 5-point definition also settles what to ask in the interview.

One thing to watch for when writing is unintended bias introduced by the wording. Suppose you assess all candidates on the same axes regardless of role, as in graduate hiring for generalist roles, and two candidates answer as follows on the axis “Gets results” (an illustrative example). Candidate A, from sales, says “I grew sales in my territory by 20% on the previous year”. Candidate B, from general administration, says “I reworked our internal application process, and applications sent back for correction almost stopped”.

If you add that figures, reactions from the people involved or changes to the work all count, A and B can both reach 5 points on the same terms. If figure-based results are genuinely needed for a role, write that into a separate axis for that role.

Preparing

Bring in your own criteria, or build them from scratch with AI support

There are two broad ways to prepare evaluation criteria: bring in criteria you already have, or build them from scratch with AI support.

What to check when bringing in performance review items is covered in Use your performance review criteria in interviews, word for word.

“HR writes the criteria” in the title means that, even when AI writes the draft, HR decides what to change and what to keep, and is accountable for the criteria. No one has a better feel for “this is how a strong performer talks” than interviewers from the business, so have them read the criteria and add examples of real answers to the level definitions. The AI draft is ready almost at once, and you can also choose hands-on support from our staff in building the criteria.

Improvement cycle

Refine the criteria by reviewing them in use

Criteria are not written once and left alone. You use them in interviews, compare human and AI scores, and rewrite them, over and over, bringing them closer to your company’s standards.

Score again with the rewritten criteria, and compare

Over the first few dozen interviews, people score the same interviews as the AI, so expect the workload to rise for a while just after rollout. During this period, do not use the AI’s scores as an input to pass or fail decisions; base decisions on human evaluations only.

With our hands-on support, repeatedly re-scoring past interviews with rewritten criteria (backtesting) can raise agreement between human and AI scores relatively early. If you fit the criteria too closely to human scores, the human side’s biases are copied into the AI as well, so read the records of the interviews where scores diverged before rewriting. When you recheck agreement, compare against scores agreed by two or more interviewers, using interviews that were not used for the rewrite. Re-scoring past interviews and checking scores against whether hires are still with the company should stay within the purposes and retention periods you notified candidates of; check this with your legal team as well.

Once the criteria settle, human scoring moves to spot-checking a sample of interviews. After you start using the AI’s scores as an input to decisions, switch to revised criteria at a changeover date set in advance (see below). Even after that, recruiters decide the outcome, reading the AI’s scores and the records.

These are the ways to rewrite the criteria in Proomi. For what each plan includes, see the pricing page. The setup fee depends on the scope of onboarding support and is given in a quotation.

Reviewing

When reviewing, check score distributions and bias in the records

When you compare human and AI scores, there are three more things worth looking at.

To check for bias, look for axes where only candidates with a particular background score low, reading the quoted statements alongside the scores. You can also compare score distributions by attributes your company lawfully holds, such as gender, without using information that should not be collected in selection. Also check the records for transcription errors caused by dialect or speaking style. The quickest way to notice bias is to have someone who has seen many interviews read the records, in addition to the person who wrote the criteria.

Decide in advance when you will revise the criteria: between rounds if you recruit in rounds, or every three months, for example, if you recruit all year. HR should keep a record of the date and content of each change and look at scores before and after it separately. Decide in advance, too, how to treat candidates turned down before a change (for example, consider re-evaluating them if the change corrects a bias, and keep the earlier results if it only tidies the wording).

HR work

Spend the time AI interviews free up on refining the criteria and on final decisions

When AI interviews take over scheduling and conducting first-round interviews and collating evaluation sheets, we recommend spending the time this frees up on refining the criteria, reading the records and comparing them with your own assessment, making final decisions, and attracting candidates. Just after rollout, people also score the same interviews as the AI, and later they spot-check a sample, so plan for that time as well. People decide the outcome, and the AI’s score is one input to that decision.

In one of our case studies, where AI was brought in mainly for the steps from document screening to first-round interviews, a staffing group listed on the Prime Market of the Tokyo Stock Exchange saw the following results and now spends the time freed up on final decisions and placements. The 32% and 28% are figures published on our case studies page and measure something different from how far scores can be trusted.

Source: Prance, Case studies https://prance.co.jp/en/case-studies/

Limits

Keep using your own data to check whether AI scores can be trusted

Comparison points

When choosing an AI interview service, check who writes the criteria

Based on publicly available information, we broadly group services into four approaches to writing criteria, each suited to different situations.

Proomi takes the “write your own” approach: you write the criteria in your own words, and the effort of writing is reduced by a draft that AI makes from the experience level and the job title, and by our hands-on support where needed. If you opt for the “write your own” approach, here are five points to check when choosing a service.

Q&A

Frequently asked questions

Can we write the evaluation criteria ourselves?

Yes, both the evaluation axes and the definition of each level, and you can also start from a draft made by AI. There are three ways to review them: rewriting them yourself in the admin console, using the operations review on the Growth plan (which requires data on whether hires are still with the company), and working with us. We explain the scope of our hands-on support when you apply for the trial and when we prepare your quotation.

How long does it take to write the criteria?

The AI draft is ready almost at once, so HR can start by editing it. Just after rollout, people score the same interviews as the AI, so the effort rises for a while. The time needed depends on the role and the number of axes, so the most reliable way to measure it is on real roles during the free trial.

If the AI’s score differs from the interviewer’s, which one counts?

People decide the outcome. Read the record and the quoted statements, check which reading fits the criteria better, and if the criteria are unclear, revise them.

Can we decide pass or fail on AI scores alone?

Have a person decide, using the score as one input. Proomi has no function that decides pass or fail automatically from scores; the outcome is entered by your own staff.

If we change the criteria, can we still compare against past scores?

Keep a record of the date and content of the change and look at scores before and after it separately. You can also re-score past interviews with the new criteria and compare (a backtest); this is done with our hands-on support.

Related

Read next

For how to write and review criteria, these articles may also help.