How to choose an AI interview tool

Context

How far the AI goes

Candidates open the URL they are sent and take the interview without waiting for an interviewer to be free. The AI asks questions set in advance and follows up on the answers. It then records an evaluation against the criteria, with its reasoning. This page is about how to decide what to hand over to it.

Scope

How much of hiring an AI interview tool takes on

AI interview tools differ in how much of hiring they take on, from simply running interviews to checking the criteria and carrying them into placement and development. The scope splits into five parts.

Vendors compete on the first two, and that is where they are hardest to tell apart. If the goal is throughput, as in high-volume graduate hiring, the first two are enough. For mid-career and specialist hiring, or hiring that spans several departments and roles, the other three are what make the difference. Buyers are starting to ask who owns the criteria; few yet ask about checking them or carrying them forward.

Note that you can own your criteria even before you have written them. We can build them with you, from the definitions your performance reviews already use and from what your interviewers actually look at. What matters is not who writes the first draft, but whether the finished criteria are yours and whether you can rewrite them afterwards.

AI interview services fall into a few broad types, by what they optimise for. Our article “AI interview services compared” classifies each of the leading services based on what its vendor publishes.

Checklist

What to put in your comparison table

When you build a table to compare services, or write a request for proposal (RFP), put these in as rows. A table that only lines up prices and features leaves out the part of the scope that price cannot compare, described above. Most of these items are there to bring that part into the table.

Comparison item	What to ask each vendor	How to read the answer
01What it was built for	Is the service built mainly to handle more interviews, or to improve the quality of evaluation?	Whether its case studies talk about saving time or about better evaluation
02How criteria are set	Can we write the names and definitions of the criteria in our own words, from scratch?	Whether you can write them from scratch, or only re-weight criteria the service provides
03Measuring current employees	Can current employees take the interview in the same format as candidates?	Whether their results can be compared with candidates’ on the same criteria
04Checking after the hire	How do we confirm that people hired on these criteria performed?	Whether it rests on the vendor’s statistics, on your own analysis, or on matching against your own hiring outcomes and tenure
05Fixing gaps between AI and people	When the AI and an interviewer disagree, who corrects it?	Whether there is a way to notice, and whether you can revise the criteria yourself
06Extending across roles and languages	Can each role have its own criteria? Can Japanese and English interviews use the same criteria?	Whether criteria can go beyond the templates provided

Price, integration with your applicant tracking system (ATS), security, and onboarding and support are basic rows in any comparison table. Vendor materials fill them in, so they are not covered here. These items go below them.

When the answer is “yes”, ask one more question: is that available now? Some services include features under development in their feature lists. If the answer column in your RFP is split into “available”, “in development” and “not offered”, the answers are easier to compare.

1What it was built for

Whether you want more throughput or better evaluation changes which service fits. Most services will say they do both, so look at their case studies rather than the answer. Case studies about cutting interviewer hours and scheduling point to throughput; case studies about consistent evaluation or how hires performed point to evaluation quality.

If you skip this: If you have not settled your own goal, the feature list just grows longer, and in the end you choose by the number of rows.

2How criteria are set

Every vendor will say “customisable”. What to find out is whether you can define the criteria themselves, or only adjust the weighting within the criteria provided. Ask whether there is a limit on the number or naming of criteria, and whether different roles can have different criteria. Brochure wording rarely shows the difference, so ask to see criteria being written in the admin console.

If you skip this: After adoption, if you can only change weights, you cannot add the points your own interviewers care most about.

3Measuring current employees

Most services are designed for use within selection. If current employees can take the same interview as candidates, you can check with real data which criteria your high performers score well on.

If you skip this: Whether the criteria fit your high performers cannot be checked until the people you hire have been in the job for a while.

4Checking after the hire

The answers fall into three groups: the vendor checks with its own statistics; the vendor hands back the data and you check; or you match results against your own hiring outcomes and tenure. If you want to check the criteria against your own hiring results, ask whether the third is possible.

If you skip this: Interview volume grows with no way to tell whether the criteria were right.

5Fixing gaps between AI and people

As you keep running interviews, the AI and your interviewers may come to disagree. What to look at is whether there is a way to notice, and whether you can revise the criteria definitions yourself once you have. If you have to wait for the vendor to update its model, you cannot decide when the fix comes.

How to test rewritten criteria on past interviews is covered in “Rewrite your criteria, then test them on past interviews”.

If you skip this: You notice the gap but cannot fix it, and interviews carry on against criteria that do not fit.

6Extending across roles and languages

Ask whether each role can have its own criteria, and whether Japanese and English interviews can run without separate standards. If criteria can only move within the templates provided, the more roles and languages you add, the more interviews run on criteria that do not fit.

If you skip this: It works for the first role, but when you extend to a second, you are left choosing between criteria that do not fit and looking for another service.

Standardisation

How to keep evaluation aligned

Evaluation is standardised once you have checked, in order, that everyone is assessed against the same criteria and that those criteria connect to your hire/no-hire decisions and to retention after joining, and you can correct them where they drift. It does not just mean using the same list of criteria. The checks come in three stages.

Stage one shows up in the score distribution per criterion. The smaller the standard deviation, the more the AI tends to give candidates similar scores. Work out whether the criterion is vaguely worded, the questions fail to probe it, or the candidates really are alike, and you can correct the design.

Show the numbers

Criterion	1 pt	2 pt	3 pt	4 pt	5 pt	Std. dev.	Modal share
Intent to stay	12	63	68	30	2	0.87	39%
Communication	4	19	64	64	24	0.94	37%
Adaptability	7	36	67	49	16	0.99	38%
Logical thinking	9	33	79	37	17	0.99	45%
Initiative	13	29	62	48	23	1.10	35%
Domain knowledge	19	44	54	36	22	1.18	31%

The figure on each bar is the share taken by the most common score. Criteria are sorted by standard deviation, smallest first.

Next to read

The leading services compared, and the drawbacks of AI interviews

A separate article classifies the leading services by what they optimise for, based on what each vendor publishes, and compares them on questions, criteria and verification, without ranking them. The same article covers the pros and cons of AI interviews themselves. Another article covers how to keep improving the candidate experience after you choose. Comparing services by who writes the criteria is covered in a column as well.