Question design and AI interviewer behaviour: keep improving them from the records

Introduction

What this article covers

In public surveys, candidates’ complaints about AI interviews sound like this: “I was asked the same question again and again,” “The interviewer moved on before I had finished,” “I couldn’t tell how it was going.” They seem to be less about what is asked than about how the conversation goes. Five aspects of how an AI interview is run are likely to shape that: how the questions are put together, how the AI interviewer conducts the conversation, how it reacts while the candidate is speaking, how it looks and sounds, and what candidates are told before and after the interview. This article covers those five aspects, a way of improving your questions and the AI interviewer’s settings by reading interview records, and the three ways to review them in Proomi: on your own in the admin console, with Prance’s operations review, and with Prance for finer adjustments.

Key points

What this article says

1. In public surveys, complaints about AI interviews include “I was asked the same question again and again,” “The interviewer moved on before I had finished,” “I couldn’t tell how it was going,” and “I didn’t know how I was being evaluated.”
2. Five aspects are likely to be involved: how the questions are put together, how the AI interviewer conducts the conversation, how it reacts during the conversation, how it looks and sounds, and what candidates are told around the interview.
3. The same complaint can have different causes, in the questions or in the AI interviewer’s behaviour, and you only see which by reading the interview records. It works best to improve the questions and the AI interviewer’s settings together, using the records.
4. In Proomi, your team can review the questions, the evaluation criteria, the avatar, the briefing before the interview, and the feedback to candidates in the admin console. The Growth plan adds an operations review. When you need finer adjustments, Prance can handle them individually.
5. Adjusting does not guarantee that the AI interviewer will never cut a candidate off or repeat a question. Check the results in the interview records.

Candidate voices

What candidates complain about

What do candidates who have taken an AI interview complain about, or feel anxious about? Public surveys include the following.

• In a survey of 308 job-seeking students graduating in 2027, published in September 2026 by Preferred Networks, 51.0% of those who had taken an AI interview said they were satisfied with the result. Among those who were not, the comments included “I was asked the same question again and again” and “The interviewer moved on to the next question halfway through what I was saying.”
• In a survey of 328 students graduating in 2027 to 2029, published in May 2026 by Leverages, the top reasons for feeling negative about AI interviews included “It seems my enthusiasm and character will not come across accurately,” “The interviewer gives no reaction, so I can’t tell how it is going and find it hard to talk,” and “It is unclear how I am being evaluated.”

Both are about how the conversation goes, and how the evaluation looks from the candidate’s side. Both surveys were run by companies that provide AI interview services, and neither measured appearance or voice on its own. Even so, they suggest that how candidates take an AI interview depends on more than the content of the questions: it also depends on how the conversation goes and how the evaluation is explained.

Aspects

Five things that shape the candidate’s experience

A candidate’s experience of an AI interview is likely to come from these five aspects working together.

Your team can review the questions on its own. For the other aspects, some can be set by your team in the admin console, and some are adjusted together with Prance. The later sections explain this.

How to improve

Improve the questions and the AI interviewer together

Suppose a candidate says, “I was asked the same question again and again.” There are several possible causes.

You cannot tell which one it is from the comment alone. The interview records show it. A different cause means a different place to fix. If you change the questions, the room for probing and the length of the interview change, so the best way of conducting the conversation changes too. If you change how the AI interviewer conducts the conversation, the same sequence of questions comes across differently. If you fix the questions and the AI interviewer’s settings separately, one fix can undo the other.

We therefore suggest improving the questions and the AI interviewer’s settings together, in this loop.

Keep repeating steps 2 to 5 as you run interviews

Reading the records and making fixes is not a one-off job. Keep at it while you run interviews, and the questions and the AI interviewer’s settings come closer to suiting your roles and your candidates.

Ways to review

Three ways to review your questions and settings

In Proomi, your team can review the questions and interview settings on its own in the admin console. When you need finer adjustments, Prance can help.

Most reviews can be done with the first and second ways. The third is for when there are still points you want to fix after reading the records.

Why

Why finer adjustments are best made with Prance, reading the records

We believe that adjusting how the AI interviewer conducts the conversation and how it reacts during it is not a job you finish by changing one number on a settings screen. There are three reasons.

First, you cannot tell whether it is working without reading the interview records. Whether a question was repeated, or a candidate was cut off, only shows in the transcript or the recording. How readily the AI interviewer should react to what a candidate says also depends on how the candidate speaks and on the quality of the connection. So the work is to read the records, change the settings, and read the records again, several times.

Second, what counts as “better” lives in the experience of your hiring team. Judgements such as “for this role I would like to probe a little more” or “this question only needs to be asked once” come from experience, and are mostly tacit knowledge that has not been put into words. Turning that tacit knowledge into words, as questions and as instructions for the AI interviewer, is the work Prance does together with you.

Third, changing settings midway makes interview conditions differ between candidates. If you change settings while a role is open, candidates before the change and after it are interviewed under different conditions. When you change something, record the date and what changed, and look at the records before and after separately. Changes to settings that cannot be made from your own screen are made by a Prance contact.

Checking

How to check the result of an adjustment

Check the result of an adjustment in the interview records.

• In the transcript, read whether questions are repeated, and whether the interview moves on before the candidate has finished
• In the recording, see whether the AI interviewer waits until the candidate has finished speaking
• Ask the candidates, in a short survey, what they thought

Adjusting does not guarantee that the AI interviewer will never cut a candidate off or repeat a question. Keep checking against the records. We suggest starting small: run it alongside your existing interviews, for some roles, at the first-interview stage, where you receive many applications.

Choosing

Four things to check when choosing an AI interview service

When you choose an AI interview service, check how it handles what affects the candidate’s experience, as well as how easy it is to set up questions and evaluation criteria. Depending on the service, behaviour may be fixed or adjustable, and the support after adoption varies widely.

1. Can the AI interviewer’s way of conducting the conversation (depth of probing, wording) be adjusted for each company? How far can you go?
2. Can you adjust how it reacts during the conversation, how it looks, and how it sounds? If these are fixed, how does it behave?
3. Who makes the adjustments? Do you change them on screen yourself, or does the provider do it?
4. After adoption, will the provider read the interview records with you and help you fix the questions and settings? What does that cover, and what does it cost?

Q&A

Frequently asked questions

Can we change the settings ourselves?

Yes. Someone on your team can set the questions, the evaluation criteria, how the avatar is used, the briefing before the interview, and whether feedback is sent to candidates, in the admin console. On the Growth plan and above, Prance also gives you an operations review summarising interview counts and score variation. How the AI interviewer conducts the conversation and how it reacts during the conversation are adjusted together with a Prance contact (quoted separately). To ask about it, use the contact form.

Is it enough to improve the questions?

If the cause lies in how the questions are put together, fixing the questions alone may be enough. But the cause may also lie in how the conversation goes, in how the AI interviewer looks and sounds, or in what candidates are told. We suggest reading the records, confirming the cause, and then fixing it.

Can feedback be sent to candidates?

Yes. Your administrator decides, for each role, whether it is sent. What the feedback says is adjusted together with a Prance contact. What to tell candidates is decided according to your own policy.

Is the avatar required?

No. You can choose, for each role, whether to use one and which one. Interviews can also be held without an avatar.

If we change settings, will candidates be interviewed under different conditions?

If you change settings while a role is open, yes. When you change something, record the date and what changed, and look at the records before and after separately.

Related pages

Read next

For more on evaluation criteria and on choosing a service, see these pages as well.