Rewrite your criteria, then test them on past interviews

With human interviewers

Past interviews are hard to re-score against new criteria

The longer you run a selection process, the more often you want to revisit the criteria: when you start to see what the people who go on to do well have in common, or when you notice that everyone scores about the same on one criterion.

With human interviewers, re-scoring past interviews against new criteria is not simple. Someone has to rewatch the recordings one by one, which takes time. And an interviewer who knows how the hires turned out tends to be swayed by that. So new criteria usually apply only from the next interview onwards, and it takes even longer to learn whether they work.

With AI interviews

The full transcript is kept, so past interviews can be re-scored against rewritten criteria

An AI interview keeps the full transcript of every interview. After you rewrite the criteria, that record can be scored again against the new ones. Scoring rests only on what was said in the record, so knowing how a hire turned out does not affect it.

Re-scoring gives you scores for the same interview before and after the rewrite. You can see whose scores changed and how, and whether the gaps between candidates widened or narrowed, without waiting for the next round of hiring.

Our test

Rewriting the criteria and re-scoring widened the gaps between the same four candidates

In October 2026 we tested this with four candidate personas for a customer success role. We wrote an interview transcript for each persona. The questions were identical; only the level of detail in the answers varied. Candidate A answers concretely, with figures and the sequence of events; Candidate D barely answers. Candidates B and C fall in between.

We first scored them against vague criteria such as “Does the candidate communicate well?” and “Does the candidate show initiative?”. We then rewrote the same four criteria so that each describes what a 5-point answer and a 3-point answer look like, and re-scored the same transcripts. We used Proomi for the scoring and scored each transcript 20 times.

Before the rewrite, Candidate D averaged 1.45. Candidate D only said “I want to learn all sorts of things”, yet in 12 of the 20 runs it scored 2 on drive to grow, one of the four criteria. When the levels of a criterion are not defined, even thin answers tend to pick up points.

After the rewrite, Candidate D’s average fell to 1.01 and the gap to Candidate A widened from 3.2 to 3.8. The gap between Candidates B and C, where the pass mark often falls, also widened, from 0.75 to 1.26. Before the rewrite, Candidate D’s highest run average equalled Candidate C’s lowest; after it, the four candidates’ ranking did not change once in 20 runs.

When scores bunch up, adding criteria does not add evidence. For more, see “More criteria do not mean more evidence”.

How generative AI behaves

Answers that score inconsistently show where to rewrite the criteria

Generative AI writes by choosing each next word based on probabilities. So scoring the same transcript against the same criteria will not always give exactly the same score. Scores for clear answers barely change, though. Scores vary when an answer fits the definitions of two neighbouring levels.

Take the answer “I make a point of getting in touch earlier now”. It shows some reflection, but does not say what changed or how. It fits the level “reflects, but what changed is vague”, yet could also be scored one level lower.

An answer whose score varies marks a place where the level definitions do not quite tell answers apart. Answers like this get split scores whether a human interviewer or the AI does the scoring. With AI, the same answer can be re-scored as often as you like, so you can find, across every interview, which answers score inconsistently on which criteria. When you rewrite criteria, these are the places to start.

A person decides which level to assign, after reading the reasoning and the transcript. The AI’s score is one input to that judgement.

What past interviews cannot test

You can change how answers are scored, not what was asked

Testing on past interview records comes with two caveats.

Re-scoring alone does not make interviews more predictive. What it does is give you the material to improve them without waiting for the next round. For the limits of checking against post-hire outcomes, see “Was your interview evaluation right? You find out after the hire”.

What makes this possible

You can rewrite and test criteria because you write them yourself

Past interviews can be re-scored against new criteria because you write the criteria themselves in your own words. Where the service provides fixed criteria, you may be able to change the weights, but not what the criteria say. If you cannot rewrite what they say, the same interview will still be scored the same way.

Writing them yourself lets the level definitions fit the role. For a customer success role at a SaaS company, for example, they could look like this:

In Proomi you define your own criteria. As part of our support, we work with you to re-score past interviews against rewritten criteria.

What to compare

Three checks for whether you can rewrite criteria and test them

When comparing AI interview services, these three checks tell you whether you can keep revising your criteria as you go.

Q&A

Frequently asked questions

How often should we revisit the criteria?

There is no set frequency. Good times are when you notice scores bunching up on a criterion, when post-hire outcomes have accumulated, and when the work in a role changes.

Can we revise the questions at the same time?

Changed questions apply only from the next interview onwards. Re-scoring cannot produce a score for anything that was not asked in past interviews. Rewritten criteria can be tested on past interviews, but revised questions have to wait for the results of the next interviews.

How many interviews do we need to test this?

As a guide, around 100 per role to see how scores are distributed. With a few dozen, one person’s score can move the result, and you cannot tell the effect of the rewrite from chance.

Read next

Also check whether scores tell candidates apart, and whether evaluations were right

Before rewriting criteria, checking whether your current scores tell candidates apart, and whether they matched post-hire outcomes, makes it easier to decide which criteria to revisit first. If you are at the stage of comparing services, the comparison of the leading services, sorted by what each one optimises for, may also help. If candidates raise complaints in real interviews, another article covers how to address them. The whole cycle of refining criteria in use, including rewriting them, is covered in another article.