Was your interview evaluation right? You find out after the hire

Terms

What structured interviews and predictive power mean

A structured interview is one where the content of the questions, their order and the criteria for evaluation are fixed in advance and applied to every candidate under the same conditions. Whether the interviewer is a person or an AI, the same questions and the same criteria are used. An AI interview whose questions and criteria are set in advance counts as a structured interview.

Predictive power is how well a score at selection tells you how someone will perform after joining. Research calls it predictive validity and expresses it as the correlation between the selection score and later performance. A correlation runs from −1 to 1. The closer to 1, the more likely it is that people who scored high at selection are rated highly after joining. The closer to 0, the less the selection score has to do with later performance.

Findings

Structured interviews average 0.42 in predictive power, but vary widely by setting

Personnel psychology has long studied the predictive power of each hiring method. In an analysis published in 2022 in the Journal of Applied Psychology, Sackett and colleagues found that structured interviews had the highest predictive power of the methods examined, with an average correlation of 0.42 with later job performance. The research covers structured interviews conducted by people. You need to check for yourself whether an AI interview reaches the same level.

But 0.42 is an average. The same analysis says predictive power differs widely by company and job, even for structured interviews. If structured interviews were used in ten workplaces, nine would be expected to reach 0.18 or more and one to fall below it.

Here is one way to picture the difference. Rank candidates by selection score, highest first, and split them exactly in half. Call the upper half the “high-scoring group” and the lower half the “low-scoring group”. In a workplace where predictive power is 0.42, about 64% of the high-scoring group go on to perform above average after joining, against about 36% of the low-scoring group. People who scored high clearly tend to perform well afterwards. Where predictive power is 0.18, the figures are about 56% and 44%. The gap is slight, and the score alone is a poor guide to later performance.

Source: Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107, 2040–2068. https://doi.org/10.1037/apl0000994. The shares for the two groups are our own conversion, assuming a symmetric, bell-shaped distribution of scores and performance, and are a guide only.

Why they differ

Predictive power varies with interview content, even among structured interviews

The paper points out that structured interviews differ in content. For candidates with no job knowledge or experience, for example, interviews may use situational questions (“what would you do in this situation?”); for candidates who do have it, they may ask about past behaviour (“what did you do?”). The paper says one should not assume the same level of predictive power from the two, because the content of the interviews is quite different.

The paper also accepts that structured interviews have a high average predictive power, but notes that the variation between settings is large and that a low value is a real risk. It says it would be desirable to identify the factors that explain that variation and give more specific estimates by job and by interview design.

A further point: when scores bunch into a narrow range, the link between score and performance looks smaller than it really is.

It follows that an interview’s predictive power depends on which job it is used for, what it contains and how it is scored. Using a structured interview does not by itself mean your interviews predict well. You need to compare your own selection results with post-hire outcomes and check whether the interview actually predicts performance.

Note: this section is based on the discussion in Sackett, Zhang, Berry, & Lievens (2022).

Making use of it

Matching AI interview records to post-hire outcomes can improve evaluation accuracy

Right after each interview, the AI records a score and its reasoning for every candidate on each criterion, in the same format. That record can be matched against post-hire outcomes without further processing.

Matching shows which criteria were tied to success after joining. Criteria that were tied to success give you grounds for weighting them more heavily in the next round of hiring. For criteria that were not, the description or the question is worth revisiting. Producing a score is not the same as being right, so adding this check lets you improve the accuracy of evaluation step by step.

The outline of the check is simple. Split candidates on each criterion into a high-scoring group and a low-scoring group, and see whether their post-hire outcomes differ. If they do, that is a sign the criterion was linked to post-hire outcomes.

Who owns the criteria

To check, the selection criteria must connect to the criteria used after joining

Suppose selection looks at “communication”, while the post-hire performance review has only “proposal skills” and “negotiation skills”. Compare the two and you cannot say which criterion was right. Matching works when the selection criteria connect to the criteria used after joining: defined in the same words, or mapped to one another.

If you define the criteria yourself, the same definitions can be used on current employees. If you use a framework designed by the service provider, you need separate criteria for employees, so hiring and evaluation tend to end up with two separate yardsticks. With two yardsticks, comparing how people were rated at hiring with how they are rated after joining takes a separate step to map one onto the other.

Limits of the check

No firm conclusion in the first year

There are three easy ways to misread the result.

The second and third shrink as you gather more records or compare like with like. Records take time to build up, so the sooner you start checking, the sooner you can draw conclusions.

Q&A

Frequently asked questions

Is there any point if only a few people join each year?

A single year gives no conclusion. Even so, keeping each criterion’s score and each person’s post-hire outcome is worth doing in itself. Once you have about three years of records, you have more to go on in reading the trend for each criterion.

Can I use resignation as the measure?

You can, but it is not conclusive on its own. People leave for many reasons other than how they were assessed at selection. Some leave for reasons beyond their control, such as family circumstances or illness, so a resignation cannot simply be read as a failure of selection. Whether someone is still there is the quickest measure to obtain, so it works well as a starting point; switch to performance reviews once they are available.

Should the recruitment team see performance review results?

It depends on how you run it. They do not need to be passed to recruiters in a form that identifies individuals; aggregating by criterion before matching is enough. The aim is to test the criteria, not to re-assess individuals.

Read next

Criteria that did not hold up can be rewritten and tested on past interviews

Criteria whose scores did not track post-hire outcomes can be rewritten, then tested by re-scoring past interviews. Before asking whether an evaluation is right, also check that its scores tell candidates apart. Checking human and AI scores against each other, before comparing with results after hiring, is covered in another article.

Related pages

How it applies, by situation

Checking against post-hire outcomes looks different in mid-career hiring, new-graduate hiring and goal reviews.