AI Candidate Evaluation: How AI Scores an Interview and Where It Falls Short

An AI candidate evaluation is a score built mostly from what a candidate said. The audio becomes a transcript, a model reads the transcript against a rubric, and out come ratings per question and an overall figure. Some tools add structured items such as a coding task or a short written answer, which are scored separately and tend to be more reliable. Almost everything useful in the score traces back to the words.

That shapes how a hiring team should read the number. It is good evidence about the content of answers, weak evidence about manner, and no evidence at all about several things recruiters care about a great deal. This guide sets out where each line falls and what has to sit between a score and a rejection email.

What does an AI interview score actually measure?

The research on automated video interviews points in one direction. Louis Hickman and colleagues, writing in the Journal of Applied Psychology in 2022, built models that assessed personality from recorded interviews. They found the models tended to be dominated by verbal behaviour, with extraversion the exception, and that models trained on ratings from interviewers showed stronger evidence of validity than models trained on what candidates said about themselves.

In practice, a well built evaluation can tell you whether the candidate answered the question that was asked, whether they named specific systems, numbers, dates and outcomes, and whether their account of a project holds together when a follow up pushes on it. Those are judgments about text against a rubric. They form the reliable core of the score.

The rubric decides how good that core is. It should be written by someone who does the job. For a payroll analyst role, a strong rubric rewards a candidate who walks through how they found and fixed a mismatch between the payroll register and the bank file, step by step. It gives little credit to a candidate who says they are detail oriented and stops there.

Which signals deserve little or no weight?

Confidence is the signal buyers ask about most and the one a system can least supply. A three second pause before an answer can look like hesitation to a model. To an interviewer with some years behind them it often looks like a person thinking before speaking. The audio is identical in both cases.

Some things were never in the input at all. A model cannot know that a two year gap on a resume was spent caring for a parent, or that the candidate left their last job because of a manager whom three other people also left. It cannot tell whether this person will speak up when a project starts to slip. Those questions belong to the human rounds, and a scorecard should leave room for them.

How should an AI score fit on a human scorecard?

Treat the AI round as something that fills in some cells on a scorecard a person signs. A mid level engineering scorecard might carry six criteria. The AI round can supply evidence for three of them, such as technical depth, problem breakdown and relevant experience, with the quote that supports each rating. The other three stay blank until a person speaks with the candidate.

Two habits help once you have read a few hundred of these. First, look at the spread of question scores before the average. A candidate who scores 90, 88, 92 and 30 is often more interesting than one who scores 70 on every question, because the 30 frequently comes from a badly worded question or one outside the role. The average hides that.

Second, watch answer length. Senior people often give short answers because they have explained the same design decision many times and have learned to compress it. A scoring setup that quietly rewards length will push your most experienced applicants down the list, and nobody will notice unless someone reads the transcripts. For how these rounds are usually set up, see our guide on how one way video interviews work.

Where does AI interview scoring break?

Speech to text is the first weak point, because every later step reads its output. Allison Koenecke and colleagues, writing in the Proceedings of the National Academy of Sciences in 2020, tested five commercial speech recognition systems on matched recordings of 73 Black speakers and 42 white speakers. The average word error rate was 0.35 for Black speakers and 0.19 for white speakers. The authors traced the gap mainly to the acoustic models, since it held even when both groups said the same phrases.

The lesson carries over to any applicant pool with a mix of accents and first languages. If the transcript gets roughly one word in three wrong for some candidates and one in five for others, the rubric is scoring two different qualities of evidence and calling it one measure. Before you go live, record twenty colleagues across the accents you actually hire, read the transcripts yourself and count the errors.

Other failure points show up again and again in practice.

• Very short answers give the model too little text, and very long ones dilute the strong parts. Both pull scores toward the middle.
• Poor audio from a shared laptop or a weak mobile connection lowers transcript quality, so the candidate is partly scored on their bandwidth.
• Specialist terms such as statutory forms, accounting standards and product names are often misheard by general speech models.
• Polished answers with no specifics can score well on a loose rubric. A follow up that asks for a number or a date usually exposes them.

Should an AI score rank candidates or filter them out?

Use it to rank. A ranking score changes the order in which a person reviews candidates, and everyone above a generous cutoff still gets read. If the model is wrong about someone, the cost is that a good candidate was read fourth instead of first.

A filtering score removes candidates before any person sees them. When the model is wrong, that candidate is gone and nobody ever finds out, so the mistake never feeds back into the process and never gets corrected. Given the accent gap in transcription described above, a filter applied to transcript based scores falls harder on some groups than on others, whatever the rubric says.

Automatic rejection thresholds are the riskiest setting in most of these products, and they are usually one switch away from being on. Leave them off. If a vendor will not let you switch them off, that tells you a good deal about how the product was designed.

What do regulators expect from the human reviewer?

Under the European Union General Data Protection Regulation, Article 22 gives people the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects. Guidance from the UK Information Commissioner names online recruiting with no human involvement as an example of such an effect. A reviewer who approves every score without reading the evidence does little to change that picture.

The EU AI Act lists recruitment and candidate selection systems as high risk in Annex III. On 29 June 2026 the Council of the European Union approved changes that start those duties for stand alone systems on 2 December 2027. Once they apply, Article 14(4) expects the people overseeing a system to understand its limits, stay aware of automation bias, which the Act describes as the tendency of automatically relying or over relying on the output, interpret what it produces correctly, and be able to disregard, override or reverse that output in any particular case.

Read that list again with a recruiter in mind. It describes a good recruiter reading an AI scorecard, so it is a sensible standard to build to now, wherever you hire.

A checklist for keeping a person accountable

1. Name one person who is accountable for every rejection the system feeds into.
2. Set the tool to rank candidates and switch off automatic rejection.
3. Require the transcript and the supporting quote next to every score.
4. Test transcription on your own accent mix before launch, and again after any vendor model change.
5. Each quarter, compare pass rates by gender, age band, region and education background.
6. Tell candidates what is assessed before the interview, and offer a human interview to anyone who asks.
7. Read the ten lowest scored transcripts every week. If you cannot see why they scored low, rewrite the rubric.
8. Once a year, check scores against how the people you hired are performing twelve months in.
9. Keep decision records for your legal claim period, then delete them on schedule.

None of this needs a large team. It needs one owner and a recurring slot in the calendar. For the wider picture on interview formats and how candidates react to them, see our guide to AI video interviews, and for the data side, candidate data privacy in hiring.

Sources

1. Journal of Applied Psychology, Hickman et al., Automated video interview personality assessments: Reliability, validity, and generalizability investigations, 2022
2. Proceedings of the National Academy of Sciences, Koenecke et al., Racial disparities in automated speech recognition, 2020
3. Information Commissioners Office, What does the UK GDPR say about automated decision making and profiling, 2024
4. European Commission AI Act Service Desk, Regulation (EU) 2024/1689, Annex III: High risk AI systems, 2024
5. European Commission AI Act Service Desk, Regulation (EU) 2024/1689, Article 14: Human oversight, 2024
6. Council of the European Union, Artificial Intelligence: Council gives final green light to simplify and streamline rules, 2026