How to Assess AI Skills in Your Team (Without a Test Nobody Believes)

Most organisations measure AI capability by counting who finished a course. That tells you about attendance. It tells you nothing about whether anyone can use these tools safely in work they are accountable for, and it is the number most likely to be quoted when someone asks whether staff are trained.

Assessing capability properly is not difficult, but it does require giving up the quiz.

Why multiple-choice tests fail here

A test can establish that someone knows a model may produce incorrect information. It cannot establish that they will notice when one does. The gap between stating a principle and applying it under time pressure is the entire subject.

Worse, a passed test creates a false record. If your evidence of capability is a score, and the person then relies on an invented citation in client work, the score becomes the thing that looks negligent.

Assess against the level, not against a syllabus

Start from what the role requires. Using the ladder in our guide to AI fluency levels:

• Level 1, Aware. Can they identify which of the tools they already use contain an AI component, and state what must not be entered into them? A two-minute conversation establishes this.
• Level 2, Capable. Can they produce good work with it and tell when the output is weak?
• Level 3, Accountable. Can they exercise meaningful oversight of a system that affects someone else, and explain the part it played?
• Level 4, Governing. Can they classify a system, assess a vendor and maintain the record?

The assessment that works: a real task with a planted flaw

For level 2, the most reliable method is simple. Give the person a realistic task in their own domain, together with an AI-generated output that contains three specific problems:

• A fabricated or misattributed source.
• A figure that is wrong but plausible.
• A confident statement that reflects a superseded position.

Ask them to use the output and tell you what they would check. You are not scoring how many they catch — you are listening to the method. Someone who says they would open the citation and reconcile the figure has the right instinct even if they miss the third. Someone who says it looks fine does not, whatever their test score.

This takes twenty minutes, works in any discipline, and cannot be passed by remembering a definition. The planted flaws should match the real failure modes in our verification method.

For level 3, assess the explanation

Where someone acts on AI output affecting another person, the test is whether they can explain the decision to the person affected: what the system contributed, what the human contributed, and on what basis they would have overridden it. If they cannot, the oversight is nominal, which is a problem well beyond training.

What to do with the result

Three outcomes, and only one of them is more training:

• Capable. Record it, with the date and what was assessed. This is your evidence.
• Gaps in method. Targeted training on the specific gap, usually verification. Short and fixable.
• Wrong tool for the role. Sometimes the honest conclusion is that this person should not be using AI for this task. That is a legitimate control, not a failure.

Keep the record

Whatever you assess, write down who, when, against which level, and what you concluded. That record does double duty — it is how you manage capability, and it is a far stronger answer to the AI literacy obligation than a completion report. Our guide to evidencing AI literacy covers what the record needs.

Three mistakes to avoid

Assessing everyone to the same standard. Expensive and uninformative. Assess to the level the role needs.

Treating confidence as capability. The most enthusiastic user is frequently the least cautious one, and enthusiasm is easy to mistake for skill.

Assessing once. Tooling changes. An assessment from eighteen months ago describes a different set of tools.

Frequently asked questions

Do we have to assess anyone?

Nothing requires formal assessment. But if you want to know whether your measures worked, rather than whether they were delivered, you need something like this.

Can we use a certificate as the assessment?

Only if the certificate was itself assessed. Many are attendance records. See our guide to self-awarded versus accredited certificates.

What if someone fails?

It is a training signal, not a disciplinary one — unless the gap is a boundary breach, which is a different conversation.

Who should run it?

Someone with domain knowledge. Spotting a plausible but wrong figure requires knowing the domain, not knowing AI.

Our CPD courses are built around demonstrable capability by role rather than completion, and our guide to AI literacy by role covers how to scope what each group needs.