How to Assess AI Skills in Your Team (Without a Test Nobody Believes)

Completion rates measure attendance, not capability. A practical way to assess AI skills using real work, and what to do with the result.

Learnsignal Education Team
21 Oct 2026
6 min read
Updated

Most organisations measure AI capability by counting who finished a course. That tells you about attendance. It tells you nothing about whether anyone can use these tools safely in work they are accountable for, and it is the number most likely to be quoted when someone asks whether staff are trained.

Assessing capability properly is not difficult, but it does require giving up the quiz.

Why multiple-choice tests fail here

A test can establish that someone knows a model may produce incorrect information. It cannot establish that they will notice when one does. The gap between stating a principle and applying it under time pressure is the entire subject.

Worse, a passed test creates a false record. If your evidence of capability is a score, and the person then relies on an invented citation in client work, the score becomes the thing that looks negligent.

Assess against the level, not against a syllabus

Start from what the role requires. Using the ladder in our guide to AI fluency levels:

  • Level 1, Aware. Can they identify which of the tools they already use contain an AI component, and state what must not be entered into them? A two-minute conversation establishes this.
  • Level 2, Capable. Can they produce good work with it and tell when the output is weak?
  • Level 3, Accountable. Can they exercise meaningful oversight of a system that affects someone else, and explain the part it played?
  • Level 4, Governing. Can they classify a system, assess a vendor and maintain the record?

The assessment that works: a real task with a planted flaw

For level 2, the most reliable method is simple. Give the person a realistic task in their own domain, together with an AI-generated output that contains three specific problems:

  • A fabricated or misattributed source.
  • A figure that is wrong but plausible.
  • A confident statement that reflects a superseded position.

Ask them to use the output and tell you what they would check. You are not scoring how many they catch — you are listening to the method. Someone who says they would open the citation and reconcile the figure has the right instinct even if they miss the third. Someone who says it looks fine does not, whatever their test score.

This takes twenty minutes, works in any discipline, and cannot be passed by remembering a definition. The planted flaws should match the real failure modes in our verification method.

For level 3, assess the explanation

Where someone acts on AI output affecting another person, the test is whether they can explain the decision to the person affected: what the system contributed, what the human contributed, and on what basis they would have overridden it. If they cannot, the oversight is nominal, which is a problem well beyond training.

What to do with the result

Three outcomes, and only one of them is more training:

  • Capable. Record it, with the date and what was assessed. This is your evidence.
  • Gaps in method. Targeted training on the specific gap, usually verification. Short and fixable.
  • Wrong tool for the role. Sometimes the honest conclusion is that this person should not be using AI for this task. That is a legitimate control, not a failure.

Keep the record

Whatever you assess, write down who, when, against which level, and what you concluded. That record does double duty — it is how you manage capability, and it is a far stronger answer to the AI literacy obligation than a completion report. Our guide to evidencing AI literacy covers what the record needs.

Three mistakes to avoid

Assessing everyone to the same standard. Expensive and uninformative. Assess to the level the role needs.

Treating confidence as capability. The most enthusiastic user is frequently the least cautious one, and enthusiasm is easy to mistake for skill.

Assessing once. Tooling changes. An assessment from eighteen months ago describes a different set of tools.

Frequently asked questions

Do we have to assess anyone?

Nothing requires formal assessment. But if you want to know whether your measures worked, rather than whether they were delivered, you need something like this.

Can we use a certificate as the assessment?

Only if the certificate was itself assessed. Many are attendance records. See our guide to self-awarded versus accredited certificates.

What if someone fails?

It is a training signal, not a disciplinary one — unless the gap is a boundary breach, which is a different conversation.

Who should run it?

Someone with domain knowledge. Spotting a plausible but wrong figure requires knowing the domain, not knowing AI.

Our CPD courses are built around demonstrable capability by role rather than completion, and our guide to AI literacy by role covers how to scope what each group needs.

This page was last updated:

Learnsignal Education Team

Expert Tutor at Learnsignal

Qualified professional with years of experience helping students advance their professional careers.

View all posts by Learnsignal Education Team

Subscribe to Our Newsletter

Join over 30,000+ Learnsignal students and get regular insights delivered to your inbox.

Related Articles

Unfair Dismissal from 1 January 2027: Six Months, Not Day One
Workplace & HR Compliance

Unfair Dismissal from 1 January 2027: Six Months, Not Day One

Almost all commentary on unfair dismissal under the Employment Rights Act 2025 says day-one rights with a statutory probation period. That is not what is happening. From 1 January 2027 the qualifying period falls from two years to six months, the compensation cap is removed, and the commencement regulations are already made. Here is the transitional rule, what it means for your probation policy, and what to do before January.

Learnsignal Education Team10 min read
Fire and Rehire Under the Employment Rights Act 2025: The Date That Actually Matters Is January
Workplace & HR Compliance

Fire and Rehire Under the Employment Rights Act 2025: The Date That Actually Matters Is January

March 2027 is a Government intention with no commencement instrument behind it. The deadline that will catch employers is 1 January 2027, when the compensatory award cap disappears.

Learnsignal Education Team11 min read
Early Years Employment Regulation Orders: The Seven Rates Irish Providers Must Pay
Workplace & HR Compliance

Early Years Employment Regulation Orders: The Seven Rates Irish Providers Must Pay

Two statutory instruments set minimum pay across the sector from 13 October 2025. Only two of the seven rates depend on a qualification, and the WRC page still shows the old figures.

Learnsignal Education Team10 min read
School-Age Childcare Has No Qualification Requirement: What S.I. 575/2018 Actually Says
Workplace & HR Compliance

School-Age Childcare Has No Qualification Requirement: What S.I. 575/2018 Actually Says

The Level 5 requirement does not reach school-age services or childminders. The law asks only that staff be suitable and competent, yet a Level 7 still unlocks a higher statutory pay rate.

Learnsignal Education Team9 min read

Ready to Start Your Learning Journey?

Join thousands of successful students who have achieved their qualifications with Learnsignal.

Ready to get started?

Join 100,000+ students across 130 countries. Choose a plan that fits your goals — cancel anytime.

View plans