Over-Reliance in AI Support: The Air Canada Chatbot Case

TL;DR

Over-reliance means treating a fluent model as if it were a policy owner. In Moffatt v. Air Canada (2024), a chatbot invented a bereavement-fare rule. The Civil Resolution Tribunal held the company responsible. That is safety, not security.

If the bot can change a customer’s rights, a human review path belongs in the policy before launch.

Key Takeaways

• Safety = protect people from the model (accidental harm: bias, hallucinations, leaking PII, over-reliance) — guardrails. Security = protect the model from attackers (adversarial: prompt injection, breaches, exfiltration). Intent is the key distinction.
• If the bot can change a customer’s rights, a human review path belongs in the policy before launch.
• Disclosure (“you are talking to AI”) is not a substitute for a correct answer.

What happened, in original words

A passenger used the airline’s chatbot. The bot described a bereavement fare that did not match the written policy. The dispute was not that a hacker poisoned the model. The dispute was that the company put a model in front of a customer and the model made a promise. The tribunal treated that promise as the company’s.

Why support teams train the wrong cert

Support leaders get sent to prompt-injection workshops. Useful if someone is attacking the bot. Useless if the bot is simply wrong. Hallucination plus over-reliance is a safety pair: the model invented a fact, and the organization had no human who had to confirm fares that move money or rights.

What to put in the support policy

Allowed topics. Forbidden topics (legal rights, refunds above a threshold, medical). Escalation to a named human. Logs. A line that staff must not paste tickets with passports into a public model. Module 0 is where you write that page.

Enter the free classroom

Watch Module 0 — Why AI Safety Is Not Security — with no login. Write and quiz stay in your classroom. Unlock M1–M5 for $199. If you are a deployer, also read the EU AI Act deployer page.