Particle.news
Download on the App Store
3 ARTICLES
·
2d ago

Auditor General Rebukes CRA’s $18 Million ‘Charlie’ Chatbot for Inaccurate Answers

The agency cites a new generative-AI version with near-90% pre-release scores, with real-world accuracy still unverified.

Image
Image
Charlie the Chatbot's "responses tended to be brief, offering limited context and minimal additional information," wrote Auditor General Karen Hogan.

Overview

  • The Auditor General’s test found Charlie answered only 2 of 6 tax questions correctly, while other public AI tools got 5 of 6 right, and its replies were brief and light on context.
  • Parliamentary documents show more than $18 million has been spent on the chatbot since 2018–19, including $13.67 million for salaries and $3.21 million for IT consultants, excluding benefits and travel.
  • The CRA upgraded Charlie in November to a generative-AI system and reported approximately 90% accuracy in pre-release testing, but said confirming live performance would require reviewing every interaction.
  • MPs challenged the CRA over why a specialized government tool performed worse than general-purpose public systems, with Liberal MP Anthony Housefather pressing the discrepancy.
  • The CRA’s 100-day service improvement update reports 70% of users reached their goal in November and says responsiveness nearly doubled after rehiring or extending 1,200 call-centre employees.