Utah's tax chatbot pilot exposed the non-deterministic problem every enterprise faces: initial LLM accuracy hit 65-70% when judged by expert panels, with another 20-25% partially correct. After months of iteration, three of four vendors delivered strong enough results for Utah to make a vendor selection and begin production deployment. Christian Napier, Director of AI for Utah's Division of Technology Services, explains why the gap between proof of concept and production is where AI budgets and timelines collapse.His team deployed Gemini across state agencies with over 9,000 active users collectively saving nearly 12,000 hours per week. Meanwhile, agency-specific knowledge chatbots struggle with optional adoption, competing against decades of human expertise.The bigger constraint isn't technical. Vendor quotes for the same citizen-facing solution dropped from eight figures to five during negotiations as pricing models shifted. When procurement cycles run 18 months and foundation models deprecate quarterly, traditional budgeting breaks.Topics discussed:Expert panel evaluation methodology for testing LLM accuracy in regulated tax advice scenariosLow-code AI platforms reaching capability limits on complex use cases requiring pro-code solutionsAvoiding $5 million in potential annual licensing costs through Google Workspace AI integration timingTracking self-reported productivity gains of 12,000 hours weekly across 9,000 active usersAI Factory process requiring privacy impact assessments and security reviews before any pilotsVendor pricing dropping from eight-figure to five-figure quotes as commercial models evolvedForcing adoption through infrastructure replacement when legacy HR platform went read-onlySeparating automation opportunities from optional tools competing with existing workflowsDigital identity requirements for future agent-to-government transactions and authorizationRegulatory relief exploration for AI applications in licensed professions like mental health

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related