AI incidents, audits, and the limits of benchmarks
· 43 min · Episode 346
Episode AI incidents, audits, and the limits of benchmarks
AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEFCON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice.
Featuring:
• Sean McGregor– LinkedIn • Chris Benson – Website, LinkedIn, Bluesky, GitHub, X • Daniel Whitenack – Website, GitHub, X
Links:
• AI Verification & Evaluation Research Institute • AI Incident Database • 38th convening of IAAI • BenchRisk • State of Global AI Incident Reporting
Upcoming Events:
• Register for upcoming webinars here!
Links in the notes 15
- https://www.linkedin.com/in/seanbmcgregor/
- https://chrisbenson.com/
- https://www.linkedin.com/in/chrisbenson
- https://bsky.app/profile/chrisbenson.bsky.social
- https://github.com/chrisbenson
- https://x.com/chrisbenson
- https://www.datadan.io/
- https://github.com/dwhitena
- https://x.com/dwhitena
- https://www.averi.org/
- https://incidentdatabase.ai/
- https://aaai.org/conference/aaai/aaai-26/iaai-26-call/
- https://www.youtube.com/watch?v=rE17hShOF_Q
- https://www.youtube.com/watch?v=BOjTPSimNJk
- https://practicalai.fm/webinars
Chapters
Transcript
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.