This workshop is part of the How to Think about Tech? The Case of 'AI Safety' study group initiated by some of the fellow candidates of the 2025/2026 Introduction to Political Technology course. It is open to faculty and fellowship candidates only - external guests by request (please contact us explaining your interest and area of expertise).

AI companies regularly release new models claiming they are “more capable” and “safer” than previous versions. But how do they know? What gets measured? What gets left out? And how do evaluation choices shape governance decisions?

This workshop provides a high-level primer on AI evaluations (“evals”) -- the practice of assessing AI systems for capability, safety, fairness, and societal impact. We'll explore:

What AI evals are and why they matter
The main categories of evaluation (capability, alignment, safety, sociotechnical)
How benchmarks, datasets, and metrics function in practice
The limits of measurement and the politics of defining “good performance”
(Optional) We’ll work through a collaborative exercise designing an evaluation for a hypothetical AI system to surface the tradeoffs between what we can measure and what actually matters.

Note - this session is designed as a collaborative technical primer. I've been studying and working with evals over the past year as part of an ML curriculum, and I’m excited to share what I’ve learned. If you have deep experience in this area, please reach out as we'd love to learn from you!

This workshop pairs with the AI Safety study group discussion on on AI Evals (https://luma.com/357o5glx) examining the governance and political implications of evaluation frameworks.

Recommended Readings (from the study group):

AI Evaluations - a technical primer workshop