Simon Willison

smevals - a small eval suite for evaluating models, prompts, and harnesses

We've been building a new system for running evals against different models, prompts, and harnesses, with the goal of being able to identify the most appropriate small and inexpensive models for different categories of task.

Scraping and analyzing submissions to Terminal Bench 2.0

How we built a Git scraping pipeline to track Terminal Bench 2.0 leaderboard changes, loaded the results into SQLite, and deployed an interactive Datasette Lite explorer via GitHub Actions.

Scenarios: Model- and harness-agnostic test scenarios for demonstrating prompt injection patterns

Introducing Scenarios, a project to simulate prompt injection attacks.