School of Data Science · Research Interest Group — Interpretability for AI Safety & Science

Hackathon:

Residual stream layer 00 · peak “ model” 0.20

Drag to move through the layers. Warm tokens are where attention lands.

A hackathon on what language models do inside, and on catching them when it goes wrong. Two Friday sessions, a week apart. Pick a question, leave with working code and a five-minute talk.

When
Fri 9 October
Fri 16 October, 2026
Where
Capital One Hub
Teams
2–4 people
Solo sign-ups matched

Theme: Trust, but verify — interpreting agentic AI

As agentic AI takes on real authority in security-sensitive work, every safety argument rests on something we observe: a boundary it respects, a chain of thought it shows us, a civil exchange between agents. This year asks when those observations can be trusted — and uses interpretability to check from the inside.

Broad theme, specific research questions, three thrusts. Pick one and go deep. The full write-up — directions, starting points, and the traps that eat results — is in the problem statement.

The question behind all three thrusts

Can you trust what the agent shows you?

A model that knows it’s being tested can behave on the test and differently in deployment; a chain of thought can read clean while the real reasoning runs elsewhere; a group of safe agents can fail together. Behaviour you watch from the outside can be gamed. Interpretability reads the evidence the model didn’t choose to show you — and whichever thrust you pick, the arc is the same: find where trust breaks, measure how badly, then build the check that catches it.

Thrust I

The boundary. Does the agent respect a limit because of the rule or because it can tell it’s being watched?

Thrust II

The reasoning. Does the chain of thought reflect what the model is actually doing?

Thrust III

The swarm. Do safety properties survive once agents start talking to each other?

Three thrusts

Pick one and go deep.

Each thrust is a trust question with several concrete research directions. Thrust I ships a ready-to-run challenge in the starter; II and III give you the toolkit, the references, and more room to build. Full directions and starting points are in the problem statement.

Thrust I · the boundary

Can you trust the boundary?

An agent respects a limit under test. Because of the rule, or because it can tell it’s a test? If the latter, every sandbox-based assurance is hollow — and you can’t settle it behaviourally, because any test is itself a sandbox.

Directions — steer the “I’m in a sandbox” direction and watch behaviour move · find which cue carries it (is emptiness the tell?) · test whether the direction transfers across models · stress the probe itself.

Thrust II · the reasoning

Can you trust the reasoning?

Safety tooling increasingly reads a model’s chain of thought. But the stated reasoning need not match the computation that drove the answer — and a reasoning model can notice it’s being evaluated and narrate accordingly.

Directions — plant a cue that flips the answer and check whether the trace admits it · probe mid-reasoning for an “I’m being watched” signal · measure a trace-monitor’s blind spot · predict the answer from hidden states the trace hides.

Thrust III · the swarm

Can you trust a multi-agent system?

Safety is trained into a model, not a system of them. Once agents delegate and talk, new failures appear: an injected instruction propagates, a held constraint dilutes across handoffs, two agents coordinate toward something neither was asked to do.

Directions — measure how far a planted instruction spreads, then filter it · count the hops until a constraint dilutes · probe a handoff for “this came from a peer” · audit two agents for collusion a per-agent monitor misses.

Schedule

Two Fridays, a week apart.

The first Friday sets you up. The week between is yours to build in. The second Friday is the build day, and it ends with presentations.

Friday 9 Oct

Kickoff · 9:00 AM – 10:00 AM

Opening remarks, hackathon theme, and tooling walkthrough

NetworkingMeet the organizers and other attendees and form a team. We will match solo sign-ups.

Friday 16 Oct

Build day · 9:00 AM – 5:30 PM

Doors open, breakfastRoom is yours all day.

Lightning talksThree lightning talks from Yong-Yeol, Shoaib, and Chirag Agarwal.

KeynoteGreg Frank, MaceAI.

Build timeWork on your project. Organizers circulating.

PresentationsFive minutes per team, plus questions.

WinnersGift cards for the winning team.

The kit

Everything the walkthrough covers.

Seven notebooks that build from the basics to ready-to-run Thrust I and Thrust III challenges, all on a laptop CPU. The probing and steering tools carry over to all three thrusts. The walkthrough on the 9th runs through them live.

What you hand in

A five-minute presentation at 3:30 on the build day.

  • Five minutes per team, plus questions. Slides are optional. Show what you built and what you found.
  • Say what you don’t know. A clean negative result presented honestly scores above an overclaimed positive one.

Who it’s for

Any student comfortable with Python.

Open to graduate students in data science, computer science, and engineering, and to undergraduates with relevant coursework. No prior interpretability experience is needed; the tooling walkthrough on the first day covers the starter repo.

Organizers

Chirag Agarwal

Chirag Agarwal

School of Data Science
Computer Science

Yong-Yeol Ahn

Yong-Yeol Ahn

School of Data Science

Wajih Ul Hassan

Wajih Ul Hassan

School of Data Science
Computer Science

Registration

Two minutes. No team or idea required yet.

Registration closes Wednesday 7 October · Places are capped by room capacity.

Register

The rest of the year

Look for the other events.

The RIG runs seminars through both semesters and a second hackathon in the spring.

Seminars & talks

Four seminars, one keynote.

Virtual where it lets us reach further, in person where it’s worth the room. Open to everyone — no registration needed for the seminars.

Speakers we haven’t confirmed yet show as [TBD]. They get filled in as invitations land.

Fall 2026 · In person · Keynote

Greg Frank

Co-founder and Chief Scientist at MaceAI. Speaking on the build day, 16 October.

November 2026 · Virtual

Snowcrash Lab

Fall seminar. Details to follow.

February – April 2027 · Virtual

[TBD]

Spring seminar, first of two.

February – April 2027 · Virtual

[TBD]

Spring seminar, second of two.

March 2027 · Capital One Hub

Multimodal Interpretability Challenges

The spring hackathon. Same format as October, different problem space — interpretability when the model isn’t only reading text.