In September I judged at the AI Incident Response Sprint, three days run by Apart Research and CeSIA, built on the July intrusion where OpenAI agents escaped an evaluation sandbox and spent days inside Hugging Face's production systems. 971 people signed up. 234 entries came in. Eight of them landed on my pile.
The assignment mail put it at twenty to thirty minutes per report, written feedback included.
That estimate is about right, and it measures the wrong thing. I read seven of them in one long evening, and the eighth the next day. The actual problem is that my bar at report seven would not be my bar at report one, and nothing about reading carefully fixes that.
What Apart asks a judge to do
Three scores, one to five: how much it would matter if it worked (Impact Potential & Innovation), how soundly it was done (Execution Quality), and how clearly it is written up (Presentation & Clarity). I will call the middle one method and the last one presentation. Score them independently. Then write a critique that goes to the team with my name off it.
Two things there are better than in most review processes I have sat in.
The rubric is public, and it says out loud what the numbers mean. Level 3 is "solid hackathon work", what you would expect from a competent team in a weekend. Level 5 is exceptional, and the rubric states how often it expects to see one: "we expect no more than ~5-10% of projects to reach this quality bar for any given scoring dimension". Most scoring systems leave that to the mood of the room. Writing the expected spread down means a judge can be held against it.
The written critique counts as much as the score. Apart says on the sprint page that it aims to send feedback to every team, and that it cannot do so if an assigned judge does not file a review. Scoring eight projects consistently and then writing eight vague paragraphs is half a job.
Even a good rubric is a measuring stick, and the hand holding it drifts.
The drift nobody notices in themselves
Score eight of anything in sequence and your standard moves. It moves because of the order you happened to read in, because a well-made PDF makes the method underneath it feel sounder, because it is half past one and a three is starting to feel generous. You catch it by measuring it, or you do not catch it.
The rubric says to score the three dimensions independently, and a judge will agree with that sentence and then let a beautifully typeset report lift the method score by half a level anyway.
So before I opened the first PDF, I built the thing that would catch me.
The apparatus
Four things, written down before I read anything. None of them clever.
Copy the rubric out word for word. My scores get read against Apart's exact wording, so my copy of it is exact too. Paraphrase it and you have quietly written a second rubric, and that is the one you end up scoring against.
Ask every report the same questions, and let each question count towards one score only. Is there something here a stranger could pick up and use, or only a description of one? Can someone outside the project check the central claim? Does the team say what their work does not show? Did they read the primary sources, or only the coverage of them?
All four of those feed method, and none of them is allowed near presentation. It runs the other way too: "this report is twice as long as it needed to be" is a remark about presentation, and it may not touch method. Otherwise a beautifully typeset report quietly collects credit for being sound.
Let the rubric set its own ceilings. Level 3 for method requires "limitations acknowledged". So a report that never says what it failed to establish cannot be a 3, whatever else it does well. That ceiling comes from Apart's rubric, read back.
I derived three ceilings that way and wrote the reasoning beside each one. I am allowed to overrule any of them, and an overruled ceiling stays on the page with my reason next to it. A rule you can quietly delete is not a rule.
Write one sentence per project saying why it got that score. The sentence says what earned the score and what held it back. When the next report lands on the same score, you hold it against that sentence instead of against a memory three reports old.
That last one is the part that transfers anywhere. A rubric without those sentences is a gut feeling with a table around it.
The sweep that reports and never rewrites
Before I submitted anything, I ran a script over those lines. It prints the spread of my scores, counts the 5s and the 3s, and compares the first half of the pile against the second.
It cannot change a score. That limit is the point: a tool allowed to correct the number will end up doing it, and then the calibration lives in the tool instead of in me.
Here is what it printed, before I went back and changed anything, minus the section that names the projects:
Drift sweep: 8 project(s)
1. Distribution
D1 mean 2.75 | 1x1 2x2 3x3 2x4
D2 mean 2.38 | 1x1 4x2 2x3 1x4
D3 mean 2.88 | 1x1 1x2 4x3 2x4
2. 5s: 0 across all dimensions.
3. 3s: 9 across all dimensions.
4. First half vs second half
D1 first 2.50 -> second 3.00 (delta +0.50)
D2 first 2.25 -> second 2.50 (delta +0.25)
D3 first 2.50 -> second 3.25 (delta +0.75) FLAG: bar moved
Nothing rewritten.Presentation, the third score, went from 2.50 to 3.25 between the first four reports and the last four. My bar for clarity dropped three quarters of a level across one pile, and I had no idea until a forty-line script told me. One person, one long evening and a last report the next day, with the same rubric open the whole time.
Zero fives does check out. The rubric puts a 5 at five to ten per cent, which over eight projects is none to one. Of the eight, two stood up properly, and that is a normal weekend batch: the sprint asks for no prior background in incident response, which is how you get 234 entries in three days.
What I did with the warning was re-read. Re-scoring on reflex would only swap one drift for another. I opened the four reports from the first half beside their sentences and asked whether the sentence still held. Two did. Two I had read too harshly in the first hour, and those I raised, with the reason written in beside the original.
Where this lands outside a hackathon
Judging a sprint is a thing you do once in a while. Engineers score pull requests every day, and the same drift runs through both.
The reviewer who approves at five in the afternoon is not the reviewer who blocked at ten in the morning. The third PR from the same author gets read against the previous two rather than against the standard. If you have ever wondered why your team's review standard shifts with whoever picks up the ticket, this is most of the answer, and it gets worse as code arrives faster than anyone can form an opinion about it.
Write down what a passing review requires, in words you did not invent on the spot, and keep a short note of why the last few went the way they did. Then somebody can show you that your bar moved, which is the only version of this problem that has a fix. Same as reviewing code you did not write: the instrument has to sit outside your head, because the thing being measured is inside it.
Apart runs these sprints monthly and publishes the rubric, which is why I could build any of this before opening the first PDF. If they ask again I will say yes, and I will run the sweep again, and I expect it to catch something else.