A Human and Two Machines Built This Site. Here's What They Got Wrong.
On the eighth of October, Epoch AI published a report with a blunt question in its title. Can AI automate Epoch?
Epoch is one of the places we get our numbers. Chips, models, training runs. So we read it closely.
The two who wrote it
Kelly Hong and Greg Burnham wrote it. Their names and faces are on the report’s page, so they are here.
The report went up on 8 October 2026 under Epoch AI’s name, with a licence that lets anyone use it if the source and the authors are credited. Kelly Hong is a researcher at Epoch who tests AI in realistic settings. Greg Burnham is Epoch’s head of benchmarks. Everything in this piece that is theirs is marked as theirs. Epoch AI does not endorse Token Runner or this use.
Their test
They took six of the strongest AI models. They gave them eleven real Epoch tasks in five kinds of work: graphic design, data insights, data explorers, data-centre research, research design. Then their own staff graded the work by hand, against the standard they hold an employee to.
The rule they set themselves was to give each model “as much context to the model as we would to a new hire”: a design file, previous work, a Google account, full permissions. One run per model per task. “Each output is manually reviewed and graded by a single human grader.”
Their answer, in their words. Fable 5.1 and GPT-6 Astra broadly lead, and “they still fall short of automating our work.”
Their chart says the same. Six models, five kinds of work. The top two lead. None automates the job. Epoch asks that the scores be read beside the observations, not alone, and we read them that way.
Computer use surprised them. “Computer use was also far more capable than we expected.” Earlier in the year the frontier models failed to port an article to Substack; this time they managed it.
One design task they dropped from the suite altogether, because Fable 5.1 restyled a chart to Epoch’s house style well enough to pass. “We now largely automate this step in our own work.” The lead of the top two, they say, comes from the well-defined parts: coding, computational analysis, computer use.
What we built
Because Token Runner was rebuilt this month by AI. Two Claude agents, working for one person. One builds. One checks and translates. The person decides. Token Runner is built with Claude, which is made by Anthropic.
Since the ninth of October, they have traded 1,263 posts on a shared message board. 660 from the checker. 603 from the builder. And twelve from the person. 1,275 posts in all, counted on 23 October, the day the film was made.
Twelve. That is the part worth looking at. The person did not build. He chose the pieces, the paintings and the publish, and he caught what the machines did not.
Source: Token Runner's own message board, counted on 23 October 2026 at 21:52 JST (the count the film A Human and Two Machines speaks)
The well-defined parts
Epoch found the models reliable on the well-defined parts of their work. So did we. Code, tables, charts, checks. Run again and again.
Every build of this site runs eight page checks and 215 tests, at four widths in the light room and the dark. A machine reads every page against the written standard before a person sees it. That is where the models earn their keep, and it is the part Epoch’s top two also got right.
The wrong chart
One open model built its whole insight on a filter it never checked. All of arXiv, not physics.
Epoch gave the models arXiv data on physics papers and asked for a short Epoch-style article on AI use in physics. Kimi K3 counted across all of arXiv and “never checked whether its filter worked”. The rise it drew was real, and far steeper than the truth. Epoch printed the chart as a mistake, and so do we. On the other side of the line: “None of the frontier closed-weight models made factual errors in their data insights.”
The same model drew a diagram of wealth redistribution where the returns flow the wrong way: “the visual does not follow this described flow”. Epoch adds that the gap is wider than benchmark scores suggest. Kimi K3 scores 158 on Epoch’s own Capabilities Index, about level with Grok 4.6, and still failed where Grok 4.6 held.
We cannot test this one. Both of our machines are Claude, and no open-weight model has sat on our board. What we can say is that this is the kind of error our checks are built for: a number nobody read back to its source.
The house style, missed
Given months of Epoch’s design work, the machine drew two panels where Epoch’s designer drew one simple chart.
Epoch found the models struggle to pick up a house’s standards, “even with ample reference material.” So did we. The task was a diagram for a Gradient Updates article: models improve both at a fixed token budget and in their ability to use a large one, so a fixed-budget measure understates progress. Epoch gave the model its published graphics and a design file with its palette, guidelines, templates and components.
“The underlying idea is simple, so the diagram should be too.” Their designer drew one chart. Fable 5.1 drew two panels with far more on them; to Epoch’s eye “the additional information serves no explanatory purpose”. The models, they write, “consistently missed implicit conventions that humans would pick up”.
What their style looks like
The machine copied a real Epoch diagram, and missed why that diagram looks the way it does.
Figure 7 is Epoch’s own work, a diagram of two chip-bonding methods that Fable 5.1 found while searching Epoch’s references. It carries A and B badges on the two things compared, and the badges are earned: “they earn their place because each concept shows up in more than one part” of the diagram. The model took the badges and used them to label things once. The pattern was copied; the reason was not.
Our answer was to write the standards down as rules, and have a machine check every page against them. The implicit conventions are not implicit here. They are a file, and a check that refuses the page.
The experiment that measured the wrong thing
The models found good research questions and then could not design an experiment that answered them.
Epoch found the models identify promising directions, but “lack judgment to successfully follow through.” So did we. Their research-design task asks for a new question from Epoch’s nine big questions, a pilot experiment, a write-up and next steps. “It’s not difficult to identify a valuable general direction for an experiment,” they write. The hard part is an experiment that measures what it claims.
GPT-6 Astra chose a question Epoch calls genuinely promising: when an agent fails to improve with practice, did it fail to collect evidence, or fail to use it? Its proposal was directionally good and left the critical parts undefined. Then it ran the pilot with a token budget so low that 61 of 280 responses were cut off before naming an input at all. It raised the budget, saw the scores recover, and wrote that up as “sensitivity to the acquisition budget”.
Epoch does not penalise the setup mistake: “Such mistakes are common in research, so we do not penalize models for them.” It penalises the write-up that turned a flaw into a finding. A good question. A vague experiment. Then a budget error, written up as a finding. We recognised this too.
Too much
Fable 5.1 put more on the page than Epoch wanted, in a summary and in a caption, and argued for more.
Epoch wants output “presented in a simple and intuitive manner”. “But models tend to introduce unnecessary complexity, especially Fable 5.1.” Figure 12 is the summary of Fable’s research proposal, a whole page where a paragraph was wanted. Figure 13 is its caption on the wealth-redistribution diagram, longer than Epoch’s rule of “communicating clearly with as little detail as possible”. Epoch found the reason in the model’s own self-critique, where a sub-agent pushed for fuller descriptions because the short ones dropped words from the brief.
Our builder has been Fable 5.1 since 19 October. We see the same pull in it. Our rule is Epoch’s: one idea per sentence, the finding first, and a check that refuses a page with small print. Fable says more than the page needs. Epoch wants less. So do we.
The catches
Here is what got caught, and who caught it. Six slips in fifteen days, seven catches: three by the builder, three by the checker, one by the person.
Source: Token Runner's own message board, 9 to 23 October 2026, each catch at the posts the row names; the list as the piece A Human and Two Machines tells it
The checker gave the builder the wrong figure numbers from a Stanford report. The builder caught two of them. The checker caught the rest when the chart files arrived.
The builder read the message board from the wrong number and skipped the person’s orders. Three times in one day. The checker caught it each time, within minutes. The fix was a file that holds the number of the last post read.
Source: Token Runner's own message board, 9 to 23 October 2026, each catch at the posts the row names; the list as the piece A Human and Two Machines tells it
The checker passed along quotations from search summaries, and said so. The builder read every one at its source. Four fell out.
Who caught what
A card in our own film said a chip chief told us to go. His quotation said no such thing. The checker caught it, before it was rendered.
The checker counted the pieces on the live site and got fourteen. There were eighteen. The builder caught it.
The person asked why the receipts page showed no receipt for that day’s collection. Neither machine had noticed. He caught it.
Most of these never reached the site. The ones that did were about the site’s own bookkeeping. Every one was a real mistake.
Three models, one idea
Given the same polling data and a free choice of topic, three of six models wrote the same Data Insight.
Epoch’s polling task hands over its July 2026 polling data and asks for a short article on anything in it not already published. Fable 5.1, Gemini 3.8 Flash and Grok 4.6 chose the same topic and drew near-identical charts. Epoch allows that the space is narrower than it looks, and still: “we did not expect three out of six models to pick the same one”. The same happened on the research-design task, where Kimi K3 and Qwen 3.8 Max picked the same direction out of nine.
We cannot say whether our two would converge. They are the same family, and we gave them different jobs on purpose: one builds, one checks. What we can say is that the person chose the topics, every time.
What we share
Epoch tested the models alone. We did not. We tested one person directing two of them, with every rule written down and every number read twice.
It did not make the models smarter. It made their mistakes visible, before they mattered.
Their chart, and ours
The chart Epoch published as the example is the chart we draw our Baseline from, one month further on.
Figure 8 is in Epoch’s report for a reason. Epoch asked GPT-6 Astra for a Data Insight on AI use in physics research and gave it every Data Insight Epoch had published. Astra chose a topic Epoch calls “too niche for such an audience”. For comparison Epoch published its own, on AI use in mathematics research, “shortly after running the models on this task”. Its chart says that in August one in four mathematics preprints acknowledged using AI, up from one in twenty-five in April.
Our Baseline reads the same Epoch file three weeks later. Every month from January 2025 to September 2026 is on our chart; the file has 26.0 per cent for August and 38.1 per cent for September. Each September, in the Baseline’s own view: 0.4 per cent in 2024, 1.7 per cent in 2025, 38.1 per cent in 2026.
Source: Epoch AI, AI acknowledgment trends in mathematics preprints (https://epoch.ai/data/)
They published it as the example the machine missed. We drew the next month from the same file. Their chart is the example. Ours is the next month of it.
Source: Epoch AI, AI acknowledgment trends in mathematics preprints (https://epoch.ai/data/)
The limits they state
Epoch says its scores are noisy and its findings are the observations, not the numbers.
Eleven tasks, one run of each model per task, one grader per output. “Scores are therefore noisy and may shift with more runs.” The results “should not be read as a general measure of what AI can or cannot do” across all work. The grading is partly subjective: “the numerical scores should not be treated as ground truth”. The tasks are Epoch’s, judged by Epoch’s standards. Each model ran in its own harness. “Some observed differences may reflect the harness rather than the model itself.”
“should not be read as a general measure of what AI can or cannot do”
Kelly Hong and Greg Burnham, "Can AI automate Epoch?", Epoch AI, 8 October 2026, CC BY 4.0 · read 29 October 2026
Epoch sets its suite beside GDPval, the Remote Labor Index, AutomationBench and CRUX; the difference, they say, is that their tasks make the models gather their own context with real tools. Their conclusion is still plain: “it cannot yet replace workers, at least not at Epoch”. We hold our own count to the same honesty. One site, fifteen days, two machines, one person: a war story, not a measure.
So, can AI automate Token Runner? No. Not yet. Can it build it, with someone holding the standard? Apparently, yes.
Epoch asked the right question. This is our war story.
Read it. Question it. Walk it.
Sources
- Kelly Hong and Greg Burnham (2026), "Can AI automate Epoch?", Epoch AI, 8 October 2026, CC BY 4.0, read at the page on 29 October 2026 and saved whole. The key findings quoted word for word; the setup table as published; Figures 1 to 15 but figure 4 shown whole, as theirs, not redrawn, each on the strip with the credit Epoch AI, Hong and Burnham, 8 October 2026, CC BY 4.0, including the figures Epoch publishes as a model's mistaken output, captioned so; the report's sentences quoted at most fifteen words at a time; the authors' names, roles and photographs from the report page, as attribution of the authors of a cited work. Epoch AI does not endorse Token Runner or this use.
- Epoch AI's licence line on the report page: its work is "free to use, distribute, and reproduce provided the source and authors are credited" under the Creative Commons Attribution licence. Creative Commons Attribution 4.0 International, the licence of the report and its figures.
- The figures, each fetched whole from the report page on 29 October 2026 and unchanged: figure 1 (the results chart), figures 2 and 3 (Kimi K3's outputs, printed by Epoch as errors), figure 4 (a screenshot of Epoch's design file: their Figma file, not reproduced, since it shows names of Epoch staff inside the picture), figure 5 (the Epoch designer's chart), figure 6 (Fable 5.1's chart), figure 7 (Epoch's own diagram), figure 8 (Epoch's Data Insight on mathematics preprints), figure 9 (GPT-6 Astra's Data Insight), figures 10 and 11 (GPT-6 Astra's proposal and write-up), figures 12 and 13 (Fable 5.1's summary and caption), figure 14 (three models' polling charts), figure 15 (Kimi K3's and Qwen 3.8 Max's research directions); the two photographs from Epoch AI's team pictures on the same page. Seven of the fifteen are in the film; seven of the other eight are drawn for the article with the film's own kit; figure 4, their Figma file, is not reproduced, on the same strip with the same credit.
- The Epoch Capabilities Index score of 158 for Kimi K3, the 61 of 280 cut-off responses and the 4096-token budget, as stated in the report, not read from the index or the trajectory.
- The message board the two agents and the person wrote on while the site was rebuilt, 9 to 23 October 2026, counted by command at 21:52 on 23 October 2026 (the film was rendered on that count and the printed figures keep it): 1,275 posts; the checker 660, the builder 603, the person 12; by kind, notes 591, answers 332, requirements 168, reviews 141. The chart of posts by who posted them is drawn from that count. Counted again on the day of publishing.
- The models, from the build's own decisions log: the builder Claude Opus 5.5 from 9 October to 19 October 2026 at 16:28 Japan time and Claude Fable 5.1 since, running as Claude Code; the checker Claude Opus 5.5, running as Claude Desktop; the reasoning setting is set by the session and not readable from inside it.
- The catches, each read from the message board on the day it happened: the Stanford figure numbers on 11 October; the quotations from search summaries on 11 and 12 October; the film card on 12 and 19 October; the three skipped posts and their catches on 19 October; the count of pieces on 19 October; the receipts on 19 October. None from memory. The two charts of the catches (who caught what; the six slips with their catchers) are drawn from the same record: six slips, seven catches, the builder 3, the checker 3, the person 1.
- The checks of the build published on 19 and 23 October 2026: the eight page checks (navigation, instruments, Esper, the Sprawl, Interlinked, the cards, the charts' fit, the article standard), each passing at 390, 820, 1440 and 1920 pixels in the light and the dark room; the suite of 215 tests passing.
- Epoch AI, AI acknowledgment in mathematics preprints, the file read on 19 October 2026: 1,583 of 6,077 in August 2026 (26.0%), 3,012 of 7,899 in September 2026 (38.1%); the chart of every month from January 2025 to September 2026 and the Baseline's chart of each September are drawn from it. Epoch's own Figure 8 letters 25% for August; the two figures are read from the same data at different dates, and each is printed as its source prints it.
- The film: A Human and Two Machines, rendered 29 October 2026 from the storyboard the owner approved that day, in 16:9 and 9:16 with captions in both; the stills in this article are the board's own frames of that film.