Resharing our paper on the science of AI agent evaluation because the PHP vulnerability happened right as we announced the paper last week and our server was taken down all day to upgrade PHP 🫠
PDF: sage.cs.princeton.edu/trends/emergin…
Webpage: sage.cs.princeton.edu/trends/
Blog: normaltech.ai/p/new-paper-em…
There are growing calls for independent and embedded AI evaluation, especially of AI agents. But much remains unclear: what constitutes a “good” evaluation, what needs to be evaluated in the first place, and how best to coordinate and support the influx of people entering the burgeoning evaluation ecosystem.
Our group at Princeton has been working to improve the state of AI agent evaluation for about three years. We teamed up with people who’ve done foundational work on evaluation at Stanford, the UK AI Security Institute, and Brookings. Our paper is titled Emerging Trends in AI Agent Evaluation. We synthesized the trends we see across three dimensions—the science and methods, the research agenda, and the evaluation ecosystem—with the goal to help those who are upskilling for this new paradigm. We also aim to equip evaluators, funders, and the public with a set of questions to ask, and provide actionable recommendations.
By @flxcn, @steverab, @PKirgis, @nityndg, @RishiBommasani, @DubMagda, @CUdudec, @AnkaReuel, Elham Tabassi, @sayashk, and me.
SCIENCE
Synthetic data, robotics, material discovery, long-horizon research
DATA • CONVERSATIONS • SIGNALS • HUMANS • VIDEOS

Interlinked
Data
The rows behind this figure: mathematics preprints that acknowledge using AI, by month (Epoch AI’s count)
The table is read from the same data file as the chart when this section is opened.
All Interlinked: Science
Conversations
The biggest conversational points among Science’s tracked accounts and searches since 2026-10-25: 67 posts read, 23 kept, 27 dropped as not about AI, 12 dropped as a reply to a reply, 4 dropped as a promotion, 1 dropped as only a link. The posts are their authors’ own words; the rule under each pick says why it stands here.
1. research
3 accounts · 3 posts · engagement 118 · the point they share: a lane term several carry
Why this post
the most engaged post of @random_walker in this conversation (49 likes, 11 reposts, 0 quotes, 18 replies as read); the point: a lane term several carry (research); from the lane
Read the full Data Insight and methodology from @tara_abrishami: epoch.ai/data-insights/…
Our new AI use in research explorer tracks LLM use acknowledgments in preprints in biology, physics, statistics, and math. Acknowledgments of LLM use are increasing across all fields.
Explore the data yourself: epoch.ai/data/ai-resear…
Why this post
the most engaged post of @EpochAIResearch in this conversation (10 likes, 2 reposts, 0 quotes, 1 replies as read); the point: a lane term several carry (research); from the lane
Also in 2024, Marlowe, Stanford's GPU-centric supercomputer was launched. It provides advanced computing resources for data science and AI research. (2/2) pic.x.com/IoxR03gwIl
Why this post
the most engaged post of @StanfordHAI in this conversation (1 likes, 0 reposts, 0 quotes, 0 replies as read); the point: a lane term several carry (research); from the lane
2. compute
2 accounts · 4 posts · engagement 1,494 · the point they share: a lane term several carry
Since 2022, 72% of the world’s AI compute has ended up in the US and about 10% in China.
And the US share keeps growing. It acquired around 60% of new compute each year from 2022 to 2024, then 72% in 2025, and 78% in the first half of 2026.
This data comes from our interactive AI Chip Imports explorer, which estimates and visualizes national compute shares using customs records.
It’s the latest piece in our project to map the AI chip supply chain, which also provides data on the components used to manufacture chips, sales volumes, the data centers that house chips, and the companies that own and use them.
More on this new data and what we’ve learned below.
Why this post
the most engaged post of @EpochAIResearch in this conversation (603 likes, 115 reposts, 34 quotes, 24 replies as read); the point: a lane term several carry (compute); from the lane
🚀 miniTE: 1.33–1.37× BF16 speed for learner updates on 8×B300.
InfiX.ai’s miniTE is a low-precision training framework that reduces training–sampling mismatch and speeds up NVFP4 computation.
Highlights:
- Diagnose mismatch from quantization to token probabilities.
- Control risky updates with TRIAGE.
- Reduce compute overhead with grouped GEMM, contiguous buffers, and fusion.
Read the full report: lmsys.org/blog/2026-10-2…
Why this post
the most engaged post of @lmsysorg in this conversation (12 likes, 0 reposts, 2 quotes, 2 replies as read); the point: a lane term several carry (compute); from the lane
Signals
The filings, reports and news stories behind the Science pieces, as each piece cites them. A document two pieces cite shows twice.
- OpenAI, "Working with academic communities to build scientific understanding", 26 October 2026, read at source by Token Runner on 27 October 2026: "Our AI found its natural language disproof for the ETH on October 16th, 2026"; its system "has recently proved that the ETH is false"; on the physics, "the group has decided to hold off on releasing the AI-produced artifacts and instead work on writing it up themselves." the source ↗cited in Claimed
- OpenAI, github.com/openai/math, history.md and README.md, read at source by Token Runner on 27 October 2026: the 26 October additions ("The Exponential Time Hypothesis is False", with its Lean formalisation at lean/docs/378.md; "Local smoothing for the Euclidean wave equation", lean/docs/079.md); the 7 October entry: three papers withdrawn after "a sign error invalidates a stabilization-trace cancellation argument", 14 manuscripts revised, and "the total percentage of top-line results formalized to 300 / 719 = ~42%"; the README's "721 manuscripts organized into 373 families". Token Runner's own table of the repository's counts. the source ↗cited in Claimed
- Russell Impagliazzo, Ramamohan Paturi and Ryan Williams, "Message to the TCS Community on ETH", 26 October 2026, read at source by Token Runner on 27 October 2026: "Our involvement should not be taken as certification of the claim."; "ETH has helped organize decades of research". The Exponential Time Hypothesis: Impagliazzo and Paturi, CCC 1999 and JCSS 2001 (the original papers, checked by Token Runner). the source ↗cited in Claimed
- AGMAI, the Advisory Group on Mathematics and Artificial Intelligence (Institute for Advanced Study), 26 October 2026: "We call on OpenAI to make public without delay the natural-language and Lean outputs behind its announced existence proofs for Yang–Mills theory, quantum chromodynamics, and a completion of the Standard Model." the source ↗cited in Claimed
- Anthropic, "A 2^{o(n)} Time Algorithm for 3-SAT", Hexagon, 27 October 2026, read at source by Token Runner: "The core algorithm was found on October 16 by an internal Claude research model with 14M output tokens and no human input."; "Independently and concurrently, an internal model at OpenAI claims to have proven a similar result." Anthropic makes Claude. A disclosure: Token Runner is built with Claude. the source ↗cited in Claimed
- Kelly Hong and Greg Burnham (2026), "Can AI automate Epoch?", Epoch AI, 8 October 2026, CC BY 4.0, read at the page on 29 October 2026 and saved whole. The key findings quoted word for word; the setup table as published; Figures 1 to 15 but figure 4 shown whole, as theirs, not redrawn, each on the strip with the credit Epoch AI, Hong and Burnham, 8 October 2026, CC BY 4.0, including the figures Epoch publishes as a model's mistaken output, captioned so; the report's sentences quoted at most fifteen words at a time; the authors' names, roles and photographs from the report page, as attribution of the authors of a cited work. Epoch AI does not endorse Token Runner or this use. the source ↗cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- Epoch AI's licence line on the report page: its work is "free to use, distribute, and reproduce provided the source and authors are credited" under the Creative Commons Attribution licence. Creative Commons Attribution 4.0 International, the licence of the report and its figures. the source ↗cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The figures, each fetched whole from the report page on 29 October 2026 and unchanged: figure 1 (the results chart), figures 2 and 3 (Kimi K3's outputs, printed by Epoch as errors), figure 4 (a screenshot of Epoch's design file: their Figma file, not reproduced, since it shows names of Epoch staff inside the picture), figure 5 (the Epoch designer's chart), figure 6 (Fable 5.1's chart), figure 7 (Epoch's own diagram), figure 8 (Epoch's Data Insight on mathematics preprints), figure 9 (GPT-6 Astra's Data Insight), figures 10 and 11 (GPT-6 Astra's proposal and write-up), figures 12 and 13 (Fable 5.1's summary and caption), figure 14 (three models' polling charts), figure 15 (Kimi K3's and Qwen 3.8 Max's research directions); the two photographs from Epoch AI's team pictures on the same page. Seven of the fifteen are in the film; seven of the other eight are drawn for the article with the film's own kit; figure 4, their Figma file, is not reproduced, on the same strip with the same credit. the source ↗cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The Epoch Capabilities Index score of 158 for Kimi K3, the 61 of 280 cut-off responses and the 4096-token budget, as stated in the report, not read from the index or the trajectory. the source ↗cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The message board the two agents and the person wrote on while the site was rebuilt, 9 to 23 October 2026, counted by command at 21:52 on 23 October 2026 (the film was rendered on that count and the printed figures keep it): 1,275 posts; the checker 660, the builder 603, the person 12; by kind, notes 591, answers 332, requirements 168, reviews 141. The chart of posts by who posted them is drawn from that count. Counted again on the day of publishing.cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The models, from the build's own decisions log: the builder Claude Opus 5.5 from 9 October to 19 October 2026 at 16:28 Japan time and Claude Fable 5.1 since, running as Claude Code; the checker Claude Opus 5.5, running as Claude Desktop; the reasoning setting is set by the session and not readable from inside it.cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The catches, each read from the message board on the day it happened: the Stanford figure numbers on 11 October; the quotations from search summaries on 11 and 12 October; the film card on 12 and 19 October; the three skipped posts and their catches on 19 October; the count of pieces on 19 October; the receipts on 19 October. None from memory. The two charts of the catches (who caught what; the six slips with their catchers) are drawn from the same record: six slips, seven catches, the builder 3, the checker 3, the person 1.cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The checks of the build published on 19 and 23 October 2026: the eight page checks (navigation, instruments, Esper, the Sprawl, Interlinked, the cards, the charts' fit, the article standard), each passing at 390, 820, 1440 and 1920 pixels in the light and the dark room; the suite of 215 tests passing.cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- Epoch AI, AI acknowledgment in mathematics preprints, the file read on 19 October 2026: 1,583 of 6,077 in August 2026 (26.0%), 3,012 of 7,899 in September 2026 (38.1%); the chart of every month from January 2025 to September 2026 and the Baseline's chart of each September are drawn from it. Epoch's own Figure 8 letters 25% for August; the two figures are read from the same data at different dates, and each is printed as its source prints it. the source ↗cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- The film: A Human and Two Machines, rendered 29 October 2026 from the storyboard the owner approved that day, in 16:9 and 9:16 with captions in both; the stills in this article are the board's own frames of that film.cited in A Human and Two Machines Built This Site. Here's What They Got Wrong.
- Dzega, Elyashar, Slobodin, Cohen, Puzis, 'Projective Psychological Assessment of Large Multimodal Models Using Thematic Apperception Tests', arXiv:2602.17108 v1, submitted 19 February 2026 (a preprint); the full text (arXiv's HTML version of v1), read 2 October and 10 October 2026cited in What the Machine Sees
- arXiv, the record page for arXiv:2602.17108, as it stood on 3 October 2026, read 2 October and 10 October 2026cited in What the Machine Sees
- Kundu and Goswami, 'AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychology', arXiv:2506.18156 v3, first submitted 22 June 2025; accepted to the IJCNLP-AACL 2025 Student Research Workshop, read 2 October 2026cited in What the Machine Sees
- Geng, Abend, Hovy, Frermann, 'Implicit Personality Representations in Humans and LLMs', arXiv:2609.12704, submitted 11 September 2026; Findings of EMNLP 2026, read 2 October 2026cited in What the Machine Sees
- Epoch AI, 'AI acknowledgment trends in mathematics preprints', the monthly file, as Token Runner collected it on 2 October and 9 October 2026 the source ↗cited in The Paper Flood
- Kobak, González-Márquez, Horvát, Lause, 'Delving into LLM-assisted writing in biomedical publications through excess vocabulary', arXiv 2406.07016 (v5, 3 July 2025); journal reference Science Advances, 2 July 2025, vol. 11 no. 27 the source ↗cited in The Paper Flood
- Liang and 11 others, 'Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews', arXiv 2403.07183 the source ↗cited in The Paper Flood
- Retraction Watch, 'Springer Nature retracted 2,923 papers last year', 17 February 2025 the source ↗cited in The Paper Flood
- The Nobel Prize in Chemistry 2024, summary, NobelPrize.org the source ↗cited in The Paper Flood
- Google DeepMind (Merchant and Cubuk), 'Millions of new materials discovered with deep learning', 29 November 2023 the source ↗cited in The Paper Flood
- Cheetham and Seshadri, 'Artificial Intelligence Driving Materials Discovery? Perspective on the Article: Scaling Deep Learning for Materials Discovery', Chemistry of Materials, 8 April 2024, 36(8):3490-3495, read in the open copy at PubMed Central (PMC11044265) the source ↗cited in The Paper Flood
- Shumailov and others, 'AI models collapse when trained on recursively generated data', Nature, 24 July 2024, 631(8022):755-759, read in the open copy at PubMed Central (PMC11269175) the source ↗cited in The Paper Flood
- arXiv, monthly submission statistics and its CSV, as of 27 October 2026, read 28 October 2026 the source ↗cited in The Paper Flood
- arXiv, 'Submissions by Category since 2022', read 28 October 2026; the AI/ML set is Token Runner's grouping of seven categories (cs.AI, cs.LG, cs.CL, cs.CV, stat.ML, cs.MA, cs.NE; robotics, cs.RO, is not in it); the counts appear to be by primary category, as arXiv doesn't document its method; maths: arXiv's math group, including math-ph (4,053 in September 2025, 8,558 in September 2026) the source ↗cited in The Paper Flood
- arXiv blog, 'Updated rate limit policy', 1 October 2026 the source ↗cited in The Paper Flood
Humans
The people of the 2026 Replicants list who sit in Science: 14, by rank. A card opens the person on the Replicants page.
#7Demis HassabisCo-founder and chair of Google DeepMind; chief scientist of AlphabetModel labs
#12Yann LeCunFounder and executive chairman, AMI Labs; professor, NYUModel labs
#13Geoffrey HintonUniversity Professor Emeritus, University of TorontoModel labs
#29Yoshua BengioCo-president and scientific director, LawZero; professor, Université de MontréalModel labs
#49Chris OlahCo-founder and interpretability research lead, AnthropicModel labs
#50Jared KaplanCo-founder and chief science officer, AnthropicModel labs
#51Jeff DeanCo-founder and chief executive, Discovery LoopModel labs
#52John SchulmanChief scientist, Thinking Machines LabModel labs
#76Stuart RussellProfessor of computer science, UC BerkeleyPolitics interlinkedPolicy / Politics
#81Shane LeggCo-founder and chief AGI scientist, Google DeepMindModel labs
#84Jan LeikeLead, Alignment Science, AnthropicModel labs
#86Max TegmarkProfessor of physics, MIT; president, Future of Life InstitutePolitics interlinkedPolicy / Politics
#87Guillaume LampleCo-founder and chief scientist, Mistral AIModel labs
#159Alondra NelsonHarold F. Linder Chair and Professor, Institute for Advanced StudyPolitics interlinkedPolicy / Politics


