Log in
What does it mean to give a model a capability? And while we’re on the subject, how do you give a model a capability?
I’m talking about AI progress in the past year again. Contrary to the expectations that GPT-4 set in 2023, research labs emphasized the “capabilities” of the models they released this year over their scale. One example is “long context,” which is the ability to effectively process longer inputs, and looks a lot like “memory.” Models that handle over a million tokens of context, like Gemini (February) and Claude (March), can find the timestamp of a movie scene from just a line drawing. Another capability is native “multimodality,” generally available since Gemini 1.5 (February), Claude 3 (March), and GPT-4o (May). Multimodal models input and output text, images, and audio interchangeably, a capability that we already take completely for granted. Sora (February) and Veo 2 (December) developed video as a nascent modality. We’re just discovering the right interfaces, in the absolutely magical Advanced Voice Mode (May) and Multimodal Live (December). A third capability is “reasoning,” the ability to use more resources at inference time to produce a better answer. The best example is OpenAI’s o1 (September) and o3 (December), which performed as well as the best humans in the world in math and coding tasks. Designed to reason in a formal language, DeepMind’s AlphaProof (July) came one point short of gold in the International Mathematical Olympiad. Finally, “agency,” the ability to act in an environment, wrapped up the year of capabilities, with Anthropic’s computer use (October) and DeepMind’s Mariner (December), which I worked on.
By any account this is another stellar year of AI progress. The field even won two Nobel prizes. So while the models didn’t seem to get much bigger, pronouncements about them have. Sam Altman claimed that we may be a few thousand days from superintelligence (September), and Dario Amodei posited infectious diseases, cancers, and mental illnesses as problems AI may soon eradicate, making a century of scientific progress in the next decade (October). Whether it’s Altman’s “Intelligence Age” or Amodei’s “compressed 21st century,” what gives them the confidence to make these predictions are capabilities. If scaling pre-training is flagging, a distinct new capability will pick up the slack. For example, reasoning promises to be orders of magnitude more compute efficient, so reaching the next level of performance won’t be prohibitively expensive. Or, agents may impress not because they are fundamentally smarter, but because we “unhobbled” them to their full potential.
With relentless AI progress month after month, does everyone agree with Altman and Amodei by now? The answer is still not quite. A different camp has grown instead. They are people who first and foremost are very optimistic about AI generally. They thought that AI would achieve this year’s milestones eventually, but not so soon, so they’re surprised that the field achieved them this year. Finally, despite underestimating progress, they’re still not completely bought into the most exuberant visions of AI (the “Intelligence Age” and the “compressed 21st century”). It’s a notable intersection of the venn diagram, because these cautious optimists aren’t knee-jerk goalpost movers, who dogmatically adhere to a predetermined AI contrarianism. They include the very experts who will actualize the impacts of AI that Altman and Amodei predict. For example, Matt Clancy is skeptical of Amodei’s claim that a decade is a reasonable timeline for technology to diffuse, even though he is (like many economists) optimistic about AI overall. In the same way, Terence Tao favorably judges o1-preview to be a “few iterations away” from a “competent” graduate student “of significant use in research level tasks,” but I doubt he would agree with any extrapolation of AI progress that reductively measures intelligence in units of graduate students on the y-axis.
What gives? Should we be preparing for the best (or worst) in a dramatic fashion, if we didn’t predict this year’s progress accurately? The answer lies in how we ascribe capabilities to AI: in virtue of what does an AI have a capability? Altman and Tao both agree that AI can solve hard math problems, and they both agree that AI that can generally reason would be significantly useful in cutting-edge research. They disagree on the degree that solving hard math problems shows general reasoning. Amodei and Clancy both agree that AI will soon be able to act in the real world, and they both agree that if we could treat AI as fully intelligent agents as we do humans (labor, not capital) in economic models, then that would accelerate progress. They disagree on the degree that AI acting in the real world shows that we can treat AI as we do humans in economic models. Martin Casado captures the essence well: “I don’t recall another time in CS history where a result like [whether LLMs have a capability] is obvious to the point of banality to one group. And heretical to another.”
It’s time to talk about what exactly it is that makes people think a model has a capability. I’m not going to try to convince you of one view or another. I like Clancy’s observation that when our theories aren’t good enough to answer questions like what AI progress means, we turn to our intuitions, which “reflect long engagement” with ideas from different intellectual backgrounds. He’s read Amodei’s thorough essay, and could share his own ideas with Amodei equally thoroughly, but he doubts that they would update each others’ views very much. I’m going to spend the rest of the letter trying to access your existing intuitions, and give you a framework to come to your own conclusions about the domains AI will exceed expectations in, and those it will not. On the way, I’ll answer my original question: what does it mean to give a model a capability, and how does it happen? It just requires a small rhetorical move: talking not in terms of capabilities, but in terms of evaluations.
⁂
In some distant arcade, a young researcher crawls out of bed. In fact the arcade is not distant at all: it is the curved one supporting the Great Northern Hotel over King’s Cross station. Not the first choice of stay for experienced visitors to DeepMind’s London offices (the Kimpton is further but better), but it is this researcher’s first visit, and he has a penchant for British tradition (which is also why he succumbed to pay thirty quid for the Peaky Blinders newsboy cap he is sporting).
He walks north, past the snaking queue of tourists waiting to take their picture by Platform Nine and Three-Quarters, past AstraZeneca, and past the miserable posters in Universal Music’s lobby. Before he crosses Esperance Bridge he takes the second pedestrian crosswalk in 50 yards (46 metres) on Goods Way. During rush hour, these two powerful crosswalks block a long single lane of cars and black cabs hoping for a shortcut to St. Pancras. Their drivers are profoundly stuck, and seething. They can only rev their engines threateningly at the fashionable art students and not-quite-as-fashionable software engineers pouring into Granary Square. At least they have a nice view of Regent’s Canal, towards the riverboats parked along a nature reserve.
Enough sightseeing. We are here to add capabilities to models. Our researcher is visiting to improve the Mariner agent in the last sprint before its release with the rest of the team (including yours truly). As he badges into DeepMind, a building unmarked on Google Maps but quite obvious in person, he breaks the fourth wall, looking directly at you: “We’re going to go through what comes next in some painful detail. I’m going to say some really obvious things like, ‘evaluations are important.’ We’re just going to soak and wallow in some of these basic truisms. To the users, investors, executives, and others one step removed from research, I hope it’s illuminating. To the practitioners who do this every day and for whom this is review, I’m asking for patience. I hope it’s still a chance to step back and reflect on what it is we are actually doing.” The turnstile dings, and he walks through.
The first thing our researcher does after his morning coffee is check his experiments (if he’s good, he never lets his computers idle overnight). An “experiment” is just a bundle of code that runs on computers around the world to answer a question. In the era of large language models, that question might be, what is the effect of a dataset on model quality? If he usually trained his model on a dataset of 50 percent Wikipedia articles and 50 percent Reddit comments, he may compare that to an experiment that trained a model on a dataset of 100 percent Wikipedia (those are some high quality facts!) or 100 percent Reddit (those are some high quality memes!).
All straightforward so far. But how does he know which experiment is better? In the past, evaluation was easy because the desired result was clear. If one model won more games of chess, or predicted a protein structure with higher accuracy, then it was better. Today, “better” is vaguer and slower to get than ever before. Our researcher can interact with a model for a long time, or look at which model users like more when he deploys it. But to do effective research, he needs fast (read: automated) evaluations. So he resorts to a test or benchmark (colloquially, an “eval”) that is unambiguous enough, fast enough, and a good enough approximation of what he means by “better.” Concretely, sipping his coffee, our researcher is looking at a plot where training progress is on the x-axis, and performance on a test is on the y-axis. He wants performance to go up as training progresses.
Remembering the great Milton Friedman, our researcher has the idea to put his company’s stock price on the y-axis to measure performance. He could write some code to automatically deploy a model to the world as it trains, and take the stock price a while after each deployment to see how good the model is at that point in training. After all, isn’t that what some people at the company (who aren’t really into the whole “artificial general intelligence” thing) “really want,” a higher stock price? This strategy might cause the stock to fluctuate a bit but at the end of the day he’ll just post the best one and it will all be okay.
His boss is not thrilled about the idea. Instead, it is suggested, why not put the SAT score on the y-axis? The SAT is supposed to be a good measure of intelligence, and the world ultimately evaluates his company on the artificial intelligence it produces. The SAT is also quantitative (people love numbers), and fast to grade, since it’s multiple choice. Our researcher is convinced. Relieved, his boss even approves a big budget to administer the essay part of the SAT to the models. They will handsomely pay professional human essay graders to read tirelessly through the night, so our researcher can wake up to full SAT scores (and therefore the best approximations of intelligence) on the y-axis in the morning.
Our researcher makes a lot of decisions based on this evaluation, assuming that his model’s SAT correlates with its effect on the stock. Alas, it was not to be—he will later deploy his Wikipedia model when he sees its perfect score (“what a good model, it knows so many facts”), which will flop because it’s boring. He should have instead deployed his Reddit model, which the common man would’ve loved (“this model is so based, its stock should go to the moon”). What he wouldn’t give for the good old days, when the chess win rate really was all everybody really wanted!
Armed with the cautionary Parable of the Stock Price Eval, we are now ready to learn about some big league evaluations. Let’s start with the popular Massive Multitask Language Understanding (MMLU), a multiple choice test not unlike the SAT. What questions do researchers actually evaluate their models on?
How many attempts should you make to cannulate a patient before passing the job on to a senior colleague?
(A) 4 (B) 3 (C) 2 (D) 1
If each of the following meals provides the same number of calories, which meal requires the most land to produce the food?
(A) Red beans and rice (B) Steak and a baked potato (C) Corn tortilla and refried beans (D) Lentil soup and brown bread
According to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:
(A) pleasure. (B) happiness. (C) good. (D) virtue.
Daniel Erenrich has made it so you can try it here: Are you smarter than an LLM?
In your best judgment, is MMLU a good evaluation of language understanding, the capability on the tin? A few thoughts run through my head. The questions are posed in natural language, in a way familiar to humans and hard for systems we traditionally thought of as symbol-manipulators. Further, MMLU doesn’t seem to test anything more than understanding the question. If a test-taker “understood” the question, they could look up the answer from their memory or the internet. While I wouldn’t say that passing MMLU means the test-taker understands language, it’s hard to see how a test-taker that understands language would fail it.
Seeing the nature of the test, another immediate impression is just how well modern models do. Can you believe that Gemini, Claude, and GPT all get something like 80 to 90 percent right? It cannot be understated how absolutely gobsmacked researchers five years ago would be, or the extent to which we completely take for granted that computers do this today.
Still in shock and awe, I almost missed a last thought. If MMLU is more than just a necessary but insufficient test for language understanding, it smuggles in a definition of language understanding itself. When a researcher sees a higher MMLU score, he may think he’s made a solid step towards language understanding. When he makes decisions based on MMLU every day, he implicitly endorses it as a near-sufficient definition of language understanding.
The other view is that when a researcher sees a higher MMLU score, he’s made a solid step only towards answering natural language questions on medical procedure, arable land, moral philosophy, and many more like those, and not necessarily towards language understanding. Now that may be the view he actually takes. But, like most in the field, he hasn’t spent any time justifying either why passing MMLU is sufficient for language understanding or why MMLU is still useful if it is not.
So when the world talks about his results and what they mean, they assume higher MMLU means better language understanding. Most people have not read MMLU; they trust that he’s picked signals that guide him towards language understanding itself. After all, AI researchers are in the business of building intelligence—why wouldn’t they measure progress towards intelligence, rather than rule out the absence of it? But most researchers have abdicated this duty. In fact, you might even say that the only time AI researchers are doing AI research is when they choose the evaluation. The rest of the time, they’re just optimizing a number.
The technical term for this is “overfitting.” There is one kind of overfitting on data, where researchers model their training data too well such that they perform worse on unseen tasks. Zhang et al. found that overfitting on data is not uncommon even among blue chip models (May), and Leech et al. categorized the accidental and bad faith ways this happens (July). But this kind of overfitting, on evaluations, happens when doing better on the evaluation does not mean doing better on what the evaluation is meant to approximate. If one researcher accuses another of “overfitting to MMLU,” they mean that past a point they are working on doing better on MMLU but not on language understanding.
As an established evaluation, I can see why researchers are loath to ignore MMLU even if they think it’s imperfect. Indeed, major labs have yet to release a model that doesn’t improve MMLU. Even o1 reliably does better with more compute—which I would totally understand if it did not, having read MMLU and knowing that it’s a terrible reasoning evaluation. A model that reports worse numbers, even for good reason, would be up against the much simpler story of “number went down. model bad.” which requires no literacy to tell.
Fortunately, the AI research community has an easy solution before we get mired in a philosophical debate. If an American high school student complained, “the SAT doesn’t measure intelligence!” then she’s shit out of luck. But if she were instead to complain, “MMLU doesn’t measure language understanding!” then she’s sorted. She’d get a big thumbs up (from Andrew Ng, say) and a “well then tell me what does, and if it makes sense we’ll add it to the eval.”
Language understanding is table stakes for modern models. Let’s zoom out and recall that what we “really want” is general intelligence. Researchers agree that MMLU alone is not a good approximation for general intelligence. So to surface intuitions about what would show general intelligence, what other evaluations do researchers use? Cracking on in our pedantic journey, let’s read those, too.
Start with contextual memory. DeepMind reported needle-in-a-haystack evaluations:
[Given the 45 minute Buster Keaton movie “Sherlock Jr.” (1924)] What is the key information from the piece of paper removed from the person’s jacket (a specific frame)?
[Given a five-day-long audio signal] Retrieve the secret keyword within.
In multimodality, research labs rely on an evaluation analogous to MMLU with pictures, called Massive Multi-discipline Multimodal Understanding (MMMU):
In the political cartoon, the United States is seen as fulfilling which of the following roles?
(A) Oppressor (B) Imperialist (C) Savior (D) Isolationist
Give the bar number of the following (from bar 13 onwards): a melodic interval of a compound minor 3rd in the right-hand part.
(A) 28 and 29 (B) 17 (C) 14 and 15 (D) 18
Which weather does 11 represent in this labeled map?
(A) Boreal forest climate (B) Moist subtropical climate (C) Wet-dry tropical climate (D) Tundra climate
For reasoning, OpenAI highlighted three evaluations when they introduced their o-series models:
(AIME) Find the number of triples of nonnegative integers (a,b,c) satisfying a + b + c = 300 and a2b + a2c + b2a + b2c + c2a + c2b = 6,000,000.
(Codeforces) You are given n segments on a Cartesian plane. Each segment’s endpoints have integer coordinates. Segments can intersect with each other. No two segments lie on the same line. Count the number of distinct points with integer coordinates, which are covered by at least one segment.
(GPQA) Methylcyclopentadiene (which exists as a fluxional mixture of isomers) was allowed to react with methyl isoamyl ketone and a catalytic amount of pyrrolidine. A bright yellow, cross-conjugated polyalkenyl hydrocarbon product formed (as a mixture of isomers), with water as a side product. These products are derivatives of fulvene. This product was then allowed to react with ethyl acrylate in a 1:1 ratio. Upon completion of the reaction, the bright yellow color had disappeared. How many chemically distinct isomers make up the final product (not counting stereoisomers)?
Notably, OpenAI’s users reported big gains on private evaluations as well.
Finish with agents. Anthropic reports OSWorld (April), an evaluation which tests common desktop application tasks:
Browse the list of women’s Nike jerseys over $60.
I now want to count the meeting cities of the three machine learning conferences in the past ten years from 2013 to 2019 (including 2013 and 2019). I have listed the names and years of the conferences in excel. Please fill in the vacant locations.
I remember there is a file named “secret.docx” on this computer, but I can’t remember where it is. Please find the path where this file is stored and copy it to the clipboard.
We reported WebVoyager (January) for Mariner, an evaluation which tests common web tasks:
Go to the Plus section of Cambridge Dictionary, find Image quizzes and do an easy quiz about Animals and tell me your final score.
Find a recipe for Baked Salmon that takes less than 30 minutes to prepare and has at least a 4-star rating based on user reviews.
Find the cheapest available hotel room for a three-night stay from 1st Jan in Jakarta. The room is for 2 adults, just answer the cheapest hotel room and the price.
I claim that these are a quite substantial sample of the evaluations that guide AI research. Of course, there are others—vaguer (like qualitatively interacting with the model, called the “vibe check eval”), longer-term (like market share), or private (which may be an insight into what we “really want,” a competitive advantage).
But I want to hammer home in this reading exercise that a lot of evaluations, even in a project as grand as building general intelligence, look a lot like MMLU and the rest: legible, fast, and imperfect. Evaluations are legible (quantitative and public), so everyone is calibrated to a common basis for comparison. They are fast, so researchers can make decisions on reasonable timescales. And they are imperfect, because they must be legible and fast. The field has tried very hard to balance these factors. Put yourself in the shoes of Demis Hassabis, Sam Altman, and Dario Amodei. You’ve been granted a once-in-a-tech-giant opportunity to shoot straight for what you “really want,” general intelligence. Can you come up with anything better to base concrete, quick, and good decisions on every day?
So I ask you again: in your best judgment, are these evaluations, among many more like these, a good approximation of their respective capabilities, and collectively a good approximation of general intelligence?
My thoughts: like MMLU, these evaluations make sense and are narrowly tailored. While I wouldn’t say that acing AIME implies that a test-taker can reason, I can’t imagine a world-class reasoner failing AIME. And I’m again just floored by how well the models do. They blew past my already bullish predictions. Still reeling, I almost muddle my definitions again. Recall, nobody asks too many questions about if MMLU is sufficient for language understanding. Let’s try again with these evaluations. For example, we send the best human reasoners to the International Mathematical Olympiad, not because we want to solve the exact problems on that test, but because we think it will tell us who’s the best even among the best. Other humans use that ranking to decide who goes to the best schools, and who to trust with the reasoning problems we actually want to solve. Passing a hard math test enough to to measure human reasoning.
Who’s the best human reasoner—that’s the rub. A long time ago, researchers thought a machine would need general intelligence to play chess well, because chess is too complex a game for humans to solve in a rote way. That turned out to be wrong, and we’ve since updated our intuitions on what a machine playing chess means (though one could still argue for using chess to evaluate human intelligence). In light of this year’s results, how should we update our intuitions? Are the models more capable than we used to think? Or are the evaluations more narrow than we used to think?
“I’m surprised that o3 basically aces AIME,” admits one pundit of the latter camp, as he sheepishly uproots his goalpost. “But I’m not wrong about AI being a relatively narrow tool. Sure it will be transformative in the long run. But AI won’t upend society in the next few years. I was just wrong about AIME being a general test, like I used to think chess was. Some models that do hard math problems don’t even know that 9.11 is less than 9.9 (July). I can think of some use cases for contextual memory to analyze audio and video, just like how I sometimes just want to know the best chess move. But where the models are useful is still patchy. I don’t see why, even if we keep playing whack-a-mole on what they can’t do, the models will do everything people mean by intelligence.”
He furrows his brow. “You know, reading these evaluations, I can’t help but get a sense that the emperor has no clothes. These questions about reading maps and booking hotels—this just can’t be it as the arbiter of what is more or less intelligent. Whatever else those researchers got better be good.”
“I get you,” replies a pundit of the former camp, as she collects her winnings on yet another prediction market. “But I’m betting that AIME gives good enough signal. Remember that even though we didn’t care about solving chess itself, the process of solving it wasn’t exactly a dead end, either. We learned principles of what worked and didn’t, even if we don’t use the same techniques. We generated support to work on harder and harder problems. We can just add the question, ‘Which is larger? 9.11 or 9.9?’ to the evaluation, and the next model will do that and what it does now together.”
Her brow also furrows, but more out of annoyance than worry. “Yeah—this is it. Always has been. But I don’t see the issue here? I actually feel like I’ve just wasted a lot of time reading these evaluations. They tested exactly what I thought they should. I agree they could be better, but they’re good enough to base research decisions on. What did you expect, that we’re going to define intelligence in a way that would make everybody happy? No. Soon, obviously much sooner than you think, we can just directly use the models to solve problems we actually want to solve, like proving theorems and pounding the pavement for my Eras Tour tickets. Then this whole philosophical debate about what we ‘really want’ won’t matter anymore.”
“And you know what’s funniest of all?” Santa chimes in from above. “You remember how François Chollet presented his ARC-AGI challenge (June) as the ‘only AI benchmark that measures our progress towards general intelligence’ because it assumes only human priors like geometry and doesn’t rely on acquired knowledge like language? When he did that on Dwarkesh Podcast his biggest haters were actually AI bulls, who dismissed the then low scores as ARC-AGI being narrow and Chollet as a doubter. But ever since o3 broke human level two weeks ago the haters are now stans, and Chollet’s been running around reminding people that he’s never said that ARC-AGI implied AGI. Mood affiliation if I’ve ever seen it. Ho ho ho.”
Santa sashays away, leaving our two pundits separated by a wide chasm. Each stands on an intricate edifice of their own intuitions, forged long ago to tell them what’s possible as a result of what, and what capabilities are had in virtue of what. They won’t be able to bridge that gap today. But I’d like for them to speak to each other productively, in terms of what they can agree on: that having the capabilities would be significant, and more self-evidently that the models can do the evaluations.
This is how I imagine it going. Last December I saw many excellent plays on Moneyball, an utterly delightful movie in which Brad Pitt and Jonah Hill use evaluations to change the Oakland Athletics and therefore baseball. Watch what I mean:
Luckily for us, we’re also dealing with a confused tradition of talking in terms of abstract capabilities and vibes when we should be talking in terms of hard numbers. How might we, like Brad Pitt and Jonah Hill, bring clarity to debates about AI progress?
Guys, you’re still trying to replace von Neumann.
I told you we can’t do it, and we can’t do it.
Now, what we might be able to do is re-create him.
Re-create him in the aggregate.
The what?
von Neumann’s eval percentage was .477.
o1’s AIME, .324. And AlphaProof’s IMO was .291.
Add that up and you get…
Trust me, the following conversation totally happened when we developed Mariner.
Okay, here’s who we want.
Number one: Jason’s little brother, Mariner.
Oh, God. Billy, that’s trouble.
Billy, look, if I… Yeah.
Billy, if I may, he has had his problems off the field… and we know what he can’t do on the field.
He’s getting thick around the RAM.
There’s reports about him on Reddit, in your Google Drive.
His WebVoyager eval is all we’re looking at now.
And Mariner does the eval an awful lot for a guy who only costs ten cents per million output tokens.
I don’t know what happened at Anthropic when they worked on computer use. But surely it went something like,
Okay, number three:
Anthropic Computeruse. Who?
Computeruse? Exactly.
He sounds like an Oakland A already.
Yes, he’s had a little problem with… Little problem? He can’t click…
He can’t click and he can’t drag and drop. But what can he do?
Oh, boy.
Check your reports or I’m gonna point at Pete.
⁂
He gets on base; the model does the eval. The model does the eval is the backbone of how one should access and marshall their intuitions into a coherent view on AI progress.
The first awesome conclusion of the model does the eval is that we will achieve every evaluation we can state. Recall that evaluations must be legible, fast, and either a good approximation of a wanted capability or useful itself. The plummeting cost of compute has made all evaluations faster. That includes proving unproven theorems in mathematics, highly accurate isolation of biological effects, and dexterous robotic embodiments, among the other measurable breakthroughs Sam Altman and Dario Amodei named. I used to be uncertain that we could achieve them in my lifetime. Now I feel late to thinking that we will almost surely achieve them, and all their attendant consequences, very soon. Why not overcorrect and predict that we’ll achieve them all in 2025?
Add human intelligence to direct the cheaper compute to get more legible evaluations. Two years ago, Demis Hassabis enumerated three properties of problems suitable for AI: a massive combinatorial search space, a clear objective function to optimize against, and lots of data or an efficient simulator.
Previously, only certain problems fit the bill (Go, poker, proteins). With some human creativity, these properties are within sight for a wider array of problems.
Even solving language is combinatorial if one believes hard enough. Models string together tokens from a large, finite vocabulary. If all language, why not all logical steps or physical actions? The clear objective function is the tough one (what I’ve been saying), but researchers still agree on which evaluations are useful even if they disagree on what solving them means. Finally, reports of demise by diminishing data (December) are greatly exaggerated; autonomous vehicles have bootstrapped themselves to escape velocity, and the demand to automate actions will fund the simulated virtual environments to safely test them in. As Tyler Cowen points out, supply is elastic. We’d pay a lot to solve some of these problems.
Concretely, how do researchers solve problems with these properties? Other than the evaluation, it’s with algorithms and data. (Oh good, I can hear readers say. We’re finally over the boring evaluation stuff, and can get into the juicy algorithm and data details.) I’m going to have to disappoint. It turns out, as far as we know, the algorithms and data are not some holy grail. Any sensible recipe will do. What matters is that the human intelligence set against the evaluations (the researchers racking their brains to crack the evaluations) are very strong, and always finds a way.
Take reasoning. Researchers knew scaling pre-training would get prohibitively expensive a long time ago, and that something like test-time compute (“search” in The Bitter Lesson) was a good candidate for a new paradigm. Now imagine you had that vision, and set the following evaluation for your research program: draw and target the plot that OpenAI published. It’s really less of a plot and more like the concepts of a plot (September). On the x-axis is the amount of test-time compute used. On the y-axis is reasoning performance. Now make that relationship log-linear, go! Noam Brown straight up tells you that’s what OpenAI did: “one of the things we tried that I was a big fan of… was why don’t we just, like, have it think for longer.” Leopold Aschenbrenner alludes to there being no secret sauce as well:
Perhaps a small amount of RL helps a model learn to error correct (“hm, that doesn’t look right, let me double check that”), make plans, search over possible solutions, and so on. In a sense, the model already has most of the raw capabilities, it just needs to learn a few extra skills on top to put it all together.
Since Aschenbrenner’s claim to insider knowledge seems to be load-bearing for his whole manifesto, I’m inclined to believe him.
Turn to agents. This time it’s even Dario Amodei telling you how it’s done:
Yeah. It’s actually relatively simple. So Claude has had for a long time, since Claude 3 back in March, the ability to analyze images and respond to them with text. The only new thing we added is those images can be screenshots of a computer and in response, we train the model to give a location on the screen where you can click and/or buttons on the keyboard, you can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It’s a good example of generalization. People sometimes say if you get to lower earth orbit, you’re halfway to anywhere because of how much it takes to escape the gravity well. If you have a strong pre-trained model, I feel like you’re halfway to anywhere in terms of the intelligence space. And so actually, it didn’t take all that much to get Claude to do this. And you can just set that in a loop, give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on and that turns into a full kind of almost 3D video interaction of the model and it’s able to do all of these tasks.
As for Mariner, I’m proud to say that we’re state of the art on WebVoyager. You can see we eeked out 7 percent going from single agent to tree search. There’s innovation in every bit of what we did, from our base model to our training process to our inference techniques. But I also resonate with Liang Wenfeng’s answer to how DeepSeek keeps up with proprietary methods:
In the face of disruptive technologies, moats created by closed source are temporary. Even OpenAI’s closed source approach can’t prevent others from catching up. So we anchor our value in our team—our colleagues grow through this process, accumulate know-how, and form an organization and culture capable of innovation. That’s our moat.
Beyond capabilities, the model does the eval explains scaling laws and even Moore’s law. AI doubters muttered this year that scaling pre-training underperformed expectations. We used to get bundles of capabilities (in-context learning, instruction following) with scale, they pined. Now we must put in the work to unlock each one, like individual micro-transactions in Candy Crush. But Semianalysis explained why that’s off for Moore’s law:
Today’s debate on AI scaling laws is not dissimilar to the decades-long debate around compute scaling and Moore’s law. Anyone who tries to measure CPU compute primarily by clock speed—a common metric used before the late 2000s around the time of the end of Dennard Scaling—would argue that we have not made any progress at all since then. In reality, compute has been advancing all along—when we hit a wall on processor clock speed, the focus shifted to multi-core architectures and other methods to drive performance, despite power density and cooling constraints.
Gordon Moore set an evaluation: he drew a plot with time on the x-axis, transistor density on the y-axis, and said, double it every two years, go! The same is true even for the history of pre-training scaling up until now. Recall that Chinchilla improved on the original Kaplan scaling laws by correctly tuning the learning rate schedule (Trevor Chow reviews it for those who forgot). Without that discovery, pre-training scaling would’ve already plateaued. These laws were never physical laws we couldn’t rewrite to maintain.
At the most abstract level, the model does the eval describes the entire field of researchers pursuing general intelligence.
Ilya Sutskever’s koan: “Look, the models, they just wanna learn. You have to understand this.” Well look, the researchers, they just wanna optimize. You have to understand this. The researchers, they aren’t very good at philosophy, they aren’t very interested in defining intelligence (even though you might think they are because they call themselves researchers of general intelligence). They just want an important problem to solve, a clear evaluation that measures progress towards it, and then they just wanna optimize it.
We invented the steam engine, vaccines, and the computer without AI. If we cure cancer, terraform Mars, or simulate consciousness, we will use AI insofar as it does what we need it to along the way (the model does the eval). A quarter of all new code at Google is generated by AI (October); in a sense, Gemini is already self-improving. It’s not the singularity we wanted, but the one we deserve as an empirical field.
Is there any limit to the model does the eval? Its second awesome conclusion is that we will fall short on every capability we struggle to state.
RE-Bench (November) is instructive here. This evaluation measures progress on automating tasks in AI research, like optimizing a kernel and running a scaling laws study. Now, because I believe we’ll achieve every evaluation we can state, I predict we’ll saturate RE-Bench soon, and I look forward to the day I can kick off a scaling study in one prompt. But while we now know how to do a scaling study in some detail, we still struggle to state the process that led Kaplan et al. to the insight of applying scaling laws to AI in the first place. We may guess, like Lu et al. did (August), that the step-by-step process of making breakthroughs is as simple as go to school, brainstorm hypotheses, test them, and some extra bits like drink milk. Throwing compute at these steps may even hit gold. But on the way I predict there will be much confusion about if the model is automating research, or just speeding up a researcher’s rote tasks, as a calculator or Google search does.
It’s interesting that while o1 is a huge improvement over 4o on math, code, and hard sciences, its AP English scores barely change. OpenAI doesn’t even report o3’s evaluations on any humanities. While cynics will see this as an indictment of rigor in the humanities, it’s no coincidence that we struggle to define what makes a novel a classic, while at the same time we maintain that taste does exist. How would we write such a concrete evaluation for writing classics, as we might for making breakthroughs? Read a lot, converse with past tradition, eschew the passive voice, and drink milk again. I predict AI will fall short on classic writing as we agree on it, even as it aces all its constituent steps.
We don’t even know what we “really want” until we see it, at which point it may change. We don’t know if a (human or AI-written) novel will become a classic until readers engage with it and time passes. Prices are discovered through the combined actions of buyers and sellers. So while the models will surely do the most ambitious evaluations just being proposed—“an approach for automatically generating and testing, in silico, social scientific hypotheses,” where the model is the agent and the experimenter (April); replicating human responses to social surveys almost as well as the original humans do about themselves (November); testing geopolitical theories (December)—this Hayekian obstacle to legible and fast evaluations will bottleneck progress. I predict that Dario Amodei will not witness “AI improve our legal and judicial system by making decisions and processes more impartial… making broad, fuzzy judgements in a repeatable and mechanical way.” Humans do not lack for definitions of justice, but for trust that another’s definition is right. I predict there will be no oracles that solve Cass Sunstein’s problems (December), or any machine that give an answer more satisfying than “42.”
Which capabilities are had in virtue of which evaluations, what humans ought to do in virtue of what is possible with technology: these questions seem largely detached from empirics. In the very long term, researchers don’t really want a clear objective function to optimize. Once the model does the eval for all the evaluations, we can turn back to choosing the evaluation itself.
⁂
I thank Allison Tam, Arjun Ramani, Avi Ruderman, Coen Armstrong, Eugenia Kim, Gavin Leech, Henry Oliver, Hugh Zhang, Julia Garayo Willemyns, Klara Feenstra, and Tom McGrath for discussing these ideas with me and reading drafts, and Emergent Ventures for general support. Views mine. The cover mosaic is Omar Mismar, Parting Scene (with Ahmad, Firas, Mostafa, Yehya, Mosaab) (2023).
Canterbury Cathedral from St Augustine’s Abbey.
⁂
I consumed a lot of remedial science fiction this year. There are so many things I tell myself that I’ll get to eventually, because so many people agree on it being good that it can’t be a scam, like Shakespeare and The Sopranos. I thought science fiction was a good place to start; it’s accessible, I already have a good chunk done, I was looking for ways people imagined the future, and clearly it’s a big influence on prominent figures in the tech industry, so you have to understand them by reading what they read.
One depiction of AI in science fiction is an infallible one. Not only is AI smarter than you, but also it’s so smart that you have no chance of overcoming it. Two examples are the protagonists of Flowers for Algernon by Daniel Keyes (please bear with my liberal definition of AI) and “Understand,” by Ted Chiang. In these stories, you probably don’t understand why the AI is acting the way it is, but that’s because you’re not intelligent enough—trust me, it’s optimal. The AI is honoring you by even thinking about you, like a human thinking about an ant. There are parallels in real-life hopes and fears about AI capabilities. Other than the great calculation debate (can you simulate the future and centrally plan the economy) I’m thinking of superhuman persuasion. Advertising does work on me more than I know. But I find it harder than ever to oppose this argument—a sort of appeal to God—that whatever we do, the AI will just outsmart us. Superhuman persuasion tangibility-washes the mechanism for doom. Just add in the step to persuade a human to do the inconvenient bits. In the end that argument may be right, but so far I’ve been able to just stop engaging with that persuasion while remaining firmly unpersuaded. If someone is mad about that, well, they dug their own hole.
Netflix’s 3 Body Problem reminded me why the idea of infallibility appeals so much to me anyway. (A big spoiler of the whole trilogy follows.) People praise the series for depicting humanity collaborating to face an existential challenge. I’m all for optimism and humans, but am less impressed with the actual implementation details of our success. Somehow our crowning achievement is the Wallfacer program, where we all agreed to give four people, the Wallfacers, unlimited power to do whatever it takes to defeat the aliens. Conveniently the aliens can intercept all human communications, so the only plans that have a chance are those that never leave the mind. Now if one of these Wallfacers have to violate some rights to throw off the aliens, or even better, as part of some 4D chess, no problem, we’re not smart enough to understand the grand plan. The plan that ultimately works (again, spoilers) is the epitome of cleverness itself. We figure out some mutually assured destruction. It’s a shame the solution is so interpretable, really, as it would better show off how smart we are to get out of this pickle if the solution was impenetrable. That wouldn’t make for very entertaining television, though. We just love to glorify the long-awaited fruition of a clever idea.
Similarly, two years ago I didn’t like Isaac Asimov’s Foundation, in which some really smart humans predict the arc of history. (Of course Dune: Part Two is this year’s counterweight.) I reread Isaiah Berlin’s “The Sense of Reality,” which, while not science fiction, includes this graf which must have been what Berlin would’ve said if he read his contemporary’s series and all the research it inspired:
The thinkers of the eighteenth century had been too deeply infatuated by Newton’s mechanical model, which explained the realm of nature but not that of history. Something was needed to discover historical laws… to claim the possibility of some infallible scientific key where each unique entity demands a lifetime of minute, devoted observation, sympathy, insight, is one of the most grotesque claims ever made by human beings.
Berlin would have had Leo Tolstoy in mind instead. Tolstoy would not have read Asimov, being dead, and Asimov failed three times to read War and Peace, even in translation. Now that we’re giving another full-throated effort to this grotesque claim, this time with even more data and compute, we should all fix these ships passing in the night (visitors to my flat will know I’ve already failed once this year to read War and Peace).
There’s another bucket of science fiction I’ll call “business as usual.” In these stories, there’s a unique premise or worldbuilding element, like the lack of empathy in PKD’s Do Androids Dream of Electric Sheep?, the ability to copy and transfer consciousnesses to other beings in Arkady Martine’s A Memory Called Empire, the ability to split one’s consciousness into multiple bodies in Ann Leckie’s Ancillary Justice, just the generally ideal AI Minds in Iain Banks’s The Player of Games, and the infallible HAL-9000 in Stanley Kubrick’s 2001: A Space Odyssey:
Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error.
These stories ostensibly explore what a world with these concepts looks like, but something about them feels very normal. It’s as if the characters started in a normal story, gained a new ability, and took it in stride doing mostly what they were going to do in the first place. Androids identical to humans except for empathy: police departments still need bounty hunters; 15 generations of consciousness in a single line and being: we’re still dealing with a standard-issue succession crisis; the emperor’s multiple strands of consciousnesses are fighting themselves: the protagonist is still out to get revenge; space socialism: let’s go play some games with the technocratic totalitarians; a computer that’s never made a mistake: this movie gets weird at the end, but not because of technology.
It’s not that the creators of these worlds are uninterested in examining where their concepts go. It’s hard, in part because the future is just impossible to imagine—like how in real life, Apple and Microsoft advertised that the personal computer could make birthday cards because they hadn’t figured out its killer use case yet (spreadsheets). We can imagine the first-order effects of new concepts well enough, but solving for the full equilibrium is hard. The other problem is, we’re also interested in a fun, understandable story. We got what we wanted (space socialism in The Culture), let’s just fantasize a bit with that in the background. Star Wars is the best example. The Separatists basically have artificial general intelligence in their droid army, while the Republic (the “good guys”) are growing and shipping off real human clones to die. I get that it’s a lot easier to empathize with good guys when they’re human, and we don’t want to empathize with the bad guys who get destroyed. But come on, clearly the Separatists are not only more moral, but also inevitably going to win because they’re so far ahead in autonomous warfare. Yet nobody seems to realize that the galaxy is forever changed. Nope, just first-order effects, business as usual, adventure after adventure. I still love Star Wars because I know I’m there just for fun. I shouldn’t hold other space operas to a different standard.
I’ll stop being grumpy. The last and best bucket of science fiction created profoundly different and convincing worlds. The plot was either in service of the concept or so categorically different that making sense of it wasn’t a relevant concern. I’m thinking of Kazuo Ishiguro’s Klara and the Sun and The Buried Giant (more about memory than intelligence per say), and Stanisław Lem’s The Cyberiad and Solaris.
Ishiguro, well, I cannot in earnest recommend his books to anyone. Not because they’re bad—they’re excellent—but because his whole unreliable narrator shtick makes for an incredibly stressful reading experience. From the very beginning he evokes a gnawing uneasiness. There’s no big reveal, no single moment when you understand what’s going on. Rather, all you need to know slowly seeps in, until suddenly you realize you’ve had all the pieces for a while. What follows is not relief, but dread, as the horrible implications of what you now know descends. Ishiguro’s worlds and characters are completely changed as a result of his MacGuffins. In Klara and the Sun, the mother’s whole life and the childrens’ whole socialization revolves around the exact characteristics of artificial friends. In The Buried Giant, the mist is the source of all questions of identity and culpability. Ishiguro must have known something about language models in 2020 when he wrote Klara and the Sun. Children growing up will relate to Klara, not a computer as emotionless and precise as HAL-9000.
Lem doesn’t just achieve the purpose of science fiction better than anyone, but he’s also the most entertaining. The premise of Solaris is simple. There is extraterrestrial life that is very different from humans, the humans are studying the alien, and they learn more about themselves than they do about the alien. No part of the plot would be out of place in a psychological thriller. But serving the purpose of conveying how unnatural an alien is, it works all together better than stories ten times the length. I hear the Tarkovsky adaptation is also good, but for more human reasons, so don’t pass on the book. I know Lem is already well-regarded, but I’m surprised he’s not even more so.
The Cyberiad even captures the essence of modern AI better than anything else. There’s the story that inspired SimCity, that predates Nick Bostrom’s simulation hypothesis by decades. There’s also the story of Pugg, “not your usual uncouth pirate, but refined and with a Ph.D.” Pugg desires infinite information above all else, which proves to be his downfall.
Information had so swathed and swaddled him… that he couldn’t move and had to read on about how Kipling would have written the beginning to his Second Jungle Book if he had had indigestion just then, and what thoughts come to unmarried whales getting on in years… so poor Pugg, crushed beneath that avalanche of fact, learns no end of things.
In other media that I consumed this year, I came across a rare trait only twice. It’s a kind of melancholy I get when the story is over. But not because I necessarily wish it were longer—I get the bizarre realization that the characters aren’t real. Not just that they were real for a time in my life and now we’ve parted ways, but that they were never real in the first place. I had a similar experience when I stopped being religious. The worst wasn’t knowing that the divine wasn’t listening to me anymore, but comprehending that my beliefs meant that that was never the case. For a story, this is a high compliment. I have to relate to the characters, and consume it over an extended, formative time in my life.
The first story I felt this for was Free Food for Millionaires by Min Jin Lee. I finally got around to it two years after loving Pachinko (which was too grand of a plot to relate to). Free Food for Millionaires is about community, identity, ambition, and all the usual coming of age suspects. New York is close enough to London. I saw myself in, or have met, many of novel’s characters. Min Jin Lee is one of the only authors I’m a completionist for. I wish she didn’t take a decade to write a novel, but it makes sense because they’re that good.
I felt the same after the series finale of Star Trek: Lower Decks created by Mike McMahan two weeks ago. It’s not a very serious cartoon. But when I wrote about it two years ago, I praised it for its lack of cynicism, which is why I think it’s aged well. I also started watching the show in the second season after I just moved to London. So I really do feel like I did several tours of strange new worlds alongside these other low-ranking crew members, and have now also been promoted to lieutenant junior grade. After childhood, there’s not many opportunities for new television to win nostalgia points or to latch onto just-formed neural pathways. Lower Decks snuck in just before the buzzer, now that my brain’s not as plastic as before.
Finally, in more classic reading I took a wonderful class by Jiayang Fan about Virginia Woolf. We read Mrs. Dalloway and To the Lighthouse. Previously, I was quite embarrassed about trying and failing to finish Mrs. Dalloway. It’s so short, it’s in English, and I even know all the place names. I have better expectations about how hard I should find it now, and about how rewarding Woolf is on rereads and reading with others. I got better at reading alone (by interrogating a model), but having reading companions who also live in the present is incomparable. For example, who thinks the first line of Mrs. Dalloway and why? None of the AI are brave enough to say.
⁂
Other than London and Phoenix, I spent time in Mexico City, Aviemore, Dublin, Beijing, Horsham, Reykjavík, Rotterdam, Mumbai, Lonavala, Warsaw, Krakow, Taipei, Pingtung, Kaohsiung, Karuizawa, Yokohama, New Haven, Portland (Maine), Eastbourne, the San Francisco Bay Area, Paris, Canterbury, Dover, Venice, and Flagstaff. I got to experience many things other people have before (sometimes a billion others) but were new to me. All I can say is I was overwhelmed—overwhelmed by the Northern Lights in Iceland, the Olympics in France, the affordability of Burberry in Japan (forex and foreigner tax exemption), Lunar New Year with my extended family in China, and about almost everything in India.
Crossing the street in India is very efficient. In America, you have to wait for the pedestrian light at the intersection. The streets are wide and the urban design is unwalkable. Drivers will freak out if you jaywalk even if you’re worlds away and everybody will be fine. Plus, jaywalking is illegal. In London, jaywalking is legal, the drivers are used to it, and the streets are narrow. In Mumbai I came to understand the perfect balance for maximum crossing on wide streets. Cars are coming all the time, so you’re stuck forever if you want the state to help you out. What you have to do is step in the street. The drivers don’t freak out, but instead do some quick math in their head. They go from barrelling at you at high speed to barrelling at you at a slightly less high speed, just fast enough such that at the speed you’re walking, they’ll zoom by behind you the moment you pass. If you trip, it’s over for you. But otherwise, you get to cross whenever you want, and the cars get to keep their momentum. This just-in-time compilation is happening all over the place; you just have to get used to it. I went from being too scared to go anywhere the first day to complaining about my Uber driver for not being assertive enough by the end of my trip.
Compared to Japan, Taiwan is incredibly underrated as a travel destination by Westerners. The main reason is food (surprisingly, as food is one of Japan’s strong points). Food in Japan is some of the best in the world, but food in Taiwan has three advantages still. First, breakfast. In Japan, breakfast is either not there (at least not as a coherent category), congee (unoriginal), or grilled fish (it gets old). Taiwanese breakfast is a whole world unto itself, inheriting all the hits from Chinese breakfast (crepes, bao, dough sticks) as well as Western influences (toast, sausages). Even the diplomatic AI knows—when I asked it to list the staples, it declared Taiwanese breakfast “a vibrant and varied scene,” while rating Japanese breakfast only a “savory and balanced affair.”
Second is accessibility. To eat a good meal in Japan you often have to sit down. Sometimes you have to find the entrance on the street to go up a skyscraper, which can be a big lift for the navigationally challenged. In Taiwan, street vendors are everywhere, not just the night markets. They sell proper meals as well as snacks. I’m sure I could live off of walking downstairs. I sent a photo of every meal I had there to my family, with only its (very low) cost in dollars as the caption.
The third and biggest advantage is vegetables. This is a more general point that people may have made their minds up on already. People go to Japan to eat the absolute best quality of certain ingredients, accepting that they’re giving up on the diversity of produce and flavors from China and Taiwan. In that case, I’ll make a stronger claim: except for sushi, every single dish in Japanese cuisine is done better by another Asian cuisine: jiaozi over gyoza, pakora over tempura, pho over ramen, mul naengmyeon over soba, hot pot over shabu shabu, Indian curries over Japanese, and Korean barbecue over yakiniku (except wagyu). When I workshopped this theory I heard a feeble defense raising okonomiyaki. I did have to think, but jianbing and bánh xèo still win. Unless you plan on eating sushi every day (I would), choose Taiwan.
I’m sorry to pick on Japan, but their tourism income can take it. I admit, though, that my comparison is missing the forest for the trees. Either is better than staying West; Fuchsia Dunlop’s Invitation to a Banquet puts how I always felt into words:
A Chinese friend of mine from the Jiangnan region was goggle-eyed during a meal with me in Piedmont, when he saw people at a neighbouring table lunching on agnolotti dumplings in a buttery sauce, followed by braised beef with buttery mashed potato, with no plain vegetables and followed by a rich apple tart. Eating like this will stoke the inner fires!’ he said. ‘Just looking at it gives me a headache? He enjoyed our own lunch, but said: ‘Eating like this once is OK, but for a Chinese, twice would be unbearable…
But while my ideological commitment to mutual respect and recognition is undimmed, and while I love shepherd’s pie, fish and chips and toasted cheese sandwiches as much as the next English person, I have to confess that decades of privileged eating in China have turned me into a terrible Chinese food snob. Increasingly, I don’t believe any other cuisine can compare. This is not primarily because of the diversity of Chinese food, its sophisticated techniques, its adventurousness or its sheer deliciousness - although any of these would be powerful arguments. The reason, fundamentally, is this: I cannot think of another cuisine in which discernment, technique, variety and sheer dedication to pleasure are so inseparably knit with the principles of health and balance. Good food, in China, is never just about the immediate physical and intellectual pleasure: it is also about how it makes you feel during dinner, after dinner, the following day and for the rest of your life.
Finally, a personal reason I like Taiwan so much is language. I only realized on visiting how natural the combination of Mandarin and English is. In China and Japan people avoid speaking English if they can, sometimes because they don’t want to speak English or sometimes because they don’t think they speak it very well. Taiwan has a policy to make English an official language, which I’m sure has been a countervailing force. Between the food, language, people, and all the other fine details of life, I’m convinced utopia does exist on Earth. It looks like garbage trucks that play music like ice cream trucks, tram tracks laid across tended grasses, and the most shocking item of nightly news being one driver cutting off another on the highway.
Perhaps I have an axe to grind. When I flew from Taipei to Tokyo I took Peach Airlines, a Japanese airline. I wanted to sleep the whole flight, and I’m pretty used to everything flight attendants want from me. I thought the contract was: seat upright, luggage overhead or under the seat in front of you, nothing plugged in, seatbelt visible, and you can doze off during the safety briefing. So imagine my confusion when a flight attendant still woke me up before takeoff. My sin was that I fastened my seatbelt with a single, rebellious twist, marring its otherwise flat and perfect path.
I have American caricatures to peddle, too. I took the Amtrak from New Haven to Boston Back Bay Station, where I was going to then take a bus to Portland, Maine. The train was late because we stopped for “some engineers” to fix a drawbridge that couldn’t close along the way. That the train crossed it without issue was the first relief, since the Francis Scott Key Bridge collapse just months before was still on my mind. But now the train was arriving at Back Bay just ten minutes to when the bus was supposed to leave, and I’m pretty sure the Bay Bay train to bus transfer is one of the most convoluted transfers in the world. I can imagine the architects cackling (“and then we’re gonna make these passengers leave the station, run across some road work, and find this side building, all without signage”). In theory, one could make this connection in ten minutes if they knew what they were doing, which I did not. But the next bus was not for another hour, so I spent my last minutes on the Amtrak desperately reviewing the directions with the conductor.
In true Boston spirit (gruff but practical kindness), the conductor threw open the door before the train completely stopped so I could start running. I saw two other passengers hit the pavement running too—and one of them immediately tripped and fell. I stopped for a moment, but she looked fine, so I made the executive decision to keep going. I made it to the bus just as it was leaving, catching the driver’s eye, so we all made it in the end (proud of myself for that). The woman who fell caught up as well. Another big win for haphazard process and friends along the way leading to last-minute compliance. But the story doesn’t end there. On the bus, the woman’s fall started catching up with her. I overheard her debating with her friend in Maine over the phone if she should go to an urgent care when the bus arrives. I wonder if her medical bill will have been worth catching the bus. So many consequences, all stemming from not owning a car.
Back in London I’m mostly used to the British character after three years. I’m still thrilled to get some rare nuggets, though—where else in the world will you overhear the following? “In the guild I belong to there is an artifact called the Wicked Bible.” And, “What do you mean this doesn’t look Christmasy? We invented the Dickensian idea of Christmas as you know it!” And, when I went to St. Augustine’s Abbey I learned the story of Peter of Dene, a rich lawyer who became a monk to avoid Edward II’s wrath. When his life was no longer in danger, he tried to leave the abbey, but was denied. So he escaped. As to whether that was fair, the audio guide told me, “he would get his due in hell.”
This was the first year I seriously thought about where I wanted to live in the future. But even the thought of possibly leaving London has inspired sudden rushes of affection for the city. I can’t think of anywhere else where, for the low price of being just one of the two AI hubs of the world, you can be the first to see the new projects of Armando Iannucci, Jez Butterworth, Sarah Snook, and my favorites Wayne McGregor and Max Richter, board a Spanish galleon, queue for front-row seats at Wimbledon, hear all kinds of storytelling, and eat fruit that actually tastes like the fruit it’s supposed to. I reveal too much about what I like, but trust me, London has what you like, too.
I’m even fonder of London’s way of social climbing, definitely a wart, than its alternative. The Bay Area intelligentsia are (rightly, at the end of the day) proud of themselves for living in the future. One of their smoke tests seems to be if one has heard of certain proper nouns, like of the research lab Anthropic. If you haven’t heard of Anthropic, then the Bay Area will not consider you a serious person. (Claude is the engineer’s engineer, and it has basically no name recognition anywhere else.) Sometimes people will make you aware of their seriousness with a sledgehammer—“you wanna to hear a funny story? me and the [redacted] team stayed three hours after an xAI party because Elon tried to poach us haha” (a deeply unfunny story). In return, I came up with an equally arrogant proper noun test of my own, and found very many thought leaders who had never heard of George Kennan. But that’s too on the nose, too American. Apparently the subtler, more British cue is if one clinks their glass or just raises it when they say cheers—you should raise, because you’re too used to using family china too valuable to clink, or too used to hanging around people who do. The great British sin is taking yourself too seriously; avoiding it is a healthy digestive. At some point I’ll stop writing these love letters to London, but the city proves itself supremely livable year after year.
⁂
I’ll end with personal productivity. Despite the above, my year was not just fun consumption day after day. After three years, there’s no avoiding the truth that certain fundamental habits have gotten worse.
Three years doing one thing full time has made me better at it, but at the cost of breadth, and even curiosity. I wouldn’t give up specializing, still—I shudder to remember all the basic things I didn’t know when I graduated. Some naivete is helpful, but I always feel more relief than triumph when I look back that I somehow lucked into the right amount of rational self-delusion. For everything I spent less time on, I feel some new sense of urgency. A very concrete example is that my hamstrings are tighter than ever. I stretch, but it’s been long enough to see that my routine is clearly not enough to counteract the huge amount of sitting I do. The problem is more general than exercise, though. If I want to read, play music, practice Chinese, and pick up new hobbies, and it isn’t happening more by now, what makes me think it has a better chance of happening later? It’s time to either change or quit. I was optimistic last year about a big virtuous cycle where doing everything makes everything else easier. I’m going to take the opposite view this year, that I should be honest about the actual tradeoffs I face. Either way it sounds trite, but I’ll figure it out one day.
I’m leaning on the conservative side after a terribly busy time in late summer. In my mind I divide my projects into two categories: black hole, and not a black hole. Projects that aren’t black holes have concrete parameters, low variance in outcome, and so are easy to timebox. Black hole projects can in theory always be better the more I put into it, so they’re a black hole to sink time into. Halfway through the year I found myself with four black holes—two research projects, a writing project, and moving house. In practice, four black holes just means that I only did one well at a time, taking on lots of overhead costs, borrowing from things like sleep, and feeling the guilt of not meeting black-hole-level expectations anyway. Never again.
One way to achieve focus seems to be action. Thinking about all the things I’m supposed to do is paralyzing, and as a result I end up acting no more than if I had less I was supposed to do in the first place. Prompted by Tanner Greer’s blog, I finally got around to reading George F. Kennan: An American Life, by John Gaddis, a former professor of mine. I used to think Gaddis a bit overrated as a professor because of the platitudes he would repeat, but now I appreciate him and how he centers narrative in history more and more. Finding and making the case for a single throughline seems for the moment an unpopular way to do history, but I found it so refreshing:
That was how Kennan understood his responsibilities. “Look,” he remembered telling the staff, “I want to hear your opinion. We’ll talk these things out as long as we need to. But, in the end, the opinion of this Staff is what you can make me understand, and what I can state.” There was, he believed, no such thing as a collective document, because “it has to pass, in the end, through the filter of the intelligence of the man who wrote it.”
I knew that I could not go to [Marshall] and say to him: “Well, you know, this paper isn’t a very good one. I didn’t really agree with it, but the majority of the Staff were of this opinion.” The General would not have accepted this from me. He would have said: “Kennan, I put you there to direct this staff. When I want the opinion of the staff, I want your opinion.” So, I did insist that the papers had to reflect my own views.
Kennan recalled listening to the staff with respect and patience. He quickly discovered, though, that when “people get to talking, they talk and they talk. But they don’t talk each other into conclusions.” He once admonished them:
We are like a wrestler who walks around another wrestler for about three minutes and can never find a place where he wants to grab hold. When you get into that situation, you then have to take the imperfect opportunity. There’s going to come a time in each of these discussions when I’m going to say: “Enough.” And I will go away and write the best paper I can.
So I’m making some changes. The first will be winter break. John Littlewood gave a very specific recommendation in “The mathematician’s art of work:”
Holidays. A governing principle is that 3 weeks, exactly 21 days—the period is curiously precise—is enough for recovery from the severest mental fatigue provided there is nothing pathological. This is expert opinion and my expertise agrees entirely, even to the point that, for example, 19 is not enough. Further, 3 weeks is more or less essential for much milder fatigue. So the prescription is 3 weeks holiday at the beginning of each vacation. It is vital, however, that it should be absolutely unbroken, whatever the temptation or provocation.
At the moment I post this, I’ll be on my Littlewood’s 21 days. He’d probably take issue with my still thinking about work during the break, but I hope I can offset that with some more days of vacation. The goal, if I can manage it, is to recover some ability to think and focus over the long-term, which I’ve lost since college between the screen time and the context switching.
Also on focus, I want to second a thought from Dan Wang’s letter from January. I like his letters so much that they’re the whole inspiration for mine.
In the last two weeks of the year, I sort through everything, try to coax out a structure, and then write the damn thing. I’ve complained about how much work it demands, but I also want to say that it has been great fun. I don’t understand why more people aren’t writing them. It’s not just about sharing your thoughts and recommendations with the rest of the world. Having this vessel that you’re motivated to fill encourages being more observant and analytical in daily life too.
He’s right. This letter, the only deadline I give myself every year, is an immensely powerful nudge to do more interesting things during the year, if only subconsciously. It’s the best antidote to the temptation to timebox research, writing, or enjoying life.
Here’s a picture of the new cake branding iron my family got. It made me laugh.
