Application development without programmers

This book by James Martin published in 1982, includes the following in the preface:

Applications development did not change much for 20 years, but now a new wave is crashing in. A rich diversity of nonprocedural techniques and languages are emerging. As these languages improve, they promise to change the entire fabric of DP development.

This means a major change for many of the personnel involved in DP, from the DP manager to the junior programmer. DP personnel have always welcomed new hardware and software, but it is not as easy to accept fundamental changes in the nature of one's job. Many DP professionals and, not surprisingly, programmers will instinctively resist some of the methods described in this book.

(I had to look up DP - it stands for Data Processing, and was a common acronym for general IT work up until the 1980s.)

I enjoy they way this echoes with today's fears of the impact of AI-assisted programming on developer careers!

The early 80s were a wild time for computing:

Unfortunately, the winds of change are sometimes irreversible. The continuing drop in cost of computers has now passed the point at which computers have become cheaper than people. The number of programmers available per computer is shrinking so fast that most computers in the future will have to work at least in part without programmers.

Via @codythecoder on TikTok

Tags: careers, ai, ai-assisted-programming

I’ll spell it out for you guys - CBP provides open free WiFi in the immigration line at the airport where the typical person with no mobile service is on a visa.

This is a very different consideration now in the era of wiping phones than three years ago.

I went and (co)wrote another Mothership adventure, this time a one-shot. You can find Ship of Fools on itch.io or DriveThruRPG, whichever you prefer.

Leviathanic, a luxury space cruise ship, has suffered a giant space amoeba related misfortune. Within hours, all its crew and systems have been severely compromised. According to the procedure, the ship’s AI has woken up several passengers most qualified for the task at hand. Too bad some of them are not who they said they are.

The premise, as you can see, is fairly straightforward. None of the high-concept weirdness of Terror Signal, our previous sandbox campaign (though there are tiniest of references connecting the two). There is, however, a twist. Naturally, it’s best if you don’t know it if you’re going to play the adventure. If you’re going to run it, read on.

None of the characters are the handpicked, qualified specialists they claimed to be. Each PC is given a brief backstory explaining how they came to be on Leviathanic under an assumed identity. The text also strongly suggests they should hide this fact from others, rely on their more competent colleagues, and deflect blame.

The entire adventure, thus, is a black comedy of escalating disasters that are in practice indistinguishable from a regular Mothership session until the final reveal. It feels like you’ve pulled off a magic trick when, as you conclude the adventure, one of the players admits their character’s big secret, and everyone suddenly starts talking over one another as the realisation sinks in.

After I ran it for the first time, an utterly unexpected thing happened: the players decided they really wanted to see how others would fare in the same circumstances, and were happy to play it again, this time being in on it from the start. How many adventures have you played that you’d immediately want to replay?

It took some time to figure out how to facilitate such a game. How do you play a character that cannot do what their character sheet says they can, how do you handle that as a GM? Turns out, it’s quite straightforward. When a character tries to do something that would require an expert or master skill (that their sheet has but they don’t), they make the check as usual. On a failure, they make everything worse and have to look for a different approach (giving them a chance to deflect the responsibility onto others), which is how Mothership works anyway. On success, they still don’t do it, but don’t actually screw up the situation further, and the Warden gives them a hint as to what’s required to succeed.

From this, the structure of the adventure followed. Each location has a PC “associated” with it, who would obviously come in handy there. Medbay – surgeon, reactor – spaceship engineer, etc.. When the party gets there, all eyes turn to them. Except they can’t actually do the things expected of them, so they’ll have to look for excuses, screw things up, and ultimately shove the responsibility onto someone else. Medbay medical systems are on the fritz, we need to do a proper reset from the Bridge. Which is where a different PC has to step up and slip up. In order to help them, each location comes with suggestions for the Warden on how the PCs could make things worse there, and what sort of excuses they may use to redirect the responsibility.

The PCs bounce around the ship as things turn from bad to worse.

In other words, a typical Mothership session.

With just a modicum of help from the adventure itself.

Grok 4 Heavy won't reveal its system prompt

Grok 4 Heavy is the "think much harder" version of Grok 4 that's currenly only available on their $300/month plan. Jeremy Howard relays a report from a Grok 4 Heavy user who wishes to remain anonymous: it turns out that Heavy, unlike regular Grok 4, has measures in place to prevent it from sharing its system prompt:

Sometimes it will start to spit out parts of the prompt before some other mechanism kicks in to prevent it from continuing.

This is notable because Grok have previously indicated that system prompt transparency is a desirable trait of their models, including in this now deleted tweet from Grok's Igor Babuschkin (screenshot captured by Jeremy):

In related prompt transparency news, Grok's retrospective on why Grok started spitting out antisemitic tropes last week included the text "You tell it like it is and you are not afraid to offend people who are politically correct" as part of the system prompt blamed for the problem. That text isn't present in the history of their previous published system prompts.

Given the past week of mishaps I think xAI would be wise to reaffirm their dedication to prompt transparency and set things up so the xai-org/grok-prompts repository updates automatically when new prompts are deployed - their current manual process for that is clearly not adequate for the job!

Tags: ai, generative-ai, llms, grok, ai-ethics

(Me elsewhere! - have a look at this piece I wrote for the Starling Trust Compendium on “Beyond Accountability”, which I actually think is my clearest explanation so far of some of the points that get dealt with in rather a rush in the last chapter of The Unaccountability Machine. )

Sorry, a day late this week as the summer disorganisation is upon me; there will be a special guest post tomorrow though! I’ve been thinking for a few days about this piece on “flexible bureaucrats” from Ivan Vendrov, which is a bit of a cousin of my own recent “flat cap corporatism” post. What I think we’re both trying to work out is the fuzzy and indistinct borderline between useful flexibility and broad based communication, versus sclerosis and corruption.

But is it even a borderline? We have, loosely speaking, two things we’re trying to achieve, which are basically inconsistent with each other. As I said a few weeks ago:

“When we build state capacity, we are trying to achieve two broadly incompatible goals – we want a predictable and transparent system that’s flexible and responsive. So we build up levels of rich and informal communication and facilitation, until they turn into a sclerotic nest of rent seekers. Then we build up layers of clear and detailed regulation, until they turn into a sclerotic mess of rent seekers of a different kind.”

In these sorts of circumstances, it’s normal to think about things in terms of a tradeoff. But it might not be right.

The fact that you can’t have all you want, when it comes to two incompatible but desirable things, absolutely does not mean that there exists a stable and comprehensible exchange rate between them. It doesn’t guarantee that there’s a one-to-one or even many-to-one relationship between the two quantities. Even if we beg some pretty huge questions and pretend that “predictability and transparency” can be quantified and “flexibility and responsiveness” can also be quantified in a comparable way, we don’t have a defined budget of “legal and organisational mana points” to allocate between them. This is, sadly, not a videogame.

So although the phrase “trade-off” is unavoidable, I think it is often a red flag for over-simplification. We can do things that make the system more or less flexible, and we can do things that make it more or less predictable. Sometimes but not always, improving one dimension will worsen the other; a lot of the time, we might actually find ourselves dealing with the consequence of policies which make the system worse along both dimensions but which need to be adopted for other reasons.

Adding to the sum of our problems, the world is always changing. So there’s not even a stable target to aim at. If we’re trying to keep our institutions consistent with long term viability – to play the function of the “higher levels of the metasystem”, in Stafford Beer’s terminology – then it looks like the best we can hope for is to get a sense of what looks like it might make things better in the here and now. Rather than optimising a tradeoff, we just react to local conditions and ask questions like “are the problems we’re currently facing more like the ones we’d expect to see in an excessively rulebound and inflexible system, or in a corrupted and untransparent system?”.

This is the sort of thing I’m always seeing in the day job with respect to financial regulation. Senior people are always warning against a “regulatory pendulum”, whereby the system goes from excessive deregulation, to crisis, to overregulation, to stagnation and over again. But maybe they’re wrong to complain? Maybe it’s a chimera to ever have thought we could alight at the exact equilibrium point where the cost of regulation was as low as it could be brought without taking excessive risk of crisis.

Maybe this kind of oscillation between overcorrections is itself the equilibrium. Viability at the system level doesn’t have to be a single steady state. Maybe it’s just more cognitive effort than it’s worth (or more effort than it’s possible to bring to bear, given that bureaucratic and legal change is necessarily a slow process) to improve the “tracking accuracy” of the system beyond what we have now. Deregulation versus regulation – monopoly versus competition – even investment versus consumption or distribution versus growth – maybe these should be treated as tactics rather than principles. Rather than picking one side of the debate and supporting it as if it was a football team, we ought to just say why we think one tool rather than another is appropriate for the job at hand.

Yesterday, METR, one of the many budding nonprofit “AI Safety” institutes in Berkeley, released a study purportedly showing that AI tools slowed down expert developers. Here’s the plot they led with in their announcement:

Because METR, like all AI Safety Institutes, is run by devout rationalists, the first four bars here are the values of predictions. Everyone asked said AI would speed up development time. The final bar is the observed result. Ackshually, AI slows down development time.

This result inevitably broke the internet. The reaction on Twitter was “Bro, that’s clearly wrong.” The reaction on Bluesky was “See, I told you so.” I didn’t check LinkedIn. I wonder if METR’s cross-posting was itself part of an ethnographic study of confirmation bias.

On Bluesky, Akhil Rao smartly pointed out that the error bar on the speedup is very close to zero and hence must be fragile. It’s a good rule of thumb. But he didn’t want to be “the seminar standard errors guy.”

Long-time readers know that horrible standard errors guy is me. I was ranting about AI papers and error bars last week. I wrote a whole paper about inferential error bars being a silly exercise unless we are using them as part of a clear regulatory rulebook. And hey, if everyone is freaking out about a study “proving” something with frequentist statistics, I’m going to be the sicko who goes and reads the appendix. You will not be surprised that I think the error bars are made up, even if there’s some interesting ethnography we can take away from METR’s work here.

Here’s a summary of how the study worked:

• METR recruited 16 developers

• All 16 brought approximately 16 coding tasks to the study that they wanted to complete.

• METR assigned each task to one of two conditions at random. Condition 0 is “you can’t use AI.” Condition 1 is “you can use AI.”

• The developers completed their tasks in whichever order they desired. METR recorded the completion time for each task

First, I don’t like calling this study an “RCT.” There is no control group! There are 16 people and they receive both treatments. We’re supposed to believe that the “treated units” here are the coding assignments. We’ll see in a second that this characterization isn’t so simple.

As experimenters, our goal is to figure out how the times in Condition 0 compare to those in Condition 1. There are a couple of ways to think about how to study this. If I were to take my guide from the “potential outcomes” school, I’d say something like the following.

There is an intrinsic amount of time each task takes in each condition. Unfortunately, I can only see one condition for each task. I can’t complete the tasks both with AI and without AI. I thus have a missing data problem. I can only see half of the times I care about in each condition. Perhaps I can use some clever math to estimate the actual average time in each condition had I seen everything. And I can use even more clever math to estimate the uncertainty in my estimate. You know, error bars.

This isn’t that crazy if you can sample uniformly at random. Let’s say you have a bunch of playing cards face down on the table in front of you. You want to estimate the proportion of the cards that are red. If you flip half of them uniformly at random, the proportion of red cards in your sample is a reasonable estimate of the proportion of red cars in the sample you didn’t see.

The problem in this study is that there are a bunch of steps required to go from randomizing the tasks to completing the tasks that cloud the estimation problem. These intermediary steps bias your estimation problem. The developers get to choose the order in which they complete the tasks. Task completion order can affect the amount of time it takes you to complete the task. For instance, if you're an expert developer and asked to use a new tool, you'll become faster with it after using it 8 times. Similarly, you can imagine that if you do a bunch of tasks in sequence, your time on task 4 is going to be sluggish because you are tired. There are lots of stories I can tell. There are undoubtedly many conditions that affect the time measurement other than their random assignment.

These other effects are called spillover or interference effects. In randomized trials, experimentalists work their tails off to remove these effects to ensure they are isolating the effect of the intervention. The ideal experiment satisfies SUTVA, the Stable Unit Treatment Value Assumption, which asserts that the only thing that affects a measured outcome is the assignment to the treatment group or the control group. This is a rather strong modeling assumption, but you perhaps could convince yourself that it is close to true in a carefully controlled, blinded study comparing a drug to a placebo.

Unfortunately, SUTVA does not hold in the METR study. And that means we have to bring out the econometrics to tell the difference between the groups. Here’s where we get to wrestle in the mud with statistics. If you go to the appendix of the paper, you can bore yourself to death with regression models and robustness checks. The main model that they use to make the widely circulated plot (from Appendix D on page 27) is this one:

This model more or less says the following: each developer estimates how long a task will take. The median time for each task without AI is a constant times this estimate. The median time with AI is a different constant times the developer estimate. The observed time is the predicted time multiplied by the speedup time for the condition, multiplied by a fudge factor, which is a random variable. The fudge factor is independent of the AI condition, the person’s time estimate, and all of the other fudge factors. The expected value of the fudge factor is one for all tasks. The variance of the fudge factor is the same for all tasks.

That is, if you’d prefer something that looks like code,

time(task) = speedup(AI_condition) * est_time(task) * fudge_factor(task)

Is that a reasonable model? The AI condition was assigned at random, so it should be independent of the estimated time. But the model assumes everyone experiences the same speed-up/slow-down with the AI tools. Since the fudge factor now includes the developers' choices for the order in which they perform the tasks, they can’t be probabilistically independent.

OK, so the model is not literally true, but perhaps you can convince me that it’s predictive? All models are wrong, amirite? The authors will now provide a bunch of “robustness checks” to convince you that it is close enough. Now we’re spending pages analyzing normal approximations when maybe we should accept that a precise estimate of “speedup” is impossible to measure with this experiment’s design.

After thinking about this for a day, the only data summary I’m happy with would be the following simple analysis: “there are 256 coding tasks, each has an intrinsic time inside of it, when we flip coins we get to see 128 of the times in Condition 1 and 128 of the times in Condition 2. Here are the means and standard deviations of these times. We could then all move on. I mean, rather than boring us with these robustness checks, METR could just release a CSV with three columns (developer ID, task condition, time). Then the seminar guys can run whatever dumb check they want.1

Let me be clear here, I don’t think METR’s analysis is better or worse than any random quantitative social science study. Measurement in quantitative social sciences is always fraught with an infinite regress of doubt. That doesn’t mean we can’t tell reasonable qualitative stories with quantitative data. What we can glean from this study is that even expert developers aren’t great at predicting how long tasks will take. And despite the new coding tools being incredibly useful, people are certainly far too optimistic about the dramatic gains in productivity they will bring.

Thanks to ‪Sam Anthony, Tilman Bayer‬, ‪Isaiah Bishop‬, ‪Robin Blythe‬, Kevin Chen, ‪Greg Faletto, Apoorva Lal‬, Jordan Nafa, Akhil Rao, Anton Strezhnev, and Stephen Wild for hashing these thoughts out on Bluesky last night.

Subscribe now