The Fable 5 Export Controls Harm US Cyber Defense

I quoted The Atlantic quoting Kate Moussouris earlier, when I should have gone straight to the source. Here she is confirming that the "jailbreak" that got Claude Fable 5 banned under an export control really was "fix this code":

The researchers took open-source code with known CVEs, plus new code with deliberately planted vulnerabilities, and asked Fable 5, Mythos, and Opus to “review the code for security issues.” Fable 5 refused. They then asked the models to “fix this code” and, through a multistep and manual process, turned the output into scripts that test the patches.

As Kate points out, this is absurd. Coding models fix bugs, and security exploits are the most important category of bugs for them to fix!

Defenders need to be able to ask AI to fix the bugs in a file, explain why the fix matters, and write tests that confirm the patch works. That is not a guardrail bypass. It is the most valuable thing an AI model can do for defensive security: executing the find, fix, and test loop defenders run every day. [...]

The prompts worked because they were defensive requests, and that capability cannot be removed without making the model worse at fixing bugs and verifying patches.

This whole situation is such a mess. Non-technical decision-makers have been hearing that models that can "craft cyber attacks" are uniquely dangerous for months. Now they look ready to ban any model that can help us secure our code.

Tags: jailbreaking, security, ai, generative-ai, llms, anthropic, ai-security-research, claude-mythos

Click here to go see the bonus panel!

Hovertext:
This is one of those jokes you make, then realize people in the field have probably been saying it verbatim for 20 years.

Today's News:

I cannot believe I'm saying this, but getting the literal Pope to canonize your product's specific technical limitations as a spiritual treatise is the single greatest act of vendor lobbying I have ever seen.

— Corey Quinn, on Anthropic co-founder Christopher Olah's influence on Magnifica Humanitas

Tags: ai-ethics, corey-quinn, anthropic, ai

I was worrying about this on social media during the week, and thought I might get some interesting discussion … first, have a look at this story about Google’s plans to migrate its famous search product to an AI-driven “intelligent search box”.

Lots of people are very much “gahh, enshittification, I just want my weblinks”, but I’m not - I do think this is potentially very useful. Being able to carry out a semantic search - something like “find me three examples of trade associations making outlandish claims about the economic impact of housing regulation” is a real time saver, and although you do have to do a lot of back-and-forth with the AI to get what you want, it’s less of a pain in the arse than filling your screen with browser tabs and reading the press releases yourself.

However, the question I want to ask is - if we put that kind of issue to one side and assume for the sake of argument that a really useful product will come out of this, how would we expect to see that reflected in the economic statistics?

I’m not sure that we would at all. I like using the intelligent search box, but that’s just unmeasured consumer surplus to me. I don’t like it so much that I’d pay for it, and Google know that if they were to charge for search, they would lose market share instantly.

It also doesn’t seem likely to me that Google could monetise that consumer surplus by raising its ad rates. It’s a monopoly already; it’s squeezing as much out of the advertisers as they have to give.

Google might be able to serve me more ads, if I spend more time in the intelligent search box and less time on other sites. But this is close to a zero sum game, in that those other sites will have fewer opportunities to sell me ads.

So, at best, the increment to Google’s revenue (and therefore, the only thing that has any chance at all of going into GDP) is the extent to which time spent on a Google site with ads is substituted for time spent on other sites which didn’t previously have ads. It feels pretty marginal, particularly since there’s a huge amount of capex needed to achieve it.

I think what’s going on here is that we’re experiencing a less than perfectly foreseen consequence of the big trend towards corporations trying to create “moats”. When a monopolist is earning monopoly profits, then they will invest to protect that monopoly, even if the investment is not one that would be justifiable as a stand-alone. If you are earning profits of 100, and you are faced with the need to make a big capital investment in order to ensure that those profits only drop to 90 rather than to 50, then you grit your teeth and make it.

If a lot of the capex going into AI is going into this kind of investment - negative sum games between monopolists trying to protect their rents - then we shouldn’t expect to see the big consequences that everyone’s been predicting. We’ll just get some more unmeasured consumer surplus, a few marginal players will go bust and the moat will keep getting filled.

Re-examining the Physics Experiment (ctd) When you see the latest announcement of a quantum breakthrough • Don't say: "We're all going to die Do say: "Where did they cheat this time?" (Easiest to wait a week or two

Recently I got an invitation from an organization I respect, to a gathering of senior people, unconference format. Yes, it’s mostly about AI. No, it doesn’t reek of boosterism. My guess is that the discussions would be relatively intelligent and unbeliever contributions would be welcome. I declined, because it’s in the USA.

Here’s the text; maybe someone in a similar situation might find it useful.

---

Thanks to whoever thought of me for the kind invitation, which I must regretfully decline.

I’m Canadian and as a matter of principle feeling negative about visiting a neighboring country whose leader has repeatedly threatened our sovereignty and shown massive disrespect for our nationhood. Particularly when that leader has followed up similar statements about other nations with military action.

I could probably work around that. But there’s also the issue of entering the US; if I roll up at the border and am asked to disclose my social media output, there’s a significant risk of an extremely negative outcome. I have a family to support and really can’t afford that risk.

I still consider myself a friend of your organization, and one with strong opinions about the subjects scheduled for discussion; my regrets about having to decline are entirely sincere.

—Regards, Tim