Ethical Applications of AI to Public Sector Problems

Jacob Kaplan-Moss developed this model a few years ago (before the generative AI rush) while working with public-sector startups and is publishing it now. He starts by outright dismissing the snake-oil infested field of “predictive” models:

It’s not ethical to predict social outcomes — and it’s probably not possible. Nearly everyone claiming to be able to do this is lying: their algorithms do not, in fact, make predictions that are any better than guesswork. […] Organizations acting in the public good should avoid this area like the plague, and call bullshit on anyone making claims of an ability to predict social behavior.

Jacob then differentiates assistive AI and automated AI. Assistive AI helps human operators process and consume information, while leaving the human to take action on it. Automated AI acts upon that information without human oversight.

His conclusion: yes to assistive AI, and no to automated AI:

All too often, AI algorithms encode human bias. And in the public sector, failure carries real life or death consequences. In the private sector, companies can decide that a certain failure rate is OK and let the algorithm do its thing. But when citizens interact with their governments, they have an expectation of fairness, which, because AI judgement will always be available, it cannot offer.

On Mastodon I said to Jacob:

I’m heavily opposed to anything where decisions with consequences are outsourced to AI, which I think fits your model very well

(somewhat ironic that I wrote this message from the passenger seat of my first ever Waymo trip, and this weird car is making extremely consequential decisions dozens of times a second!)

Which sparked an interesting conversation about why life-or-death decisions made by self-driving cars feel different from decisions about social services. My take on that:

I think it’s about judgement: the decisions I care about are far more deep and non-deterministic than “should I drive forward or stop”.

Jacob:

Where there’s moral ambiguity, I want a human to own the decision both so there’s a chance for empathy, and also for someone to own the accountability for the choice.

That idea of ownership and accountability for decision making feels critical to me. A giant black box of matrix multiplication cannot take accountability for “decisions” that it makes.

Tags: jacob-kaplan-moss, ai, ethics

I was doing this talk, which I’d done dozens of times before. Good, well-practiced deck. I was speaking to CTO-types (current and aspirational) as a favor to a friend. This was a monthly morning coffee chat for this crew, and they invited folks like me to speak. It was on a weekday morning near Slack in downtown San Francisco.

No problem.

Practiced talk, small group, low stakes. I was editing the title slide to update the location and name of the event. No practice necessary; again, I’d done it before.

The piece was based on this piece called Stables and Volatiles. The brief pitch. Stables are those who happily work with direction and appreciate that there appears to be a plan and the calm predictability of a well-defined schedule. Volatiles are the opposite. Read the piece; I like it.

Finished quickly. I was told this was more about a discussion than a presentation. Fine with me. Q&A tells me precisely how well the deck landed, and I’d done this talk enough to believe the Q&A would be rich. Healthy banter. It started that way. Questions about my first stint at Apple and whether well-known people were Stables or Volatiles, but then Leo in the Back Row lost his shit.

“This is bullshit. It’s a false dichotomy.” Leo, the CTO in the Back Row, was pissed about my presentation. For those without ChatGPT at the ready, a false dichotomy is “the fallacy of presenting only two choices, outcomes, or sides to an argument as the only possibilities, when more are available.”

After some back and forth, I told Leo, the CTO in the Back Row, that, like most of my writing, I liked to describe humans in stark, clever ways. This often took the form of a “THIS or THAT” black-and-white structure, but I was 100% clear that the answer to humans was a hard-to-define grey area. My job was to get you to think, not to define every possible configuration of human behavior.

I’d delivered that answer before, and it worked, but Leo, the CTO in the Back Row, was having none of it. He was still angry and — now, I am guessing — because I’d wasted his time. He was promised a structured model, and I delivered confusing poetry.

Leo, the CTO in the Back Row, was the Police. And the Police don’t like poetry.

Guess What, Leo

I have another false dichotomy for you: the Poets and Police.

Poets:

• Finish things. Usually.
• Use rich language to describe abstract situations.
• Believe well-formed, highly descriptive ideas make the world an understandable place.
• Are fine with ambiguity because they understand it’s all just shades of grey.
• Fall in love with ideas. They’ll fall in love with a single choice word.
• Like to use the word “feel” because feelings are distilled intuition expressing themselves as inspiration.
• Love thoughtful compliments.

Police:

• Finish things. Wow, they finish things.
• Crave well-defined structure and rules.
• Believe rules make the world an understandable and measurable place.
• Hate ambiguity because it provides no direction.
• Deeply enjoy both debating and enforcing those rules. They believe this is how you make future measurable progress.
• Will debate a single word that is out of place until clarity is achieved. (Poets do this, too.)
• Are excellent at measuring anything with metrics. Are unlikely to believe unmeasurable truth.
• Never use the word “feel” because feelings are irrelevant to getting the job done.
• Appreciated well-defined accomplishment.

Two things.

First, as a Poet, I know I am describing the Police from my perspective. Police will profoundly disagree with many of the attributes I describe. I am eagerly listening.

Second, yes, this article is similar to my much earlier piece on Organics and Mechanics, but I feel it’s stronger writing.

Third, I am making up a third thing. To annoy the Police who were keeping count because that is how we Poets roll.

Success is Both

As a self-declared Poet, I can confidently describe the Police because it is a job requirement that develops strong working relationships with these essential humans. I need them because the Police do the challenging work of keeping the trains on time. This isn’t simply holding conductors to a schedule but also maintaining the trains, taking care of the track, and ensuring we have a qualified staff of humans to do all this work. Oh, and how about a budget? How are we going to afford all of this? Someone needs to build a credible business plan for this train company so we can afford to keep the trains on time.

As a self-declared Poet, we also need to understand the aspirational goals of this train company. I also understand the importance of consistently sharing this vision with everyone. I know we need to listen because we need to understand how the company feels. I’m adept at organizing teams of humans with differing ideas and skills. It’s an endless puzzle that I enjoy attempting to solve. I love celebrating our victories. I feel our failures deeply, but I know that with the Police, we will learn from these failures.

Listen. Leo. The CTO. You there — in the back row. I get why you’re mad. See, while we differ in how we view the work, we are the same regarding what’s essential. We want the team to succeed, and we want them to advance. I’ve learned some of my favorite moves watching you work, Leo. I’ll work hard to try not to waste your time with too much poetry if you work hard to understand that poetry is part of how we describe and achieve the impossible.

Oh bother.

At first, I struggled to understand why anyone would want to write this way. My dialogue with ChatGPT was frustratingly meandering, as though I were excavating an essay instead of crafting one. But, when I thought about the psychological experience of writing, I began to see the value of the tool. ChatGPT was not generating professional prose all at once, but it was providing starting points: interesting research ideas to explore; mediocre paragraphs that might, with sufficient editing, become usable. For all its inefficiencies, this indirect approach did feel easier than staring at a blank page; “talking” to the chatbot about the article was more fun than toiling in quiet isolation. In the long run, I wasn’t saving time: I still needed to look up facts and write sentences in my own voice. But my exchanges seemed to reduce the maximum mental effort demanded of me.

— Cal Newport

Tags: writing, generative-ai, chatgpt, ai, llms

Y'know, the topic of fraudulent research has been coming up a lot here recently, so I wanted to bring up one more aspect of the problem while it's on my mind. So many of these cases have turned on faked graphical representations of data - Western blots that have been cut-'n-pasted to say just what you want them to say without any of that inconvenient Other Stuff, photomicrographs of cells and tissue slices that illustrate Great Things Happening where they actually weren't. That sort of thing.

And these have been relatively easy to spot, when you actually bear down and try to spot them. One of the big reasons that they haven't been is a sort of presumption of scientific innocence, of good faith. As has been said here and in many other places, when you review a manuscript that a journal has sent you for possible publication, you tend to ask questions like: Do these data fit the conclusions that the authors are drawing? Have the right experiments been run and are they presented in enough detail to be believable and reproducible? Are the conclusions statistically plausible? Are there other possible explanations that the authors are overlooking? If so, are there experiments that could be run to nail those loose ends down? Have they fairly represented the state of the art, and cited the relevant papers so that someone could use this paper to get a foothold on the literature in this area? 

What we don't tend to ask is: Are the authors flat-out lying? Did they make all this stuff up? Are these Western blots actually mosaics of six others, slapped together by software manipulation to illustrate a so-called definitive experiment that wasn't even run in the first place? Are there blobs of duplicated pasted stuff scattered through those pictures of the cells, to make it look as if something worked when nothing ever did? And those graphs - are they maybe the same damn graphs that the authors have published before, in other journals, to illustrate other supposed data, but just recycled with the labels changed? Well, are they?

Those are very different questions, and they come from a different place entirely. You can see why we don't (or haven't been) approaching manuscripts from that direction, because if we're going to have to do that, then you have two separate labor-intensive review tasks in front of your for each paper. I mean, if you assume good faith, the workload diminishes greatly. First you'd need a forensic criminal report on the presence of fakery - then, if the paper passes that, you do the traditional internal-scientific-consistency stuff. Problem is, if you skip the first step a well-faked paper is probably going to pass that second review and get published. After all, if the authors are making stuff up, they'll make it up in a way that gives them a solid-looking manuscript with well-supported conclusions, right? That's how so much of this gets through - see that terrible Masliah example for a painful illustration. 

Now, when I say "relatively easy to spot", that's once you take off your we're-all-scientists-here glasses and put on your keep-your-hands-where-I-can-see-them ones. When you start looking for deliberate duplications you have a decent chance of seeing them; they have to be really egregious to leap out at you if you're not thinking about fraud in the first place. You get better with practice, and there are some people (Elisabeth Bik!) who are very good at it indeed. There are software solutions these days that assist with finding duplications in figures, too. But is all that about to go away?

I say that because of the rise of AI-generated fakery. Bik herself told me when I spoke with her earlier this year that she's very worried about this, and you can see why. Think about it: fakery is what AI image-generating systems do; it's their entire raison d'être. You give them prompts so that they will whip up something plausible, something that looks like what you wanted. So whip me up a good looking Western blot, or maybe just a selection of good-looking bands that I can use in my Frankensteined Westerns where they won't appear duplicated! Here, imaging model, here's a big pile of stained tissue slice images - make me some more that look like that, only with the features in them that I want people to think that I actually had. You can do that for me, right?

Right. They can. And I'm sure they are, right now. And they are going to be a lot harder to pick up. This is part of the general problem of AI-generated slop contaminating things - a good example is the open-source effort to use text from the internet to determine the frequency of word usage in English. It's now been shut down, because there is too much AI-generated content out there skewing the results. There are now piles of AI-generated books being published, full of remasticated trash written in reasonably grammatical sentences, and if you use a mushroom identification guide churned out in this manner you are taking your life in your hands. And now we get to experience this lovely effect in the scientific literature: a tidal wave of superficially attractive gibberish, with the actually worthwhile information gradually dissolving in it. We're all going to regret this.