"5+ years experience with GenAI" says the company that's totally not a distant third place in hyperscale cloud AI providers.

"But doctor," sobbed the patient, "I *AM* Andrey Markov!"

openelm/README-pretraining.md

Apple released something big three hours ago, and I'm still trying to get my head around exactly what it is.

The parent project is called CoreNet, described as "A library for training deep neural networks". Part of the release is a new LLM called OpenELM, which includes completely open source training code and a large number of published training checkpoint.

I'm linking here to the best documentation I've found of that training data: it looks like the bulk of it comes from RefinedWeb, RedPajama, The Pile and Dolma.

A bad survey won’t tell you it’s bad. It’s actually really hard to find out that a bad survey is bad — or to tell whether you have written a good or bad set of questions. Bad code will have bugs. A bad interface design will fail a usability test. It’s possible to tell whether you are having a bad user interview right away. Feedback from a bad survey can only come in the form of a second source of information contradicting your analysis of the survey results.

Most seductively, surveys yield responses that are easy to count and counting things feels so certain and objective and truthful.

Even if you are counting lies.

— Erika Hall

I have aphantasia—no ability to create images in my mind, or to remember things in a visual way.

For well over half of my life I lived under the assumption that suggestions to visualize things were entirely metaphorical, and also that I was incredibly stupid in certain ways, finding things that everyone else seemed to find easy a challenge. I struggle to learn graphical interfaces, I’ve mixed up speakers and their talks at conferences, I spend a lot of time staring at icons on my phone—as far as I’m concerned every blue icon leads to the same app.

I store information as lists. For people I see a lot, I could probably draw a reasonably accurate picture of them, based on the list of data I’ve collected. However that information can never be as detailed as being able to recall an image could be. For example, perhaps you are someone I work with. I see you several times a week on video calls. I know lots of things about how you look, including that you wear glasses. However, your glasses were unremarkable to me and so I’ve not stored any particular information about them. You could dramatically change your style of glasses, I’d not notice. I’m not comparing a visual memory of you from a few days ago, I just know you wear glasses.

If I’m visiting a new place and I go for a run, I collect data points along the way to navigate back. I will not remember how the route looked. This means that I can be incredibly good at taking people somewhere that I’ve only been once, I’ve got turn by turn directions. However, if I am walking with someone and talking, or for some reason don’t actively collect information, I’m doomed. The other week I lost an entire car park, never mind my car, due to having been distracted by a message after leaving the car park and failed to collect any information about what it looked like.

I dream, but not in pictures. For example, I have a recurring dream where I’m back in a theatre, putting on pointe shoes. I know it’s a theatre because of the smell, I feel the roughness of the shoes as I put my feet into them, the creak of the leather sole as I roll through my foot. I can feel that dream as I write about it, but there’s no image involved.

When I learned about aphantasia about ten years ago, suddenly so many things made sense. It’s harder to remember certain types of things if you have no visual memory of them. I’m not an awful person because I didn’t remember that I’d met a person before, there was no way for them to look familiar to me. However, I also think it’s at the root of some of the things I’m really good at.

My lists of information, are closer to a relational database than just a set of lists. It’s no surprise to me now that I always enjoyed working with databases and could design a complex schema without needing to sketch out a diagram. I’m constantly making connections between these bits of information. Many of these connections are just amusing to me, but other times they bring up interesting paths to investigate.

I context switch very easily, I can jump between these information sets without losing my train of thought.

I can write entire articles, documents, or conference talks in my head while out for a run. It’s usually quicker for me to create content in this way than sit at a computer and think about it. I can come in from a run and type out 2,000 words, transferring what I’ve written in my head to the document.

Having discovered this about myself, I’ve found plenty of other people who experience the world in the same way, probably unsurprisingly as people with aphantasia tend to be drawn to computing and science. I find it fascinating that we are all experiencing the world so differently, and how that can so fundamentally impact the things we find easy, or difficult.

If you have been reading my Substack (or my new book, which made the New York Times bestseller list!), you know that one of the things I find most exciting about AI is its potential for expanding the number of people able to produce and share useful innovations. I am particularly interested in how prompts can work as a sort of “program in prose” that can be tailored to many different industries and fields, and which can be created by non-technical experts.

This works because LLMs are extremely powerful tools that are remarkably easy to use, and whose capabilities can be wielded by anyone who can write or speak. And, while there are reasons to build more complex technological solutions, simple prompts given to a GPT-4 class chatbot can allow non-programmers to accomplish impressive things. Plus, while barriers to access remain, it helps that you can use LLMs on a mobile phone, and that powerful models are rapidly getting cheaper to use (the big news last week was the release of Llama 3, the open source AI from Meta that is already better than ChatGPT-3.5, and which will dramatically lower the prices for advanced AI access). This makes LLMs a good base on which to innovate.

But what does democratizing innovation with AI prompts actually look like? We have a new paper out today that tries to show one potential path. It specifically focuses on education, but I think the lessons are broadly applicable.

We took a context we know well, interactive teaching tools and pedagogy, and created research-based exercises that would have been difficult or impossible to do before the advent of generative AI. For example, the ability to generate a compelling educational simulation, with appropriate teaching support, using just a few paragraphs of text is an entirely new capability that AI has enabled. We have piloted a range of similar AI experiences in recent months, and they show considerable promise. While more experiments are needed on their effectiveness, these new tools expand the capabilities available to educators.

While I won’t discuss it here, in the paper we cover how to assign AI projects to students, as well as the ethical concerns, risks, and AI literacy issues that instructors need to consider. But, for this post, I want to concentrate on the exercises themselves, and specifically how they can help us understand the steps we can take in making AI prompts broadly useful. There is not just one way to use prompts in innovative ways, instead there multiple approaches that can work.

Share

Level 1: Use a pre-made prompt

Just like instructors often share lesson plans, or members of organizations might share tips for success, people can also share prompts. This is a useful starting point for would-be-innovators; someone else has done the work of creating and testing a prompt for you. By using other people’s prompts, you can see what an AI system can (or can’t) do.

For example, here is a goal-play prompt you can try (GPT, Gemini Advanced link, and the paper has the full text of all the prompts). In a goal play, the AI sets up a scenario in which the student knows something the character in the simulation does not and guides the character using what they know, generally a framework they have to apply. In this case, students must apply a goal-setting framework to help a fictional character set goals. For fun, I decided to help out poor Hamlet, Prince of Denmark.

This alternate version of Hamlet might be a good scenario for students to practice goal-setting, but literature instructors are probably rolling their eyes. The scenario simplifies a complex Shakespeare play into a mere transactional situation (and doesn’t even use iambic pentameter). An exercise that is great in one setting can be terrible in another. That is why using someone else’s prompts is often a starting place for your own innovation, rather than an end point. Your environment and needs will often push you to create something to fill the voids that pre-made prompts cannot. And that’s a good thing! Prompting is not as hard as people think, and doing it well starts with experimentation.

Level 2: Customize or build a prompt

One way to experiment with advanced uses is to start with a pre-made prompt. In the paper, we provide annotated versions of all of the prompts we developed, so that you can see how you might need to customize one for yourself.

Ultimately, you may need to create a new prompt from scratch to accomplish your goals. As I have discussed, prompting is not an exact science, but you may be surprised about how effective prose instructions are. Still, any prompt requires testing and revision, so we developed a set of tests and remediation steps that might be useful to anyone trying to build their own prompts from scratch. Trial and error is going to be necessary to make something that works consistently.

Once you have created a new prompt, you should consider how to share it (our prompt library is here). In fact, if you want to encourage innovation, organizations need to consider, from the start, how they will create communities of practice around AI tools that also enable prompt sharing.

Level 3: Create tools that make tools

The next level of innovating through prompting is to help other people create their own innovations based on your knowledge. To do this, you need to develop tooling - prompts that can make prompts. This allows people to share not just a specific prompt, but their skill in prompt crafting. We call these kinds of prompts blueprints, and here is one that creates tutors (GPT, Gemini Advanced). You can take the text it creates and use it as a prompt to create a solid AI tutor tailored to a specific topic. This works much better than many generic tutor prompts because it is so focused and can be more easily tested by an expert instructor.

OpenAI’s GPT maker tool is an example of a blueprint, but we think much more tailored blueprints will be useful for many educators and organizations. They can help people build the prompts they need for specific situations.

Level 4: Just tell the AI what you want.

I suspect that many of the previous levels of prompt development will eventually become obsolete for most people. While there will still be value in creating prompts, increasingly AIs will just prompt themselves to solve your problem based on your goals.

Here is an example of me using Devin, an early AI agent powered by GPT-4. Rather than creating the perfect prompt, I can simply tell Devin “create and deploy a website that teaches 11th grade American history students about the 1st red scare, make it interactive. look up appropriate AP standards for what should be taught. make sure this is really good and students can use it easily”

It took it from there, doing research on AP standards, gathering information, and launching a website, as you can see. While I certainly wouldn’t trust Devin in its current state to produce accurate and hallucination-free results, you can see how the future might just be letting the AI do things, and sharing the results that work well.

Sharing what we learn

The lesson of innovation is that technology is only really useful when it is used. This might seem like a tautology, but it isn’t. When a new technology is introduced, people adapt it to solve the needs they have at hand, rather than simply following what the manufacturer of the technology intended. This leads to a lot of unexpected uses as the new technology is pushed to solve all sorts of novel problems. Right now, AI is in the hands of millions of educators, and they can use it to solve their own problems, in their own classrooms.

Once an exclusive privilege of million-dollar budgets and expert teams, education technology now rests in the hands of educators. While it's important to remain vigilant for potential hallucinations, errors, and biases, AI enables teachers to craft personalized prompts tailored to their local contexts, significantly bolstering their resources in the pursuit of quality education. But the only way to make sure that this revolution is positive and inclusive is to share what works.

Innovation requires cooperation. Nobody knows exactly where AI is useful, and where it is not, in advance. I have yet to see robust communities of educators (or many other professions) coming together to share what they create. Policymakers and institutional leaders can help make that happen. In the meantime, we hope that our prompts and blueprints can serve as the basis for ethical experimentation. Just make sure to share what you learn!

Buy my book

Subscribe now

Llama 3 was released on Thursday. Early indications are that it's now the best available openly licensed model - Llama 3 70b Instruct has taken joint 5th place on the LMSYS arena leaderboard, behind only Claude 3 Opus and some GPT-4s and sharing 5th place with Gemini Pro and Claude 3 Sonnet. But unlike those other models Llama 3 70b is weights available and can even be run on a (high end) laptop!

My LLM command-line tool and Python library provides access to dozens of models via plugins. Here are several ways you can use it to access Llama 3, both hosted versions and running locally on your own hardware.

• Llama-3-8B-Instruct locally with llm-gpt4all
• Fast API access via Groq
• Local Llama 3 70b Instruct with llamafile
• Paid access via other API providers

Llama-3-8B-Instruct locally with llm-gpt4all

If you want to run Llama 3 locally, the easiest way to do that with LLM is using the llm-gpt4all plugin. This plugin builds on the excellent gpt4all project by Nomic AI, providing a quantized (q4) Llama 3 8B instruct model that needs 8GB of RAM and requires a 4.34GB download.

If you don't have LLM installed yet, install it like this:

brew install llm

Or like this:

pipx install llm

Install the plugin:

llm install llm-gpt4all

The first time you run a prompt it will download the model:

llm -m Meta-Llama-3-8B-Instruct "Three great names for a pet emu"

What an interesting question! Here are three name suggestions for your pet emu:

1. Emmet: A playful take on the word "emu" that suits a friendly and curious bird.
2. Gizmo: Emus are known for their quirky behavior, so Gizmo is a fitting name to capture their unique personality.
3. Aussie: As an Australian native species, naming your emu Aussie would be a nod to its heritage.

Remember to choose a name that suits your pet's temperament and characteristics!

Each time you run a command like this the model will be loaded into memory from scratch. To keep it resident in memory, use the llm chat command:

llm chat -m Meta-Llama-3-8B-Instruct

Here's an example session:

Chatting with Meta-Llama-3-8B-Instruct
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
> javascript function to reverse a string
%
Here is the JavaScript code that reverses a given string:
```
function reverseString(str) {
  return str.split("").reverse().join("");
}
```

You can use this function by passing your desired input string as an argument, like so:

```javascript
let reversedStr = reverseString("Hello World");
console.log(reversedStr); // Output: "dlroW olleH"
```
In the code above:
- `str.split("")` splits the original string into individual characters.
- `.reverse()` reverses the order of those characters.
- `.join("")` combines them back together as a single string.

This function works for any input string, not just "Hello World". Try it out with different strings to see how it handles punctuation and spaces!
> capitalize it too
%
Here's an updated JavaScript function that not only reverses the input string but also capitalizes each character:
```javascript
function reverseAndCapitalize(str) {
  return str.split("").reverse().map(function(char) { 
    if (char.match(/[a-z]/i)) {
      return char.toUpperCase();
    } else {
      return char;
    }
  }).join("");
}
```

(I'm not sure I'd write reverseAndCapitalize() quite like that, but still a fun example!)

Consult the LLM documentation for more details on how to use the command-line tool.

Fast API access via Groq

Groq serve openly licensed LLMs at ludicrous speeds using their own custom LPU (Language Processing Unit) Inference Engine. They currently offer a free preview of their API: you can sign up and obtain an API key to start using it.

You can run prompts against Groq using their OpenAI compatible API endpoint.

Edit the file ~/Library/Application Support/io.datasette.llm/extra-openai-models.yaml - creating it if it doesn't exist - and add the following lines to it:

- model_id: groq-openai-llama3
  model_name: llama3-70b-8192
  api_base: https://api.groq.com/openai/v1
  api_key_name: groq
- model_id: groq-openai-llama3-8b
  model_name: llama3-8b-8192
  api_base: https://api.groq.com/openai/v1
  api_key_name: groq

This tells LLM about those models, and makes them accessible via those configured model_id values.

Run this command to confirm that the models were registered correctly:

llm models | grep groq

You should see this:

OpenAI Chat: groq-openai-llama3
OpenAI Chat: groq-openai-llama3-8b

Set your Groq API key like this:

llm keys set groq
# <Paste your API key here>

Now you should be able to run prompts through the models like this:

llm -m groq-openai-llama3 "A righteous sonnet about a brave owl"

Groq is fast.

There's also a llm-groq plugin but it hasn't shipped support for the new models just yet - though there's a PR for that by Lex Herbert here and you can install the plugin directly from that PR like this:

llm install https://github.com/lexh/llm-groq/archive/ba9d7de74b3057b074a85fe99fe873b75519bd78.zip
llm keys set groq
# paste API key here
llm -m groq-llama3-70b 'say hi in spanish five ways'

Local Llama 3 70b Instruct with llamafile

The Llama 3 8b model is easy to run on a laptop, but it's pretty limited in capability. The 70b model is the one that's starting to get competitive with GPT-4. Can we run that on a laptop?

I managed to run the 70b model on my 64GB MacBook Pro M2 using llamafile (previously on this blog) - after quitting most other applications to make sure the 37GB of RAM it needed was available.

I used the Meta-Llama-3-70B-Instruct.Q4_0.llamafile Q4 version from jartine/Meta-Llama-3-70B-Instruct-llamafile - a 37GB download. I have a dedicated external hard disk (a Samsung T7 Shield) for this kind of thing.

Here's how I got it working:

curl -L -o Meta-Llama-3-70B-Instruct.Q4_0.llamafile 'https://huggingface.co/jartine/Meta-Llama-3-70B-Instruct-llamafile/resolve/main/Meta-Llama-3-70B-Instruct.Q4_0.llamafile?download=true'
# That downloads 37GB - now make it executable
chmod 755 Meta-Llama-3-70B-Instruct.Q4_0.llamafile
# And start it running:
./Meta-Llama-3-70B-Instruct.Q4_0.llamafile

A llamafile is an executable that runs on virtually any platform - see my previous notes on Cosmopolitan and Actually Portable Executable for more on how that works.

This will take quite a while to start, because it needs to load that full 37GB of binary content into memory. Once it's finished loading a local web server becomes available at http://127.0.0.1:8080/ - this serves a web UI you can use to interact with the model, and also serves another OpenAI-compatible API endpoint.

The easiest way to access this from LLM is to install the llm-llamafile plugin:

llm install llm-llamafile

All this plugin does is configure a model called llamafile that attempts to access the model hosted on port 8080. You can run prompts like this:

llm -m llamafile "3 neat characteristics of a pelican"

Here are three neat characteristics of a pelican:

1. Unique Beak: Pelicans have a distinctive beak that is shaped like a scoop or a basket. This beak is specially designed to catch fish, and it can hold up to 3 gallons of water! The beak is also very sensitive, which helps pelicans detect the presence of fish in the water.

2. Waterproof Feathers: Pelicans have a special coating on their feathers that makes them waterproof. This is essential for their fishing lifestyle, as they need to be able to dive into the water without getting weighed down by wet feathers. The coating is made up of a waxy substance that helps to repel water.

3. Pouch-Like Throat: Pelicans have a unique throat pouch that allows them to catch and store fish. When they dive into the water, they use their beak to scoop up fish, and then they store them in their throat pouch. The pouch can expand to hold multiple fish, and the pelican can then swallow the fish whole or regurgitate them to feed their young. This pouch is a key adaptation that helps pelicans thrive in their aquatic environment.

If you don't want to install another plugin, you can instead configure the model by adding this to your openai-extra-models.yaml file:

- model_id: llamafile
  model_name: llamafile
  api_base: http://localhost:8080/v1
  api_key: x

One warning about this approach: if you use LLM like this then every prompt you run through llamafile will be stored under the same model name in your SQLite logs, even if you try out different llamafile models at different times. You could work around this by registering them with different model_id values in the YAML file.

Paid access via other API providers

A neat thing about open weight models is that multiple API providers can offer them, encouraging them to aggressively compete on price.

Groq is currently free, but that's with a limited number of free requests.

A number of other providers are now hosting Llama 3, and many of them have plugins available for LLM. Here are a few examples:

• Perplexity Labs are offering llama-3-8b-instruct and llama-3-70b-instruct. The llm-perplexity plugin provides access - llm install llm-perplexity to install, llm keys set perplexity to set an API key and then run prompts against those two model IDs. Current price for 8b is $0.20 per million tokens, for 80b is $1.00.
• Anyscale Endpoints have meta-llama/Llama-3-8b-chat-hf ($0.15/million tokens) and meta-llama/Llama-3-70b-chat-hf ($1.0/million tokens) (pricing). llm install anyscale-endpoints, then llm keys set anyscale-endpoints to set the API key.
• Fireworks AI have fireworks/models/llama-v3-8b-instruct for $0.20/million and fireworks/models/llama-v3-70b-instruct for $0.90/million (pricing). llm install fireworks, then llm keys set fireworks to set the API key.
• OpenRouter provide proxied accessed to Llama 3 from a number of different providers at different prices, documented on their meta-llama/llama-3-70b-instruct and meta-llama/llama-3-8b-instruct pages (and more). Use the llm-openrouter plugin for those.
• Together AI has both models as well. The llm-together plugin provides access to meta-llama/Llama-3-8b-chat-hf and meta-llama/Llama-3-70b-chat-hf.

I'm sure there are more - these are just the ones I've tried out myself. Check the LLM plugin directory for other providers, or if a provider emulates the OpenAI API you can configure with the YAML file as shown above or described in the LLM documentation.

That's a lot of options

One key idea behind LLM is to use plugins to provide access to as many different models as possible. Above I've listed two ways to run Llama 3 locally and six different API vendors that LLM can access as well.

If you're inspired to write your own plugin it's pretty simple: each of the above plugins is open source, and there's a detailed tutorial on Writing a plugin to support a new model on the LLM website.