Compression made the information age possible by stripping things down to fit the pipes. Expansion made the AI age possible by blowing data back up again. Both operations leave marks; we’ve learned to spot compression artifacts, but we’ve only just begun to reckon with expansion artifacts. Until we do, there’s a lot of risk to manage.
People talk about the effectiveness (or lack thereof) of large language models as though all tasks are comparable. But it strikes me that there are three broad categories of work that large language models are applied to:
Compression is when you feed a large language model something big that you want to make small. Summarise this book. Give me the gist of this meeting. Large language models are generally pretty good at this, which makes sense given that they themselves are kind of like compressed artifacts.
Transformation is when large language models convert from one format into another. Turn this audio into text. Turn this jumble of data into structured JSON. A large language model can handle these tasks pretty well. There’ll probably be a few errors so make sure that’s not a deal-breaker.
Expansion is when you give a large language model a prompt to generate something from scratch. An image. A presentation. An email. A poem. This is where slop lives. The output inevitably betrays its origins, glistening with a sheen of mediocrity.
Laurie spotted this three-way split a while back:
Is what you’re doing taking a large amount of text and asking the LLM to convert it into a smaller amount of text? Then it’s probably going to be great at it. If you’re asking it to convert into a roughly equal amount of text it will be so-so. If you’re asking it to create more text than you gave it, forget about it.
I hope that when the bubble finally bursts, we’ll see the surviving large language models put to work on the first two categories. The boring stuff. The work that’s tedious for humans.
But tedious is as tedious does. Something I consider drudgery might be the very thing that gives you life. Like Giles says:
I have a feeling that everyone likes using AI tools to try doing someone else’s profession. They’re much less keen when someone else uses it for their profession.
The big exception seems to be programming. Apparently there are plenty of coders who never before expressed an interest in being managers who are now happily hanging up their coding spurs in favour being the overseer of non-human workers.
It’s a reasonable outlook. It could even be considered a user-centred approach. Users don’t care about the elegance of your code; they care about accomplishing their tasks.
Programming is something of an exception to the efficacy of large language models in general. Instead of relying on the subjectivity of painting, poetry, or prose, programming can be objectively tested. Throw enough money at the worst people in the world and they’ll give you tokens you can use to get the machines to test their own output. So you can get a large language model to create something reasonably good from scratch as long as that something is code.
If you had asked me about the threat model of large language models two years ago, I probably would’ve been worried for artists, writers, and musicians. I thought that software had enough inherent complexity to be relatively safe.
Now my opinion has completely reversed. Software is almost certainly the killer app for large language models.
I think the artists, writers, and musicians will be okay, or at least as okay as they ever were. It turns out that humans like things made by other humans.
And y’know what? If I had to choose which endeavour I’d rather see automated away—programming or art—it’s no competition.
Don’t get me wrong—it would be nice if everyone got paid for doing what they enjoy. It’s just that I’m okay with software engineers not being at the front of that line.
I remember when I first started getting paid money to make websites. “Really?” I thought, “Someone is willing to pay me to do something I’d do anyway?” I kept waiting for the jig to be up. Instead I saw my profession grow and expand.
Perhaps there’s a long-overdue compression happening.
Or maybe it’s more like a transformation.
A very astute framing by Ted Chiang—large language models as a form of lossy compression for text.
When we’re dealing with sequences of words, lossy compression looks smarter than lossless compression.
A lot of uses have been proposed for large language models. Thinking about them as blurry JPEGs offers a way to evaluate what they might or might not be well suited for.
There’s a new image format on the browser block and it’s very performant indeed. Jake has all the details you didn’t ask for.
This is good to know! Because of a bug in Google App Engine, Brid.gy won’t work for sites using Brotli compression on HTML.
A deep, deep, deep dive into the JPEG format. Best of all, it’s got interactive explanations you can tinker with, a la Nicky Case or Bret Victor.
I’ve been playing around with the newly-released Squoosh, the spiritual successor to Jake’s SVGOMG. You can drag images into the browser window, and eyeball the changes that any optimisations might make.
On a project that Cassie is working on, it worked really well for optimising some JPEGs. But there were a few images that would require a bit more fine-grained control of the optimisations. Specifically, pictures with human faces in them.
I’ve written about this before. If there’s a human face in image, I open that image in a graphics editing tool like Photoshop, select everything but the face, and add a bit of blur. Because humans are hard-wired to focus on faces, we’ll notice any jaggy artifacts on a face, but we’re far less likely to notice jagginess in background imagery: walls, materials, clothing, etc.
On the face of it (hah!), a browser-based tool like Squoosh wouldn’t be able to optimise for faces, but then Cassie pointed out something really interesting…
When we were both at FFConf on Friday, there was a great talk by Eleanor Haproff on machine learning with JavaScript. It turns out there are plenty of smart toolkits out there, and one of them is facial recognition. So I wonder if it’s possible to build an in-browser tool with this workflow:
Maybe the selecting/blurring part would need canvas? I don’t know.
Anyway, I thought this was a brilliant bit of synthesis from Cassie, and now I’ve got two questions:
A handy in-browser image compression tool. Drag, drop, tweak, and export.
Handy web-based tools—compress HTML, CSS, and JavaScript, and convert files from one format to another.
Following on from Amber’s introduction, here’s a really in-depth look at image formats, compression and optimisation techniques from Addy.
This is a really nicely put together little web book released under a Creative Commons licence.
Google have released this encoder for JPEGs which promises 20-30% smaller file sizes without any perceptible loss of quality.
The history of the GIF—a tale of licensing, compression, and standards.
Some good thinking from Jason here. In a roundabout way, he’s saying that when it comes to responsive images—as with just about every other aspect of web development—the answer is …it depends.
You’ve probably seen this already, but it’s really worth bearing in mind: when you’re scaling up JPGs for retina display you can safely reduce the image quality by quite a lot—to the point of getting the exact same file size as a higher quality image that’s half the size.
This looks like a really handy tool for reducing the file size of JPEGs without any perceptible loss of quality (in much the same way that ImageOptim works for PNGs)—available as a Mac app or an installable web service.
An in-depth analysis (graphs! data!) of how popular sites are using—or not using—compression.