AI Did the Work. What Did You Learn?

AI can write code, explain a paper, and quickly put together a working tool. As results get easier to obtain, it’s natural to wonder whether we still need to spend so much time learning. I’m interested in what happens afterward: when a similar problem comes up again, will we understand it any better?

In a 2023 interview, CNBC’s David Faber asked Elon Musk what career advice he could offer children entering the workforce as AI improved. Musk paused, acknowledged the difficulty, and suggested pursuing interesting, fulfilling work that was useful to society. The uncertainty extends to learning: will the time we invest today still be worth something tomorrow?

I recently read a short post by Simon Willison in SuperMuse about three blog posts that had influenced him. One was Joel Spolsky’s 2002 essay, The Law of Leaky Abstractions. I’ve long enjoyed Joel’s writing and had read this one years ago. Simon’s post sent me back to it.

Reading Joel Again

Simon said Joel’s blog post encouraged him to keep learning about the layers beneath his work, in case an abstraction leaked.

An abstraction gives us a simpler way to work with something complicated.

Objects bundle state and behavior behind methods, hiding their internal implementation. Relational databases let us work with tables and queries without managing every detail of file storage and disk access. Object-relational mappers let us work with those databases through objects in our code, handling the mapping to tables and generating common SQL queries. But the hidden details can still affect the result. When they do, the abstraction “leaks.”

When the Details Come Back

Why isn’t one emoji always one character long?

A username field seems easy to limit: check the string’s length. Yet in JavaScript, "😄".length is 2. The property counts UTF-16 code units, not the characters a person sees. Compound emoji can involve even more code units. MDN explains the distinction.

If a counter is wrong or truncation breaks a character, we need to understand encoding and character boundaries. A property named “length” hasn’t settled what a user means by “one character.”

Why can a few lines of database code make a page slow?

An object-relational mapper lets us access records much like ordinary objects. Fetch ten books, then read each book’s author: straightforward enough. Without eager loading, however, that may mean one query for the books and ten more for their authors. This is the N+1 problem, illustrated in the Rails guide.

The framework saves us from writing SQL by hand. It doesn’t remove the cost of database round trips. To investigate the slowdown, we need to see what queries actually run. The few lines on the screen may hide most of the work.

Why isn’t retrying always the answer to a timeout?

Suppose an application sends a request to create an order but never receives a response. The server might not have received the request. Or it might have created the order and lost the response on the way back. Without protection against duplicates, trying again could create a second order.

We need to understand what each side knows and how to retry one operation without repeating its effect. Stripe’s documentation on idempotent requests describes one approach. A convenient function call still depends on uncertain communication.

These situations share a pattern: to understand what’s happening, we often have to look one layer deeper.

Joel puts it this way:

“So the abstractions save us time working, but they don’t save us time learning.”

The tool removes repetitive work. Knowing how to call its interface doesn’t mean we understand what happens underneath.

A Higher Abstraction Still Leaks

From the user’s perspective, AI-assisted programming adds another layer of abstraction. We describe what we want in natural language; AI translates that into a programming language, and frameworks and systems execute it. A request as simple as “build an order page” can leave AI making decisions about data structures, queries, and request handling.

A higher abstraction can still leak. A page that loads successfully may slow down as its data grows or create duplicate orders when requests arrive together. Once those details affect the result, we have to look at the code, database, and communication between systems. Without some understanding of the underlying mechanisms and experience in software engineering, it can be hard to locate the problem or judge whether a fix is sound.

In this sense, Joel’s law of leaky abstractions still applies to AI-assisted programming. Natural language makes it easier to begin; the rules governing the system remain. Understanding the underlying system helps us know what to question—and how to check it.

This new layer can make us more productive without giving us a corresponding understanding of the work. Getting things done faster doesn’t mean learning happens automatically, or becomes easy. AI can explain mechanisms and help devise experiments, but learning them thoroughly still takes reasoning, practice, and revising our judgments in response to feedback.

A Correct Answer Can Leave Us No Wiser

There’s a subtler possibility: the AI is correct, the task succeeds, and we learn very little.

In an interview this August, Andrew Ng discussed cognitive offloading: delegating some thinking can also remove opportunities to practise. He described asking AI how frontend or backend components worked, finishing the project, then facing a similar task six months later without remembering the answer. He had to ask again.

That isn’t evidence that AI damaged his memory. The more immediate concern is that the knowledge may never have been thoroughly understood or retained.

A study in PNAS tested AI-assisted maths learning with nearly a thousand students at a Turkish high school. Students using a standard GPT-4 chat interface did better during assisted practice, but scored about 17% below the no-AI control group on subsequent independent exams. A tutor designed to offer teacher-informed hints and withhold direct answers largely avoided that harm, though it did not significantly outperform the control group on the independent exams.

One study doesn’t cover every learning situation. But it makes an important distinction concrete: performance on a task and learning from it are different outcomes.

A clear explanation and a successful attempt can feel like understanding. A changed condition—or the need to explain the idea from scratch—can reveal what we’re still missing.

Our End of the Lever

These days, AI is often described as a lever. It lets one person do work that once required more time and more people. At our end of that lever is what we bring to it.

That weight includes knowledge, experience, and familiarity with the problem. We might question whether an article’s evidence supports its conclusion, or distinguish a user’s actual difficulty from a need we’ve imagined. Those insights help us ask AI more useful questions and turn a first answer into something worth using.

Experience doesn’t guarantee good judgment, and beginners can use AI to accomplish things they couldn’t before. We can add to the weight at our end as we go. Each thing we understand a little better gives us more to try next time.

Two Extremes to Avoid

Taking learning seriously can turn into an insistence on doing everything the hard way: every line of code must be handwritten for the work to count. But understanding a network request doesn’t require implementing the entire protocol stack from scratch. Tools can spare us repetitive work and leave more time for the questions worth studying.

The other extreme is outsourcing everything, including judgment and verification. If the result runs, we stop asking why; if it fails, we keep asking AI to try again. We may finish plenty of tasks without becoming better prepared to investigate when an abstraction leaks.

I’d rather use the tools and learn as I go. Let AI help with the work while keeping opportunities to question, verify, and try things independently. The problems in front of us can guide what we need to understand next.

See the Whole, Then Explore the Details

Leaky abstractions give us a reason to understand what lies underneath. But we can also lose the forest for the trees. Start with the problem we’re trying to solve and how its concepts fit together; that helps us decide which details deserve attention. Once we understand a detail, step back and ask where it belongs in the larger picture.

AI can help us build that picture: outlining a subject’s central ideas, comparing similar problems across fields, and connecting new knowledge to what we already know. Treat this as an initial map, then check and revise its connections through original sources and experiments. Our notes can record how our understanding changes along the way. Learning should leave us something to think with next time.

Reading with Muse

In SuperMuse, AI Muse can accompany that process. Discuss a passage together, then let it test your reasoning with a question, a counterexample, or a scenario with a tempting wrong answer. Having to explain a choice can reveal what we haven’t quite worked out.

After reading, bring new highlights and old notes together with Muse, and decide which connections hold up. There’s a Chinese phrase I like: 温故而知新—finding new understanding by revisiting what we’ve learned. A new question can give an old note fresh meaning. Keeping reading, notes, and discussion together gives us a chance to make that knowledge our own.

AI can help us get the work done. I’d like us to come away understanding a little more, too.