Most of what I know about working with AI, I learned by watching.
The software foundations came first: school, self-teaching, curiosity, my first job. When AI started being a thing, I applied what I already knew to it, because there was nothing else to apply. No rules, no guidance. Just a new technology that I knew would change everything.
In the early open-source days, working with the first Llama 13B, you did not have a choice. The context window was two thousand tokens. Every token you sent mattered, and every token that came back told you something. If the model gave a strange answer, the cause was usually sitting right there in the prompt: an example pulled the output sideways, a loose word got interpreted literally, something important fell off the end when the window filled up. With this technology the smallest thing can change the result. So I built a habit early: look at the output, then trace back how we got there.
Being able to summarize large amounts of text, or hold a chatbot conversation that stayed coherent and useful for the task, took real skill back then, and most of that skill was observation. You watched what worked, you watched what broke, and you adjusted. Nobody could tell you the rules because nobody knew them yet.
The models got better and the windows got bigger, and I kept the habit. Of everything I picked up in those days, that habit is the one that has carried the furthest.
Watching got harder right when it mattered more
Today the tools do so many things at once that watching is genuinely difficult. One request can fan out into dozens of steps: file reads, searches, tool calls, agents spinning up other agents. The two thousand token days forced you to see everything because there was almost nothing to see. Now there is too much to see, and most of it happens while you are not looking.
You could conclude that watching stopped being practical. I think the opposite is true. The more a system does on your behalf, the more important it is to know what it actually did. We are not just dealing with model calls anymore; we are dealing with agents and harnesses that handle a lot of the plumbing for you.
So the habit had to become infrastructure. The way I think about it now has two stages. Observation comes first: you notice things by hand, the way I did when the windows were small. That manual noticing is how you learn what is worth paying attention to. Then you build observability: instruments that watch those exact things consistently, on a schedule, whether or not you remember to check.
I run a weekly telemetry report over my agent fleet that does exactly this. It did not start as a system. It started as noting behavior down when I saw it: an instruction here, an ad hoc script there, a reminder to myself to go look again. Only after I had built the pieces and understood how they worked together did it become a weekly report that runs on its own. And it has already taught me the follow-up lesson: the instrument needs watching too. One week the report showed my agents’ scored task outcomes noticeably worse than the month before. It looked like a real decline. Before reacting to it, we checked the measurement itself, and the change turned out to be in how outcomes were being labeled, not in how the agents were behaving. I wrote about that general rule in Tests Green Is Not the Truth: the measuring code is code, and code has bugs.
Checking the instrument you just built sounds like extra work. It is less work than trusting a broken chart that your next decisions ride on.
Asking why
Observation on its own is passive. What made it compound was curiosity.
When an agent makes a decision I did not expect, I want to know why. Sometimes I literally ask it. Sometimes I read the transcript and reconstruct the reasoning. Either way, the question is the same one I was asking in the small-window days: how did we get here from there? This is also why it matters to have a decent foundation in whatever you are working on. Without one, it is the blind leading the blind.
In asking, you learn. That learning gives you the knowledge you need to actually understand the system, and understanding is the thing you can turn into action. There is a version of working with these tools where you never ask, you just retry until the output looks right (cough cough, vibe coding). It works, in the short term. But you end the day with a result and no understanding, and tomorrow you start from zero again. I think of that as knowledge debt, the human version of technical debt: skip the understanding now, and you will pay for it later, with interest, when you have to come back and relearn work you already did.
Ask enough of those questions and the answers start to accumulate. You stop seeing individual outcomes and start seeing the modes: the ways this kind of task tends to fail, the conditions where it tends to go well. It is the same way you learn a coworker’s strengths and blind spots, not from a resume but from working beside them.
And once you know the modes for a class of problems, you can start predicting. You can look at a planned build and say where it will probably crack. You can read a task description and know which step the agent will fumble. You start exercising an operator’s judgment in a way that directs outcomes. Those predictions are worth more than the satisfaction of being right, because you can hand them to your agents as signals: watch for this, verify that, do not trust this kind of result without a second look. The agent arrives at a sounder answer in less time, not because it got smarter, but because it inherited my experience.
That is one of the core parts of my memory system. Every time something fails in a new way, the lesson gets written down where the agents can find it. A trap that has been documented costs you once. A trap that has not costs you every time you meet it. I wrote about that system in The Library That Pays You Back, and this is the reason it exists: experience that stays in one person’s head does not compound. Teams have a name for that, tribal knowledge, and it is a risk. Experience that is written where the fleet can read it does compound. It is the same instinct I have always worked by: leave the place better than you found it, for your team, your future self, or the next developer.
Building around it, and knowing when not to
Understanding eventually turns into building. And every build that has worked for me started the same way: by breaking the problem down until I found a piece small enough to solve completely.
Solve that smallest unit, prove it, then build up toward the greater system. When I wanted to trust a swarm of agents doing research, I did not start with the swarm; I started with one agent, one question, one brief, and got that unit right first (The Brief Is an API is that story). The system is just the subsystems with the connections engineered.
The measure that a loop actually closed, for me, is that the system needs less from me than it did before. Less input, less context, less hovering, more trust. If I automated something and I am still babysitting it, the loop is not closed; I just moved the work.
But there is a counterweight to all this building, and it is one life taught me early: pragmatism as restraint.
The temptation with every problem is to solve the general version of it immediately. Build the framework, handle the cases you have not hit yet, scale it before it needs to scale. I try to do the opposite. Solve the problem in front of you, cheaply, and watch what happens. It saves time, and more importantly you learn from the small version things you would have guessed wrong about the big one.
So I wait for the pattern to prove itself. My working rule is roughly three occurrences: do not build the general solution until you have watched the problem show up enough times to know its real shape. Emergence tells you what actually recurs. Pragmatism keeps you from solutioning ahead of the evidence. When the pattern does become prominent, then address it properly, with a solution fitted to the shape you observed rather than the shape you imagined at the start.
And even then, the solution you ship will not be perfect. It cannot be; you built it from the occurrences you have seen so far, and the world will produce ones you have not. Which is why the loop does not end at shipping. You monitor. You check in. The imperfect solution hands you right back to observability, and the loop keeps going, and the system keeps learning.
Why I automate
The last step of almost every one of these loops is automation. Not because automating things is fun (it is). I automate because I want consistency, and because I want my work to compound. A task done by hand happened once, and it happened however I happened to do it that day. A task automated happens every time, the same way, whether I am at the desk or not. The consistency is what makes the results trustworthy, and the trustworthiness is what lets me stack the next loop on top of this one instead of re-checking the last one forever.
That stacking is the part I care about most. The telemetry report watches the fleet weekly without me. The memory system briefs agents on old traps without me. Each closed loop is a small promise the system keeps on its own, and every kept promise makes the next delegation easier to make. That is how trust with these systems actually gets built, in my experience: not from a benchmark or a demo, but from watching something do the right thing enough times that you stop needing to watch (How to Trust a Swarm walks through what that took for a whole team of agents at once).
I learned this technology by watching every token because the window was too small for anything else. The window is enormous now, the tools run whole workflows on their own, and the lesson did not change: observe, ask why, learn the modes, solve the smallest unit, build up, hold back until the pattern is real, and then automate it so it keeps happening without you.
None of those loops is impressive on its own. Most of them are small. But they are still running, and they add to each other, and that is the point. I would rather have a small system that still works next year than an impressive one I have to rebuild every month.
That is the whole method: close the loop, one step at a time, and then go start the next one.