AgentsAgents are writing more interface code than ever, which sounds like great news for design systems. Agents love to reuse code and design systems love to be used. But, like most things, it’s not that simple. Agents fumble with design systems all the time. They hallucinate props, they fixate on wrong things in the docs, they ignore the right things… It’s buckets of fun.

We hit this problem too at Sanity. We’ve maintained our design system (Sanity UI) for years to help keep our interfaces consistent. We’ve spent a lot of effort to make our system work better with agents. Turns out, the answer wasn’t “just throw documentation at the problem.” Rather, it was how the documentation is formatted.

As you may suspect, we had to make a tool to tackle this issue. I’ve been using it for a few months now. I’ve laughed, I’ve cried, and I’ve got opinions.

Now’s as good a time as ever to share our learnings, the open-source tool, and the specification for design system documentation.

(Yes, all em-dashes in this blog post are hand-typed and free-range farm raised.)

YouYou can do a lot with Sanity—with agents, APIs, and with user interfaces. And despite of agents showing up at work, user interfaces are still incredibly important for content work. Sanity has been around since 2017, so it’s no surprise that we have a lot of interface code. But managing UI across many contributors is notoriously hard. Drift sets in fast. Before you know it, the experience is inconsistent with the added bonus of debt. Design systems help avoid that. They define interface design standards and make them actionable through reusable components.

We’re working on the next version of our design system—Sanity UI. It should be the best way for anyone or anything to build Sanity interfaces. That’s not always the case (yet)—specifically for agents. But it’s easy to notice something isn’t working—figuring out why is a whole other thing. We needed a way to expose the problems in gory detail. So we built a tool to test agents on our design system. It spins up agents, gives them the same prompt, then evaluates the results.

Oh, and it’s available for you to try. Now, keep in mind this is a work-in-progress tool. It only supports Claude models and has plenty of sharp edges. But it’s a starting point for any weary design systems traveler.

II write every post about AI with this preamble. This is all new. I’m flying blind, and I’m OK with that. I’m sharing this so we can learn together.

Please don’t read any of this as a declaration of rightness.

HowHow the tool works is not brain surgery[1]. It accepts agent tests aka “evals.” A test defines the design system to use, setup instructions and training data. Every test spins up an n number of agents—each with the same prompt. Agents use the prompt to build UI. The results are measured, and a report pops out the other side. That’s it. That’s the loop.

OK, fine, there’s more to it.

You can choose specific models. The prompt can be custom or agent-generated. Tests can use markdown or an MCP server for training. There’s plenty of doodads to play with—if that’s your sort of thing.

All that is great, but it wouldn’t be useful without the evaluation. The final report provides a lot of info. It lists the number of tries to get a clean build, accessibility issues, lines-of-code, performance metrics, visual diffs per iteration, token spend, and more. It also collates feedback from agents on what was confusing or tripped them up (more on that later).

There’s a decent amount in this tool, but the core is simple. Test, assess, repeat.

TheThe first couple weeks using this tool wasn’t fun. The new design system didn’t have many components, so we covered gaps with the current version. The docs were also new—meaning they were rough. And this whole agent test harness thing was new to us. The wins were few and far between. Just getting a single successful build felt like a miracle. The agent feedback (politely) told me to find a new line of work.

Close

The issue causing this pain was documentation, or a lack of it. Sanity UI isn’t a huge design system with gobs of training data in the wild. If our docs aren’t helping, agents are on their own.

We started giving agents markdown docs, but the results varied heavily per test. And the docs became rough to manage over time. We weren’t sure markdown formatted docs was the answer. I’d been noodling on a schema for design system documentation in my spare time. It gives you a JSON-based schema for defining parts and concerns within a design system.

{
  "kind": "use-cases",
  "purpose": "Buttons trigger immediate actions within a surface.",
  "items": [
    {
      "description": "When the user needs to trigger an action such as submitting a form.",
      "stance": "recommended"
    },
    {
      "description": "When the action navigates to a different page.",
      "stance": "discouraged",
      "alternative": {
        "identifier": "link",
        "rationale": "Links carry native navigation semantics."
      }
    }
  ]
}

We decided to give it a real shot. That ended up being a turning point.

Close

Subtle improvements began showing after each test. Feedback and error logs got clearer. We started to know what the issues were. As big problems got fixed, feedback improved, and shipping refinements got faster. Make no mistake, we were light years away from good. There was light at the end of the tunnel though. The agents decided I could keep my job.

We’re about four months into using this tool. It’s still a daily-driver, but the results are more stable. Our new design system is still mid-swing, so there’s always something changing. I test changes against a pool of agents whenever we ship a new component, change our API, or update our docs. Sometimes it’s to make sure nothing broke. Other times it’s to sniff out new areas of improvement. Most of the time it’s to make our documentation is that much better.

TraditionalTraditional design system feedback cycles can be glacial. This tool made feedback loops nearly instant. This enabled a (very) tight iteration cycle.

It looks like this:

1. Agents slurp the docs (via markdown, examples, MCP, or whatever else)
2. Tests’ feedback and results give hints for where the docs failed
3. Use those hints to fix the docs
4. Test again
5. Measure the before/after
6. Do it again

You used to hope doc improvements were actually improvements. That reality hasn’t changed for people using the system, but it has for agents. We can tell if a change worked in a fraction of the time. But the trick is how to know what to change—and how.

Which brings me to agent feedback. There’s the kind of feedback you get from your dentist. Like, when they tell you to floss. That’s the kind of thing you take at face value. Then there’s the kind of feedback a smoke detector gives. It beeps when there’s smoke. It doesn’t say why there’s smoke, or where to put out the fire.

It does a good job reporting symptoms, but not the cause. And definitely not the solution. Don’t get me wrong—the agent feedback was essential to accelerating this flywheel. But it depended on thoughtful, human translation.

SoSo what did we get after all this work? Agents are better able to build a working Sanity UI interface with the new system. Across all models. By a long shot.

We wanted to make sure that our results weren’t some weird one-off that was out of touch with reality. So we ran the full suite through AILF[2]—our internal eval harness that scores how well AI coding agents can actually build with Sanity UI, based purely on its docs. It tests agents on tasks with no docs, with docs handed to them directly, and with agents finding docs on their own. Across 76 test tasks, the library landed a 90/100 score, with a +34 point gap between agents with no docs and agents with docs provided. So, yeah, the docs are doing some heavy lifting.

We’ve even tested our new version of Sanity UI against the big systems like Polaris, Carbon, and Gestalt (my past fling). We don’t want to get ahead of ourselves, but the new Sanity UI appears to be hanging with them.

Anytime you’re playing with agents, you’re playing with money. This is something you just have to take into consideration when using a tool like this. It’s also exactly why the tool measures token usage. The trick is to run enough tests to get signal, but not enough to burn cash.

But the thing is, providers charge you the same amount whether an agent was able to perform a task or not. So if agents can build with the design system more effectively, that’s cash saved from the proverbial woodchipper. And that impact multiplies based on how many agents use the system. Here’s the cost breakdown for the results above:

I’m clearly biased, but I think these tests have been worth every penny given the improved success rates, faster completion times, and lower token spend.

I’ve found tests with 3-5 iterations are enough to know if things are working. For more definitive results I can bump it up to 10. There’s also interesting results to be found by tuning effort (which the tool allows you to configure). There’s infinite variables to consider based on the size of your documentation or what you’re testing agents against. Your mileage will vary. I guess what I’m saying is, don’t run a test with 1,000 iterations.

As you’d expect, tests on low cost models cost less. Tests on high cost models cost more. More or less. The prompt matters a lot as well—which you also know. Here’s the average costs we’re seeing per test iteration.

• Haiku: $0.27 per iteration
• Sonnet 4.6: $0.75 per iteration
• Opus 4.8: $1.10 per iteration
• Sonnet 5: $1.60 per iteration (interesting outlier)
• Fable 5: $2.52 per iteration (yikes)

II could see teams treating faster iteration loops as a cue for continuous documentation. Because I tried it. I experimented with agent-driven documentation that would continually address agent feedback. More context means better results, right?

I justified the approach since humans wouldn’t ever see this documentation. Hell, maybe agents prefer to read what agents write. I ran the feedback loop for a few days to see if improvements would follow. They didn’t. The results plateaued. There was so much documentation that it became noise. Agents would latch on to different content or choose to ignore parts. Why? No idea.

Turns out more isn’t better—better is better. I know, shocking. So, on to Plan B.

MoarMoar docs hit a ceiling. We weren’t going to context-bomb our way to success. So we took the opposite approach—we tested workflows where agents were asked to do less.

Yeah, we get the value of agents writing code. But they produce highly varied output when left to their own devices (that’s a pun, right?). Every agent makes a dashboard in their own special way. We wanted to see if we could lean on agents less. Not to stop agents from writing code altogether. But we think a little can go a long way.

We built an MCP server to support this theory. It both delivers our docs and provides code assistance tools. Three in particular are worth noting: an agent-wizard for configuring a component, a lint/autofix function, and a code chunk provider. I’ve ordered them from worst to best.

The interactive tool attempts to fix the countless hallucination of prop names and values. The tool walks an agent through each component’s props and available values, then returns the final code. Does it work? Not really. Agents seem to get bored and nope out of the process midway through. If I had to guess, this one has the biggest chance of getting canned.

The linter is better but still needs work. We added an MCP tool to lint/autofix agents’ code. Agents were repeating similar errors all the time. So our lint rules focused on those. Agents didn’t have to diagnose common errors. Instead, good old-fashioned lint rules take the job. This meant agents didn’t need to be perfect, just close enough for linting to do the rest. It sounds great—at least to me—but the results are still spotty.

The last tool has turned out to be the biggest winner—and most obvious. The tool provides chunks of reusable code for common use cases. We added a Chunks schema to our schema and wired it up to the MCP. Agents get a list of the chunks available to use along with tips on when/why/how to use. They can add that code directly into the project. It’s like Stack Overflow all over again.

{
  "kind": "chunk",
  "identifier": "empty-state",
  "name": "Empty state",
  "description": "A centered message with an action for when a list has nothing to show.",
  "code": {
    "language": "tsx",
    "code": "<Flex align=\"center\" direction=\"column\" gap={3} padding={6}>\n  <Text muted>No documents yet</Text>\n  <Button text=\"Create document\" tone=\"primary\" />\n</Flex>"
  },
  "documentBlocks": [
    {
      "kind": "use-cases",
      "items": [
        {
          "description": "Fill an empty list or search result instead of rendering nothing.",
          "stance": "recommended"
        }
      ]
    }
  ],
  "agentDocumentBlocks": [
    {
      "kind": "checklist",
      "items": [
        {
          "label": "Replace the placeholder message and button label with copy for the actual context.",
          "level": "must"
        }
      ]
    }
  ]
}

People might ask why anyone should have chunks instead of components? Solid question. Not every chunk of code needs to be a component. Some are too big/unwieldy to be one. Also, making a component takes a decent amount of time/care. Chunks are a way to get reusable code in the hands of people/agents now. It’s also a great low-key way to submit contributions…

And here’s how these tools impact results:

Now, I’d love to say we’ve figured all of this out and already rode into the sunset. But the results say otherwise. I went into this thinking that these tools would shave token spend. Nope. It’s actually the opposite sometimes. This seems like a failure on the surface. But I’m not so sure just yet…

II ran into an interesting kink while producing test results for this blog post. I was running side-by-side comparisons of our current system against the new system with docs, chunks and linters enabled. I noticed that the results for the new system were good, but not really good. I brushed that aside as run-of-the-mill variance you can get from agent tasks.

At some point it dawned on me that running side-by-side tests where the new system gets docs, chunks and lint rules isn’t really a side-by-side test. So, I disabled chunks and linting from the new system’s tests. And they tanked. Which, you know, was great for my blood pressure.

It turns out that agents were actually struggling with a couple of recently added components. The chunks and linting were acting as a semi-protective membrane from total failure. That’s why we saw small, but noticeable declines in the initial tests. This is great because those tools can help protect against issues where agents struggle with documentation. This is also horrible because it hides when agents struggle with documentation.

We were able to address the root cause that caused confusion and the results went back up to really good. Now I run all documentation tests with chunks and linting off. But in no way am I getting rid of them because it turns out they have another upside.

It’sIt’s easy to measure if an agent can get a web app to build. It’s much harder to measure if what got built is any good. Now, I don’t think it’s possible to measure quality. But you can measure elements of it. And while token spend didn’t improve, all sorts of quality-related elements did.

Three quality markers we can measure are accessibility violations, number of inline styles, and percentage of times the interface is responsive across devices. Chunks and linting showed a pretty stark improvement against agents with no assistance.

So, while it’s not a win across the board, using linters and chunks improves accessibility test results as well as how often the UI works responsively. And this shouldn’t come as a surprise. Chunks represent our ability to tell agents follow an exact pattern that follows design guidelines. Linters turn non-deterministic no-nos into “the right way”. So while I’m confused-in-a-bad-way about token spend, I’m not-confused-in-a-good way about quality markers.

And then there’s the eyeball test. This is the least objective measure, but also maybe the most important. None of the builds are winning any design awards. Still, using chunks and linting shows a noticeable leap in consistency and visual execution .

Close

Close

Close

Close

Close

Close

Perfect? Nah—not even close. But we’re seeing continual improvement. And I’ll take it.

II find it fun that a lot of the lessons are more of the same for design systems.

Documentation is always vital, but it can be misunderstood or ignored. On-rails, turnkey solutions remove foot guns. More is rarely the best option. We’re learning these timeless lessons all over again. It’s interesting how much stays the same. In many ways, this tool helped us reach that conclusion much faster.

This tool showed us once again that agents aren’t magic. They aren’t useless either. They’re effective—in the right situation, at the right time, and in the right amount. And we’re learning more every day. It’s why I know Sanity UI and agents will get along just fine—soon enough.

And if you try out this design system tool, do let us know how it worked for you. You can find us in Discord and on most of the socials.