By Even Westvang, Rostislav Melkumyan, Knut Melvær
Published
Agents as Sanity documents, baked with Knowledge Bases, routed by a classifier and kept alive by Temporal. Here’s what’s under the hood of @s.
Jump to section
“Where are we most inconsistent across Sanity?”
“What on earth is panna-cotta?”
“Based on what we know about OKRs and product planning, which product teams have the most dependencies / interdependencies?”
“What does "dataset" mean? and what about a resource?”
“Who is best to ask about a really gnarly GROQ query I need to optimise?”
“Any closed-lost accounts where the original champion has moved to a new company?”
These were some of the many questions that were answered by our new super agent at Sanity, @s, during our all hands hackathon.
We built a custom harness that lets anyone at Sanity quickly stand up an agent with skills, tools, and knowledge bases that can be accessed in Slack (and soon elsewhere). It’s dogfooding Sanity as the agent configurations live in a Studio and dataset. Which means that anyone in the company can create and iterate on agents without having access to git or fire up a coding agent.
At our recent company all-hands in the south of Portugal, we asked everyone in our now 300 person company to build agents, across product, marketing, sales, customer solutions, finance, and people teams.
In this post, we want to unpack how it actually works, what useful/inspiring/whimsical agents folks built, and what we learned from this exercise.
Even and Rosti were responsible for both making the harness and running the hackathon. The recipe was pretty simple:
- One system prompt, carefully crafted
[1]Read How to write for an agent for how we think about crafting agent instructions. - One or more knowledge bases, containing the facts it needs to do the thing
- One or more tools, so it can actually do things
- Skills, for defining operational procedures for more specialized tasks
Traditionally, our all hands hackathons have been cross-functional, as coming together and building something is always fun. This time, however, we focused on departmental teams as we wanted to capture more of the actual day-to-day chores.
Rosti built a custom harness that allows you to make agents as quickly as you can fill out its instructions, give it access to resources, and publish a document in Sanity. It’s deployed in our company Slack and can be either invoked by tagging @s or invited to auto-respond in predefined channels. It’s inspired by Vercel’s super agent @v, the routing agent that they talk about in their eve launch blog post. While @s doesn’t use eve (it probably could!), it does use Vercel’s Chat SDK and other handy packages.
The idea is to avoid having people remember a bunch of different agents in the day-to-day. They can interact with @s by stating their problem, and it will route it to the right agent, or ask the user to specify from a selection of 3 which it is. If an agent never turns up, then it’s on its creator to create a better description.
There are plenty of agent harnesses we like out there. Like Mastra, which we use for the Content Agent, or the above-mentioned eve. However, we had multiple reasons to build our own harness:
- We wanted to define agents as Sanity documents. Most harness frameworks assume agents to be defined as code/markdown files, or ship their own studios. We wanted it to be dead simple without requiring git access.
- Agent turns need to outlive the process running them. We needed runs to be durable while the codebase was rapidly evolving. Defining this ourselves also allowed us to easily add scheduling for agents.
- We can try new things the week they ship. When TypeSafe’s Jev came along, we could adopt it in our router two weeks into the project. Maintaining our own harness allows us to be early adopters and not have to wait for framework maintainers to do it for us.
The harness uses libraries like AI SDK, Chat SDK, it’s deployed in clusters on Google Cloud Platform, and uses Temporal Cloud for durable runs. We give it access to inference via OpenRouter with zero data retention. We’re also using Vercel Sandbox to give agents that do data work their own computer in the thread.
There are different approaches for how to make multi-agent harnesses. Until recently, the intuitive approach was to have the conductor agent use LLM reasoning to decide which agent to delegate a task to. While that can make sense in many cases, we wanted to explore a different approach that has been made more feasible with the new(ish) “decision models”
This makes the initial step way faster and cheaper. All it does is to take the message
This means that agents that are too vaguely and/or broadly defined will either never or always be chosen. But agent authors will also quickly discover this and be incentivized to improve their descriptions. We saw that during the hackathon where HAL, a meeting hygiene agent, started to answer everyone because it was instructed to reply to questions about “workstreams.” It was very quickly noticed and fixed.
Even though “everyone is a developer” nowadays, with a harness, specific agents aren’t really made out of code. They are made out of instructions and configurations. Sure, we could probably have added our whole company to GitHub and have everyone from our account executives to accountants mint PRs to the superagent repo and spend their tokens on resolving merge conflicts, rebasing, and whatnot.
Nah, we’re not going to do that.
The feedback loop would still be slow and the barrier-to-entry way higher than just logging into a Sanity Studio in our dashboard and just defining the agent right there, or even skip that part and make one via the Sanity MCP that’s connected to the same project. Now they can test changes to the agent as quickly as publishing a document (which is pretty darn quick).
Agents aren’t made out of code. They’re made out of instructions.
Rosti modelled an agent as having a name, description (for triggering it), model choice, owner, instruction, skills, knowledge bases, and tools. It also has advanced options for controlling who it can answer and being able to spin up a sandbox that can run code.
When the agent document is published, it gets instantly available to the superagent, as do any subsequent updates. This makes the feedback loop when tweaking its instructions and capabilities very fast. Moreover, it makes it fairly easy to query agents whenever you need to, and the Sanity MCP can easily access their configurations too.
Another advantage over defining agents as markdown files in a repo, is that you can set up helpful guardrails for people who don’t want to wait for a CI/CD run to fail, but rather, get instant feedback when something is not ideal.
From this first basic version, we can later add more governance options and workflows for requiring certain evals, review, and so on. You can even query agent configurations with GROQ wherever you need to list them out, for example as a dynamic and for-ever up-to-date list in our Intranet.
There was one coding element of the hackathon and how the agent works. Tools, that is, the agent’s capabilities beyond pure information processing, do need to live as code in its repo. Rosti gave the superagent a set of basic tools like looking up employees with roles and departments and searching the web. You can also ask the agent to code new tools and connectors for itself, and open a PR for review. So part of the hackathon was also to add new tools to the repo. When they get merged, they are made available in the tools drop down in the studio.
At the end of the hackathon we’d gone from 9 to 98 tools
At Sanity, we manage our org- and team-wide skills also from a Sanity Studio
Agents pick their skills in their document, the same way they pick Knowledge Bases. When a message comes in, the agent only gets a short catalog of its skills and a skill-read tool for loading the full procedure when the task calls for it. That keeps the instructions lean, and the how-to only gets pulled in when it’s needed.
One of the main goals with this hackathon was for everyone in the company to use Knowledge Bases, the newish feature for Sanity Context. You create them through defining a purpose and sources, which can be content in a dataset, an external URL, and files. From those, we create a “LLM wiki” that extracts all the facts that can be quickly and efficiently looked up by agents.
Compiling a Knowledge Base only takes a few minutes of selecting its sources, defining its purpose, and having some patience while it builds. Getting an agent to use it well took a bit more thought.
We didn’t want agents on @s to search everything it has access to on every question. So when a question is passed to an agent, the harness will fetch the outline of each Knowledge Base it has access to and add it to its instructions: the title, the purpose, the paths to every entry. We did experiment with including each entry’s summary too, but the add to the context window didn’t really lead to better answers.
From here, agents have two tools for retrieving context out of a Knowledge Base:
knowledge-base-read: does what it says on the tin, fetches whole entries, which includes sources for each “fact.” We capped the read at about 15k tokens as we saw answers started to degrade after that.knowledge-base-search: when the outline isn’t enough to go on, the agent can use GROQ to search with BM25 scoring, with some extra weights for keywords that’s mentioned in the title vs its body
If the agent isn’t successful in retrieving what it needs from a knowledge base, it’s instructed to say that it wasn’t able to find the answer, rather than invent one. That mostly seemed to work (we saw few hallucinations during the hackathon testing).
During the day of the hackathon, around 80 commits landed in the @s repo with new tools and other tweaks. Every merge to main restarts the agent pods. It would suck if that also means that any agent that was working and being tested just died half-way.
Temporal is already part of our platform stack at Sanity, so it was a good choice for making agent runs durable. So every message to @s will start a Temporal workflow, using the Slack thread ID to keep it “idempotent.” This makes it possible for a worker to pick up an agent run if its original worker dies midway.
A few things we learned along the way using Temporal for this:
- Retry less than the default. Out of the box, Temporal will retry an activity forever. A tool that can’t reach its system (which frankly happens a lot while developing these things) would just give you a forever spinner in Slack. It’s better to let the tool call fail and let the agent tell the user that it happened. Three attempts is enough for a blip.
- Streaming needs a fallback. Slack supports streaming answers from agents now, but if the pod doesn’t confirm that the stream has ended, the workflow will just post the complete answer.
- Stop means stop. Slack has a new affordance to stop agent runs in its UI, which should ultimately also stop the Temporal workflow. It is a bit annoying that this affordance is close to where you expand a thread though. We saw a lot of accidental stops during the company testing.
- Limits are checked between steps. We gave turns 40 steps, a million tokens, and ten minutes. Checking between steps allows us to let the agent stop with an answer instead of being cut off mid-call.
This setup pretty much gave us background tasks and scheduled agents for free. A task can run up to an hour and have 200 steps, and you can steer it by replying in the thread in Slack. It continues as a new workflow before it gets too big for Temporal Cloud. We should also add that everything Temporal stores is encrypted with AES-256-GCM before it leaves our cluster, since it holds whole conversations and whatever the tools return.
So that’s pretty much the shape of the agent harness. It’s fairly basic as of now, and there’s obviously tons of things we can do to extend it, like supporting more triggers, more automated governance around shipping agents internally, you name it.
It’s pretty interesting to ask a whole company to build agents for work they find… annoying. Meaning that the asks happen often enough and take enough manual labour to be worth giving over to an agent. Regardless if the agents work or not, it is a bit like holding a mirror up to your own company.
It’s a bit like holding a mirror up to your own company.
So here’s a smol sample of some agents that turned out pretty useful and probably will live on after the post-hackathon agent purge:
- Sherlog tells you whether a production service is healthy and when a problem started. It has a single tool,
clickhouse-investigatethat queries our logs, plus three Knowledge Bases and a skill for how to investigate. It was helpful on its first day as we experienced a real outage in one of our services (unrelated to the hackathon). - Demo Concierge was the customer solutions team's attempt to streamline bootstrapping custom demos for incoming customers. It finds a demo template, provisions a working copy with a new organization, a studio, and a website. It’s largely scripted so the agent can run on haiku and be cheap.
- Smokey (not the bear) helps support engineers decide whether a ticket should become an incident. It has nine tools across Plain, incident.io, Omni, and BigQuery to help support engineers very quickly connect the smoking dots.
- Can I Expense will now automatically reply to anyone who asks about expense policy, which now exists in a Knowledge Base, in our #finance-and-accounting channel. And no, it isn’t automatically set to always answer “no.”
- Who is uses a Knowledge Base about what folks are working with and answers any question starting with “who” with names, and nothing but names. Ask it something that isn’t about work (“who’s the best dancer?”) and it picks a seemingly random team member from Slack and tags them with a trollface.
The main learning is that it’s totally reasonable and very interesting to ask a whole company to make agents. We also learned that it probably takes a bit more time than just a couple of hours to make a workstream fully agentic.
About 42 of the 61 agents entered at the hackathon worked fully or partly. Those who didn’t largely failed on not having access to resources they needed to get stuff done. So it could be easy enough to understand what type of context they needed, but it wasn’t always straightforward to establish the access (in a secure and sensible way).
The router only knows an agent by its description. Since Jev returns probabilities (and confidence) it forced us to write agent descriptions that were specific enough for the agent to be chosen for the right tasks without overlapping too much with potentially similar agents.
Since Knowledge Bases are compiled using inference, they need to be given a clear purpose so that the right information from its sources are collected and structured in the right way. So when teams found that their Knowledge Bases lacked certain necessary facts, that was largely because they hadn’t been specific enough in the purpose they gave it.
@s is just a thin layer: a classifier in the front, agents as documents, durable runs on Temporal, and tools in a repo. Most of its value sits inside the documents that folks made, the Knowledge Bases they generated, and the collective experience we all got from getting together to make agents. We expect the harness to shift like the ship of Theseus, but we wouldn’t be surprised if a lot of the Knowledge Bases settle and change way less (apart from keeping the facts up to date, of course).
If you're building something similar, you don't need our harness to get the knowledge part. Context MCP serves the same outline, read and search tools to whatever agent framework you're on.
- New to Sanity? Start a new project with Sanity Context switched on.
- Already on Sanity? Knowledge Bases are in beta. An organization admin enables them on the Labs page in Manage.
- Want the why? Read In defense of agent sprawl for what the agents do and why we stopped worrying about agent sprawl.
More from Engineering