Taking Karpathy’s wiki to work
The idea of the “LLM knowledge base” is powerful, but how do you bring it into organizational contexts?
Knut Melvær
Principal Developer Marketing Manager
Published
If you have been handed the challenge of making an agent that actually knows your business (and makes no mistakes), you’re probably deep in making decisions about RAG pipelines and committing changes to
If this is you then you are focusing on the wrong layer. We have so many ways of handing context to agents, so the crucial question is no longer how they get it. It’s who owns what becomes context in the first place, and how do we make sure it’s consistent and correct? That’s a
I will leave you with a mental model for how to approach agentic context at scale, inspired by Andrej Karpathy’s post about “LLM Knowledge Bases.” And six questions to run against your setup.
The last couple of years, we have spent a lot of time discussing how to do agentic engineering. But it feels like we have spent less time discussing how we make agentic experiences work for those without “engineer” in their job description. Every time I see someone on X proclaim that “MCP is dead, let’s just use CLIs,” I get this feeling that they don’t realize that people who don’t have a CLI within claw’s reach use agents too.
Using agents is not the same as making agents, even though there is a lot of overlapping “stuff.”
Like, how do you make an agent that knows that you just sold out of a certain shoe, or what your new return policy is? How do you enable your wider organization to collaborate, maintain, govern, and work on an agentic context without giving them the keys to GitHub or redeploying a whole stack for every little change?
By context, I mean the information an agent needs to solve a problem or do a task. In most cases, this context will be content. In the sense that someone has authored it (maybe using AI), published it, and probably maintains and governs it.
Going beyond your personal use, agentic context starts to require owners, processes, audits, governance, distribution, and interoperability. Especially as agents start to do business-critical work, maybe even autonomously. Of course, nothing about these requirements is new. What is new is that the efficiency of agents and the promise of automation have created a huge incentive to figure it out.
As engineers, we now have an opportunity to figure out how we equip wider teams in our organizations to build and govern internal and customer-facing agentic experiences.
In April 2026, AI researcher and educator Andrej Karpathy (founding member of OpenAI, at Anthropic since May) posted his thoughts about “LLM Knowledge Bases” on X. It’s closing in on 22 million views and has spawned a lot of conversation, plus a few opportunistic GitHub repos in the replies.
In the post, he reflects on having agents build small wikis (a.k.a knowledge bases) when doing research by having them compile raw sources into a structure. “Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly,” he writes. Just as we all seem to be doing test-driven-development now (or rather, we have our agents doing it for us), we’ll finally have our wikis up-to-date (everyone who has been near a team’s Notion or Confluence the last decade knows what I’m talking about).
If we look closer, these knowledge bases aren’t just a bunch of files in folders. Karpathy has his agents traverse them by reading auto-maintained index files and brief summaries, then drilling in. Agents can do this part efficiently up to a certain point of scale. Since Large Language Models have a lot (as in “a shit ton”) of built-in domain knowledge, they often don’t need more than that list of titles and descriptions to quickly home in on whatever they need to know about a thing.
Karpathy’s post outlines, in miniature, the stuff you need to solve for when context is maintained by a team/s:
- Ingestion: The raw material that the knowledge base is built from
- IDE: The software that lets you browse/view/interact with the knowledge base
- Q&A: How your agent is able to ask questions and get answers
- Output: How you render whatever the agent is figuring out
- Linting: How you maintain the internal consistency of the knowledge base, and prevent entropy
- Extra tools: Whatever else you add to ingest, search, update, maintain, lint the knowledge base
He also recognizes that his setup is working at a small scale, roughly a hundred articles and 400k words, and that the tooling is a stopgap: “I think there is room here for an incredible new product instead of a hacky collection of scripts.”
Anyone who has worked in a company with an intranet, or any kind of internal information system, knows this: Content decays. Facts often live across multiple sources. When a fact is updated one place and not in another, you get drift, and facts becoming falsehoods. A lot of institutional knowledge doesn’t make it to its canonical dwelling, but remains as lore-somewhere-in-a-Slack-thread or is laid to rest in someone’s AI-generated meeting note that no one reads. And some facts expire without folks noticing. Engineers tend to call this entropy, and editorial teams call it drift.
Karpathy’s curated sources of articles, repos, papers, and datasets are less prone to the lossy reality of organizational “facts.” And when we use agents to compile facts from the org, it can get even harder to keep track of what ends up in the knowledge base, since LLMs tend to synthesize and write something that sounds plausible.
So you need a system where you can direct which sources get used, set rules for how agents compile them, trace every fact back to its sources, and persist the decisions that resolve conflicts once they're found.
In fact, letting agents compile knowledge bases from sources in your org can be a great way to reveal contradictions, entropy, and drift in content. Ideally, you should use this process to fix those things upstream. It gets especially valuable when you do it for your public-facing content, like fixing contradictions in your docs, support material, product information, and so on. Because that’s the version your customers, and their agents, are already reading.
What do you need to consider when you go from “works on my computer” to making an LLM-wiki organizational agentic infrastructure? Let’s view it through the structure that Karpathy laid out to ask some questions:
- Ingestion: Who can decide and manage what sources are used for the context? Do these changes have an audit trail?
- IDE: How does one configure, audit, and approve changes to its setup?
- Q&A: How is the knowledge made available for the wider org and customers? Can you Bring Your Own Agent (BYOA)?
- Output: Is there a single source of truth for the context? Is it clear where it’s derived from?
- Linting: When the model flags a contradiction or a stale claim, who decides which version is true? And does that decision survive the next rebuild, or does the wiki forget it?
- Extra tools: Who builds and runs the plumbing, like search, permissions, versioning, and backups? And who's on call when it breaks?
Start by thinking about this as a content operations problem. That allows you to lean on patterns that have been developed for decades.
This whole list maps to questions and areas that anyone who has ever implemented a CMS for a team can recognize. What changed is that Large Language Models, and the agents built on them, change how much content operations work we can hand off, and how fast.
You might argue that for an AI Content Operating System, all problems are likely to look like content problems. Fair. We build and sell in exactly this space, so consider this a disclaimer of sorts. But then again, it also means that we have seen these challenges in practice a lot. With customers and real-life implementations big and small.
The somewhat mind-bendy perspective is that we had great success in approaching content operations more as developer operations (DevOps), where by treating content as data, we are able to automate and distribute it in more efficient and reliable ways than traditional page-oriented CMSes have. But! We’ll also argue the reverse: to scale how we maintain and govern agentic context, you have to first look at it as a content operations problem, rather than a DevOps problem.
Let me give you an example. If you look at context as a DevOps problem, it feels intuitive to keep context as markdown files in a repository alongside your code published and maintained through GitHub (or any of the new challengers). This works to an extent.
Let’s say that you want to publish your company policy on your website (which also has an agent-friendly markdown version), have it available for a support agent, and the agent that automatically triages refunds. How does that get updated? And where?
The intuitive DevOps approach is that files in a repo are the source of truth and everything else derives from it. That works as long as the folks responsible have the access and have the git know-how (even via agents). But the moment someone without a git setup needs to interface with this, they now have to work through an engineer, agentic or not. Legal doesn't review code line diffs; they review documents, with effective dates, approvals, and a record of who signed off.
Sure, you can build all of that on top of git: scheduled merges, release branches for policies, an audit trail stitched together from commit messages. At some point it's worth noticing what you've built: a content management system, inside your repo, that you now maintain, constrained by git’s assumption that what it tracks is code. Most DevOps infrastructure carries the same assumption: what changes is business logic as code, and not knowledge as context for agents.
Scroll through the replies and forks under Karpathy's gist and you can watch teams arrive at the same requirements independently. Concurrent ingests forked one team's wiki, so they invented draft states and locking. Rebuilds kept overwriting human corrections, so they invented pinned overrides with provenance. Internal pages leaked into an external assistant, so they compiled a separate wiki per audience. Nobody in the thread says "content management system." Every fix feels like one.
So the question isn't whether you'll run a content operation for your agents. You will. The question is whether you build it by accident inside a repo, one incident at a time, or stand it up on purpose, with owners, review, and decisions that survive a rebuild.
We've spent a decade arguing content works better treated as data than as “posts and pages”; it’s why content on our platform is queryable, searchable, vectorized, and served over both HTTP and MCP. For agent context, it is the same argument, but with higher stakes. A stale page used to mislead the occasional human who found it. Behind an agent, it misleads confidently, at machine speed, in every conversation.
Karpathy’s closing line was that there’s room for an incredible product to replace his “hacky collection of scripts” approach. Our bet is that the product isn't a smarter wiki. It's the operation around it.
We've been building for exactly this, and we look forward to showing you later this quarter. Until then, ask the questions against your own setup. If every answer comes back to one person (and that person is you), you are in Karpathy’s position, and files in a repo might be fine. If it doesn’t, I’d like to hear how you’re handling it, and where it strains.