How VillageSQL does documentation
How VillageSQL writes its docs: a public repo, an AI agent that runs every example it writes on a live server, and a person who approves.
We've heard positive feedback about VillageSQL's documentation, so we wanted to share a bit more on how we build it and how we hope to evolve it. VillageSQL Server tracks upstream MySQL and adds an extension framework, so the documentation has two moving targets. We pull in upstream changes with each MySQL release, and VillageSQL's own changes land daily in the dev branch and ship with each release. We believe that it's important for a complex project to provide accurate documentation that is updated as frequently as code ships. This is easier said than done. We use AI extensively to generate our docs, but we believe it's critical to have a human as a review gate before any change is shipped. AI makes mistakes, so we make it prove every change to itself and to humans by testing against a running server.
Frequent readers of the VillageSQL docs will also notice a recent redesign. The goal with that was to make the site easier to navigate. As with everything we do, we'd love your feedback.
The docs are a public repository
The documentation lives in villagesql-docs on GitHub as a public repo under the same rules as the server repo. Anyone can submit a change as a pull request. If you want to know more about VillageSQL's approach to open source, read our post "How VillageSQL does open source".
The docs have English, Japanese, Korean, Chinese (Simplified), and Brazilian Portuguese versions. The translations are the product of AI. The current set was translated by Claude Fable 5.1 and then reviewed by Gemini 4, which compares each translated page with the English source page. A finding from the Gemini reviewer is verified by a third agent, Claude Fable 5.1, before it's applied, because the reviewer is as capable of being wrong as the translator. We have experimented with locally running models as both the translator and reviewer but found that hosted models hit the right price/performance bar for our current needs. We are grateful to the Brazilian MySQL community for feedback on the Portuguese translations.
Why we like Mintlify
The docs site is hosted on Mintlify. We like Mintlify because their platform is flexible, intuitive for humans and agents to use, and their team is responsive and inquisitive. We picked Mintlify when we launched the documentation site and chose to stay on it through the recent redesign.
The docs are just markdown files, so they are simple for humans and agents to work with. Mintlify automatically publishes a machine-readable index of the docs for AI agents. For a complex piece of software like a database, making the docs accessible to both humans and agents matters.
We are starting to experiment with Mintlify's advanced features like Autopilot, which can read changes in the server repo and open pull requests against our docs repo, as a complement to the homegrown workflow below.
An AI agent with a human as the gate
Every weekday, an agent looks for docs pages that have fallen behind the VillageSQL Server code and VillageSQL-authored extensions. It then drafts suggested changes. The server pages have a copy for each MySQL version we track (8.4 and 9.7). The agent covers those pages, the server README, and the README of every VillageSQL-authored extension.
When we started down this path, the workflow ended at this draft stage. We quickly discovered that even AIs make mistakes (sarcasm). So, we introduced a workflow step that has mattered a lot: the agent pulls every SQL statement and command out of its own drafts and runs each one on a live VillageSQL Server build that is current up to the commit of the change being tested. If the output on the page doesn't match the output from the server, one of them is wrong, and the agent works out which before finalizing the docs change.
Live verification catches what the agent initially got wrong. It doesn't catch what the agent didn't think to check, so a critic agent with none of the authoring agent's context checks the draft against its sources and flags any claim that has no source. The authoring agent gets two rounds to fix what the critic raises. Whatever still stands after that goes to a human to make a judgment call. During drafting, the work is split across smaller helper agents: each one owns one page or one pull request, and a coordinating agent reads their findings. Through trial and error, we have found that giving agents discrete tasks keeps each one's context small, and in our experience that means fewer invented claims when dealing with a complex codebase.
Next comes the human review. The agent summarizes the workflow run: what changed, why, and what pull requests it recommends. A person says yes or no. On yes, the agent commits and opens draft pull requests, or adds commits to pull requests that are still in review. The agent groups the docs changes into one pull request per engineer whose server change triggered them, so the engineer who made the change can review it. As soon as that human review is complete and the pull request merges, the live docs site is rebuilt with the update.
This workflow has evolved significantly over time. It started life as a simple workflow running on a laptop and is now an Anthropic Managed Agent, running on a Mac mini that interacts with its human reviewers via Slack. Some revisions are about accuracy, such as the critic pass and reading each pull request diff three times independently before trusting it. We can't seem to get better than an 80% accuracy score (tested by planting defects) in a single session, which is why we run so many passes over a single change.
How the agent learns
The agent starts every run drawing on a set of instructions about VillageSQL that has been built up through the many iterations of the workflow. There are three kinds of files. A rules file says how things are done: what a docs change looks like, where a new feature's documentation goes, and how error text is captured. A lessons file holds an accumulated list of assumptions and mistakes past agents have made. A history file holds the story behind each lesson, so a rule that looks arbitrary can be understood. When a person corrects the agent, the correction goes into those files before the session ends so the next run learns from it.
Guardrails on every page
Instructions tell the agent what to do. Checks help keep it in line when it doesn't follow them. Like instructions, most of the checks we use come from trial and error.
Every docs page about extensions, the SDK, or the server declares its audience in its front matter: someone writing an extension, someone contributing to the server, or someone using the database. The audience field forces the question "who is this for?" at the moment of writing. This helps prevent the AI from conflating topics on a page. For example, the raw ABI is what the SDK connects to, and the SDK is what extension authors connect to, so a page for extension authors should not drift into the raw ABI. That distinction was confusing to agents initially.
Other checks read the code blocks on a page and compare them against a list of banned phrases and past mistakes. Documented table columns and system variables are checked against a running server. A code block that names a variable the server doesn't have, or uses a MySQL concept that isn't relevant to VillageSQL-specific documentation, gets flagged by name. Checks also look for AI-slop language we want to keep out of the docs, such as transition language ("the server now calls", "previously the extension"), because a reference page describes the current state, and a reader who never saw the old page does not need the correction.
The strictest rule is about error text. Any error message, error number, or warning on a page must be pasted verbatim from a running server. An agent never types one from memory or infers one from the source code.
Some interesting lessons we have taught the agent
Below are some examples of lessons we have had to "teach" the agent.
- A stale server produces confident, wrong findings. Before anything is verified, we compare when the server was built with when the code last changed. A result from a stale build is reproducible and often false, which is the worst kind of wrong.
- AI adds. On its own, it rarely removes. When an agent finds a wrong sentence, its instinct is to add a note beside it rather than fix it. We make it change the sentence, and every review pass on our docs has asked for removals.
- A fix in one copy is a bug in the others. We maintain docs for multiple versions in multiple languages. A fix that lands in just one copy and misses the rest leaves the docs disagreeing with themselves. A rule is that a fix goes to every copy.
The workflow is a means to an end
An open process that continuously improves the docs is the goal, so users have the latest as soon as code ships. Our current workflow is just a means to an end. AI tooling is improving fast enough that the best way to achieve our goals a year from now probably doesn't exist yet. We'll keep the rules, the lessons, and the checks, because those are ours, and we'll swap the tooling under them whenever something better comes along. If you'd like to contribute to the VillageSQL documentation, you can do so in the public repo.