MySQL

Announcing VillageSQL Server 0.0.7

Adam Levin

Adam Levin

— 8 min read
Share
Announcing VillageSQL Server 0.0.7

We are pleased to announce the launch of VillageSQL Server 0.0.7. You now have a License to Query (please note this post contains lots of James Bond 007 database puns). In this release, the mysql-8.4 build tracks MySQL Server 8.4.11, the mysql-9.7 build tracks MySQL Server 9.7.2, and the percona-8.4 build carries Percona Server 8.4.11-11.

Highlights of this release include the following:

  • The server can now describe a custom index, store its data, report it, and run a nearest-neighbor search through it. This is the foundation of custom indexing and vector search we demoed at Percona Live Amsterdam 2026.
  • CLONE now checks that the recipient has your extensions before it copies anything.
  • An extension can give every connection its own copy of a setting.
  • Every bundled extension, and the server itself, runs under Valgrind and ThreadSanitizer every night.

Custom indexes defined by extensions

MySQL gives you B-tree, R-tree, hash, and full-text indexes, and that list is fixed. If you want a vector index, you need to wait for a future MySQL release that ships one, and MySQL Community Server doesn't ship one in any version. VillageSQL Server custom indexes let you define an index yourself as an extension. The extension supplies the index type, the storage, the build, and the search, and the server treats the result as a first-class index. This release lays the foundation for that (the Quantum of SQL).

SHOW CREATE TABLE renders a custom index as DDL, with its type, the distance profile for each column (hnsw_cosine here), and the WITH parameters the extension defined:

  KEY `idx_title` (`title_vec` `vsql_vector`.`hnsw_cosine`) USING EXTENDED(`vsql_vector`.`hnsw`) WITH (`ef_construction` = 64, `m` = 8),
  KEY `idx_body` (`body_vec` `vsql_vector`.`hnsw_l1`) USING EXTENDED(`vsql_vector`.`hnsw`),

The server doesn't interpret m or ef_construction on its own. It stores them because the extension asked it to, and prints them back as DDL you can replay. SHOW INDEX and INFORMATION_SCHEMA.STATISTICS now report the extension's index type name. The server also stores the data a custom index needs. A vector column lives outside the row, in storage the extension manages, and InnoDB now indexes it through a reference, so you can index a column far wider than the usual key-length limit.

The server can also plan a search through a custom index. The query shape is a distance function over the indexed column in ORDER BY, with a LIMIT. The server routes that query to the extension's index instead of sorting the whole table. The classic optimizer reports the access type as index_distance, and EXPLAIN FORMAT=TREE prints Custom index distance scan on idx_title. The hypergraph optimizer picks the same plan.

The server runs the whole scan: it picks the index, asks the extension for the nearest entries, and fetches each row by the primary key the index stored. That means the index has to store the primary key. The vsql-vector-pk extension does, and its HNSW index answers these queries end to end on this release. It is a variant of vsql-vector whose index stores the primary key alongside each entry.

The remaining work on custom indexing is index maintenance. On a table with a custom index, the server currently refuses an UPDATE that changes the primary key or a column in any index, ordinary indexes included, and an ALTER TABLE that changes the primary key. It also refuses a DELETE (DELETE Another Day) against an index that identifies its entries by its own handles rather than by row key, because the server can't yet tell the index which entry to drop. The HNSW indexes in vsql-vector and vsql-vector-pk both work this way, so neither extension supports DELETE yet. Other ALTER TABLE operations, including a full table rebuild, keep the index.

At Percona Live in Amsterdam on September 9 we demoed vector search in MySQL on a development build. An HNSW index defined entirely by the vsql-vector extension answered a nearest-neighbor query from the graph instead of scanning the table, and we measured recall against the same rows. The metadata, the storage, the DDL, and the server side of that scan all ship in this release.

CLONE now checks for your extensions

CLONE INSTANCE copies a running server's data into another server. We extended its compatibility checks to cover extensions, alongside the existing checks for plugins, character sets, and configuration. Extension bundles (.veb files) live outside the data directory, so a clone doesn't copy them, but the system table that records which extensions are installed is copied.

We check the donor's extension list during the handshake, before any data moves. Every extension installed on the donor has to be present in the recipient's veb_dir with a matching version and a matching .veb checksum, which is the same bar the server applies to its own extensions at startup.

This check matters beyond clone. An extension can define a column type whose stored bytes mean nothing without the extension that wrote them. The extension is part of the server's state, so every operational path that copies, moves, or upgrades a server has to account for it. Clone is the first path we covered. Backup and restore, upgrades across codebases, and dependencies between extensions come next.

Extension functions can compute their own return type

The vsql_tvector example extension defines TVECTOR(N), a fixed-width vector type. A function that joins a TVECTOR(3) and a TVECTOR(4) end to end should return a TVECTOR(7). The server's built-in type rules can't produce that answer: they copy one argument's parameters onto the return value, and they reject a call whose arguments disagree.

An extension can now register a function with bind_and_check_types, and the server calls that hook while it resolves the statement:

make_func<&tvector_concat>("tvector_concat")
    .bind_and_check_types<&tvector_concat_bind>()

The hook receives each argument's kind, the type parameters already resolved for it, and the literal text when an argument is a string constant. It returns the type parameters for the result, optionally new parameters for each argument, or an error. The hook replaces the built-in rules for that function, so a call the server would have rejected is allowed and the return type is the one the extension computed.

vsql_tvector gains 7 functions that use it, covering a computed return dimension, 2 that fix the dimension and inherit the element type, 1 that inherits the dimension and fixes the element type, 2 reductions to a scalar, and an accessor. You can use these as examples when building your own extension.

Per-session extension settings

An extension can declare per-session variables through the new vsql::session_var capability. Every connection holds its own value, SET SESSION changes it, and the value the extension declared is the global default. INT and STR are supported. Extension system variables, which hold one value for the whole server, work as they did before.

The vector work is why this exists. HNSW search has a dial that trades recall against speed, and that choice belongs to the query, so vsql-vector declares the dial per session:

SET SESSION vsql_vector.ef_search = 100;

For vsql-vector, the default is 40, and a value outside 1 to 65536 is clamped to the nearest bound. Turn it up and the search reads more of the graph, returns better answers, and takes longer. One connection can run an analytics query at 400 while every other connection stays at the default. A function reads the calling session's value through int_var(). The capability is preview, so it needs vsql_allow_preview_extensions=ON.

System variables with a fixed list of values

For Your Enums Only. A vsql::sys_var variable can now be an enum: a fixed list of names, stored as an index. The server validates the name at SET time and rejects anything off the list, so the extension no longer has to check a free-form string after the fact.

Statement events identify the client

The Spy Who Logged Me. The vsql::preview::statement_event capability now reports the client process id, the connector library, and the application name, each NULL when the client didn't send it. If you're trying to be The Man with the Golden Query and writing per-query telemetry, that's what attributes a slow query to the service that ran it.

Breaking changes

DDL that the server used to accept can now be rejected. The server validates type parameters before it hands them to an extension, and for the TYPE, TYPE(N), and TYPE('k=v,...') forms it rejects a duplicate parameter name, a nameless parameter, and a parameter with no value. After the extension's resolve_params callback returns, the server also rejects a rewrite that dropped a parameter. Adding parameters and changing values is still legal. Check your DDL and your extensions against these rules before you upgrade.

The preview authentication capability changed too. auto_grant is gone, replaced by roles_mode, which tells the server whether to activate, grant, or sync the roles an identity provider supplies. An authentication extension built against the 0.0.6 preview needs a source change to compile. vsql-oauth2 already has, and its own vsql_oauth2.auto_grant setting still exists.

Strengthening testing

Tomorrow Never Deadlocks. Every night we run the unit tests, the full VillageSQL test suite, and every bundled extension under both Valgrind memcheck and ThreadSanitizer, so the server and the extensions you get with a release have been through both. We also used gcov to find code the suites never reached and wrote tests for it, and moving our CI runners to Ubuntu 26.04 turned up undefined behavior in our type-parameter parsing along with an upstream bug in the Group Replication plugin.

New extensions

Three extensions joined this cycle:

  • vsql-duckdb (From Amsterdam with Select): query Parquet, CSV, and JSON files from an ordinary MySQL connection, without exporting the data first or standing up a second query engine. It embeds DuckDB in the server process and reads the files where they already live, in an s3:// bucket or behind an https:// URL. Reading files off the server's own disk works too, once you turn on vsql_duckdb.allow_local_files. It's a port of the PostgreSQL extension pg_duckdb, narrowed to what the extension framework supports today. It uses preview capabilities, so it needs vsql_allow_preview_extensions=ON, and you build it from source. It also needs a server newer than 0.0.6, so this is the first release it runs on.
  • vsql-vector (Dr. No-Scan): the SVECTOR vector type, stored as packed floats in InnoDB pages through the column storage interface, the distance functions to go with it, and the HNSW index type the demo ran on. Its index does not store a row reference, so on this release the server builds and keeps it current on INSERT but cannot answer a query through it. It needs vsql_allow_preview_extensions=ON, and you build it from source.
  • vsql-vector-pk (The Vector Who Loved Me): the same SVECTOR type and HNSW index, with the primary key stored alongside each entry. That is what the nearest-neighbor scan described above needs, so this is the variant that answers queries on this release. A server runs either vsql-vector or vsql-vector-pk, never both. It needs vsql_allow_preview_extensions=ON, and you build it from source.

What's ahead

The next release is more hardening and production readiness. We're testing VillageSQL against popular ORMs, the JDBC and ODBC connectors, the Windows client, and backup and restore. We're building performance tests for a server running extensions, so that when someone asks what an extension costs in terms of performance we have a number instead of an opinion. Custom indexes carry on in parallel, on the query side and the write side, until an extension's index runs end to end on a released server without the preview flag.

Special thanks

The native RPM and DEB packaging in this release came from Evgeniy Patlan at Percona. The packages take over an existing MySQL install (On Her Majesty's Secret Server) rather than sitting beside one, sharing its binary names, data directory, systemd unit, and port, and the build dependencies resolve on EL8, EL9, EL10, Ubuntu 24.04, Ubuntu 26.04, Debian 12, and Debian 13. No workflow publishes them yet, so the release artifacts are unchanged for now, but the sources and the build scripts are in the tree.

There's more in this release than fits here. The release notes carry the per-PR detail. Our roadmap is published on GitHub, and we'd like to hear what you're building and what we should work on next, either on GitHub or on Discord. If you want to support VillageSQL, please star the repo.

Read more