VillageSQL

Announcing VillageSQL Server 0.0.4

Adam Levin

Adam Levin

— 6 min read
Share
Announcing VillageSQL Server 0.0.4

VillageSQL Server 0.0.4 is now available (and marked as the stable release for installs). 0.0.4 takes the parameterized types and SQL integration from 0.0.3 and extends them in two directions:

  • smarter server reasoning (inference rules, custom aggregates)
  • operational completeness (config, metrics, secrets management)

Extensions stop being things you install and hope work; they become things you can configure, monitor, and manage like any other part of the server.

Area What's new Why it matters
VEF v3 New vsql namespace, compile-time validation, custom aggregate functions Catch signature errors at build time; define aggregation logic for custom types
Type system Parameter inference, variable-length types without parameters Server enforces type constraints; extensions write less validation code
Runtime operations Extension registry, preview capabilities Configure, observe, and manage extensions like the rest of MySQL

VEF v3 and aggregate functions

The VEF C++ SDK moves to the vsql namespace, bringing vsql::make_extension(), vsql::make_func<&impl>(), and vsql::make_aggregate_func<>() under one top-level home. Internal wiring like type wrapping, string handling, and function table registration is handled by the SDK rather than spread across extension code.

Extension authors get compile-time constraint validation. Wrong argument types and unsatisfied builder requirements surface during a build or at install time. That matters in practice — a bad function signature caught during development is a five-second fix; the same error caught in production is a bug report.

A note on SDK versioning: "v2" has been renumbered as v3. Going forward, odd-numbered SDKs are stable; even-numbered are development-only. The v1 SDK is still supported. The server rejects v2, so extensions built against v2 need to be rebuilt against v3.

0.0.3 made existing aggregates like GROUP_CONCAT work on custom types. 0.0.4 goes further: extensions can now define their own aggregation logic. The vsql::make_aggregate_func<>() builder uses the same method-chaining interface as scalar functions:

vsql::make_aggregate_func<VectorSumState, &avg_vector_result>("avg_vector")
    .returns("my_vector")
    .param("my_vector")
    .clear<&avg_vector_clear>()
    .accumulate<&avg_vector_accumulate>()
    .build()

Any GROUP BY aggregate for a custom type follows the same pattern. An extension that defines a vector type can now also define how the server accumulates those values across rows — custom statistics, vector aggregation, domain-specific folding — without any special casing in server code.

Type system improvements

Custom types can include values in their declaration — a dimension, a precision, a scale — that become part of the type itself. That moves constraint enforcement into the type contract, where the server can catch violations rather than leaving it to extension code or the user. VECTOR(1536) and VECTOR(3) are handled as different types; the dimension is part of what the type is, not a limit applied on top of a generic container.

Two rules landed in 0.0.4. First, when multiple VDF arguments share the same abstract type, the server enforces at query time that all carry the same concrete type — mixing TVECTOR(3) with TVECTOR(1536) is an error, not a silent wrong result. Second, when a VDF's return type is the same abstract type as an argument, the server infers the return parameters from the input — an add_vectors function can declare VECTOR(N) as its return type and have N resolved automatically, without callers re-specifying the dimension. Extension authors write less validation code; users get errors at query time rather than wrong results that propagate silently.

0.0.4 also extends variable-length support to types whose size varies row by row, without needing a parameter. A roaring bitmap, a sparse vector, a compressed blob — extensions can now define types where each value has its own length, and the server tracks that length per row rather than fixing it at the type level.

Runtime operations

A new system table lists every installed extension with its name, negotiated VEF protocol version, and JSON metadata:

SELECT extension_name, negotiated_protocol, registration_json
FROM INFORMATION_SCHEMA.EXTENSION_REGISTRATION;

EXTENSION_REGISTRATION gives you a live inventory of your extension runtime from a standard SQL query — what's loaded, at what protocol version, and what capabilities each extension negotiated at startup. You can audit your extension environment, spot protocol version mismatches before they become problems, and confirm that capabilities registered as expected, all without leaving the MySQL client.

Preview capabilities

Preview features are those that are available for early use and feedback but are not yet part of the stable SDK surface. We want extension authors to be able to try these features, but we have not committed to keeping them stable. Extensions that use any preview capability will not load unless the server has preview extensions enabled.

Six capabilities are now available in preview: column storage, config variables, background threads, status variables, keyring, and SQL query. MySQL already exposes the middle three as Component Services, but now VEF extensions can access that same infrastructure.

  • Column storage — extensions can define their own binary on-disk layout in InnoDB rather than routing data through VARBINARY. For types like dense vectors that have a natural packed representation, that's both a correctness story and a storage efficiency one.
  • Config variables — extensions can register named configuration variables that appear in SHOW VARIABLES and accept SET GLOBAL, with on-change callbacks to react immediately when a value changes.
  • Background threads — extensions can register OS-level worker threads; the server manages their lifecycle and gives them database access for scheduled work like cache warming or metrics aggregation.
  • Status variables — extensions can expose read-only runtime metrics through SHOW STATUS, readable by any MySQL-compatible monitoring tool with no extra configuration.
  • Keyring access — extensions can store and retrieve secrets through MySQL's keyring subsystem, keeping credentials out of SHOW VARIABLES and integrated with whatever keyring backend the server uses.
  • SQL queries — extensions can execute SQL queries against the running server from within their callbacks, with results returned row by row and proper error propagation.

Installing 0.0.4

0.0.4 tracks Oracle MySQL 8.4.9, incorporating all upstream fixes from that branch.

Run the install script at villagesql.com/shell.

Upgrading from 0.0.3: Databases without extensions installed upgrade automatically on first start — just point the new binary at your existing datadir and it picks up where 0.0.3 left off.

If you have extensions installed, uninstall them on the 0.0.3 server before upgrading, then reinstall on 0.0.4:

-- On 0.0.3, before upgrading:
UNINSTALL EXTENSION vsql_ai VERSION '0.0.3';
-- repeat for each installed extension

-- After upgrading to 0.0.4:
INSTALL EXTENSION vsql_ai;

0.0.4 adds UNINSTALL EXTENSION name VERSION 'x.y.z' syntax to target a specific installed version when more than one is registered.

The server records extension VEB hashes at install time. When the VEB changes between versions, the old hash won't resolve and the server won't start. Uninstalling before the upgrade clears the record; reinstalling afterwards picks up the new VEB. Bundled extensions ship already rebuilt against SDK v3 and any third-party extension built against v2 will be rejected on load and need to be rebuilt against v3.

What's ahead

Two large efforts are well underway and are the focus of what comes next.

The first is a custom index API. Extensions will be able to register their own index types, backed by custom storage structures, with dedicated CREATE INDEX syntax that routes to the extension's implementation. For a vector extension, that means a native approximate nearest neighbor index rather than a full-table scan on every similarity query. It also opens up domain-specific index structures that a B-tree can't express.

The second is out-of-process extensions. The background threads in this release run inside the server process — they're fast, but a badly-behaved extension thread can affect server stability. Out-of-process extensions run in a completely separate process. If the extension crashes, the server keeps running. It also removes the requirement that extensions be compiled C++ loaded into the server's address space, which opens the door to extensions in other languages.

Special thanks

VillageSQL would like to thank the external contributors pyshx, anand-trehan, and renecannao for their contributions and feedback in this release.

These are some of the highlights of the 0.0.4 release and a summary of what we're working on next. There is more in this release. Check the Release Notes for additional details. Please let us know what extensions you're building and what we should focus on building next. Our entire roadmap is published on GitHub. If you want to get involved in the community, we appreciate code contributions. You can discuss the roadmap with us on GitHub or on Discord.

Read more