Preview Capabilities - VillageSQL Server for MySQL

Preview capabilities are server-provided features exposed to extensions before their APIs are finalized. An extension that declares a preview capability requires vsql_allow_preview_extensions = ON to install (see Enabling the Preview Tier) — extensions that don’t use preview capabilities install normally regardless of this setting.

Preview capability APIs are not stable. An extension built against a preview capability may fail to load after a server update. When a capability stabilizes, its header moves to a versioned stable SDK path.

Enabling the Preview Tier

Set vsql_allow_preview_extensions = ON with SET PERSIST before installing any extension that uses a preview capability:

SET PERSIST vsql_allow_preview_extensions = ON;

SET GLOBAL is rejected for this variable — the server requires SET PERSIST so the setting survives restart. Extensions with preview capabilities are loaded at startup, so the variable must be ON when the server starts. If you’re launching mysqld directly (for example, from an install script that starts the server for the first time), pass the flag on the command line instead — mysqld-auto.cnf won’t exist yet to carry the persisted value:

mysqld --vsql_allow_preview_extensions=ON

To disable:

SET PERSIST vsql_allow_preview_extensions = OFF;

This fails if any extension using a preview capability is currently installed. Uninstall those extensions first, then turn the setting off.

Capability Index

Capability	Header	Status
`vsql::preview::column_store`	`<villagesql/preview/storage_builder.h>`	Preview
`vsql::preview::keyring`	`<villagesql/preview/keyring.h>`	Preview
`vsql::preview::sql_query`	`<villagesql/preview/sql_query.h>`	Preview
`vsql::preview::status_var`	`<villagesql/preview/status_var.h>`	Preview
`vsql::preview::storage`	`<villagesql/preview/storage_builder.h>`	Preview
`vsql::preview::sys_var`	`<villagesql/preview/sys_var.h>`	Preview
`vsql::preview::thread_worker`	`<villagesql/preview/thread_worker.h>`	Preview

Registration Pattern

To use a preview capability, declare a capability object by value at file scope and pass it by reference to .with() inside make_extension(). The server populates the object’s abi pointer during registration:

#include <villagesql/preview/keyring.h>
#include <villagesql/vsql.h>

using KeyringCapability = vsql::preview_keyring::KeyringCapability;

static KeyringCapability g_keyring;

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .func(/* ... */)
        .with(g_keyring))

.with(capability) tells the server which capabilities the extension requires. If vsql_allow_preview_extensions is OFF when the extension is installed, the server rejects the install with an error naming the capability.

Every capability object declared in an extension must be passed to .with() exactly once. At load time, the server cross-checks every declared capability instance against what .with() received and fails INSTALL EXTENSION if the rule is violated:

Declared but never passed to .with(): capability '<Type>' was declared but never passed to .with(); every CapabilityBase-derived static must be registered via .with(cap) in the extension builder
Same instance passed to .with() more than once: capability '<Type>' passed to .with() more than once
Object passed to .with() is not a capability: .with() received an object that does not inherit vsql::detail::CapabilityBase; not a registered capability

The full error surfaces as: Failed to load VEF extension '<name>': vef_register returned an error: <message above>.

Keyring Access

The keyring capability (vsql::preview::keyring) lets extensions read and write secrets stored in the MySQL keyring component. Extensions use it for things like API keys, encryption keys, or other secrets that shouldn’t live in SQL tables. The capability name VEF_PREVIEW_KEYRING_NAME is "vsql::preview::keyring". A keyring component must be installed on the MySQL server for reads and writes to succeed. Without one, operations return KeyringCapability::Status::UNAVAILABLE.

Status Values

KeyringCapability::Status is a scoped enum returned by read() (inside ReadResult) and write():

Status	Meaning
`Status::OK`	Operation succeeded.
`Status::NOT_FOUND`	The key does not exist (read only).
`Status::UNAVAILABLE`	No keyring component is installed.
`Status::ERROR`	Other failure.

Declaring the Capability

Include the header, declare a capability object at file scope, and pass it to .with():

#include <villagesql/preview/keyring.h>
#include <villagesql/vsql.h>

using KeyringCapability = vsql::preview_keyring::KeyringCapability;

static KeyringCapability g_keyring;

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(g_keyring))

The g_keyring object is populated by the server at load time. The read() and write() methods return Status::UNAVAILABLE at runtime when no keyring component is installed — check that status on each call rather than gating on a separate availability probe.

Reading and Writing

struct KeyringCapability::ReadResult {
  KeyringCapability::Status status;
  std::string value;
};

[[nodiscard]] KeyringCapability::ReadResult
KeyringCapability::read(std::string_view data_id,
                        std::string_view auth_id = {}) const;

[[nodiscard]] KeyringCapability::Status
KeyringCapability::write(std::string_view data_id,
                         std::string_view auth_id,
                         std::string_view data) const;

data_id is the key identifier. auth_id is the owning user — pass an empty string (or omit it on read, which defaults to {}) to read or write internal keys not associated with a specific user. read returns a ReadResult by value. Bind it with structured bindings:

auto [status, value] = g_keyring.read("my_secret");
if (status == KeyringCapability::Status::OK) {
  // value contains the secret bytes
}

On any status other than Status::OK, value is empty. write returns Status directly and stores data under data_id / auth_id.

Complete Example

This is a simplified version of the vsql_keyring_reader test extension that ships with the server. It registers 2 VDFs: keyring_read and keyring_store.

#include <villagesql/preview/keyring.h>
#include <villagesql/vsql.h>

using namespace vsql;
using KeyringCapability = vsql::preview_keyring::KeyringCapability;

static KeyringCapability g_keyring;

void keyring_read(StringArg data_id, StringArg auth_id, StringResult out) {
  if (data_id.is_null()) { out.set_null(); return; }

  const auto [status, value] =
      g_keyring.read(data_id.value(), auth_id.is_null() ? "" : auth_id.value());
  if (status == KeyringCapability::Status::UNAVAILABLE) {
    out.error("No keyring component is installed");
    return;
  }
  if (status != KeyringCapability::Status::OK) { out.set_null(); return; }

  auto buf = out.buffer();
  size_t len = std::min(value.size(), buf.size());
  memcpy(buf.data(), value.data(), len);
  out.set_length(len);
}

void keyring_store(StringArg data_id, StringArg auth_id, StringArg value,
                   IntResult out) {
  if (data_id.is_null() || value.is_null()) { out.set(1); return; }

  KeyringCapability::Status status = g_keyring.write(
      data_id.value(), auth_id.is_null() ? "" : auth_id.value(), value.value());
  if (status == KeyringCapability::Status::UNAVAILABLE) {
    out.error("No keyring component is installed");
    return;
  }
  out.set(status == KeyringCapability::Status::OK ? 0 : 1);
}

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .func(make_func<&keyring_read>("keyring_read")
                  .returns(STRING).param(STRING).param(STRING).build())
        .func(make_func<&keyring_store>("keyring_store")
                  .returns(INT).param(STRING).param(STRING).param(STRING).build())
        .with(g_keyring))

Status Variables

The status_var capability (vsql::preview::status_var) lets an extension expose long long and double counters as MySQL status variables. The extension owns the storage and writes to it; the server reads through the pointers each time the status variable is queried. Build the capability with vsql::preview_status_var::make_capability(), passing a braced list of descriptors from make_int(name, value_ptr) or make_double(name, value_ptr). The template deduces the count from the braced list, so no explicit size is required.

Complete Example

#include <villagesql/preview/status_var.h>
#include <villagesql/vsql.h>

namespace sv = vsql::preview_status_var;

static long long g_hits   = 0;
static long long g_misses = 0;

static auto STATUS_VARS = sv::make_capability({
    sv::make_int("ext_hits",   &g_hits),
    sv::make_int("ext_misses", &g_misses)});

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(STATUS_VARS))

make_int requires a long long *; make_double requires a double *. Those are the only two types supported.

Accessing from SQL

After INSTALL EXTENSION my_ext, the variable is visible with the extension name as a prefix:

SHOW GLOBAL STATUS LIKE 'my_ext%';

Variable_name       Value
my_ext.ext_hits     0
my_ext.ext_misses   0

Concurrent increments from multiple query threads using a non-atomic ++ may occasionally be lost; this is acceptable for approximate call counters exposed via SHOW STATUS.

System Variables

The sys_var capability (vsql::preview::sys_var) lets an extension register MySQL system variables backed by extension-owned storage. Three types are supported: BOOL (bool *), INT (long long *), and STR (char **). INT descriptors also carry min_val and max_val bounds; all descriptors carry a default value and a comment. Build the capability with vsql::preview_sys_var::make_capability() and the matching factory functions make_bool, make_int, and make_str. The capability object also exposes get() and set() for programmatic access from extension code. Both return false on success. To react to value changes, chain .on_change<&fn>() on a descriptor. The callback receives a sv::SysVarChange with var_name() and typed accessors (as_int(), as_real(), as_str()). The capability object must have static storage duration. MySQL writes directly to the storage pointers when the user sets a variable.

Factory	Storage type	Extra parameters
`sv::make_bool`	`bool *`	`def_val`
`sv::make_int`	`long long *`	`def_val`, `min_val`, `max_val`
`sv::make_str`	`char **`	`def_val`

Complete Example

#include <villagesql/preview/sys_var.h>
#include <villagesql/vsql.h>

namespace sv = vsql::preview_sys_var;

static bool      g_enabled   = true;
static long long g_threshold = 1000;
static char     *g_log_file  = nullptr;

static void on_threshold_change(sv::SysVarChange c) {
  // c.var_name() identifies the variable; c.as_int() returns the new value
}

static auto SYS_VARS = sv::make_capability({
    sv::make_bool("enabled",      "Enable feature",  &g_enabled,   true),
    sv::make_int ("threshold_ms", "Threshold in ms", &g_threshold, 1000, 0, 3600000)
        .on_change<&on_threshold_change>(),
    sv::make_str ("log_file",     "Log file path",   &g_log_file,  "/tmp/myext.log")});

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(SYS_VARS))

Accessing from SQL

After INSTALL EXTENSION my_ext, variables are accessible using the extension name as a component prefix:

SELECT @@global.my_ext.threshold_ms;
SET GLOBAL my_ext.threshold_ms = 500;
SET GLOBAL my_ext.log_file = '/var/log/myext.log';

Reading and Writing from Extension Code

For INT and BOOL variables, read the global storage pointer directly — MySQL updates those atomically. To update a variable through MySQL (so locking, range validation, and persistence are handled by the server), call SYS_VARS.set(extension_name, var_name, scope, value). Both set and get return false on success.

bool err = SYS_VARS.set("my_ext", "threshold_ms", nullptr, value);

The scope argument controls persistence:

Scope	Behavior
`nullptr`	Update running value only, not persisted.
`"PERSIST"`	Update running value and write to `mysqld-auto.cnf`.
`"PERSIST_ONLY"`	Write to `mysqld-auto.cnf` only; takes effect on next restart.

Thread Worker

The thread worker capability (vsql::preview::thread_worker) lets an extension run a background thread driven by the server. The thread is started and stopped via a control system variable that the server registers at extension load; the server invokes the extension’s work function on a periodic timer, on file-descriptor readiness, or in response to enable/disable events. The capability name VEF_PREVIEW_THREAD_WORKER_NAME is "vsql::preview::thread_worker".

Declaring the Capability

Include the header, declare a ThreadWorkerCapability instantiated on your work function at file scope, and pass it to .with():

#include <villagesql/preview/thread_worker.h>
#include <villagesql/vsql.h>

static vef_next_wakeup_t my_work(vef_wakeup_reason_t reason,
                                 struct vef_thread_handle_t *thread,
                                 void *arg) {
  // ...
  return {};
}

static vsql::preview_thread_worker::ThreadWorkerCapability<&my_work>
    g_worker{"suffix"};

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(g_worker))

The work function is supplied as a non-type template argument (ThreadWorkerCapability<&my_work>), so it must be a function with the signature shown below. The first constructor argument is the thread-name suffix; the optional second argument overrides the control sys var name.

Work Function Signature

typedef vef_next_wakeup_t (*vef_work_fn_t)(vef_wakeup_reason_t reason,
                                           struct vef_thread_handle_t *thread,
                                           void *arg);

reason indicates why the server called the function. thread is the server-owned handle for this worker (NULL on the initial VEF_WAKEUP_ENABLE call — see below). arg is the opaque pointer registered on the descriptor; it is passed through unchanged.

Wakeup Lifecycle

The server calls the work function with one of four reasons:

Reason	Meaning
`VEF_WAKEUP_ENABLE`	Worker was just enabled (control sys var flipped ON). The return value sets the initial `poll_fd` and `sleep_ms`.
`VEF_WAKEUP_PERIODIC`	Periodic timer fired (`sleep_ms` elapsed).
`VEF_WAKEUP_POLL_FD`	A watched file descriptor became readable.
`VEF_WAKEUP_DISABLE`	Worker disabled (control sys var OFF) or server shutting down. The return value is ignored.

The thread parameter is NULL when the reason is VEF_WAKEUP_ENABLE, because the thread handle does not exist yet at that point. For the other three reasons, thread is non-null.

Wakeup Return Value

typedef struct {
  unsigned int sleep_ms;
  int poll_fd;
} vef_next_wakeup_t;

The work function returns a vef_next_wakeup_t to update the next wakeup configuration. A zero value in either field means “keep the current setting” — return a value-initialized struct (return {};) to leave both unchanged. To set a new poll file descriptor, return its value (must be greater than zero). To clear an existing poll file descriptor, return -1 in poll_fd. The return value is ignored when the reason is VEF_WAKEUP_DISABLE.

Thread Name and Control Variable

Two fields on the descriptor control naming:

suffix — the thread-name suffix. The server prepends the extension name, producing thread names like my_ext/monitor.
var_name — optional. When non-null, the server registers this exact name as the control system variable. When null, the server uses the default pattern {suffix}_enabled.

The control variable is a server-registered system variable. Enable the worker with SET GLOBAL {suffix}_enabled = ON; set it OFF to stop it.

Complete Example

A minimal extension with a single periodic worker that increments a heartbeat counter on each timer tick.

#include <villagesql/preview/thread_worker.h>
#include <villagesql/vsql.h>

#include <atomic>

static std::atomic<unsigned long long> g_heartbeat{0};

static vef_next_wakeup_t heartbeat_work(vef_wakeup_reason_t reason,
                                        struct vef_thread_handle_t *thread,
                                        void *arg) {
  switch (reason) {
    case VEF_WAKEUP_ENABLE:
      return {1000, 0};  // tick every 1000 ms, no poll fd
    case VEF_WAKEUP_PERIODIC:
      g_heartbeat.fetch_add(1, std::memory_order_relaxed);
      return {};  // keep current sleep_ms and poll_fd
    case VEF_WAKEUP_POLL_FD:
      return {};  // not used in this example
    case VEF_WAKEUP_DISABLE:
      return {};  // ignored
  }
  return {};
}

static vsql::preview_thread_worker::ThreadWorkerCapability<&heartbeat_work>
    g_worker{"heartbeat"};

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(g_worker))

With this extension installed (and vsql_allow_preview_extensions = ON), the server registers a heartbeat_enabled system variable. Enable the worker with:

SET GLOBAL heartbeat_enabled = ON;

SQL Query

The sql_query capability (vsql::preview::sql_query) lets an extension execute SQL statements from a background thread. Queries run inside the server through the capability vtable — extensions do not link against any MySQL client library. The capability name VEF_PREVIEW_SQL_QUERY_NAME is "vsql::preview::sql_query".

A SQL session must be opened from a thread-worker callback using that callback’s vef_thread_handle_t *. open() is not valid from VDFs or from arbitrary extension-created threads — it requires the worker session context.

Declaring the Capability

Include the header, declare a SqlQueryCapability at file scope, and pass it to .with(). It is typically registered alongside a ThreadWorkerCapability, since sessions are opened from the worker callback:

#include <villagesql/preview/sql_query.h>
#include <villagesql/preview/thread_worker.h>
#include <villagesql/vsql.h>

static vsql::preview_sql_query::SqlQueryCapability g_sql;

static vef_next_wakeup_t my_work(vef_wakeup_reason_t reason,
                                 struct vef_thread_handle_t *thread,
                                 void *arg) {
  auto session = g_sql.open(thread);
  if (!session) return {};
  // ...
  return {};
}

static vsql::preview_thread_worker::ThreadWorkerCapability<&my_work>
    g_worker{"sql_demo"};

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(g_worker)
        .with(g_sql))

g_sql.open(handle) returns a Session. Check it with operator bool before use; an invalid Session indicates the capability vtable was not bound or the server could not allocate a session. The Session is move-only and closes itself on destruction.

Executing Queries

A Session produces a SqlQuery via session.sql(sv). The query can be run in two modes:

execute() — runs the statement and buffers the full result set in a Result. Iterate rows by calling next() at the caller’s pace.
for_each(fn) — runs the statement and invokes fn once per row as rows are produced, without buffering. The returned Result carries diagnostics only (no rows).

Both return a Result. A non-null Result does not mean the statement succeeded — call has_error() to find out. Buffered (execute):

auto result = session.sql("SELECT id, name FROM t").execute();
if (result.has_error()) {
  // result.error().message holds the server error string.
  return {};
}
while (result.next()) {
  long long id          = result.column_int(0);
  std::string_view name = result.column_str(1);
  // ...
}

column_str() returns a string_view that is valid only until the next next() call or until Result is destroyed. Copy it if a longer lifetime is needed. A string_view with data() == nullptr indicates SQL NULL. Streaming (for_each):

auto status = session.sql("SELECT 1").for_each(
    [](const auto &row) {
      // row.column_int(0), row.column_str(1), etc.
    });
if (status.has_error()) {
  // status.error().message
}

The Row passed to the callback is valid only for the duration of the call — do not store references to it across rows. The Result returned by for_each holds no buffered rows; next() on it will not yield data. Use it only for has_error(), error(), warning_count(), and warning(i).

Diagnostics

Both execute() and for_each() surface diagnostics through the returned Result. A diagnostic is one Diag:

struct Diag {
  uint32_t errno_;
  vef_sql_diag_severity_t severity;   // NOTE | WARNING | ERROR
  std::string_view sqlstate;          // 5-char SQLSTATE
  std::string_view message;           // may be empty
};

Field	Meaning
`errno_`	MySQL error number. `0` on a default-constructed `Diag` returned when there is no error.
`severity`	`VEF_SQL_DIAG_NOTE`, `VEF_SQL_DIAG_WARNING`, or `VEF_SQL_DIAG_ERROR`.
`sqlstate`	5-character SQLSTATE.
`message`	Server-supplied diagnostic message; may be empty.

Result exposes:

bool         Result::has_error() const;
Diag         Result::error() const;
unsigned int Result::warning_count() const;
Diag         Result::warning(unsigned int i) const;

error() returns a default-constructed Diag (errno_ == 0) when the statement succeeded. warning(i) returns a default-constructed Diag when i >= warning_count(). The sqlstate and message views point into storage owned by the Result and become invalid when the Result is destroyed — copy them if they need to outlive it.

Complete Example

A worker that runs one buffered query and one streaming query on each tick, logging diagnostics from both:

#include <villagesql/preview/sql_query.h>
#include <villagesql/preview/thread_worker.h>
#include <villagesql/vsql.h>

static vsql::preview_sql_query::SqlQueryCapability g_sql;

static vef_next_wakeup_t sql_demo_work(vef_wakeup_reason_t reason,
                                       struct vef_thread_handle_t *thread,
                                       void *arg) {
  if (reason == VEF_WAKEUP_ENABLE) return {5000, 0};
  if (reason != VEF_WAKEUP_PERIODIC) return {};

  auto session = g_sql.open(thread);
  if (!session) return {};

  // Buffered: read a small result set.
  auto rs = session.sql("SELECT id, name FROM mydb.t LIMIT 10").execute();
  if (rs.has_error()) {
    auto e = rs.error();
    // Log e.errno_, e.sqlstate, e.message somewhere extension-owned.
  } else {
    while (rs.next()) {
      long long id          = rs.column_int(0);
      std::string_view name = rs.column_str(1);
      (void)id; (void)name;
    }
  }

  // Streaming: process rows without buffering.
  auto status = session.sql("SELECT v FROM mydb.t").for_each(
      [](const auto &row) {
        long long v = row.column_int(0);
        (void)v;
      });
  for (unsigned i = 0; i < status.warning_count(); ++i) {
    auto w = status.warning(i);
    (void)w;
  }
  return {};
}

static vsql::preview_thread_worker::ThreadWorkerCapability<&sql_demo_work>
    g_worker{"sql_demo"};

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(g_worker)
        .with(g_sql))

Column Storage

Column storage lets an extension register a custom binary on-disk layout directly with InnoDB for one of its custom types, instead of routing the type’s bytes through the row’s VARBINARY payload. Use it when your type needs an on-disk shape VARBINARY cannot express — for example, a packed array of floats that must live in dedicated pages. This is a capability feature: it enables new storage layouts, not a tuning knob for existing ones.

Column storage is a preview ABI — under active development and may change between releases. It currently covers row-level persistence only; indexing over custom-stored columns is not yet available.

Declaring the Capabilities

Two preview capabilities work together:

vsql::preview::storage — opens access to InnoDB storage infrastructure (mini-transactions, segments, pages). Declare a StorageCapability at file scope.
vsql::preview::column_store — binds a per-type storage implementation to one of the extension’s custom types. Declare a ColumnStoreCapability at file scope using make_column_store<Ctx>(TYPE).…build().

Both must be passed to .with() on make_extension():

#include <villagesql/preview/storage_builder.h>
#include <villagesql/preview/storage_api.h>
#include <villagesql/vsql.h>

namespace storage = vsql::preview_storage;
using vsql::preview_storage_builder::ColumnStoreCapability;
using vsql::preview_storage_builder::make_column_store;
using vsql::preview_storage_builder::StorageCapability;

struct MyCtx {
  storage::Space::Ref space = 0;
  storage::Segment::PageRef root_page = storage::Page::INVALID_REF;
};

static auto STORAGE = StorageCapability{};

static constexpr auto kMyStorage =
    make_column_store<MyCtx>(MY_TYPE)
        .create<&MyStorage::create>()
        .drop<&MyStorage::drop>()
        .load<&MyStorage::load>()
        .insert<&MyStorage::insert>()
        .select<&MyStorage::select>()
        .mark_delete<&MyStorage::mark_delete>()
        .purge<&MyStorage::purge>()
        .build();

static auto COLUMN_STORE = ColumnStoreCapability().column_store(kMyStorage);

using namespace vsql;

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .with(STORAGE)
        .with(COLUMN_STORE)
        .type(MY_TYPE))

make_column_store<MyCtx>(MY_TYPE) ties the implementation to one custom type registered on the same extension. All seven slots are required at build() time because each maps to a distinct point in the column lifecycle that InnoDB will reach during normal operation.

The Seven Storage Functions

Every function takes storage::Column::StorageCtx<MyCtx>*, whose user() accessor returns the extension’s per-column state and whose arena() provides server-managed allocation for auxiliary objects. Every function returns false on success and true on error, writing a message into error_msg (capacity error_msg_len) so failures surface to the SQL client.

// CREATE TABLE / ALTER TABLE ADD COLUMN.
// col_len is the type's persisted length. Reserve segments here and store
// space + root_page in ctx->user() so DML functions can reach them.
bool create(storage::Column::StorageCtx<MyCtx>*, storage::Space::Ref,
            storage::Segment::TrxRef, uint32_t col_len,
            char* error_msg, uint32_t error_msg_len);

// DROP TABLE / ALTER TABLE DROP COLUMN.
// Release any segments reserved in create(). Arena memory is freed by the
// server after this call returns.
bool drop(storage::Column::StorageCtx<MyCtx>*, storage::Segment::TrxRef,
          char* error_msg, uint32_t error_msg_len);

// Called when the server reattaches to existing storage (e.g. after restart).
// Recover space and root_page from the StorageRef set in create().
bool load(storage::Column::StorageCtx<MyCtx>*, storage::Column::StorageRef,
          char* error_msg, uint32_t error_msg_len);

// INSERT. col_data is the encoded value; rowid_prefix identifies the owning
// row. Write into your storage layout and return a Column::Ref the server
// stores in the row payload in place of the value bytes.
bool insert(storage::Column::StorageCtx<MyCtx>*, storage::MtrCtx::Ref,
            storage::Segment::TrxRef, storage::Column::Data col_data,
            storage::Column::Data rowid_prefix, storage::Column::Ref* col_ref,
            char* error_msg, uint32_t error_msg_len);

// SELECT. Given the Column::Ref produced by insert, populate col_data and
// rowid_prefix, and report the writing transaction and delete-mark status.
bool select(storage::Column::StorageCtx<MyCtx>*, storage::MtrCtx::Ref,
            storage::Column::Ref, storage::Column::Data* col_data,
            storage::Column::Data* rowid_prefix, storage::Segment::TrxRef*,
            bool* delete_marked, char* error_msg, uint32_t error_msg_len);

// DELETE (in-transaction). Set or clear the delete-mark flag. The actual
// bytes must remain readable until purge() runs.
bool mark_delete(storage::Column::StorageCtx<MyCtx>*, storage::MtrCtx::Ref,
                 storage::Segment::TrxRef, storage::Column::Ref,
                 bool delete_mark, char* error_msg, uint32_t error_msg_len);

// InnoDB purge. Reclaim storage for entries whose deleting transaction is
// no longer visible to any active snapshot.
bool purge(storage::Column::StorageCtx<MyCtx>*, storage::MtrCtx::Ref,
           storage::Segment::TrxRef, storage::Column::Ref,
           char* error_msg, uint32_t error_msg_len);

mark_delete and purge are distinct because InnoDB MVCC requires deleted rows to remain readable by older snapshots until purge runs.

Per-Column Context and the Arena

The SDK default-constructs MyCtx before calling either create or load — ctx->user() is already populated when your function is entered. MyCtx must be default-constructible; the SDK calls T() with no arguments. Use ctx->user() directly to initialize state. Do not call ctx->arena().construct<MyCtx>() — that allocates a second, unused instance and ctx->user() does not point to it.

bool MyStorage::create(storage::Column::StorageCtx<MyCtx>* ctx,
                       storage::Space::Ref space, storage::Segment::TrxRef trx,
                       uint32_t col_len,
                       char* error_msg, uint32_t error_msg_len) {
  storage::Segment::PageRef root;
  if (storage::Segment::create(space, 1, trx, root) != storage::Error::SUCCESS) {
    snprintf(error_msg, error_msg_len, "%s", storage::last_error().data());
    return true;
  }

  ctx->user()->space = space;
  ctx->user()->root_page = root;
  // Encode space and root into StorageRef so load() can recover both.
  ctx->set_ref((static_cast<storage::Column::StorageRef>(space) << 32) |
               static_cast<storage::Column::StorageRef>(root));
  return false;
}

load follows the same pattern — ctx->user() is pre-populated and storage_ref carries the packed value stored by ctx->set_ref() in create:

bool MyStorage::load(storage::Column::StorageCtx<MyCtx>* ctx,
                     storage::Column::StorageRef storage_ref,
                     char* error_msg, uint32_t error_msg_len) {
  ctx->user()->space =
      static_cast<storage::Space::Ref>(storage_ref >> 32);
  ctx->user()->root_page =
      static_cast<storage::Segment::PageRef>(storage_ref & 0xFFFFFFFF);
  ctx->set_ref(storage_ref);
  return false;
}

Use ctx->arena() only to allocate auxiliary objects that are too large or dynamic to embed directly in MyCtx. The SDK destroys the arena — and calls ~MyCtx() — automatically after drop returns, regardless of whether drop succeeds.

InnoDB Access Utilities

Include <villagesql/preview/storage_api.h> for the InnoDB primitives. All page reads and writes must occur inside a mini-transaction:

storage::MtrCtx mtr;
storage::MtrCtx::Ref mtr_ref = mtr.start();
if (mtr_ref == nullptr) { /* OOM — handle error */ return true; }
// ... page operations ...
mtr.commit();

Committing the mini-transaction releases page latches and writes the redo log records that make changes durable. Segments are reserved at create time — see the create and load examples in Per-Column Context above for the complete setup pattern. During DML operations, get a segment reference from the root page to allocate new pages:

storage::Page root;
root.load(ctx->user()->space, ctx->user()->root_page,
          storage::Page::Latch::EXCLUSIVE, mtr_ref);
storage::Segment::Ref seg = storage::Segment::get_header(root, 0);
storage::Page data_page;
data_page.load_new(seg, mtr_ref);  // allocates a fresh page

Pages are read with a shared latch and written with an exclusive one. Pass mtr_ref to write calls so InnoDB logs the change:

storage::Page page;

// Read
page.load(ctx->user()->space, page_num, storage::Page::Latch::SHARED, mtr_ref);
uint32_t v = page.read_integer_4(storage::Page::HEADER_SIZE + offset);

// Write
page.load(ctx->user()->space, page_num, storage::Page::Latch::EXCLUSIVE, mtr_ref);
page.write_integer_4(storage::Page::HEADER_SIZE + offset, v, mtr_ref);

Page layout constants:

Constant	Value	Notes
`storage::Page::HEADER_SIZE`	`38`	Extension data begins at this offset.
`storage::Page::TRAILER_SIZE`	`8`	Do not write past `page_size - TRAILER_SIZE`.
`storage::Page::get_size(space)`	runtime	Use instead of hard-coding 16384.

Reading or writing inside the header or trailer regions corrupts the page — InnoDB uses those byte ranges for its own bookkeeping and checksum.

​Enabling the Preview Tier

​Capability Index

​Registration Pattern

​Keyring Access

​Status Values

​Declaring the Capability

​Reading and Writing

​Complete Example

​Status Variables

​Complete Example

​Accessing from SQL

​System Variables

​Complete Example

​Accessing from SQL

​Reading and Writing from Extension Code

​Thread Worker

​Declaring the Capability

​Work Function Signature

​Wakeup Lifecycle

​Wakeup Return Value

​Thread Name and Control Variable

​Complete Example

​SQL Query

​Declaring the Capability

​Executing Queries

​Diagnostics

​Complete Example

​Column Storage

​Declaring the Capabilities

​The Seven Storage Functions

​Per-Column Context and the Arena

​InnoDB Access Utilities

Enabling the Preview Tier

Capability Index

Registration Pattern

Keyring Access

Status Values

Declaring the Capability

Reading and Writing

Complete Example

Status Variables

Complete Example

Accessing from SQL

System Variables

Complete Example

Accessing from SQL

Reading and Writing from Extension Code

Thread Worker

Declaring the Capability

Work Function Signature

Wakeup Lifecycle

Wakeup Return Value

Thread Name and Control Variable

Complete Example

SQL Query

Declaring the Capability

Executing Queries

Diagnostics

Complete Example

Column Storage

Declaring the Capabilities

The Seven Storage Functions

Per-Column Context and the Arena

InnoDB Access Utilities