C++ Development

This guide is the deep reference for writing C++ VDF implementations. It is the companion to Creating Extensions in C++, which covers the end-to-end build steps, and C++ Testing, which covers the test-and-iterate loop.

VEF Protocol 3 is stable as of v0.0.4. Protocol 4 is under development and is available only via opt-in dev ABI headers (-DVSQL_USE_DEV_ABI=ON). Extensions built against the old Protocol 2 are rejected by the server and must be rebuilt.

Writing Extension Functions

Extension functions are written in C++ and registered with VEF. Include a single header to access the full SDK:

#include <villagesql/vsql.h>

Exception Safety

Containing exceptions is the extension’s responsibility. The SDK does not wrap your implementation in a try/catch, and neither does the server, so an exception that escapes an entry point unwinds across the C ABI boundary with no handler anywhere above it and the server process terminates. Ordinary SQL input reaches this: std::stoi() throws on a non-numeric string, and nlohmann::json::dump() throws on text that is not valid UTF-8. Wrap the body of every entry point you register: VDFs, aggregate result functions, and the from_string/to_string operations of a custom type.

#include <villagesql/vsql.h>
#include <string>

using namespace vsql;

void to_int_impl(StringArg in, IntResult out) {
  if (in.is_null()) { out.set_null(); return; }
  try {
    out.set(std::stoi(std::string(in.value())));  // throws on bad input
  } catch (const std::exception &e) {
    out.warning(e.what());
  } catch (...) {
    out.warning("unexpected error");
  }
}

Catch ... in addition to const std::exception &: a dependency may throw a type that does not derive from std::exception, and that one still terminates the server. Choose between warning() and error() on the same basis as any other failure — warning() for bad input the statement should survive, error() where continuing is unsafe.

Argument and Result Types

VDF parameters and results are passed as type-safe argument and result types. The framework detects them from your function signature and adapts automatically — the make_func registration syntax is unchanged. Argument types: IntArg, RealArg, StringArg, CustomArg — each provides is_null() and value(). For parameterized custom types, CustomArgWith<P> adds a params() accessor that returns the cached parsed params struct (see Parameterized Types). Result types: IntResult, RealResult, StringResult, CustomResult — each provides set_null(), warning(msg), and error(msg). Scalar results also provide set(value). Buffer results provide buffer() and set_length(len). StringResult also provides set(std::string_view), which copies up to buffer().size() bytes from the view and sets the length in one call. For parameterized custom types, CustomResultWith<P> adds a params() accessor. Span type: value() and buffer() on the byte-oriented argument and result types return a vsql::Span<T> — a non-owning view over a contiguous run of T with data(), size(), empty(), begin()/end(), and operator[]. Under C++20 it is an alias for std::span<T>; under C++17 this SDK provides a minimal compatible implementation, so the same code compiles in either standard. It is available through <villagesql/vsql.h>. warning(msg) returns SQL NULL for the row and appends a SQL warning. That NULL is an ordinary value from the server’s point of view, so it only becomes a statement error the way any NULL does: writing it into a NOT NULL column fails under STRICT_TRANS_TABLES, the same as INSERT INTO t VALUES (NULL) would. Writing it into a nullable column succeeds, warning and all. Use warning(msg) for recoverable bad input, such as an unparseable string in an encode function. Use error(msg) for corrupt stored data or any condition where continuing is unsafe. The message for both is truncated to fit the server’s internal error buffer if necessary. Scalar example — add two integers:

using namespace vsql;

void add_impl(IntArg a, IntArg b, IntResult out) {
  if (a.is_null() || b.is_null()) { out.set_null(); return; }
  out.set(a.value() + b.value());
}

// Registration is unchanged:
make_func<&add_impl>("add").returns(INT).param(INT).param(INT).build();

Binary example — transform a custom type buffer in place:

using namespace vsql;

void rot13_impl(CustomArg in, CustomResult out) {
  if (in.is_null()) { out.set_null(); return; }
  auto src = in.value();   // vsql::Span<const unsigned char>
  auto dst = out.buffer(); // vsql::Span<unsigned char>
  for (size_t i = 0; i < src.size(); i++) { dst[i] = transform(src[i]); }
  out.set_length(src.size());
}

For StringResult and CustomResult, write into buffer(), then call set_length() with the number of bytes written. buffer().size() is the maximum capacity. For VDFs that return a custom type (returns(CUSTOM(MYTYPE))), the server sizes the result buffer to the resolved return type’s persisted_length automatically — extension authors do not need to declare .buffer_size(...) on the function builder for this case. If prerun grows the buffer further, that larger size is preserved. This is what lets, for example, SVECTOR::from_string('[…1024 floats…]') encode a wide vector without the result buffer running out of room. You can use different styles across functions in the same extension — each function’s style is determined by its own signature.

Aggregate VDFs

Aggregate VDFs accumulate state across rows within each GROUP BY group and return a single result per group, like SQL SUM or COUNT. Use make_aggregate_func<State, &result_fn>("name") to register one. The State type is the per-group accumulation buffer; prerun and postrun are auto-generated to allocate and delete it. The result function must have the signature void(const State&, ResultType) where ResultType is one of IntResult, RealResult, StringResult, CustomResult, or CustomResultWith<P>. Call out.set(value) to return a value or out.set_null() to return SQL NULL. Both .clear<>() and .accumulate<>() are required. The builder enforces this at compile time (via build()), and the server validates it again at INSTALL EXTENSION time — clear resets state, accumulate folds rows, and the result function reads the final state.

#include <villagesql/vsql.h>
#include <optional>

using namespace vsql;

// State type: nullopt means no non-NULL rows seen yet.
using SumState = std::optional<long long>;

void my_clear(SumState &s) { s = std::nullopt; }
void my_acc(SumState &s, IntArg v) {
  if (!v.is_null()) s = s.value_or(0) + v.value();
}
void my_result(const SumState &s, IntResult out) {
  if (!s.has_value()) { out.set_null(); return; }
  out.set(s.value());
}

// Registration:
// make_aggregate_func<SumState, &my_result>("my_sum")
//     .returns(INT)
//     .param(INT)
//     .clear<&my_clear>()
//     .accumulate<&my_acc>()
//     .build()

How the builder methods work:

make_aggregate_func<State, &result_fn>() auto-generates prerun and postrun (value-initializes and deletes State).
.clear<&fn>() registers your void(State&) reset function.
.accumulate<&fn>() registers your void(State&, TypedArgs...) fold function. TypedArgs are deduced from the function signature (IntArg, StringArg, etc.). Call .accumulate<>() after all .param(TYPE) calls: the builder checks the typed-argument count against the declared SQL parameter count at compile time, so calling it earlier fails to build.
The result type (IntResult, RealResult, etc.) is deduced from the result function signature.

For a counter that never returns NULL, use a plain state type:

using CountState = long long;
void count_clear(CountState &s) { s = 0; }
void count_acc(CountState &s, IntArg v) { if (!v.is_null()) s++; }
void count_result(const CountState &s, IntResult out) { out.set(s); }

A StringResult aggregate VDF returns text: the result reports the utf8mb4_bin charset and collation, so clients display it as characters rather than hex — the same as the scalar VDF STRING path. It also honors .max_result_length(n) — which requires the opt-in dev-ABI headers (-DVSQL_USE_DEV_ABI=ON) and Protocol 4 — the same way, sizing a materialized aggregate result (a GROUP BY/DISTINCT temp table, CREATE TABLE ... SELECT, or UNION) so it isn’t truncated at the argument width. See Custom Buffer Sizes for the sizing rules and cap.

Per-Statement State (Prerun and Postrun)

Some VDFs need state that spans every row a single query touches — a call counter, a cached result, an open resource. Allocate it in a prerun hook, access it from the VDF body, and release it in a postrun hook. Both hooks run once per statement; the VDF body runs once per row. Register them with .prerun<&Hook>() and .postrun<&Hook>(). The required signatures are:

Hook	Required signature
Prerun	`void(vsql::PrerunArgs, vsql::PrerunResult)`
Postrun	`void(vsql::PostrunArgs)`

Use PrerunResult::set_user_data(void*) to stash state; use PostrunArgs::delete_state<T>() to release it. If prerun calls set_user_data(new T{}), postrun must call delete_state<T>() — the SDK does not auto-free. PrerunArgs::type_at(i) exposes the declared SQL type of each argument before any rows are read; the predicates is_int(), is_real(), is_str(), is_custom() on the returned PrerunArgType mirror the column types. Use this in prerun to validate argument types or call PrerunResult::request_buffer_size(n) to size the result buffer.

#include <villagesql/vsql.h>
using namespace vsql;

struct CallCounter { long long n = 0; };

void ba_call_index_prerun(PrerunArgs, PrerunResult out) {
  out.set_user_data(new CallCounter{});
}

void ba_call_index(CallCounter &state, IntResult out) {
  state.n++;
  out.set(state.n);
}

void ba_call_index_postrun(PostrunArgs args) {
  args.delete_state<CallCounter>();
}

// Registration:
// make_func<&ba_call_index>("ba_call_index")
//     .returns(INT).no_params()
//     .prerun<&ba_call_index_prerun>()
//     .postrun<&ba_call_index_postrun>()
//     .build()

Varargs VDFs

A varargs VDF accepts any number of arguments of any SQL type. Declare one with .varargs() on the func builder, which is mutually exclusive with .no_params() and .param(TYPE). The body receives a vsql::VarArgs argument instead of the usual fixed-arity argument types.

Varargs registration requires VEF Protocol 3. Older servers reject the extension at install time.

The framework cannot validate argument count or types for varargs VDFs. Pair every varargs registration with a prerun hook that calls PrerunResult::error() on invalid input or PrerunResult::request_buffer_size(n) to size the result buffer. Iterate over arguments with range-for. Each AnyArg element requires a type check before reading its value:

Predicate	Accessor	Return type
`is_int()`	`as_int()`	`long long`
`is_real()`	`as_real()`	`double`
`is_str()`	`as_str()`	`std::string_view`
`is_custom()`	`as_custom()`	`vsql::Span<const unsigned char>`

Check is_null() before any accessor — all four are undefined on a null argument. PrerunArgType::custom_type() returns the argument’s custom type name as a std::string_view — empty for non-custom arguments. is_custom() alone accepts every custom type registered on the server; pairing it with a custom_type() comparison narrows a varargs call to one specific type:

#include <villagesql/vsql.h>
#include <cstring>
using namespace vsql;

// POINT2D stores two little-endian int32 (x, y) in 8 bytes — registration
// omitted here; see Custom Types for make_type<>().

void point_path_prerun(PrerunArgs args, PrerunResult out) {
  if (args.size() == 0) {
    out.error("point_path requires at least one point");
    return;
  }
  for (size_t i = 0; i < args.size(); i++) {
    auto t = args.type_at(i);
    if (!t.is_custom() || t.custom_type() != "POINT2D") {
      out.error("point_path: every argument must be a POINT2D");
      return;
    }
  }
  out.request_buffer_size(16 + args.size() * 32);
}

void point_path(VarArgs args, StringResult out) {
  auto buf = out.buffer();
  size_t off = 0;
  bool first = true;
  for (auto a : args) {
    if (a.is_null()) { out.set_null(); return; }
    auto bytes = a.as_custom();
    int x, y;
    memcpy(&x, bytes.data(), 4);
    memcpy(&y, bytes.data() + 4, 4);
    if (!first) {
      memcpy(buf.data() + off, " -> ", 4);
      off += 4;
    }
    first = false;
    off += static_cast<size_t>(
        snprintf(buf.data() + off, buf.size() - off, "(%d,%d)", x, y));
  }
  out.set_length(off);
}

// Registration:
// make_func<&point_path>("point_path")
//     .returns(STRING).varargs()
//     .prerun<&point_path_prerun>()
//     .build()

SELECT point_path(POINT2D::from_string('0,0'), POINT2D::from_string('1,2'));
-- (0,0) -> (1,2)
SELECT point_path('1,2');
-- ERROR 1123 (HY000): Can't initialize function 'point_path'; point_path: every argument must be a POINT2D

#include <villagesql/vsql.h>
#include <cstring>
using namespace vsql;

constexpr size_t kBytearrayLen = 4;

void ba_concat_all_prerun(PrerunArgs args, PrerunResult out) {
  if (args.size() == 0) {
    out.error("ba_concat_all requires at least one argument");
    return;
  }
  for (size_t i = 0; i < args.size(); i++) {
    auto t = args.type_at(i);
    if (!t.is_custom() && !t.is_str()) {
      out.error("ba_concat_all: argument " + std::to_string(i) +
                " must be BYTEARRAY");
      return;
    }
  }
  out.request_buffer_size(args.size() * kBytearrayLen);
}

void ba_concat_all(VarArgs args, StringResult out) {
  auto dst = out.buffer();
  size_t off = 0;
  for (auto a : args) {
    if (a.is_null() || !a.is_custom()) { out.set_null(); return; }
    auto bytes = a.as_custom();
    std::memcpy(dst.data() + off, bytes.data(), bytes.size());
    off += bytes.size();
  }
  out.set_length(off);
}

// Registration:
// make_func<&ba_concat_all>("ba_concat_all")
//     .returns(STRING).varargs()
//     .prerun<&ba_concat_all_prerun>()
//     .build()

Deciding a Function’s Type Parameters at Resolution Time

The server’s built-in rules usually work out a return type’s parameters from its arguments’ declared types. Sometimes they can’t. For example, tvector_concat(TVECTOR(M), TVECTOR(N)) returns a TVECTOR(M+N). The built-in rules reject two TVECTOR arguments whose dimensions differ, and they can copy an argument’s parameters to the return type but never add them. For cases like this, register a bind-and-check-types hook with .bind_and_check_types<&Hook>() on the func builder, with the signature:

void my_hook(vsql::BindArgs args, vsql::BindResult out);

.bind_and_check_types() and .varargs() are mutually exclusive on the same function — a varargs VDF has no fixed argument list for the hook to reason about. Declaring .varargs() before .bind_and_check_types<&Hook>() in the builder chain is a compile-time static_assert failure. Declaring them in the other order compiles, but the extension is rejected when the server loads it: registration walks every function’s final built state and rejects the first one that is both varargs and has a bind hook, with:

VDF '%s' declares both .varargs() and .bind_and_check_types(). This is not currently supported.

.bind_and_check_types<&Hook>() and <villagesql/vsql/bind_check_types.h> are declared only in the opt-in dev ABI headers (-DVSQL_USE_DEV_ABI=ON) and require VEF Protocol 4. A default stable-ABI build has neither, so registering a hook fails to compile.

The server calls the hook between two passes it makes over a function’s arguments: the first pass checks each argument’s base type, the hook runs next, and the second pass applies the hook’s answer for each argument where it gave one. With a hook installed, the first pass does not collect shared parameters between arguments of the same custom type, so none of them are required to agree with each other. An argument whose own type is not already resolved, such as a bare string literal, gets its parameters from the hook alone. A parameterized return type’s parameters also come from the hook, in place of the built-in rule that copies them from an argument. The vsql_tvector example extension in the server source tree registers such a hook on tvector_concat:

#include <villagesql/vsql.h>

// Excerpt from villagesql/examples/vsql-tvector/src/tvector.cc. That file
// also has the standard library includes, the constants kDefaultBytesPerElem
// (4) and kTVectorMaxDimension (4096), the TVECTOR type, the tvector_concat
// function body, and the helpers common_element_width() and
// bind_params_for_vector(), which read an argument's resolved params or count
// the elements of a constant literal.

struct TVectorParams {
  int64_t dimension;
  size_t bytes_per_elem;  // 4 for float, 8 for double

  static TVectorParams parse(const std::map<std::string, std::string> &params) {
    auto dim_it = params.find("dimension");
    int64_t dim = strtoll(dim_it->second.c_str(), nullptr, 10);
    size_t bytes = kDefaultBytesPerElem;
    auto type_it = params.find("type");
    if (type_it != params.end() && type_it->second == "double") bytes = 8;
    return TVectorParams{.dimension = dim, .bytes_per_elem = bytes};
  }

  // Inverse of parse: render a typed TVectorParams back into the canonical
  // key/value string form. Used by the SDK when the server needs to publish
  // inferred params (e.g., from a constant-string from_string) back in the
  // same shape that parse() consumes.
  static void to_strings(const TVectorParams &p,
                         std::map<std::string, std::string> &out) {
    out["dimension"] = std::to_string(p.dimension);
    out["type"] = (p.bytes_per_elem == 8) ? "double" : "float";
  }
};

// bind_and_check_types for tvector_concat.
//
// Two jobs the built-in rules cannot do:
//   1. permit arguments whose dimensions differ (parameter sharing rejects it);
//   2. compute the return dimension as M+N (the server return rule only
//      copies).
//
// A third job appears when an argument is a constant: a bare '[1,2,3]' has no
// type context, and with a hook installed nothing donates params to it, so the
// hook counts the elements itself and answers with set_arg(). The element type
// comes from whichever side the server already resolved, which is what the
// built-in rules would have donated.
void tvector_concat_bind(vsql::BindArgs args, vsql::BindResult out) {
  if (args.size() != 2) {
    out.error("expects 2 arguments");
    return;
  }

  std::string error_msg;
  size_t bpe = 0;
  if (common_element_width(args, bpe, error_msg)) {
    out.error(error_msg);
    return;
  }

  TVectorParams first{}, second{};
  bool publish_first = false, publish_second = false;

  if (bind_params_for_vector(args, 0, bpe, first, publish_first, error_msg) ||
      bind_params_for_vector(args, 1, bpe, second, publish_second, error_msg)) {
    out.error(error_msg);
    return;
  }

  // Hand back the params of whichever argument the server could not resolve.
  if (publish_first) out.set_arg(0, first);
  if (publish_second) out.set_arg(1, second);

  const int64_t total = first.dimension + second.dimension;
  if (total > kTVectorMaxDimension) {
    out.error("combined dimension " + std::to_string(total) +
              " exceeds the maximum of " +
              std::to_string(kTVectorMaxDimension));
    return;
  }
  out.set_return(TVectorParams{.dimension = total, .bytes_per_elem = bpe});
}

// Registration, inside make_extension():
// .func(make_func<&tvector_concat>("tvector_concat")
//     .returns(TVECTOR)
//     .param(TVECTOR)
//     .param(TVECTOR)
//     .bind_and_check_types<&tvector_concat_bind>()
//     .deterministic()
//     .build())

The hook lets two vectors of different dimensions combine, and it counts the elements of a constant literal. A source build of the server includes vsql_tvector; the release tarballs do not.

INSTALL EXTENSION vsql_tvector;
CREATE TABLE v2 (v vsql_tvector.TVECTOR(2));
CREATE TABLE v3 (v vsql_tvector.TVECTOR(3));
INSERT INTO v2 VALUES ('[1,2]');
INSERT INTO v3 VALUES ('[3,4,5]');

SELECT vsql_tvector.tvector_concat(v2.v, v3.v) AS c,
       vsql_tvector.tvector_dim(vsql_tvector.tvector_concat(v2.v, v3.v)) AS dim
FROM v2, v3;

+-------------+------+
| c           | dim  |
+-------------+------+
| [1,2,3,4,5] |    5 |
+-------------+------+

SELECT vsql_tvector.tvector_concat(v, '[6,7]') AS c FROM v3;

+-------------+
| c           |
+-------------+
| [3,4,5,6,7] |
+-------------+

Extension Load and Unload Hooks

An extension sometimes has work to do once per load rather than once per call — picking a CPU-specific implementation, reading its own configuration once, or building a lookup table the VDFs then read. Register that work with .on_init<&Fn>() on the extension builder, and the matching teardown with .on_deinit<&Fn>(). Both take a function with the signature void(), and must be registered together — declaring one without the other fails to compile. An extension that reaches the server with a mismatched pair some other way is rejected at load instead:

ERROR 3219 (HY000): Failed to load VEF extension '<name>': extension registers
an on_init hook without an on_deinit

.on_init<&Fn>() and .on_deinit<&Fn>() are declared only in the opt-in dev ABI headers (-DVSQL_USE_DEV_ABI=ON). A default stable-ABI build has neither method, so registering a hook fails to compile.

on_init runs once per load — at INSTALL EXTENSION, and again on every server startup — after the server has validated the extension and populated its required capabilities, so a registered system variable already holds its configured value. At startup specifically, on_init is deferred until persisted system-variable values have been applied, so it never sees a compiled-in default in place of a value set with SET PERSIST. It never runs for an extension the server rejected. on_deinit runs at unload — UNINSTALL EXTENSION or server shutdown — before the server depopulates those same capabilities, so it sees the same live state on_init did.

#include <villagesql/vsql.h>
#include <cmath>
using namespace vsql;

static float *g_sqrt_table = nullptr;

void build_table() {
  g_sqrt_table = new float[256];
  for (int i = 0; i < 256; i++) g_sqrt_table[i] = std::sqrt(float(i));
}

void free_table() {
  delete[] g_sqrt_table;
  g_sqrt_table = nullptr;
}

void fast_sqrt(IntArg n, RealResult out) {
  if (n.is_null() || n.value() < 0 || n.value() > 255) {
    out.set_null();
    return;
  }
  out.set(g_sqrt_table[n.value()]);
}

VEF_GENERATE_ENTRY_POINTS(
    make_extension()
        .on_init<&build_table>()
        .on_deinit<&free_table>()
        .func(make_func<&fast_sqrt>("fast_sqrt")
                  .returns(REAL)
                  .param(INT)
                  .build()))

After INSTALL EXTENSION my_extension, SELECT fast_sqrt(16) returns 4 — a result that is only reachable if build_table already ran.

VEF_GENERATE_REGISTRATION

VEF_GENERATE_REGISTRATION creates an internal _vef_do_register() helper that performs extension registration but does not define the extern "C" entry points. Use it when you need to customize vef_register behavior — for example, to patch descriptors after registration in a test build. For normal extensions, use VEF_GENERATE_ENTRY_POINTS instead.

VEF_GENERATE_REGISTRATION(
    make_extension()
        .func(make_func<&my_impl>("my_func").returns(INT).build()))

// Then define your own extern "C" vef_register/vef_unregister that call
// _vef_do_register() and optionally modify the result.

Custom Type Operations

For the full type operation builder reference — encode, decode, compare, hash, intrinsic defaults, and parameterized types — see Type Operations.

Preview Capabilities

The following VEF capabilities are available as opt-in preview headers. ABI and API are still under active development; see Preview Capabilities for the full reference.

Extension system variables — Preview Capabilities → System Variables
Extension status variables — Preview Capabilities → Status Variables
Keyring access — Preview Capabilities → Keyring Access
Column storage — Preview Capabilities → Column Storage

Inspecting Extension Registration Metadata

INFORMATION_SCHEMA.EXTENSION_REGISTRATION exposes the in-memory VEF registration struct for each loaded extension as a JSON document. Use it to verify that the server parsed your extension’s functions, types, and system variables correctly after INSTALL EXTENSION.

SELECT EXTENSION_NAME, NEGOTIATED_PROTOCOL, REGISTRATION_JSON
FROM INFORMATION_SCHEMA.EXTENSION_REGISTRATION
WHERE EXTENSION_NAME = 'my_ext';

Column	Type	Description
`EXTENSION_NAME`	`VARCHAR(64)`	Name of the installed extension.
`NEGOTIATED_PROTOCOL`	`BIGINT UNSIGNED`	VEF protocol version negotiated between the extension and the server.
`REGISTRATION_JSON`	`VARCHAR(65535)`	JSON serialization of the `vef_registration_t` struct, including `funcs` and `types` arrays and an `sdk_version` key reporting the SDK version the extension was built against.

Getting Started

Server Development

Using Extensions

Creating Extensions

Writing Extension Functions

Exception Safety

Argument and Result Types

Aggregate VDFs

Per-Statement State (Prerun and Postrun)

Varargs VDFs

Deciding a Function’s Type Parameters at Resolution Time

Extension Load and Unload Hooks

VEF_GENERATE_REGISTRATION

Custom Type Operations

Preview Capabilities

Inspecting Extension Registration Metadata

See Also

​Writing Extension Functions

​Exception Safety

​Argument and Result Types

​Aggregate VDFs

​Per-Statement State (Prerun and Postrun)

​Varargs VDFs

​Deciding a Function’s Type Parameters at Resolution Time

​Extension Load and Unload Hooks

​VEF_GENERATE_REGISTRATION

​Custom Type Operations

​Preview Capabilities

​Inspecting Extension Registration Metadata

​See Also

Writing Extension Functions

Exception Safety

Argument and Result Types

Aggregate VDFs

Per-Statement State (Prerun and Postrun)

Varargs VDFs

Deciding a Function’s Type Parameters at Resolution Time

Extension Load and Unload Hooks

VEF_GENERATE_REGISTRATION

Custom Type Operations

Preview Capabilities

Inspecting Extension Registration Metadata

See Also