Development

This guide covers writing VDF implementations and running regression tests for VillageSQL extensions.

If you are contributing to the VillageSQL server itself (not building an extension), see Clone and Build from Source, which covers the full server developer workflow including running tests with mysql-test-run.pl directly.

Setting Up Your Environment

To develop and test extensions, you need a built VillageSQL server. Follow the Clone and Build from Source guide to compile the server binaries. Once you have a build, use the villagesql CLI to manage a local dev server instance. Run all commands from the directory where VillageSQL was installed.

Starting a Local Dev Server

Initialize and start a server instance:

./villagesql init    # initialize database and seed bundled extensions
./villagesql start   # start the server (default port 3307)
./villagesql status  # check the server is running
./villagesql connect # open a mysql shell
./villagesql stop    # stop the server

To set a root password on init:

./villagesql init --password
./villagesql start

Pass --dir <path> before any command to manage multiple independent instances, or use --here to create a server directory in the current working directory:

./villagesql --here init
./villagesql --here start

Managing Extension Files

Before installing an extension via SQL, its .veb file must be present on the server. The CLI manages the server’s lib/veb/ directory:

./villagesql veb add /path/to/my_extension.veb  # copy a .veb to the server
./villagesql veb ls                              # list available .veb files
./villagesql veb rm my_extension                # remove a .veb file

.veb files placed in lib/veb/ before init are seeded automatically. After adding a file, install the extension via SQL:

INSTALL EXTENSION my_extension;

Writing Extension Functions

Extension functions are written in C++ and registered with VEF. Include a single header to access the full SDK:

#include <villagesql/extension.h>

Typed Wrappers (Recommended)

Typed wrappers provide a type-safe interface for VDF parameters and results. The framework detects wrapper types in your function signature and adapts automatically — the make_func registration syntax is unchanged. Input wrappers: IntArg, RealArg, StringArg, CustomArg — each provides is_null() and value(). For parameterized custom types, CustomArgWith adds a params() accessor that returns the cached parsed params struct (see Parameterized Types). Result wrappers: IntResult, RealResult, StringResult, CustomResult — each provides set_null(), warning(msg), and error(msg). Scalar results also provide set(value). Buffer results provide buffer() and set_length(len). For parameterized custom types, CustomResultWith adds a params() accessor. error(msg) sets the function result to a SQL error with the given message string. Use it for invalid input that should surface as an error to the caller, as opposed to set_null() which silently produces NULL. The message is truncated to fit the server’s internal error buffer if necessary. Scalar example — add two integers:

using namespace villagesql;

void add_impl(IntArg a, IntArg b, IntResult out) {
  if (a.is_null() || b.is_null()) { out.set_null(); return; }
  out.set(a.value() + b.value());
}

// Registration is unchanged:
make_func<&add_impl>("add").returns(INT).param(INT).param(INT).build();

Binary example — transform a custom type buffer in place:

using namespace villagesql;

void rot13_impl(CustomArg in, CustomResult out) {
  if (in.is_null()) { out.set_null(); return; }
  auto src = in.value();   // villagesql::Span<const unsigned char>
  auto dst = out.buffer(); // villagesql::Span<unsigned char>
  for (size_t i = 0; i < src.size(); i++) { dst[i] = transform(src[i]); }
  out.set_length(src.size());
}

For StringResult and CustomResult, write into buffer(), then call set_length() with the number of bytes written. buffer().size() is the maximum capacity. You can use different styles across functions in the same extension — each function’s style is determined by its own signature.

Raw ABI Style (Deprecated)

Prefer typed wrappers for new code. The raw ABI style passes pointers to the underlying C structs directly:

void add_impl(vef_context_t* ctx,
              vef_invalue_t* a, vef_invalue_t* b,
              vef_vdf_result_t* result) {
  result->int_value = a->int_value + b->int_value;
  result->type = VEF_RESULT_VALUE;
}

Registration is the same as with typed wrappers.

VDF Return Codes

Set result->type (or call the corresponding typed-wrapper method) to indicate the outcome of a VDF call:

Code	Value	Typed wrapper method	Behavior
`VEF_RESULT_VALUE`	0	`set(v)` / `set_length(n)`	Returns the computed value.
`VEF_RESULT_NULL`	1	`set_null()`	Returns SQL NULL.
`VEF_RESULT_WARNING`	2	`warning(msg)`	Returns NULL for this row and adds a SQL warning. Execution continues. In strict mode (`STRICT_TRANS_TABLES`), MySQL promotes the warning to an error on INSERT/UPDATE.
`VEF_RESULT_ERROR`	3	`error(msg)`	Aborts statement execution immediately.

Use VEF_RESULT_WARNING for recoverable bad input (e.g., an unparseable string passed to an encode function). Use VEF_RESULT_ERROR for conditions that make it unsafe to continue (e.g., corrupt stored data).

Type Operation Builders

Only needed if your extension defines a custom column type. If you’re writing functions only, skip ahead to Running Regression Tests. Custom types require four operations the engine calls internally: encode (string → binary), decode (binary → string), compare, and hash. Implement them against these C++ signatures (all defined in func_builder.h):

Fixed-Length Types

// Encode: string -> binary. false = success, true = error.
// Set *length = SIZE_MAX to produce SQL NULL.
using TypeEncodeFunc = bool (*)(std::string_view from,
                                villagesql::Span<unsigned char> buf,
                                size_t *length);

// Decode: binary -> string. false = success, true = error.
using TypeDecodeFunc = bool (*)(villagesql::Span<const unsigned char> data,
                                villagesql::Span<char> out,
                                size_t *out_len);

// Compare: returns -1, 0, or 1 (used for ORDER BY and indexes).
using TypeCompareFunc = int (*)(villagesql::Span<const unsigned char> a,
                                villagesql::Span<const unsigned char> b);

// Hash: returns hash code (used for hash joins).
using TypeHashFunc = size_t (*)(villagesql::Span<const unsigned char> data);

Register each operation with make_type_encode, make_type_decode, make_type_compare, or make_type_hash, passing the function pointer and the type name constant. The type descriptor references them by name:

constexpr const char *MYTYPE = "mytype"; // replace with your type name

VEF_GENERATE_ENTRY_POINTS(
    make_extension("my_ext", "1.0.0")
        .type(make_type(MYTYPE)
                  .persisted_length(8)
                  .max_decode_buffer_length(64)
                  .encode("mytype_encode")    // Name of the encode VDF
                  .decode("mytype_decode")    // Name of the decode VDF
                  .compare("mytype_compare")  // Name of the compare VDF
                  .hash("mytype_hash")        // Optional: name of the hash VDF
                  .build())
        .func(make_type_encode<&my_encode>("mytype_encode", MYTYPE))
        .func(make_type_decode<&my_decode>("mytype_decode", MYTYPE))
        .func(make_type_compare<&my_compare>("mytype_compare", MYTYPE))
        .func(make_type_hash<&my_hash>("mytype_hash", MYTYPE)))

Parameterized Types

Variable-length types need the column’s declared parameters at encode, decode, compare, and hash time to determine allocation sizes and layout. Define a params struct with a parse function, register it on the type builder with .params<P, &ParseFunc>(), and use const P& as the first argument of your type operation functions. The SDK caches the parse result per unique parameter combination, so the parse function runs at most once per type instantiation.

struct MyTypeParams {
  int64_t dimension;
  static MyTypeParams parse(const std::map<std::string, std::string> &p) {
    return {.dimension = stoll(p.at("dimension"))};
  }
};

bool mytype_encode(const MyTypeParams &params, std::string_view from,
                   villagesql::Span<unsigned char> buf, size_t *length) {
  size_t bytes = (size_t)params.dimension * 4;
  if (buf.size() < bytes) return true;
  // ... parse from, write to buf ...
  *length = bytes;
  return false;  // success
}

bool mytype_decode(const MyTypeParams &params,
                   villagesql::Span<const unsigned char> data,
                   villagesql::Span<char> out, size_t *out_len) {
  // ... read params.dimension floats from data, format into out ...
  *out_len = bytes_written;
  return false;  // success
}

int mytype_compare(const MyTypeParams &params,
                   villagesql::Span<const unsigned char> a,
                   villagesql::Span<const unsigned char> b) {
  // Returns -1, 0, or 1.
}

size_t mytype_hash(const MyTypeParams &params,
                   villagesql::Span<const unsigned char> data) {
  // Returns hash code.
}

Register .params<>() on the type builder alongside the operation VDF names. Both are required: .params<>() binds the parse function at startup; make_type_encode (and the other make_type_* functions) detect const P& and route through the cache automatically.

VEF_GENERATE_ENTRY_POINTS(
    make_extension("my_ext", "1.0.0")
        .type(make_type(MYTYPE)
                  .persisted_length(8)
                  .max_decode_buffer_length(64)
                  .encode("mytype_encode")
                  .decode("mytype_decode")
                  .compare("mytype_compare")
                  .params<MyTypeParams, &MyTypeParams::parse>()
                  .build())
        .func(make_type_encode<&mytype_encode>("mytype_encode", MYTYPE))
        .func(make_type_decode<&mytype_decode>("mytype_decode", MYTYPE))
        .func(make_type_compare<&mytype_compare>("mytype_compare", MYTYPE)))

The parameterized variants — TypeEncodeWithParamsFunc, TypeDecodeWithParamsFunc, TypeCompareWithParamsFunc, and TypeHashWithParamsFunc — are defined in func_builder.h. make_type_encode, make_type_decode, make_type_compare, and make_type_hash accept both fixed-length and variable-length forms.

Running Regression Tests

Run extension regression tests using the MySQL Test Runner from your VillageSQL build directory.

Running the Full Suite

To run all tests for your extension:

cd $BUILD_HOME
./mysql-test/mysql-test-run.pl --suite=/path/to/your/extension/test --parallel=auto

Running Individual Tests

To run a single test case, specify the suite path and test name:

cd $BUILD_HOME
./mysql-test/mysql-test-run.pl --suite=/path/to/your/extension/test my_test_name

Creating New Tests

When adding new features or fixing bugs, you should add corresponding regression tests.

Test Location

Tests are located in mysql-test/suite/villagesql/. Choose the appropriate sub-directory based on the SQL statement or feature being tested.

Test files end with .test and go in the t/ directory.
Expected result files end with .result and go in the r/ directory.

Test File Conventions

VillageSQL tests often involve the custom COMPLEX type. We follow a specific pattern to ensure tests are skipped gracefully if the feature is not yet fully implemented:

# Description of the test
# Grammar: (Optional) SQL syntax reference

--source include/villagesql/not_implemented_complex_type.inc

--source include/villagesql/install_complex_extension.inc

# ... Your Test Code Here ...
CREATE TABLE t1 (val COMPLEX);
INSERT INTO t1 VALUES ('(1.0,2.0)');
SELECT * FROM t1;
DROP TABLE t1;

--source include/villagesql/uninstall_complex_extension.inc

Steps to Add a Test

Create the .test file in the appropriate t/ directory (e.g., mysql-test/suite/villagesql/select/t/my_new_test.test).
Create an empty .result file in the corresponding r/ directory (e.g., mysql-test/suite/villagesql/select/r/my_new_test.result).

Run the test with --record to generate the expected output:

cd $BUILD_HOME
./mysql-test/mysql-test-run.pl --suite=/path/to/your/extension/test --record my_new_test

Verify the output in the generated .result file to ensure it matches your expectations.

Debugging Tests

If a test fails, the test framework provides detailed logs.

Test output: Check mysql-test/var/log/my_new_test.log.
Server error log: Check mysql-test/var/log/mysqld.1.err.
Diff: The framework outputs a diff between the actual output and the expected .result file.

To run a test with extra debug information:

cd $BUILD_HOME
./mysql-test/mysql-test-run.pl --verbose --suite=/path/to/your/extension/test my_new_test

Getting Started

Extensions

Using Extensions

Creating Extensions

Setting Up Your Environment

Starting a Local Dev Server

Managing Extension Files

Writing Extension Functions

Typed Wrappers (Recommended)

Raw ABI Style (Deprecated)

VDF Return Codes

Type Operation Builders

Fixed-Length Types

Parameterized Types

Running Regression Tests

Running the Full Suite

Running Individual Tests

Creating New Tests

Test Location

Test File Conventions

Steps to Add a Test

Debugging Tests

Getting Started

Extensions

Using Extensions

Creating Extensions

Documentation Index

​Setting Up Your Environment

​Starting a Local Dev Server

​Managing Extension Files

​Writing Extension Functions

​Typed Wrappers (Recommended)

​Raw ABI Style (Deprecated)

​VDF Return Codes

​Type Operation Builders

​Fixed-Length Types

​Parameterized Types

​Running Regression Tests

​Running the Full Suite

​Running Individual Tests

​Creating New Tests

​Test Location

​Test File Conventions

​Steps to Add a Test

​Debugging Tests

Setting Up Your Environment

Starting a Local Dev Server

Managing Extension Files

Writing Extension Functions

Typed Wrappers (Recommended)

Raw ABI Style (Deprecated)

VDF Return Codes

Type Operation Builders

Fixed-Length Types

Parameterized Types

Running Regression Tests

Running the Full Suite

Running Individual Tests

Creating New Tests

Test Location

Test File Conventions

Steps to Add a Test

Debugging Tests