Extension Architecture

Understanding VillageSQL’s extension architecture helps debug issues and optimize performance when building extensions.

Terminology

VEB (VillageSQL Extension Bundle) - The .veb file format, a tar archive containing manifest, library, and metadata
VEF (VillageSQL Extension Framework) - The SDK and C++ API for authoring extensions
VDF (VillageSQL Defined Function) - Functions registered through the VEF SDK using VEF_GENERATE_ENTRY_POINTS()
UDF (User-Defined Function) - Traditional MySQL functions registered via CREATE FUNCTION ... SONAME (legacy approach, still supported)

VDF Function Lookup

VDFs support both qualified and unqualified function calls:

-- Unqualified lookup
SELECT complex_abs(value) FROM table;

-- Qualified lookup
SELECT vsql_complex.complex_abs(value) FROM table;

Resolution order for unqualified function calls:

System functions (built-in MySQL)
UDFs (traditional user-defined functions)
VDFs (extension functions) - only if exactly one function with that name exists
Stored functions (created with CREATE FUNCTION)

Performance: For hot code paths, use qualified calls (extension_name.function_name()) to skip the resolution chain and directly invoke the extension function.

VEB File Format

VillageSQL extensions are distributed as .veb (VillageSQL Extension Bundle) files - tar archives containing:

extension_name.veb (tar archive)
├── manifest.json       # Extension metadata (required)
└── lib/
    └── extension.so    # Compiled shared library (required)

manifest.json Schema

{
  "name": "extension_name",          // Required: lowercase_with_underscores
  "version": "1.0.0",                // Required: semantic version
  "description": "Brief description",
  "author": "Your Name",
  "license": "GPL-2.0"
}

name: Must match extension name used in SQL (lowercase_with_underscores)
version: Semantic version (MAJOR.MINOR.PATCH)
description, author, license: Optional metadata

Extension Lifecycle

Installation Flow

INSTALL EXTENSION name
    ↓
1. Validate .veb exists in veb_dir
    ↓
2. Calculate SHA256 hash of .veb
    ↓
3. Expand to _expanded/{name}/{sha256}/
    ↓
4. Parse and validate manifest.json
    ↓
5. Load .so library (dlopen with RTLD_LOCAL)
    ↓
6. Call vef_register() entry point
    ↓
7. Register VDFs and custom types
    ↓
8. Persist registration and update cache
    ↓
Success

Rollback: If any step fails, all changes are undone and the .so is unloaded. Symbol Isolation: Extensions are loaded with RTLD_LOCAL flag, ensuring symbols from one extension don’t conflict with symbols from other extensions. This prevents naming collisions when multiple extensions use common library names or function names.

The veb_dir system variable points to the directory where .veb extension files are stored.

Uninstallation Flow

UNINSTALL EXTENSION name
    ↓
1. Check for column dependencies
    ↓
2. Call vef_unregister() cleanup hook
    ↓
3. Drop registered VDFs
    ↓
4. Drop custom types
    ↓
5. Remove extension registration and update cache
    ↓
6. Unload .so library (dlclose)
    ↓
7. Keep _expanded directory (for reinstall)
    ↓
Success

Dependency prevention: Cannot uninstall if table columns use the extension’s custom types.

Expansion Directory Structure

VillageSQL expands .veb files to support multiple versions:

veb_dir/
├── extension_name.veb
└── _expanded/
    └── extension_name/
        ├── abc123.../              # SHA256 of v1.0.0 .veb
        │   ├── manifest.json
        │   └── lib/extension.so
        └── def456.../              # SHA256 of v2.0.0 .veb
            ├── manifest.json
            └── lib/extension.so

Why SHA256 directories?

Test new versions without overwriting
Enable rollback
Prevent “same version, different code”

Cleanup: Orphaned SHA256 directories are removed on server restart.

Victionary Caching Layer

VictionaryClient maintains in-memory caches of system metadata for O(log n) lookups.

Cached Tables

SystemTableMap<ExtensionEntry> m_extensions;
SystemTableMap<CustomTypeEntry> m_types;
SystemTableMap<CustomColumnEntry> m_columns;
SystemTableMap<PropertyEntry> m_properties;

Cache Operations

Operation	Locking	Performance
Read (resolve type)	Read lock	O(log n) map lookup
Write (install extension)	Write lock	O(log n) insert + disk I/O
Server startup	N/A	Full table scan into memory

Cache invalidation: Automatic during DDL operations (INSTALL/UNINSTALL EXTENSION). Memory overhead: ~100 bytes per entry.

Custom Type System

Type Resolution

CREATE TABLE t (col COMPLEX)
    ↓
1. Parser encounters COMPLEX
    ↓
2. PT_custom_type::create()
    ↓
3. ResolveTypeToContext(extension_name, type_name)
    ↓
4. VictionaryClient::lookup_type() → O(log n)
    ↓
5. Find TypeDescriptor in cache
    ↓
6. Create Field with implementation_type

Implementation Types

Custom types map to MySQL storage types:

Custom Type	MySQL Implementation	Bytes
COMPLEX	MYSQL_TYPE_VARCHAR	16
UUID	MYSQL_TYPE_VARCHAR	16
INET6	MYSQL_TYPE_VARCHAR	16
JSON_SCHEMA	MYSQL_TYPE_BLOB	Variable

Concurrency & Transaction Behavior

Thread Safety Model

Extension functions are called in a per-row execution model:

Isolated per-row execution: Each function call gets its own result buffer (thread-safe by design)
Prerun/Postrun hooks: Per-statement setup/cleanup, called once per SQL statement
No guaranteed isolation: Multiple connections may call your functions concurrently
Best practice: Avoid global state; use function parameters and return values

VillageSQL does not guarantee thread isolation for extension functions. If you use global variables or shared state, protect them with mutexes or locks.

Transaction Behavior

Extension functions should follow these best practices:

Design functions to be stateless when possible
Avoid persistent side effects (file writes, external API calls) in functions
If using prerun/postrun state, handle cleanup appropriately

Performance Considerations

Optimization: Use prerun hooks to cache expensive per-statement setup instead of repeating work per-row.

Custom Type Performance

-- Slow: UDF call per row
SELECT * FROM signals WHERE complex_abs(impedance) > 100;

-- Fast: Computed column with index
ALTER TABLE signals
ADD COLUMN impedance_magnitude DOUBLE AS (complex_abs(impedance)) STORED,
ADD INDEX(impedance_magnitude);

SELECT * FROM signals WHERE impedance_magnitude > 100;

Security & Debugging

Security Model

Trust model: Extensions run with full server privileges.

No sandboxing or permission system
Extensions can read any file, access network, execute code
Trust implications: Only install extensions from trusted sources

Installation security: Runs as villagesql_extension_installer user (context switch).

Debugging Extensions

Enable verbose logging:

mysqld --log-error-verbosity=3

GDB debugging:

gdb -p $(pidof mysqld)
(gdb) break my_func_init
(gdb) continue

Check dependencies:

# Linux
ldd /path/to/extension.so

# macOS
otool -L /path/to/extension.so

Common errors:

Undefined symbol: Check extern "C" linkage
Cannot open shared object: Check library dependencies
Crash on UDF call: Check NULL pointer handling

Schema Validation

On server startup, SchemaManager validates system table schemas:

Open each VillageSQL system table
Check column count and names
Validate column types
Verify primary keys
Check indexes

Failure scenarios:

Missing table → Create from villagesql_schema.sql
Wrong schema → Error and refuse to start
Version mismatch → Run upgrade scripts

Server version:

SELECT VERSION();

Source builds include the git commit hash:

8.4.8-villagesql-0.0.3

Next Steps

Create Extension

Build your first extension

System Reference

System tables and views

Examples

Study vsql_complex implementation

Managing Extensions

Monitor and troubleshoot

Getting Started

Extensions

Using Extensions

Creating Extensions

Terminology

VDF Function Lookup

VEB File Format

manifest.json Schema

Extension Lifecycle

Installation Flow

Uninstallation Flow

Expansion Directory Structure

Victionary Caching Layer

Cached Tables

Cache Operations

Custom Type System

Type Resolution

Implementation Types

Concurrency & Transaction Behavior

Thread Safety Model

Transaction Behavior

Performance Considerations

Custom Type Performance

Security & Debugging

Security Model

Schema Validation

Next Steps

Create Extension

System Reference

Examples

Managing Extensions

​Terminology

​VDF Function Lookup

​VEB File Format

​manifest.json Schema

​Extension Lifecycle

​Installation Flow

​Uninstallation Flow

​Expansion Directory Structure

​Victionary Caching Layer

​Cached Tables

​Cache Operations

​Custom Type System

​Type Resolution

​Implementation Types

​Concurrency & Transaction Behavior

​Thread Safety Model

​Transaction Behavior

​Performance Considerations

​Custom Type Performance

​Security & Debugging

​Security Model

​Schema Validation

​Next Steps

Create Extension

System Reference

Examples

Managing Extensions

Terminology

VDF Function Lookup

VEB File Format

manifest.json Schema

Extension Lifecycle

Installation Flow

Uninstallation Flow

Expansion Directory Structure

Victionary Caching Layer

Cached Tables

Cache Operations

Custom Type System

Type Resolution

Implementation Types

Concurrency & Transaction Behavior

Thread Safety Model

Transaction Behavior

Performance Considerations

Custom Type Performance

Security & Debugging

Security Model

Schema Validation

Next Steps