> ## Documentation Index
> Fetch the complete documentation index at: https://villagesql.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Extension Architecture
> How VillageSQL's extension system works internally
Understanding VillageSQL's extension architecture helps debug issues and optimize performance when building extensions.
The extension author journey is straightforward: you write C++ or Rust functions
using the corresponding SDK, compile them into a shared library, and package that library
with a manifest into a `.veb` file. When you run `INSTALL EXTENSION`, VillageSQL
loads your library, calls your registration code, and makes your functions
immediately available as SQL — callable from any query as if they were built into
the server. No server restart required. The rest of this page explains how each
step of that process works.
## Terminology
* **VEB** (VillageSQL Extension Bundle) - The `.veb` file format, a tar archive containing manifest, library, and metadata
* **VEF** (VillageSQL Extension Framework) - The C++ and Rust SDKs for authoring extensions
* **VDF** (VillageSQL Defined Function) - Functions registered through VEF — via `VEF_GENERATE_ENTRY_POINTS()` in C++ or the `extension!` macro in Rust
***
## VDF Function Lookup
VDFs support both qualified and unqualified function calls:
```sql theme={null}
-- Unqualified lookup
SELECT complex_abs(value) FROM table;
-- Qualified lookup
SELECT vsql_complex.complex_abs(value) FROM table;
```
**Resolution order:**
1. System functions (built-in MySQL)
2. UDFs (traditional MySQL user-defined functions via `CREATE FUNCTION ... SONAME`)
3. VDFs (extension functions) - only if exactly one function with that name exists
4. Stored functions (created with `CREATE FUNCTION`)
If more than one extension registers a function with the same name, the unqualified call is ambiguous and the fully qualified name (`extension_name.function_name()`) must be used.
**Performance:** For hot code paths, use qualified calls (`extension_name.function_name()`) to skip the resolution chain and directly invoke the extension function.
***
## VEB File Format
VillageSQL extensions are distributed as `.veb` (VillageSQL Extension Bundle) files - tar archives containing:
```
extension_name.veb (tar archive)
├── manifest.json # Extension metadata (required)
└── lib/
└── extension.so # Compiled shared library (required)
```
### manifest.json Schema
```json theme={null}
{
"name": "extension_name", // Required: lowercase_with_underscores
"version": "1.0.0", // Required: semantic version
"description": "Brief description",
"author": "Your Name",
"license": "GPL-2.0"
}
```
* **name:** Must match extension name used in SQL (lowercase\_with\_underscores)
* **version:** Semantic version (MAJOR.MINOR.PATCH)
* **description, author, license:** Optional metadata
## Extension Lifecycle
### Installation Flow
```
INSTALL EXTENSION name
↓
1. Validate .veb exists in veb_dir
↓
2. Calculate SHA256 hash of .veb
↓
3. Expand to {datadir}/.veb_expansion_cache/{name}/{sha256}/
↓
4. Parse and validate manifest.json
↓
5. Load .so library (dlopen with RTLD_LOCAL)
↓
6. Call vef_register() entry point
↓
7. Register VDFs and custom types
↓
8. Persist registration and update cache
↓
Success
```
**Rollback:** If any step fails, all changes are undone and the `.so` is unloaded.
**Symbol Isolation:** Extensions are loaded with `RTLD_LOCAL` flag, ensuring symbols from one extension don't conflict with symbols from other extensions. This prevents naming collisions when multiple extensions use common library names or function names.
The `veb_dir` system variable points to the directory where `.veb` extension files are stored.
### Uninstallation Flow
```
UNINSTALL EXTENSION name
↓
1. Check for column dependencies
↓
2. Call vef_unregister() cleanup hook
↓
3. Drop registered VDFs
↓
4. Drop custom types
↓
5. Remove extension registration and update cache
↓
6. Unload .so library (dlclose)
↓
7. Keep .veb_expansion_cache directory (for reinstall)
↓
Success
```
**Dependency prevention:** Cannot uninstall if table columns use the extension's custom types.
***
## Expansion Directory Structure
VillageSQL expands .veb files into the MySQL data directory to support multiple versions:
```
datadir/
└── .veb_expansion_cache/
└── extension_name/
├── abc123.../ # SHA256 of v1.0.0 .veb
│ ├── manifest.json
│ └── lib/extension.so
└── def456.../ # SHA256 of v2.0.0 .veb
├── manifest.json
└── lib/extension.so
```
**Why SHA256 directories?**
* Test new versions without overwriting
* Enable rollback
* Prevent "same version, different code"
**Cleanup:** Orphaned SHA256 directories are removed on server restart.
***
## Victionary Caching Layer
VictionaryClient maintains in-memory caches of system metadata for O(log n) lookups.
### Cache Operations
| Operation | Locking | Performance |
| ------------------------- | ---------- | --------------------------- |
| Read (resolve type) | Read lock | O(log n) map lookup |
| Write (install extension) | Write lock | O(log n) insert + disk I/O |
| Server startup | N/A | Full table scan into memory |
**Cache invalidation:** Automatic during DDL operations (INSTALL/UNINSTALL EXTENSION).
**Memory overhead:** \~100 bytes per entry.
***
## Custom Type System
### Type Resolution
```cpp theme={null}
CREATE TABLE t (col COMPLEX)
↓
1. Parser encounters COMPLEX
↓
2. PT_custom_type::create()
↓
3. ResolveTypeDescriptor(extension_name, type_name)
↓
4. VictionaryClient::type_descriptors().get_prefix_committed() → O(log n)
↓
5. Find TypeDescriptor in cache
↓
6. AcquireOrCreateTypeContext(descriptor, parameters, mem_root)
↓
7. Create Field with implementation_type
```
### Implementation Types
Custom types map to MySQL storage types:
| Custom Type | MySQL Implementation | Bytes |
| ------------ | -------------------- | -------- |
| COMPLEX | MYSQL\_TYPE\_VARCHAR | 16 |
| UUID | MYSQL\_TYPE\_VARCHAR | 16 |
| INET6 | MYSQL\_TYPE\_VARCHAR | 16 |
| JSON\_SCHEMA | MYSQL\_TYPE\_BLOB | Variable |
***
## Concurrency & Transaction Behavior
### Thread Safety Model
Extension functions are called in a per-row execution model:
* **Isolated per-row execution:** Each function call gets its own result buffer (thread-safe by design)
* **Prerun/Postrun hooks:** Per-statement setup/cleanup, called once per SQL statement
* **No guaranteed isolation:** Multiple connections may call your functions concurrently
* **Best practice:** Avoid global state; use function parameters and return values
VillageSQL does not guarantee thread isolation for extension functions. If you use global variables or shared state, protect them with mutexes or locks.
### Transaction Behavior
Extension functions should follow these best practices:
* Design functions to be stateless when possible
* Avoid persistent side effects (file writes, external API calls) in functions
* If using prerun/postrun state, handle cleanup appropriately
***
## Performance Considerations
**Optimization:** Use prerun hooks to cache expensive per-statement setup instead of repeating work per-row.
### Custom Type Performance
```sql theme={null}
-- Slow: VDF call per row
SELECT * FROM signals WHERE complex_abs(impedance) > 100;
-- Fast: Computed column with index
ALTER TABLE signals
ADD COLUMN impedance_magnitude DOUBLE AS (complex_abs(impedance)) STORED,
ADD INDEX(impedance_magnitude);
SELECT * FROM signals WHERE impedance_magnitude > 100;
```
***
## Security & Debugging
### Security Model
**Trust model:** Extensions run with full server privileges.
* No sandboxing or permission system
* Extensions can read any file, access network, execute code
* **Trust implications:** Only install extensions from trusted sources
**Installation security:** Runs as `villagesql_extension_installer` user (context switch).
**Enable verbose logging:**
```bash theme={null}
mysqld --log-error-verbosity=3
```
**GDB debugging:**
```bash theme={null}
gdb -p $(pidof mysqld)
(gdb) break my_func_init
(gdb) continue
```
**Check dependencies:**
```bash theme={null}
# Linux
ldd /path/to/extension.so
# macOS
otool -L /path/to/extension.so
```
**Common errors:**
* **Undefined symbol:** Check library dependencies with `ldd` (Linux) or `otool -L` (macOS)
* **Cannot open shared object:** Check library dependencies are present and correctly linked
* **Crash on VDF call:** Check NULL pointer handling
***
## Next Steps
Build your first extension
System tables and views
Study vsql\_complex implementation
Monitor and troubleshoot