> ## Documentation Index
> Fetch the complete documentation index at: https://villagesql.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Extension Architecture

> How VillageSQL's extension system works internally

Understanding VillageSQL's extension architecture helps debug issues and optimize performance when building extensions.

***

## Terminology

* **VEB** (VillageSQL Extension Bundle) - The `.veb` file format, a tar archive containing manifest, library, and metadata
* **VEF** (VillageSQL Extension Framework) - The SDK and C++ API for authoring extensions
* **VDF** (VillageSQL Defined Function) - Functions registered through the VEF SDK using `VEF_GENERATE_ENTRY_POINTS()`
* **UDF** (User-Defined Function) - Traditional MySQL functions registered via `CREATE FUNCTION ... SONAME` (legacy approach, still supported)

***

## VDF Function Lookup

VDFs support both qualified and unqualified function calls:

```sql theme={null}
-- Unqualified lookup
SELECT complex_abs(value) FROM table;

-- Qualified lookup
SELECT vsql_complex.complex_abs(value) FROM table;
```

**Resolution order for unqualified function calls:**

1. System functions (built-in MySQL)
2. UDFs (traditional user-defined functions)
3. VDFs (extension functions) - only if exactly one function with that name exists
4. Stored functions (created with `CREATE FUNCTION`)

**Performance:** For hot code paths, use qualified calls (`extension_name.function_name()`) to skip the resolution chain and directly invoke the extension function.

***

## VEB File Format

VillageSQL extensions are distributed as `.veb` (VillageSQL Extension Bundle) files - tar archives containing:

```
extension_name.veb (tar archive)
├── manifest.json       # Extension metadata (required)
└── lib/
    └── extension.so    # Compiled shared library (required)
```

### manifest.json Schema

```json theme={null}
{
  "name": "extension_name",          // Required: lowercase_with_underscores
  "version": "1.0.0",                // Required: semantic version
  "description": "Brief description",
  "author": "Your Name",
  "license": "GPL-2.0"
}
```

* **name:** Must match extension name used in SQL (lowercase\_with\_underscores)
* **version:** Semantic version (MAJOR.MINOR.PATCH)
* **description, author, license:** Optional metadata

## Extension Lifecycle

### Installation Flow

```
INSTALL EXTENSION name
    ↓
1. Validate .veb exists in veb_dir
    ↓
2. Calculate SHA256 hash of .veb
    ↓
3. Expand to _expanded/{name}/{sha256}/
    ↓
4. Parse and validate manifest.json
    ↓
5. Load .so library (dlopen with RTLD_LOCAL)
    ↓
6. Call vef_register() entry point
    ↓
7. Register VDFs and custom types
    ↓
8. Persist registration and update cache
    ↓
Success
```

**Rollback:** If any step fails, transaction rolls back and .so is unloaded.

**Symbol Isolation:** Extensions are loaded with `RTLD_LOCAL` flag, ensuring symbols from one extension don't conflict with symbols from other extensions. This prevents naming collisions when multiple extensions use common library names or function names.

> [!NOTE]
>
> The `veb_dir` system variable points to the directory where `.veb` extension files are stored.

### Uninstallation Flow

```
UNINSTALL EXTENSION name
    ↓
1. Check for column dependencies
    ↓
2. Call vef_unregister() cleanup hook
    ↓
3. Drop registered VDFs
    ↓
4. Drop custom types
    ↓
5. Remove extension registration and update cache
    ↓
6. Unload .so library (dlclose)
    ↓
7. Keep _expanded directory (for reinstall)
    ↓
Success
```

**Dependency prevention:** Cannot uninstall if table columns use the extension's custom types.

***

## Expansion Directory Structure

VillageSQL expands .veb files to support multiple versions:

```
veb_dir/
├── extension_name.veb
└── _expanded/
    └── extension_name/
        ├── abc123.../              # SHA256 of v1.0.0 .veb
        │   ├── manifest.json
        │   └── lib/extension.so
        └── def456.../              # SHA256 of v2.0.0 .veb
            ├── manifest.json
            └── lib/extension.so
```

**Why SHA256 directories?**

* Test new versions without overwriting
* Enable rollback
* Prevent "same version, different code"

**Cleanup:** Orphaned SHA256 directories are removed on server restart.

***

## Victionary Caching Layer

VictionaryClient maintains in-memory caches of system metadata for O(log n) lookups.

### Cached Tables

```cpp theme={null}
SystemTableMap<ExtensionEntry> m_extensions;
SystemTableMap<CustomTypeEntry> m_types;
SystemTableMap<CustomColumnEntry> m_columns;
SystemTableMap<PropertyEntry> m_properties;
```

### Cache Operations

| Operation | Locking | Performance |
| - | - | - |
| Read (resolve type) | Read lock | O(log n) map lookup |
| Write (install extension) | Write lock | O(log n) insert + disk I/O |
| Server startup | N/A | Full table scan into memory |

**Cache invalidation:** Automatic during DDL operations (INSTALL/UNINSTALL EXTENSION).

**Memory overhead:** \~100 bytes per entry.

***

## Custom Type System

### Type Resolution

```cpp theme={null}
CREATE TABLE t (col COMPLEX)
    ↓
1. Parser encounters COMPLEX
    ↓
2. PT_custom_type::create()
    ↓
3. ResolveTypeToContext(extension_name, type_name)
    ↓
4. VictionaryClient::lookup_type() → O(log n)
    ↓
5. Find TypeDescriptor in cache
    ↓
6. Create Field with implementation_type
```

### Implementation Types

Custom types map to MySQL storage types:

| Custom Type | MySQL Implementation | Bytes |
| - | - | - |
| COMPLEX | MYSQL\_TYPE\_VARCHAR | 16 |
| UUID | MYSQL\_TYPE\_VARCHAR | 16 |
| INET6 | MYSQL\_TYPE\_VARCHAR | 16 |
| JSON\_SCHEMA | MYSQL\_TYPE\_BLOB | Variable |

***

## Concurrency & Transaction Behavior

### Thread Safety Model

Extension functions are called in a per-row execution model:

* **Isolated per-row execution:** Each function call gets its own result buffer (thread-safe by design)
* **Prerun/Postrun hooks:** Per-statement setup/cleanup, called once per SQL statement
* **No guaranteed isolation:** Multiple connections may call your functions concurrently
* **Best practice:** Avoid global state; use function parameters and return values

> [!WARNING]
>
> VillageSQL does not guarantee thread isolation for extension functions. If you use global variables or shared state, protect them with mutexes or locks.

### Transaction Behavior

Extension functions should follow these best practices:

* Design functions to be stateless when possible
* Avoid persistent side effects (file writes, external API calls) in functions
* If using prerun/postrun state, handle cleanup appropriately

***

## Performance Considerations

**Optimization:** Use prerun hooks to cache expensive per-statement setup instead of repeating work per-row.

### Custom Type Performance

```sql theme={null}
-- Slow: UDF call per row
SELECT * FROM signals WHERE complex_abs(impedance) > 100;

-- Fast: Computed column with index
ALTER TABLE signals
ADD COLUMN impedance_magnitude DOUBLE AS (complex_abs(impedance)) STORED,
ADD INDEX(impedance_magnitude);

SELECT * FROM signals WHERE impedance_magnitude > 100;
```

***

## Security & Debugging

> [accordion] Security Model
>
> **Trust model:** Extensions run with full server privileges.
>
> * No sandboxing or permission system
> * Extensions can read any file, access network, execute code
> * **Trust implications:** Only install extensions from trusted sources
>
> **Installation security:** Runs as `villagesql_extension_installer` user (context switch).

> [accordion] Debugging Extensions
>
> **Enable verbose logging:**
>
> ```bash theme={null}
> mysqld --log-error-verbosity=3
> ```
>
> **GDB debugging:**
>
> ```bash theme={null}
> gdb -p $(pidof mysqld)
> (gdb) break my_func_init
> (gdb) continue
> ```
>
> **Check dependencies:**
>
> ```bash theme={null}
> # Linux
> ldd /path/to/extension.so
>
> # macOS
> otool -L /path/to/extension.so
> ```
>
> **Common errors:**
>
> * **Undefined symbol:** Check `extern "C"` linkage
> * **Cannot open shared object:** Check library dependencies
> * **Crash on UDF call:** Check NULL pointer handling

***

## Schema Validation

On server startup, SchemaManager validates system table schemas:

```cpp theme={null}
1. Open each VillageSQL system table
2. Check column count and names
3. Validate column types
4. Verify primary keys
5. Check indexes
```

**Failure scenarios:**

* Missing table → Create from villagesql\_schema.sql
* Wrong schema → Error and refuse to start
* Version mismatch → Run upgrade scripts

**Server version:**

```sql theme={null}
SELECT VERSION();
```

Source builds include the git commit hash:

```
8.4.6-villagesql-0.0.2
```

***

## Next Steps

> [card] [Create Extension](https://villagesql.com/docs/en/8.4/0.0.2/create.md)
>
> Build your first extension

> [card] [System Reference](https://villagesql.com/docs/en/8.4/0.0.2/reference.md)
>
> System tables and views

> [card] [Examples](https://villagesql.com/docs/en/8.4/0.0.2/examples.md)
>
> Study vsql\_complex implementation

> [card] [Managing Extensions](https://villagesql.com/docs/en/8.4/0.0.2/managing.md)
>
> Monitor and troubleshoot


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.