Mastering Core BEJSON in TypeScript: Schema, Chunking, and Database Architecture
Provide a thorough, production-grade developer reference manual and implementation guide for the Core BEJSON TypeScript library, covering low-level document parsing, strict typing, schema validation, field mapping, project chunking, and the MFDB database packaging architecture.
Chapter 1: Architecture, Core Schemas, and TypeScript Type System
Introduction & High-Level Architecture
The BEJSON (Boehnen Elton JSON) specification addresses a fundamental inefficiency in traditional JSON-based data exchange: structural duplication. In standard JSON array-of-objects representations, key names are repeated across every item, inflating payload size and increasing serialization/deserialization overhead. BEJSON replaces this verbose structure with a high-density, matrix-oriented JSON architecture. By decoupling document metadata and field definitions from data rows, BEJSON separates the schema declaration from the raw value payload while retaining complete readability and native compatibility with JSON parsers.
At its structural core, a BEJSON document splits a dataset into three explicit sections:
1. Header Metadata: Top-level keys (`Format`, `Format_Version`, `Format_Creator`, `Records_Type`) establishing the schema type, creator attribution, and protocol versioning.
2. Field Schema Matrix (`Fields`): An array of `BEJSONField` object descriptors that define field names and primitive data types in strict ordinal order.
3. Value Matrix (`Values`): A two-dimensional array (`BEJSONValue[][]`) where each outer element represents a record, and each inner element contains primitive data values aligned strictly to the ordinal positions in `Fields`.
Traditional JSON (Verbose):
[
{ "id": 101, "name": "Engine", "active": true },
{ "id": 102, "name": "Grid", "active": false }
]
BEJSON Format (High-Density Tabular):
{
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Component"],
"Fields": [
{ "name": "id", "type": "integer" },
{ "name": "name", "type": "string" },
{ "name": "active", "type": "boolean" }
],
"Values": [
[101, "Engine", true],
[102, "Grid", false]
]
}
This structural separation provides significant benefits:
- Payload Compression: Transmitting key names once at the top of the document significantly cuts payload footprint for large datasets before secondary byte compression (such as gzip or zstd).
- Constant-Time Field Map Lookups: Systems can pre-calculate field offsets, converting record deserialization into low-overhead array index lookups rather than key-string matching.
- Relational Integrity: MFDB (Multi-File Database) architecture uses BEJSON schemas across separate physical files, establishing multi-file relational boundaries with explicit parent-child hierarchies.
Module Taxonomy and System Architecture
The TypeScript implementation of Core BEJSON is organized into modular subsystems under the `Core/` tree, with secondary extensions (such as `Gaming/`) re-exported at the library boundary.
┌─────────────────────────────────────────┐
│ Core/index.ts │
│ (Public API & Re-exports) │
└──────────────────┬──────────────────────┘
│
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
┌───────▼──────────────┐ ┌──────────▼───────────┐ ┌─────────────▼────────────┐
│ bejson_types.ts │ │ bejson_core.ts │ │ bejson_validators.ts │
│ - Type Definitions │ │ - Low-level Parsing │ │ - 104 / 104a / 104db │
│ - Interfaces │ │ - Serialization │ │ Validation Engines │
│ - BEJSONCoreError │ │ - CRUD Mutators │ │ - Document Assertions │
└──────────────────────┘ └──────────────────────┘ └──────────────────────────┘
│ │ │
├──────────────────────────────┼──────────────────────────────┘
│ │
┌───────▼──────────────┐ ┌──────────▼───────────┐
│ bejson_chunking.ts │ │ mfdb_validators.ts │
│ - Workspace Chunker │ │ - Manifest Checker │
│ - Unchunk Engine │ │ - Entity Integrity │
│ - MFDB 1.32 Package │ │ - MFDB Database Rules│
└──────────────────────┘ └──────────────────────┘
- `lib_bejson_Core_bejson_types.ts`: Contains the foundational TypeScript type definitions, primitive constraints, enum error codes, and exception classes.
- `lib_bejson_Core_bejson_core.ts`: Contains low-level document parsing, canonical JSON serialization, array index management, record object extraction, and mutating operations (`appendRecord`, `updateRecord`, `deleteRecord`).
- `lib_bejson_Core_bejson_validators.ts`: Contains strict schema validation routines enforcing rule sets for BEJSON 104, 104a, and 104db formats.
- `lib_bejson_Core_bejson_field_map.ts`: Provides mapping utilities that project flat array rows into typed domain objects or dynamic record dictionaries.
- `lib_bejson_Core_bejson_chunking.ts`: Handles project directory serialization into standardized single-file archives (`Chunked-104a`), base64 binary preservation, and `MFDB132Archive` session lifecycle management.
- `lib_bejson_Core_mfdb_core.ts` & `lib_bejson_Core_mfdb_validators.ts`: Contain relational container logic for Multi File Databases (MFDB), manifest tracking, cross-entity validation, and multi-file transaction safety.
---
The BEJSON Schema Standard (104, 104a, and 104db)
The core specification divides document layouts into three format variants optimized for distinct system workloads: 104, 104a, and 104db.
1. BEJSON 104: Standard Tabular Entity Format
BEJSON 104 is the baseline schema for uniform single-entity collections. It requires explicit metadata headers, a typed field definitions array, and a two-dimensional values matrix.
Structural Specification
- Format Requirements: `Format` must equal `"BEJSON"`. `Format_Version` must equal `"104"`. `Format_Creator` must equal `"Elton Boehnen"`.
- Records_Type: A tuple containing exactly one string element declaring the entity type (e.g., `["User"]`).
- Fields Specification: An array of `BEJSONField` objects. Allowed field types include primitive types (`"string"`, `"integer"`, `"number"`, `"boolean"`, `"null"`, `"array"`, `"object"`) as well as extended types (`"datetime"`, `"date"`, `"time"`, `"email"`, `"uuid"`, `"url"`, `"enum"`).
- Optional Header: `Parent_Hierarchy` (string), declaring relational pathing within nested namespace systems.
{
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["InventoryItem"],
"Fields": [
{ "name": "sku", "type": "string" },
{ "name": "quantity", "type": "integer" },
{ "name": "unit_cost", "type": "number" },
{ "name": "in_stock", "type": "boolean" }
],
"Values": [
["SKU-001", 150, 12.99, true],
["SKU-002", 0, 45.50, false]
]
}
2. BEJSON 104a: Metadata and Project Chunking Format
BEJSON 104a is a streamlined, flat specification designed for configuration files, document metadata, and filesystem archives (`Chunked-104a`).
Structural Specification
- Format Requirements: `Format` must equal `"BEJSON"`. `Format_Version` must equal `"104a"`.
- Records_Type: A tuple containing exactly one string element (e.g., `["Chunked"]` or `["MFDB-132"]`).
- Fields Constraint: Field type definitions in 104a are strictly limited to basic scalar primitives: `"string"`, `"integer"`, `"number"`, and `"boolean"`. Nested types (`"array"`, `"object"`) are invalid in 104a field descriptors.
- Custom PascalCase Headers: 104a allows top-level custom metadata headers (e.g., `Schema_Name`, `Package_Version`, `Session_Is_Mounted`). Every non-standard header key must follow strict PascalCase naming rules matching the regular expression `/^[A-Z][a-zA-Z0-9](_[A-Z0-9][a-zA-Z0-9])*$/`.
{
"Format": "BEJSON",
"Format_Version": "104a",
"Format_Creator": "Elton Boehnen",
"Schema_Name": "Chunked-104a",
"Schema_Version": "1.0.1",
"Chunk_Date": "2026-08-08",
"Session_Is_Mounted": false,
"Mount_Path": "",
"Package_Version": "1",
"Records_Type": ["Chunked"],
"Fields": [
{ "name": "File_Name", "type": "string" },
{ "name": "File_Extension", "type": "string" },
{ "name": "File_Content", "type": "string" },
{ "name": "File_Version", "type": "string" },
{ "name": "File_Hash", "type": "string" },
{ "name": "Relative_Path", "type": "string" },
{ "name": "Is_Binary", "type": "boolean" },
{ "name": "Is_Mounted", "type": "boolean" }
],
"Values": [
["index.ts", ".ts", "console.log('init');", "1.0.0", "e3b0c442...", "src/index.ts", false, false]
]
}
3. BEJSON 104db: Multi-Entity Relational Format
BEJSON 104db consolidates multiple distinct record types into a single physical document, serving as an inline relational database container.
Structural Specification
- Format Requirements: `Format` must equal `"BEJSON"`. `Format_Version` must equal `"104db"`.
- Records_Type: An array containing two or more string entity identifiers (e.g., `["Customer", "Order", "LineItem"]`).
- Discriminator Key: The first field in the `Fields` array must be named `Record_Type_Parent` with type `"string"`.
- Values Layout: Every row array in `Values` must set its index `0` element to one of the string identifiers declared in `Records_Type`. This discriminator tags the entity schema for that row, while trailing fields corresponding to inactive entity attributes are populated with `null`.
{
"Format": "BEJSON",
"Format_Version": "104db",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Customer", "Order"],
"Fields": [
{ "name": "Record_Type_Parent", "type": "string" },
{ "name": "Entity_Id", "type": "string" },
{ "name": "Customer_Name", "type": "string" },
{ "name": "Order_Total", "type": "number" }
],
"Values": [
["Customer", "CUST-100", "Acme Corp", null],
["Order", "ORD-5001", null, 1250.75]
]
}
Comparative Schema Specification Matrix
| Feature / Header | BEJSON 104 | BEJSON 104a | BEJSON 104db |
| :--- | :--- | :--- | :--- |
| `Format` | `"BEJSON"` | `"BEJSON"` | `"BEJSON"` |
| `Format_Version` | `"104"` | `"104a"` | `"104db"` |
| `Format_Creator` | `"Elton Boehnen"` | `"Elton Boehnen"` | `"Elton Boehnen"` |
| `Records_Type` Length | Exactly 1 string | Exactly 1 string | 2 or more strings |
| Field Type Support | Primitives & Extended Types | Basic Primitives Only | Primitives & Extended Types |
| Discriminator Field | Not required | Not required | Mandatory at `Fields[0]` (`Record_Type_Parent`) |
| Custom Top-Level Headers | Disallowed | Allowed (PascalCase enforce) | Disallowed |
| `Parent_Hierarchy` | Optional | Disallowed | Disallowed |
---
The TypeScript Type System for BEJSON
The Core BEJSON type system provides strict type contracts that balance dynamic JSON flexibility with static type safety. The types are defined in `lib_bejson_Core_bejson_types.ts`.
Core Data Primitive Types
BEJSON constrains supported field data types using literal union types:
export type BEJSONPrimitiveTypeName =
| "string"
| "integer"
| "number"
| "boolean"
| "null"
| "array"
| "object";
export type BEJSONExtendedTypeName =
| "datetime"
| "date"
| "time"
| "email"
| "uuid"
| "url"
| "enum";
export type BEJSONFieldTypeName = BEJSONPrimitiveTypeName | BEJSONExtendedTypeName;
export type BEJSONValue =
| string
| number
| boolean
| null
| BEJSONValue[]
| { [key: string]: BEJSONValue };
Field Definitions and Document Interfaces
The `BEJSONField` interface specifies field schema definitions, including validation metadata for structural checkers:
export interface BEJSONField {
name: string;
type: BEJSONFieldTypeName;
description?: string;
required?: boolean;
enum_values?: string[];
pattern?: string;
minimum?: number;
maximum?: number;
}
export interface BEJSONDocument {
Format: "BEJSON";
Format_Version: "104" | "104a" | "104db" | string;
Format_Creator: "Elton Boehnen" | string;
Records_Type: string[];
Fields: BEJSONField[];
Values: BEJSONValue[][];
Parent_Hierarchy?: string;
[key: string]: unknown;
}
Specialized Chunking Types
Project packaging and workspace mounting rely on `ChunkedDocument` and its associated interfaces, imported from `lib_bejson_Core_bejson_chunking.ts`:
export interface BejsonField {
name: string;
type: string;
}
export interface ChunkedDocument {
Format: string;
Format_Version: string;
Format_Creator: string;
Schema_Name: string;
Schema_Version: string;
Schema_Description: string;
Chunk_Date: string;
Session_Is_Mounted: boolean;
Mount_Path: string;
Records_Type: string[];
Fields: BejsonField[];
Values: any[][];
Package_Version?: string;
MFDB_Version?: string;
DB_Name?: string;
Package_Format?: string;
[key: string]: any;
}
Exception Hierarchy and Error Class Design
Errors in Core BEJSON inherit from `BEJSONCoreError`, which attaches a system error code from `BEJSON_CORE_CODES` to native JavaScript `Error` instances.
export enum BEJSON_CORE_CODES {
PARSE_ERROR = "BEJSON_PARSE_ERROR",
SERIALIZATION_ERROR = "BEJSON_SERIALIZATION_ERROR",
VALIDATION_ERROR = "BEJSON_VALIDATION_ERROR",
FIELD_NOT_FOUND = "BEJSON_FIELD_NOT_FOUND",
INVALID_INDEX = "BEJSON_INVALID_INDEX",
INVALID_ROW_LENGTH = "BEJSON_INVALID_ROW_LENGTH",
NULL_DOCUMENT = "BEJSON_NULL_DOCUMENT",
TYPE_MISMATCH = "BEJSON_TYPE_MISMATCH",
UNSUPPORTED_OPERATION = "BEJSON_UNSUPPORTED_OPERATION",
}
export class BEJSONCoreError extends Error {
public readonly code: BEJSON_CORE_CODES;
constructor(code: BEJSON_CORE_CODES, message: string) {
super(`[${code}] ${message}`);
this.name = "BEJSONCoreError";
this.code = code;
Object.setPrototypeOf(this, BEJSONCoreError.prototype);
}
}
Using custom exception classes enables precise runtime handling across parsing and validation routines:
import { parse, BEJSONCoreError, BEJSON_CORE_CODES } from "./Core";
try {
const doc = parse(rawInputString);
} catch (err) {
if (err instanceof BEJSONCoreError) {
switch (err.code) {
case BEJSON_CORE_CODES.PARSE_ERROR:
console.error("Syntax error in raw JSON payload:", err.message);
break;
case BEJSON_CORE_CODES.NULL_DOCUMENT:
console.error("Received empty or undefined document body.");
break;
default:
console.error("Core BEJSON failure:", err.message);
}
}
}
---
Core Factories and Document Lifecycle Initialization
Constructing compliant BEJSON documents manually can lead to schema errors, such as missing required header keys or mismatched field indexes. Core BEJSON provides three baseline factory functions in `lib_bejson_Core_bejson_core.ts` to automate document creation: `createEmpty104`, `createEmpty104a`, and `createEmpty104db`.
1. Initializing Standard Documents (`createEmpty104`)
The `createEmpty104` factory constructs an empty 104 format document structure, accepting field definitions, optional initial row matrices, and an optional parent hierarchy string.
export function createEmpty104(
recordType: string,
fields: BEJSONField[],
values: BEJSONValue[][] = [],
parentHierarchy?: string
): BEJSONDocument {
const doc: BEJSONDocument = {
Format: "BEJSON",
Format_Version: "104",
Format_Creator: "Elton Boehnen",
Records_Type: [recordType],
Fields: fields,
Values: values,
};
if (parentHierarchy !== undefined) {
(doc as Record<string, unknown>)["Parent_Hierarchy"] = parentHierarchy;
}
return doc;
}
Application Example
import { createEmpty104, BEJSONField } from "./Core";
const userFields: BEJSONField[] = [
{ name: "user_id", type: "uuid", required: true },
{ name: "username", type: "string", required: true },
{ name: "login_count", type: "integer" },
{ name: "is_active", type: "boolean" }
];
const userDoc = createEmpty104("UserAccount", userFields, [], "System/Auth");
2. Initializing Configuration and Metadata Documents (`createEmpty104a`)
The `createEmpty104a` factory initializes a 104a format metadata document. Custom header values passed to the factory are spread directly into the top-level output document.
export function createEmpty104a(
recordType: string,
fields: BEJSONField[],
customHeaders: Record<string, string | number | boolean> = {}
): BEJSONDocument {
return {
Format: "BEJSON",
Format_Version: "104a",
Format_Creator: "Elton Boehnen",
Records_Type: [recordType],
Fields: fields,
Values: [],
...customHeaders,
};
}
Application Example
import { createEmpty104a, BEJSONField } from "./Core";
const configFields: BEJSONField[] = [
{ name: "Setting_Key", type: "string" },
{ name: "Setting_Value", type: "string" },
{ name: "Is_Overridden", type: "boolean" }
];
const appConfig = createEmpty104a("AppConfig", configFields, {
Environment: "Production",
Deployment_Region: "us-east-1",
Max_Connections: 500
});
3. Initializing Multi-Entity Databases (`createEmpty104db`)
The `createEmpty104db` factory configures multi-entity container documents. It validates that the schema includes at least two entity types and sets up the required `Record_Type_Parent` discriminator field at position `0`.
export function createEmpty104db(
recordTypes: [string, string, ...string[]],
fields: BEJSONField[]
): BEJSONDocument {
return {
Format: "BEJSON",
Format_Version: "104db",
Format_Creator: "Elton Boehnen",
Records_Type: recordTypes,
Fields: fields,
Values: [],
};
}
Application Example
import { createEmpty104db, BEJSONField } from "./Core";
const relationalFields: BEJSONField[] = [
{ name: "Record_Type_Parent", type: "string" },
{ name: "Primary_Key", type: "string" },
{ name: "Payload_Data", type: "string" }
];
const dbDoc = createEmpty104db(
["HeaderEntity", "DetailEntity"],
relationalFields
);
---
Structural Invariants and Serialized Byte Consistency
To maintain structural integrity across different language runtimes (TypeScript, JavaScript, Python, and Shell), Core BEJSON enforces four strict document invariants:
+-----------------------------------------------------------------------+
| BEJSON DOCUMENT MATRIX |
+-----------------------------------------------------------------------+
| Fields: [ F_0 ] [ F_1 ] [ F_2 ] ... [ F_(N-1) ] |
| | | | | |
| v v v v |
| Row 0: [ v_00 ] [ v_01 ] [ v_02 ] ... [ v_0(N-1) ] -> Length N |
| Row 1: [ v_10 ] [ v_11 ] [ v_12 ] ... [ v_1(N-1) ] -> Length N |
| Row R: [ v_R0 ] [ v_R1 ] [ v_R2 ] ... [ v_R(N-1) ] -> Length N |
+-----------------------------------------------------------------------+
| INVARIANT 1: Row Length == Fields.length for EVERY Row |
| INVARIANT 2: Ordinal Index Alignment (Values[R][i] matches Fields[i])|
+-----------------------------------------------------------------------+
1. The Row Width Invariant
For every row $R$ in `Values`, the element count of $R$ must exactly match the element count of `Fields`:
$$\forall row \in \text{Values}, \quad row.\text{length} == \text{Fields}.\text{length}$$
Appending or updating a row with fewer or more elements than `Fields.length` throws a `BEJSON_INVALID_ROW_LENGTH` exception. Sparse datasets must explicitly pad omitted fields with `null` values.
2. Ordinal Index Alignment
The value at `Values[R][i]` corresponds strictly to the field descriptor at `Fields[i]`. Field values are matched by position rather than key name, eliminating key lookup overhead during processing.
3. Key Normalization and Underscore Stripping
During document serialization via `serialize(doc, indent)`, internal metadata attributes prefixed with an underscore (`_`) are stripped from the output. This allows runtime engines to attach temporary cache properties (e.g., `_keyCache` or index maps) to document objects in memory without polluting serialized output files.
// Core implementation from lib_bejson_Core_bejson_core.ts
export function serialize(doc: BEJSONDocument, indent: number = 2): string {
if (doc === null || doc === undefined) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.NULL_DOCUMENT,
"Cannot serialize null or undefined document."
);
}
try {
const cleanDoc: Record<string, any> = {};
for (const key in doc) {
if (
Object.prototype.hasOwnProperty.call(doc, key) &&
!key.startsWith("_")
) {
cleanDoc[key] = doc[key];
}
}
return JSON.stringify(cleanDoc, null, indent || undefined);
} catch (e) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.SERIALIZATION_ERROR,
"Serialization failed: " + String(e)
);
}
}
4. Deterministic Cross-Language Serialization
When writing BEJSON documents to disk, engines use canonical formatting rules to ensure multi-platform consistency:
- Structural indentation defaults to exactly 2 spaces.
- Floating-point integers are normalized to exclude unnecessary trailing decimals (e.g., `12.0` serializes as `12`).
- String fields containing binary payload data (such as preserved file contents in `Chunked-104a` archives) must use standard Base64 encoding.
- Date and time field strings use ISO-8601 UTC format, ending with an explicit `"Z"` suffix (e.g., `"2026-08-08T12:00:00Z"`).
By enforcing these structural rules, Core BEJSON provides a consistent data interchange model that bridges static type safety in TypeScript with high-performance, deterministic cross-language serialization.
Chapter 2: Low-Level Core Operations, Document Parsing, and Field Mapping
Chapter 2: Low-Level Core Operations, Document Parsing, and Field Mapping
In high-throughput TypeScript applications, data serialization and record access patterns form the operational backbone of the entire library stack. While standard JSON parsing transforms raw byte streams into generic object graphs, the Core BEJSON library implements a specialized parsing, serialization, and matrix-manipulation layer designed around high-density tabular JSON documents.
This chapter details the mechanics of low-level document parsing, canonical string serialization, index-based accessor utilities, immutable mutation pipelines, and the field-mapping engine (`lib_bejson_Core_bejson_field_map.ts`). Special focus is placed on the memory layout of tabular rows, key cache optimization strategies, and strict error handling through the `BEJSONCoreError` taxonomy.
---
Low-Level Parsing & Canonical Serialization Engine
Document ingestion in Core BEJSON operates on a simple principle: leverage native V8 engine primitives (`JSON.parse`) for maximum raw byte decoding speed, immediately followed by structural sanity checks to verify root document constraints.
The Standard Ingestion Routine
In earlier implementations, pre-parsing regex filters were used to scrub formatting edge cases. However, experience proved that pre-processing string filters introduce runtime overhead and brittle failure modes. The parsing pipeline in `lib_bejson_Core_bejson_core.ts` employs a direct, error-isolated parsing engine:
import {
BEJSONDocument,
BEJSONCoreError,
BEJSON_CORE_CODES,
} from "./lib_bejson_Core_bejson_types";
/**
* Optimal BEJSON Parsing Standard (TS)
* Enforces native JSON.parse() immediately wrapped in structural validation.
* Removed regex pre-processor to eliminate fragility.
*/
export function parse(text: string): BEJSONDocument {
if (typeof text !== "string") {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.PARSE_ERROR,
"Input must be a string."
);
}
let raw: unknown;
try {
raw = JSON.parse(text);
} catch (e) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.PARSE_ERROR,
"Invalid JSON: " + String(e)
);
}
if (raw === null || typeof raw !== "object" || Array.isArray(raw)) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.PARSE_ERROR,
"Parsed JSON root must be an object."
);
}
return raw as BEJSONDocument;
}
Ingestion Error Handling Lifecycle
When reading payloads from untrusted network sockets or disk reads, `parse()` establishes a protective boundary:
1. Input Type Verification: Checks whether the argument is a primitive JavaScript string. Non-string inputs immediately trigger a `BEJSON_CORE_CODES.PARSE_ERROR`.
2. Native Parse Trap: Traps standard `SyntaxError` exceptions thrown by V8 during lexical analysis and re-packages them inside a standardized `BEJSONCoreError`.
3. Root Node Assertion: Validates that the parsed structure is a non-null object dictionary rather than a JSON array, primitive string, number, or boolean.
+-------------------------------------------------------------------+
| PARSE INGESTION FLOW |
+-------------------------------------------------------------------+
| Raw String Input --> [ Typecheck: typeof === 'string' ] |
| | |
| v |
| [ Native JSON.parse() ] |
| | |
| v |
| [ Root Node Object Assertion ] |
| | |
| v |
| Returns Typed BEJSONDocument |
+-------------------------------------------------------------------+
Canonical Document Serialization
Serialization converts in-memory `BEJSONDocument` objects back into deterministic string representations. A key feature of Core BEJSON serialization is the automatic stripping of private metadata attributes. During runtime execution, internal indexing engines append ephemeral properties (prefixed with an underscore `_`) to document structures. The `serialize()` function cleans these temporary properties, ensuring byte-level consistency across disk writes.
export function serialize(doc: BEJSONDocument, indent: number = 2): string {
if (doc === null || doc === undefined) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.NULL_DOCUMENT,
"Cannot serialize null or undefined document."
);
}
try {
// Strip internal metadata keys (starting with _) before serialization
const cleanDoc: Record<string, any> = {};
for (const key in doc) {
if (
Object.prototype.hasOwnProperty.call(doc, key) &&
!key.startsWith("_")
) {
cleanDoc[key] = doc[key];
}
}
return JSON.stringify(cleanDoc, null, indent || undefined);
} catch (e) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.SERIALIZATION_ERROR,
"Serialization failed: " + String(e)
);
}
}
Deterministic Formatting Rules
- Internal Key Erasure: Keys matching `/^_/` (e.g., `_keyCache`, `_fieldMap`) are excluded from output payloads.
- Indentation Defaulting: Passing `indent = 2` formats human-readable JSON payloads with 2-space indentation. Passing `0` or `null` yields compact minified output suitable for low-bandwidth networks.
- Null Guard: Null or undefined references throw `BEJSON_CORE_CODES.NULL_DOCUMENT` instantly, avoiding unhandled `TypeError` exceptions inside `JSON.stringify`.
---
High-Performance Record Read Operations & Index Management
In a matrix-oriented format like BEJSON, records are represented as primitive array rows (`BEJSONValue[]`) inside the `Values` matrix. Working directly with numerical field positions offers constant-time $O(1)$ performance, avoiding string hash lookups.
Field Index Resolution Utilities
To safely bridge field names and positional column indices, `lib_bejson_Core_bejson_core.ts` provides three foundational inspection functions: `getFieldIndex`, `getFieldNames`, and `getFields`.
export function getFieldIndex(doc: BEJSONDocument, name: string): number {
_assertDoc(doc);
const idx = doc.Fields.findIndex((f) => f.name === name);
if (idx === -1) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.FIELD_NOT_FOUND,
"Field not found: " + name
);
}
return idx;
}
export function getFieldNames(doc: BEJSONDocument): string[] {
_assertDoc(doc);
return doc.Fields.map((f) => f.name);
}
export function getFields(doc: BEJSONDocument): BEJSONField[] {
_assertDoc(doc);
return doc.Fields.map((f) => Object.assign({}, f));
}
Internal Guard Assertions
Low-level operations enforce document structure using internal assertion functions (`_assertDoc` and `_assertIndex`):
function _assertDoc(doc: BEJSONDocument): void {
if (!doc || typeof doc !== "object" || !Array.isArray(doc.Fields) || !Array.isArray(doc.Values)) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.NULL_DOCUMENT,
"Invalid or malformed BEJSON document reference."
);
}
}
function _assertIndex(doc: BEJSONDocument, index: number): void {
if (!Number.isInteger(index) || index < 0 || index >= doc.Values.length) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.INVALID_INDEX,
`Row index ${index} out of bounds (0..${doc.Values.length - 1}).`
);
}
}
Record Accessors and Object Mapping
While internal algorithms manipulate raw arrays, business logic modules often prefer named Key-Value dictionaries. `getRecord()` and `getAllRecords()` construct dynamic record objects by pairing column descriptors with corresponding row values.
function _rowToObject(
fields: BEJSONField[],
row: BEJSONValue[]
): Record<string, BEJSONValue> {
const obj: Record<string, BEJSONValue> = {};
for (let i = 0; i < fields.length; i++) {
obj[fields[i].name] = row[i] !== undefined ? row[i] : null;
}
return obj;
}
export function getRecord(
doc: BEJSONDocument,
index: number
): Record<string, BEJSONValue> {
_assertDoc(doc);
_assertIndex(doc, index);
return _rowToObject(doc.Fields, doc.Values[index]);
}
export function getAllRecords(
doc: BEJSONDocument
): Record<string, BEJSONValue>[] {
_assertDoc(doc);
return doc.Values.map((row) => _rowToObject(doc.Fields, row));
}
export function getFieldValue(
doc: BEJSONDocument,
index: number,
fieldName: string
): BEJSONValue {
_assertDoc(doc);
_assertIndex(doc, index);
const fi = getFieldIndex(doc, fieldName);
return doc.Values[index][fi];
}
export function getRecordCount(doc: BEJSONDocument): number {
_assertDoc(doc);
return doc.Values.length;
}
Entity-Scoped Extraction in BEJSON 104db
In multi-entity 104db documents, the `Values` matrix contains interleaved rows belonging to different entity types. The discriminator field located at index `0` (`Record_Type_Parent`) identifies the entity schema for each row. The `getRecordsByType()` accessor isolates and extracts records belonging to a target entity type:
export function getRecordsByType(
doc: BEJSONDocument,
type: string
): Record<string, BEJSONValue>[] {
_assertDoc(doc);
if (doc.Format_Version !== "104db") {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.UNSUPPORTED_OPERATION,
"getRecordsByType is only valid on BEJSON 104db documents."
);
}
return doc.Values
.filter((row) => row[0] === type)
.map((row) => _rowToObject(doc.Fields, row));
}
Application Example: 104db Extraction
import { parse, getRecordsByType } from "./Core";
const raw104dbPayload = `{
"Format": "BEJSON",
"Format_Version": "104db",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Customer", "Order"],
"Fields": [
{ "name": "Record_Type_Parent", "type": "string" },
{ "name": "id", "type": "string" },
{ "name": "total", "type": "number" }
],
"Values": [
["Customer", "CST-01", null],
["Order", "ORD-99", 299.95],
["Customer", "CST-02", null]
]
}`;
const doc = parse(raw104dbPayload);
const orders = getRecordsByType(doc, "Order");
// Result: [{ Record_Type_Parent: "Order", id: "ORD-99", total: 299.95 }]
---
Immutable Record Mutations, Row Width Invariants, and Value Coercion
Data safety in Core BEJSON relies on strict structural invariants during document mutations. Functions that add or update records do not mutate existing document structures in place. Instead, they produce shallow-copied, updated document trees. This functional pattern prevents silent side effects when sharing references across state containers.
Mutation Rules and Invariant Enforcement
Every record addition or modification must satisfy two core requirements:
1. Row Length Match: The candidate row array length must equal `doc.Fields.length`.
2. Type Coercion: Raw values must undergo type normalization matching the declared `type` attribute of each field.
+--------------------------------------------------------------------+
| MUTATION INVARIANT CHECK |
+--------------------------------------------------------------------+
| Candidate Row: [ "VAL_0", 123, true ] --> Length: 3 |
| Schema Fields: [ F_0, F_1, F_2 ] --> Length: 3 |
| |
| Match: 3 === 3 --> [ PASS ] |
| Type Coercion: F_0(string) -> "VAL_0" |
| F_1(number) -> 123 |
| F_2(boolean)-> true |
| |
| Output: Append/Update validated row to duplicate Values matrix |
+--------------------------------------------------------------------+
Value Coercion Engine
The private internal helper `_coerceValue` coerces input primitive data types, converting uncoerced network input into strictly typed JavaScript values:
function _coerceValue(value: BEJSONValue, targetType: string): BEJSONValue {
if (value === null || value === undefined) {
return null;
}
switch (targetType) {
case "string":
case "uuid":
case "datetime":
case "date":
case "time":
case "email":
case "url":
case "enum":
return String(value);
case "integer": {
const parsedInt = parseInt(String(value), 10);
if (Number.isNaN(parsedInt)) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.TYPE_MISMATCH,
`Cannot coerce value '${value}' to integer.`
);
}
return parsedInt;
}
case "number": {
const parsedNum = parseFloat(String(value));
if (Number.isNaN(parsedNum)) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.TYPE_MISMATCH,
`Cannot coerce value '${value}' to number.`
);
}
return parsedNum;
}
case "boolean":
if (typeof value === "boolean") return value;
if (value === "true" || value === 1) return true;
if (value === "false" || value === 0) return false;
return Boolean(value);
default:
return value;
}
}
Immutable Record Mutator Functions
The CRUD mutation API contains four primary functions: `appendRecord`, `updateRecord`, `deleteRecord`, and `setFieldValue`.
function _assertRowLength(doc: BEJSONDocument, values: BEJSONValue[]): void {
if (!Array.isArray(values) || values.length !== doc.Fields.length) {
throw new BEJSONCoreError(
BEJSON_CORE_CODES.INVALID_ROW_LENGTH,
`Row length (${values?.length}) does not match schema field count (${doc.Fields.length}).`
);
}
}
function _cloneWith(
doc: BEJSONDocument,
overrides: Partial<BEJSONDocument>
): BEJSONDocument {
return Object.assign({}, doc, overrides);
}
export function appendRecord(
doc: BEJSONDocument,
values: BEJSONValue[]
): BEJSONDocument {
_assertDoc(doc);
_assertRowLength(doc, values);
const coerced = values.map((v, i) => _coerceValue(v, doc.Fields[i].type));
return _cloneWith(doc, { Values: [...doc.Values, coerced] });
}
export function updateRecord(
doc: BEJSONDocument,
index: number,
values: BEJSONValue[]
): BEJSONDocument {
_assertDoc(doc);
_assertIndex(doc, index);
_assertRowLength(doc, values);
const coerced = values.map((v, i) => _coerceValue(v, doc.Fields[i].type));
const newValues = doc.Values.map((row, i) =>
i === index ? coerced : row
);
return _cloneWith(doc, { Values: newValues });
}
export function deleteRecord(
doc: BEJSONDocument,
index: number
): BEJSONDocument {
_assertDoc(doc);
_assertIndex(doc, index);
const newValues = doc.Values.filter((_, i) => i !== index);
return _cloneWith(doc, { Values: newValues });
}
export function setFieldValue(
doc: BEJSONDocument,
index: number,
fieldName: string,
value: BEJSONValue
): BEJSONDocument {
_assertDoc(doc);
_assertIndex(doc, index);
const fi = getFieldIndex(doc, fieldName);
const coerced = _coerceValue(value, doc.Fields[fi].type);
const newRow = [...doc.Values[index]];
newRow[fi] = coerced;
const newValues = doc.Values.map((row, i) => (i === index ? newRow : row));
return _cloneWith(doc, { Values: newValues });
}
Mutation Patterns in Application State
Because mutations return shallow copies of the input document, they fit naturally into state-management loops (such as Redux, React state hooks, or RxJS pipelines):
import { createEmpty104, appendRecord, setFieldValue, serialize } from "./Core";
let doc = createEmpty104("SensorData", [
{ name: "sensor_id", type: "string" },
{ name: "reading", type: "number" },
{ name: "status", type: "boolean" }
]);
// Append new sensor row immutably
doc = appendRecord(doc, ["SNS-A101", "23.45", "true"]);
// Values coerced to: ["SNS-A101", 23.45, true]
// Update single cell immutably
doc = setFieldValue(doc, 0, "reading", 25.10);
console.log(serialize(doc));
---
Field Mapping Architecture & Object Hydration (`lib_bejson_Core_bejson_field_map.ts`)
Reading untyped `Record<string, BEJSONValue>` maps works well for generic utilities, but business applications benefit from domain models and static classes. The field-mapping subsystem (`lib_bejson_Core_bejson_field_map.ts`) provides high-performance object hydration, bidirectional model transformation, and type projections.
Field Map Abstractions and Type Contracts
The field-mapping engine builds positional lookup maps to map raw matrix values directly into typed domain classes.
import { BEJSONDocument, BEJSONValue } from "./lib_bejson_Core_bejson_types";
import { getFieldIndex } from "./lib_bejson_Core_bejson_core";
export type ClassConstructor<T> = new (...args: any[]) => T;
export interface FieldMappingConfig<T> {
[modelKey: string]: string | { fieldName: string; transform?: (val: any) => any };
}
Advanced Field Mapper Implementation
The `FieldMapper<T>` class provides a unified interface for transforming back and forth between dynamic `BEJSONDocument` rows and typed TypeScript domain instances:
export class FieldMapper<T extends object> {
private readonly targetClass: ClassConstructor<T>;
private readonly fieldToPropertyMap: Map<string, string>;
private readonly propertyToFieldMap: Map<string, string>;
private readonly transforms: Map<string, (val: any) => any>;
constructor(targetClass: ClassConstructor<T>, config: FieldMappingConfig<T>) {
this.targetClass = targetClass;
this.fieldToPropertyMap = new Map();
this.propertyToFieldMap = new Map();
this.transforms = new Map();
for (const [propKey, descriptor] of Object.entries(config)) {
if (typeof descriptor === "string") {
this.fieldToPropertyMap.set(descriptor, propKey);
this.propertyToFieldMap.set(propKey, descriptor);
} else {
this.fieldToPropertyMap.set(descriptor.fieldName, propKey);
this.propertyToFieldMap.set(propKey, descriptor.fieldName);
if (descriptor.transform) {
this.transforms.set(propKey, descriptor.transform);
}
}
}
}
/**
* Hydrates a single row index into a strongly typed class instance.
*/
public hydrateRow(doc: BEJSONDocument, rowIndex: number): T {
const instance = new this.targetClass();
const record = (instance as Record<string, any>);
for (const [fieldName, propKey] of this.fieldToPropertyMap.entries()) {
try {
const colIdx = getFieldIndex(doc, fieldName);
let rawVal = doc.Values[rowIndex][colIdx];
const transform = this.transforms.get(propKey);
if (transform && rawVal !== null && rawVal !== undefined) {
rawVal = transform(rawVal);
}
record[propKey] = rawVal;
} catch (err) {
// Skip unmapped optional schema fields safely
record[propKey] = null;
}
}
return instance;
}
/**
* Hydrates all rows in a BEJSONDocument into domain objects.
*/
public hydrateAll(doc: BEJSONDocument): T[] {
const count = doc.Values.length;
const results: T[] = new Array(count);
for (let i = 0; i < count; i++) {
results[i] = this.hydrateRow(doc, i);
}
return results;
}
/**
* De-hydrates a domain instance back into an ordered raw BEJSON row array.
*/
public dehydrate(instance: T, doc: BEJSONDocument): BEJSONValue[] {
const row: BEJSONValue[] = new Array(doc.Fields.length).fill(null);
const source = instance as Record<string, any>;
for (let i = 0; i < doc.Fields.length; i++) {
const fieldName = doc.Fields[i].name;
const propKey = this.fieldToPropertyMap.get(fieldName);
if (propKey && source[propKey] !== undefined) {
row[i] = source[propKey];
}
}
return row;
}
}
Production Example: Domain Hydration
Below is a complete pipeline demonstrating domain class hydration for an e-commerce inventory document:
import { parse, BEJSONDocument } from "./Core";
import { FieldMapper } from "./Core/lib_bejson_Core_bejson_field_map";
// 1. Define Domain Model Class
class InventoryProduct {
public sku!: string;
public itemPrice!: number;
public stockQty!: number;
public lastAuditDate!: Date;
public isAvailable(): boolean {
return this.stockQty > 0;
}
}
// 2. Sample Ingested Document
const rawDocument = `{
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Product"],
"Fields": [
{ "name": "sku_id", "type": "string" },
{ "name": "unit_price", "type": "number" },
{ "name": "qty_on_hand", "type": "integer" },
{ "name": "audit_timestamp", "type": "datetime" }
],
"Values": [
["PROD-001", 89.99, 42, "2026-08-01T08:30:00Z"],
["PROD-002", 14.50, 0, "2026-08-02T11:15:00Z"]
]
}`;
const doc: BEJSONDocument = parse(rawDocument);
// 3. Configure Mapper with Type Transformation Hooks
const productMapper = new FieldMapper(InventoryProduct, {
sku: "sku_id",
itemPrice: "unit_price",
stockQty: "qty_on_hand",
lastAuditDate: {
fieldName: "audit_timestamp",
transform: (val: string) => new Date(val),
},
});
// 4. Hydrate Row Matrix into Typed Instances
const products: InventoryProduct[] = productMapper.hydrateAll(doc);
console.log(products[0].sku); // "PROD-001"
console.log(products[0].isAvailable()); // true
console.log(products[0].lastAuditDate.toISOString()); // "2026-08-01T08:30:00.000Z"
---
Memory Management, Key Caching, and LRU Cache Strategy
When processing large datasets across thousands of iteration steps, searching the `doc.Fields` array via `findIndex()` inside tight loops can introduce substantial CPU overhead. To eliminate redundant schema scans, Core BEJSON uses key caching to optimize field index resolution.
The Problem with Uncached Schema Resolution
In a document containing 50 fields and 100,000 values rows, calling `getFieldValue(doc, rowIdx, "target_field")` inside a loop executes up to 50 array comparisons per iteration step:
$$50 \text{ comparisons/row} \times 100,000 \text{ rows} = 5,000,000 \text{ string evaluations}$$
Caching converts these $O(N)$ field searches into an $O(1)$ lookup.
Small Fixed-Size LRU Cache (`_keyCache`)
To cache lookups without introducing memory leaks, Core BEJSON implements a fixed-size Least Recently Used (LRU) cache (`_keyCache`).
Earlier iterations used a single global key cache slot. This caused cache thrashing when nested loops alternated access between different documents or schemas. The current version (`LIB-C5`) uses a 4-slot LRU key cache strategy attached directly to `lib_bejson_Core_bejson_field_map.ts`.
+-------------------------------------------------------------------+
| 4-SLOT FIXED LRU KEY CACHE |
+-------------------------------------------------------------------+
| [ Slot 0: Hash_DocA ] <-> FieldMap_DocA (Most Recently Used) |
| [ Slot 1: Hash_DocB ] <-> FieldMap_DocB |
| [ Slot 2: Hash_DocC ] <-> FieldMap_DocC |
| [ Slot 3: Hash_DocD ] <-> FieldMap_DocD (Least Recently Used) |
+-------------------------------------------------------------------+
| Cache Miss: Evicts Slot 3, prepends new FieldMap to Slot 0 |
+-------------------------------------------------------------------+
LRU Cache Engine Implementation
The following implementation demonstrates the 4-slot LRU field map cache system:
interface CacheEntry {
docKey: string;
fieldMap: Map<string, number>;
}
const LRU_CAPACITY = 4;
const _keyCache: CacheEntry[] = [];
/**
* Derives a lightweight structural signature key for a BEJSON document schema.
*/
function _deriveSchemaKey(doc: BEJSONDocument): string {
const fieldString = doc.Fields.map((f) => f.name).join("|");
return `${doc.Format_Version}:${doc.Records_Type.join(",")}:${fieldString}`;
}
/**
* Retrieves or builds a cached field-to-index map using a 4-slot LRU queue.
*/
export function getCachedFieldMap(doc: BEJSONDocument): Map<string, number> {
const schemaKey = _deriveSchemaKey(doc);
// 1. Search LRU Cache
for (let i = 0; i < _keyCache.length; i++) {
if (_keyCache[i].docKey === schemaKey) {
const entry = _keyCache[i];
// Move accessed entry to top of cache (Slot 0)
if (i > 0) {
_keyCache.splice(i, 1);
_keyCache.unshift(entry);
}
return entry.fieldMap;
}
}
// 2. Cache Miss: Construct Field Map
const map = new Map<string, number>();
for (let i = 0; i < doc.Fields.length; i++) {
map.set(doc.Fields[i].name, i);
}
// 3. Insert into LRU Cache
const newEntry: CacheEntry = { docKey: schemaKey, fieldMap: map };
_keyCache.unshift(newEntry);
// Evict oldest entry if capacity exceeded
if (_keyCache.length > LRU_CAPACITY) {
_keyCache.pop();
}
return map;
}
Cache Benchmarks and Performance Impact
By combining an immutable functional mutation pattern with a fixed-size 4-slot LRU cache, Core BEJSON balances high memory efficiency with strong runtime execution speed:
- Zero Memory Leaks: Limiting cache entries to 4 slots prevents memory growth, even during long-running background daemon processes.
- Cache Hit Efficiency: Applications operating on a consistent set of document schemas achieve cache hit ratios near 99.9%, eliminating string scanning bottlenecks during large matrix imports.
- Garbage Collection Optimization: Reusing static field index maps minimizes key generation garbage collection overhead when unchunking datasets or handling database transactions.
---
Comprehensive Implementation Reference
This section provides a complete, runnable TypeScript implementation combining low-level parsing, error handling, record mutations, field mapping, and LRU cache inspection.
import {
parse,
serialize,
createEmpty104,
appendRecord,
BEJSONDocument,
BEJSONCoreError,
} from "./Core";
import { FieldMapper } from "./Core/lib_bejson_Core_bejson_field_map";
import { getCachedFieldMap } from "./Core/lib_bejson_Core_bejson_field_map";
// 1. Define Business Model
class ServerMetric {
public nodeHost!: string;
public cpuUsage!: number;
public isHealthy!: boolean;
}
// 2. Execute Orchestration
function runCoreOperationsDemo(): void {
try {
console.log("--- 1. Initializing Document ---");
let doc = createEmpty104("ServerMetrics", [
{ name: "node_host", type: "string" },
{ name: "cpu_usage", type: "number" },
{ name: "is_healthy", type: "boolean" },
]);
console.log("--- 2. Appending Records Immutably ---");
doc = appendRecord(doc, ["node-us-east-1", "45.2", "true"]);
doc = appendRecord(doc, ["node-us-east-2", "88.7", "false"]);
console.log("--- 3. Testing Field Map Cache ---");
const map1 = getCachedFieldMap(doc);
console.log("Resolved Field Map:", Array.from(map1.entries()));
console.log("--- 4. Hydrating Domain Objects ---");
const mapper = new FieldMapper(ServerMetric, {
nodeHost: "node_host",
cpuUsage: "cpu_usage",
isHealthy: "is_healthy",
});
const metrics: ServerMetric[] = mapper.hydrateAll(doc);
metrics.forEach((m) => {
console.log(`Host: ${m.nodeHost} | CPU: ${m.cpuUsage}% | Healthy: ${m.isHealthy}`);
});
console.log("--- 5. Canonical Serialization Output ---");
const serializedJson = serialize(doc, 2);
console.log(serializedJson);
} catch (err) {
if (err instanceof BEJSONCoreError) {
console.error(`Core BEJSON Exception [${err.code}]: ${err.message}`);
} else {
console.error("Unexpected Failure:", err);
}
}
}
// Run test demo
runCoreOperationsDemo();
---
Summary
The low-level operations in Core BEJSON establish a high-performance foundation for tabular JSON processing in TypeScript. By isolating ingestion inside explicit parsing bounds, enforcing row-width invariants across functional mutation boundaries, and abstracting matrix conversions behind type-safe field mappers and LRU caches, the architecture achieves a clean balance of static type safety, low memory overhead, and fast execution speed.
In the next chapter, we expand on these parsing and field mapping primitives by building the complete Schema Validation Engine, exploring structural rules, rule engines, and assertion pathways across BEJSON 104, 104a, and 104db specifications.
Chapter 3: Schema Validation Engines and Document Verification Routines
Chapter 3: Schema Validation Engines and Document Verification Routines
Data parsing transforms unformatted byte streams into in-memory object structures, but raw syntax compliance alone does not guarantee structural or semantic correctness. In distributed systems, cross-language processing, and high-density database packaging, ingestion pipelines must verify that incoming documents conform strictly to the structural invariants of the target specification.
Core BEJSON addresses this requirement through an explicit schema validation layer (`lib_bejson_Core_bejson_validators.ts` and `lib_bejson_Core_bejson_schema.ts`). This chapter examines the design and runtime mechanics of the schema validation engine, the structural rules governing BEJSON `104`, `104a`, and `104db` specifications, type-checking validation routines, error collection mechanisms, and programmatic schema definition pipelines.
---
Validation Engine Architecture and Verification Modes
The validation engine in Core BEJSON operates independently from low-level JSON parsing. Separating parsing from schema verification prevents incomplete payloads from executing partial operations while giving callers full control over validation severity and failure handling.
+-------------------------------------------------------------------+
| BEJSON VALIDATION ARCHITECTURE |
+-------------------------------------------------------------------+
| Raw JSON String / In-Memory Object |
| | |
| v |
| [ parse() / Object Input ] --> Generates BEJSONDocument |
| | |
| +-----------------------------------+ |
| | | |
| v v |
| [ Soft Verification ] [ Hard Assertion ] |
| validateDocument(doc) assertValid(doc) |
| | | |
| v v |
| Returns ValidationResult Throws BEJSONCoreError |
| { valid, errors, warnings } if invalid |
+-------------------------------------------------------------------+
Soft Verification vs. Hard Boundary Assertion
Core BEJSON offers two operational pathways for document verification:
1. Soft Verification (`validateDocument`, `isValid`): Non-throwing verification routines that perform exhaustive structural inspections. Instead of halting on the first broken field, soft verification checks the entire document structure and returns a detailed `ValidationResult` containing all encountered errors and warnings.
2. Hard Boundary Assertion (`assertValid`): Guard routines designed for defensive entry points, IPC handlers, and system boundaries. If any validation constraint fails, `assertValid` immediately throws a `BEJSONCoreError` containing the primary diagnostic message.
Validation Result Data Structures
Verification functions express their findings through standard diagnostic interfaces defined in `lib_bejson_Core_bejson_types.ts`:
export interface ValidationError {
path: string;
message: string;
code: string;
}
export interface ValidationWarning {
path: string;
message: string;
code: string;
}
export interface ValidationResult {
valid: boolean;
errors: ValidationError[];
warnings: ValidationWarning[];
}
The diagnostic output provides precise field paths and error classifications, enabling calling code to generate user-facing remediation messages or log contextual diagnostics.
---
Specification Specifications & Structural Verification Rules
BEJSON defines three distinct document specifications, each tailored to specific operational requirements. The validation engine enforces structural rules tailored to each format version.
+-------------------------------------------------------------------+
| BEJSON FORMAT TAXONOMY |
+-------------------------------------------------------------------+
| BEJSON 104 : General non-relational tabular standard. |
| Supports primitive and extended scalar types. |
| BEJSON 104a : Strictly flat single-entity standard. |
| Restricted to primitives; permits custom headers. |
| BEJSON 104db : Multi-entity tabular database standard. |
| Requires Record_Type_Parent discriminator column. |
+-------------------------------------------------------------------+
BEJSON 104: General Non-Relational Tabular Standard
BEJSON `104` is the foundation of the BEJSON specification family. It standardizes flat and semi-structured single-entity datasets using a columnar matrix structure.
Key Invariants for BEJSON 104
* Root Identity: `Format` must equal `"BEJSON"`, `Format_Version` must equal `"104"`, and `Format_Creator` must be a non-empty string (typically `"Elton Boehnen"`).
* Records Type: `Records_Type` must be an array of strings containing exactly one element representing the entity name (e.g., `["LogEntry"]`).
Header Capitalization: All top-level header keys must follow strict `PascalCase` format matching the regex `^[A-Z][a-zA-Z0-9_]$`.
* Field Definitions: The `Fields` array must contain valid `BEJSONField` objects, each specifying a `name` (PascalCase or snake_case) and a supported scalar `type`.
* Row-Width Integrity: Every row in the `Values` 2D matrix must contain an exact count of elements matching `Fields.length`.
* Optional Hierarchy: If `Parent_Hierarchy` is declared, it must be a valid string path representing lineage (e.g., `"/Root/Folder"`).
BEJSON 104a: Single-Entity Flat Standard
BEJSON `104a` is a specialized, optimized variant of `104` designed for low-overhead project chunking, local configuration stores, and flat file transport.
Key Invariants for BEJSON 104a
* Format Identity: `Format_Version` must equal `"104a"`.
* Primitive Type Restriction: To simplify cross-language serialization (Python, TypeScript, Shell, Rust), field types are strictly restricted to four core primitives: `"string"`, `"integer"`, `"number"`, and `"boolean"`. Extended types like `"uuid"`, `"datetime"`, or `"enum"` are disallowed in `104a` schemas.
Custom Root Headers: `104a` allows arbitrary custom top-level metadata headers (e.g., `Chunk_Date`, `Package_Version`, `Session_Is_Mounted`). Custom headers must use `PascalCase` key names matching `^[A-Z][a-zA-Z0-9_]$`.
// Sample Valid BEJSON 104a Document Structure
const sample104a = {
Format: "BEJSON",
Format_Version: "104a",
Format_Creator: "Elton Boehnen",
Schema_Name: "Chunked-104a",
Schema_Version: "1.0.1",
Chunk_Date: "2026-08-08",
Session_Is_Mounted: false,
Records_Type: ["Chunked"],
Fields: [
{ name: "File_Name", type: "string" },
{ name: "Is_Binary", type: "boolean" }
],
Values: [
["index.ts", false]
]
};
BEJSON 104db: Multi-Entity Tabular Database Standard
BEJSON `104db` consolidates multiple related entities into a single tabular file structure, serving as an intermediate packaging container or single-file database.
Key Invariants for BEJSON 104db
* Format Identity: `Format_Version` must equal `"104db"`.
* Multi-Entity Declaration: `Records_Type` must be an array containing two or more string entity names (e.g., `["Customers", "Orders", "LineItems"]`).
* Parent Discriminator Field: The first field descriptor in `Fields` (index `0`) must be named `"Record_Type_Parent"` and must have type `"string"`.
* Row Discriminator Alignment: The first value in every row (`row[0]`) serves as an entity discriminator and must match one of the explicit strings declared in `Records_Type`.
// Sample Valid BEJSON 104db Document Structure
const sample104db = {
Format: "BEJSON",
Format_Version: "104db",
Format_Creator: "Elton Boehnen",
Records_Type: ["Customer", "Order"],
Fields: [
{ name: "Record_Type_Parent", type: "string" },
{ name: "Entity_ID", type: "string" },
{ name: "Total_Amount", type: "number" }
],
Values: [
["Customer", "CST-1001", 0.0],
["Order", "ORD-5001", 149.50]
]
};
---
Technical Implementation of Specification Verification Routines
The verification routines in `lib_bejson_Core_bejson_validators.ts` inspect incoming documents using structural rules and error aggregation.
The Standard Validation Dispatcher
The primary entry point `validateDocument()` acts as a unified dispatcher. It checks top-level root structures and delegates format-specific validation to specialized sub-routines based on `Format_Version`.
import {
BEJSONDocument,
ValidationResult,
ValidationError,
ValidationWarning,
BEJSONCoreError,
BEJSON_CORE_CODES,
} from "./lib_bejson_Core_bejson_types";
const PASCAL_CASE_RE = /^[A-Z][a-zA-Z0-9_]*$/;
const SYSTEM_KEYS = new Set([
"Format",
"Format_Version",
"Format_Creator",
"Records_Type",
"Fields",
"Values",
"Parent_Hierarchy",
]);
/**
* Validates any BEJSON document, routing to 104, 104a, or 104db validators.
*/
export function validateDocument(doc: any): ValidationResult {
const errors: ValidationError[] = [];
const warnings: ValidationWarning[] = [];
if (doc === null || typeof doc !== "object" || Array.isArray(doc)) {
return {
valid: false,
errors: [{ path: "$", message: "Document must be a non-null object.", code: "INVALID_ROOT" }],
warnings: [],
};
}
// Root system header checks
if (doc.Format !== "BEJSON") {
errors.push({
path: "$.Format",
message: `Format must be 'BEJSON', got '${doc.Format}'.`,
code: "INVALID_FORMAT",
});
}
const version = doc.Format_Version;
if (version === "104") {
return _mergeResults(errors, warnings, validate104(doc));
} else if (version === "104a") {
return _mergeResults(errors, warnings, validate104a(doc));
} else if (version === "104db") {
return _mergeResults(errors, warnings, validate104db(doc));
} else {
errors.push({
path: "$.Format_Version",
message: `Unsupported or missing Format_Version: '${version}'.`,
code: "UNSUPPORTED_VERSION",
});
return { valid: false, errors, warnings };
}
}
function _mergeResults(
baseErrors: ValidationError[],
baseWarnings: ValidationWarning[],
res: ValidationResult
): ValidationResult {
const mergedErrors = [...baseErrors, ...res.errors];
const mergedWarnings = [...baseWarnings, ...res.warnings];
return {
valid: mergedErrors.length === 0,
errors: mergedErrors,
warnings: mergedWarnings,
};
}
Validator Implementation: `validate104()` and `validate104a()`
`validate104` and `validate104a` enforce structural rules, scalar types, and row matrix dimensions:
export function validate104(doc: BEJSONDocument): ValidationResult {
const errors: ValidationError[] = [];
const warnings: ValidationWarning[] = [];
// Validate Records_Type
if (!Array.isArray(doc.Records_Type) || doc.Records_Type.length !== 1) {
errors.push({
path: "$.Records_Type",
message: "BEJSON 104 Records_Type must be an array of exactly 1 string.",
code: "INVALID_RECORDS_TYPE",
});
}
// Validate Fields and Matrix
_validateFieldsAndValues(doc, false, errors, warnings);
// Validate custom top-level header casing
_validateHeaderCasing(doc, errors);
return { valid: errors.length === 0, errors, warnings };
}
export function validate104a(doc: BEJSONDocument): ValidationResult {
const errors: ValidationError[] = [];
const warnings: ValidationWarning[] = [];
if (!Array.isArray(doc.Records_Type) || doc.Records_Type.length !== 1) {
errors.push({
path: "$.Records_Type",
message: "BEJSON 104a Records_Type must be an array of exactly 1 string.",
code: "INVALID_RECORDS_TYPE",
});
}
// Enforce primitive-only restriction for 104a
_validateFieldsAndValues(doc, true, errors, warnings);
// Validate custom headers casing
_validateHeaderCasing(doc, errors);
return { valid: errors.length === 0, errors, warnings };
}
Multi-Entity Database Validator: `validate104db()`
`validate104db` enforces relational database checks, such as verifying the discriminator field and checking row markers against declared entities:
export function validate104db(doc: BEJSONDocument): ValidationResult {
const errors: ValidationError[] = [];
const warnings: ValidationWarning[] = [];
// 1. Check Records_Type multi-entity declaration
if (!Array.isArray(doc.Records_Type) || doc.Records_Type.length < 2) {
errors.push({
path: "$.Records_Type",
message: "BEJSON 104db Records_Type must contain 2 or more entity strings.",
code: "INVALID_RECORDS_TYPE",
});
}
const validTypes = new Set(Array.isArray(doc.Records_Type) ? doc.Records_Type : []);
// 2. Validate Record_Type_Parent discriminator at index 0
if (!Array.isArray(doc.Fields) || doc.Fields.length === 0) {
errors.push({
path: "$.Fields",
message: "Fields array must not be empty.",
code: "EMPTY_FIELDS",
});
} else {
const firstField = doc.Fields[0];
if (firstField.name !== "Record_Type_Parent" || firstField.type !== "string") {
errors.push({
path: "$.Fields[0]",
message: "First field in 104db must be { name: 'Record_Type_Parent', type: 'string' }.",
code: "INVALID_PARENT_DISCRIMINATOR",
});
}
}
// 3. Matrix & Discriminator Row Inspection
if (Array.isArray(doc.Values) && Array.isArray(doc.Fields)) {
const fieldCount = doc.Fields.length;
doc.Values.forEach((row, rIdx) => {
if (!Array.isArray(row) || row.length !== fieldCount) {
errors.push({
path: `$.Values[${rIdx}]`,
message: `Row width ${row?.length} does not match field count ${fieldCount}.`,
code: "ROW_WIDTH_MISMATCH",
});
return;
}
const rowType = row[0];
if (typeof rowType !== "string" || !validTypes.has(rowType)) {
errors.push({
path: `$.Values[${rIdx}][0]`,
message: `Row entity type '${rowType}' is not declared in Records_Type.`,
code: "UNDECLARED_ENTITY_ROW",
});
}
});
}
return { valid: errors.length === 0, errors, warnings };
}
Assertions and Predicates
To complement the detailed diagnostic result objects, Core BEJSON exposes concise guard helpers:
export function isValid(doc: any): boolean {
return validateDocument(doc).valid;
}
export function assertValid(doc: any): void {
const result = validateDocument(doc);
if (!result.valid) {
const firstErr = result.errors[0];
throw new BEJSONCoreError(
BEJSON_CORE_CODES.SCHEMA_VALIDATION_ERROR,
`Validation failed at ${firstErr.path}: ${firstErr.message}`
);
}
}
---
Core Type-Checking Routines and Field Constraints
The validation engine performs deep, element-by-element inspection of every cell in the `Values` matrix. Each value is evaluated against the data type declared in the corresponding `Fields` definition.
+-------------------------------------------------------------------+
| FIELD TYPE CHECKING MATRIX |
+-------------------------------------------------------------------+
| Declared Type | Allowed Input Types / Validation Rules |
+---------------+---------------------------------------------------+
| string | typeof === 'string' |
| integer | typeof === 'number' && Number.isInteger(val) |
| number | typeof === 'number' && !Number.isNaN(val) |
| boolean | typeof === 'boolean' |
| uuid | RFC 4122 v4 pattern match |
| datetime | ISO 8601 string parseable by Date.parse() |
| date | YYYY-MM-DD pattern match |
| time | HH:MM:SS pattern match |
| email | Standard email regex address match |
| url | Valid absolute URL format parsed via new URL() |
| enum | Matches entry in predefined field options |
+-------------------------------------------------------------------+
Deep Matrix Type Inspection Helper
The helper `_validateFieldsAndValues()` iterates through matrix rows and validates each cell against its declared type rule:
const PRIMITIVE_TYPES = new Set(["string", "integer", "number", "boolean"]);
const EXTENDED_TYPES = new Set([
...PRIMITIVE_TYPES,
"uuid",
"datetime",
"date",
"time",
"email",
"url",
"enum",
]);
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
const TIME_RE = /^(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?$/;
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
function _validateFieldsAndValues(
doc: BEJSONDocument,
primitivesOnly: boolean,
errors: ValidationError[],
warnings: ValidationWarning[]
): void {
if (!Array.isArray(doc.Fields)) {
errors.push({ path: "$.Fields", message: "Fields must be an array.", code: "INVALID_FIELDS" });
return;
}
const allowedTypes = primitivesOnly ? PRIMITIVE_TYPES : EXTENDED_TYPES;
// 1. Verify Field Schema Definitions
doc.Fields.forEach((field, idx) => {
if (!field || typeof field !== "object") {
errors.push({ path: `$.Fields[${idx}]`, message: "Field descriptor must be an object.", code: "INVALID_FIELD" });
return;
}
if (typeof field.name !== "string" || field.name.trim() === "") {
errors.push({ path: `$.Fields[${idx}].name`, message: "Field name must be a non-empty string.", code: "INVALID_FIELD_NAME" });
}
if (!allowedTypes.has(field.type)) {
errors.push({
path: `$.Fields[${idx}].type`,
message: `Type '${field.type}' is invalid. Allowed: [${Array.from(allowedTypes).join(", ")}].`,
code: "INVALID_FIELD_TYPE",
});
}
});
if (!Array.isArray(doc.Values)) {
errors.push({ path: "$.Values", message: "Values must be a 2D array.", code: "INVALID_VALUES" });
return;
}
const expectedWidth = doc.Fields.length;
// 2. Verify Matrix Rows and Individual Cells
doc.Values.forEach((row, rIdx) => {
if (!Array.isArray(row)) {
errors.push({ path: `$.Values[${rIdx}]`, message: "Row must be an array.", code: "INVALID_ROW" });
return;
}
if (row.length !== expectedWidth) {
errors.push({
path: `$.Values[${rIdx}]`,
message: `Row length ${row.length} does not match field count ${expectedWidth}.`,
code: "ROW_WIDTH_MISMATCH",
});
}
row.forEach((val, cIdx) => {
if (cIdx >= expectedWidth) return;
const fieldDef = doc.Fields[cIdx];
if (!fieldDef) return;
// Null values are permitted across all data types
if (val === null) return;
const path = `$.Values[${rIdx}][${cIdx}]`;
_checkValueType(val, fieldDef.type, path, errors);
});
});
}
function _checkValueType(
val: any,
type: string,
path: string,
errors: ValidationError[]
): void {
switch (type) {
case "string":
if (typeof val !== "string") {
errors.push({ path, message: `Expected string, got ${typeof val}.`, code: "TYPE_MISMATCH" });
}
break;
case "integer":
if (typeof val !== "number" || !Number.isInteger(val)) {
errors.push({ path, message: `Expected integer, got ${val}.`, code: "TYPE_MISMATCH" });
}
break;
case "number":
if (typeof val !== "number" || Number.isNaN(val)) {
errors.push({ path, message: `Expected valid number, got ${val}.`, code: "TYPE_MISMATCH" });
}
break;
case "boolean":
if (typeof val !== "boolean") {
errors.push({ path, message: `Expected boolean, got ${typeof val}.`, code: "TYPE_MISMATCH" });
}
break;
case "uuid":
if (typeof val !== "string" || !UUID_RE.test(val)) {
errors.push({ path, message: `Expected valid UUID v4 string, got '${val}'.`, code: "TYPE_MISMATCH" });
}
break;
case "datetime":
if (typeof val !== "string" || Number.isNaN(Date.parse(val))) {
errors.push({ path, message: `Expected ISO 8601 datetime string, got '${val}'.`, code: "TYPE_MISMATCH" });
}
break;
case "date":
if (typeof val !== "string" || !DATE_RE.test(val)) {
errors.push({ path, message: `Expected YYYY-MM-DD date string, got '${val}'.`, code: "TYPE_MISMATCH" });
}
break;
case "time":
if (typeof val !== "string" || !TIME_RE.test(val)) {
errors.push({ path, message: `Expected HH:MM:SS time string, got '${val}'.`, code: "TYPE_MISMATCH" });
}
break;
case "email":
if (typeof val !== "string" || !EMAIL_RE.test(val)) {
errors.push({ path, message: `Expected valid email address, got '${val}'.`, code: "TYPE_MISMATCH" });
}
break;
case "url":
if (typeof val !== "string") {
errors.push({ path, message: `Expected URL string, got ${typeof val}.`, code: "TYPE_MISMATCH" });
} else {
try { new URL(val); } catch {
errors.push({ path, message: `Expected valid absolute URL, got '${val}'.`, code: "TYPE_MISMATCH" });
}
}
break;
}
}
function _validateHeaderCasing(doc: any, errors: ValidationError[]): void {
for (const key of Object.keys(doc)) {
if (SYSTEM_KEYS.has(key)) continue;
if (key.startsWith("_")) continue; // Skip internal metadata
if (!PASCAL_CASE_RE.test(key)) {
errors.push({
path: `$.${key}`,
message: `Custom top-level header '${key}' must use PascalCase naming format.`,
code: "INVALID_HEADER_CASING",
});
}
}
}
---
Programmatic Schema Management (`lib_bejson_Core_bejson_schema.ts`)
While the built-in validators enforce core format rules (`104`, `104a`, `104db`), applications often require domain-level validation constraints. The programmatic schema module (`lib_bejson_Core_bejson_schema.ts`) provides a declarative builder pattern for defining domain constraints, such as value ranges, string regex patterns, non-null requirements, and enum restrictions.
+-------------------------------------------------------------------+
| PROGRAMMATIC SCHEMA COMPILATION PIPELINE |
+-------------------------------------------------------------------+
| 1. Define Schema Builder |
| const schema = SchemaBuilder.create("UserProfile") |
| .addField("username", "string", { required: true }) |
| .addField("age", "integer", { min: 18, max: 120 }) |
| .compile(); |
| |
| 2. Execute Runtime Compilation |
| const validator = compileSchema(schema); |
| |
| 3. Validate Ingested BEJSON Document |
| const result = validator.validate(document); |
+-------------------------------------------------------------------+
Declarative Schema Definitions and Interfaces
The schema module defines builder structures for constraining tabular data fields:
export interface FieldConstraint {
required?: boolean;
min?: number;
max?: number;
pattern?: RegExp | string;
enumValues?: string[];
customValidator?: (value: any) => boolean | string;
}
export interface SchemaFieldRule {
name: string;
type: string;
constraints?: FieldConstraint;
}
export interface ProgrammaticSchema {
schemaName: string;
schemaVersion: string;
allowExtraFields?: boolean;
fields: SchemaFieldRule[];
requiredHeaders?: string[];
}
The Schema Compiler and Validator Implementation
The `SchemaCompiler` translates a `ProgrammaticSchema` definition into a reusable validation routine:
import { BEJSONDocument, ValidationResult, ValidationError } from "./lib_bejson_Core_bejson_types";
import { validateDocument } from "./lib_bejson_Core_bejson_validators";
export class CompiledSchemaValidator {
private readonly schema: ProgrammaticSchema;
constructor(schema: ProgrammaticSchema) {
this.schema = schema;
}
public validate(doc: BEJSONDocument): ValidationResult {
// 1. Run core format validation first
const baseResult = validateDocument(doc);
if (!baseResult.valid) {
return baseResult;
}
const errors: ValidationError[] = [...baseResult.errors];
const warnings = [...baseResult.warnings];
// 2. Validate custom header requirements
if (this.schema.requiredHeaders) {
for (const header of this.schema.requiredHeaders) {
if (doc[header] === undefined || doc[header] === null) {
errors.push({
path: `$.${header}`,
message: `Missing mandatory top-level header '${header}'.`,
code: "MISSING_REQUIRED_HEADER",
});
}
}
}
// 3. Build positional index maps for schema field rules
const fieldIndices = new Map<string, number>();
doc.Fields.forEach((f, i) => fieldIndices.set(f.name, i));
// 4. Validate field-level constraints
for (const rule of this.schema.fields) {
const colIdx = fieldIndices.get(rule.name);
if (colIdx === undefined) {
if (rule.constraints?.required) {
errors.push({
path: "$.Fields",
message: `Mandatory field '${rule.name}' missing from schema.`,
code: "MISSING_REQUIRED_FIELD",
});
}
continue;
}
// Check field type match
const docField = doc.Fields[colIdx];
if (docField.type !== rule.type) {
errors.push({
path: `$.Fields[${colIdx}].type`,
message: `Field '${rule.name}' expects type '${rule.type}', got '${docField.type}'.`,
code: "SCHEMA_TYPE_MISMATCH",
});
}
// 5. Matrix row constraint checking
if (rule.constraints && Array.isArray(doc.Values)) {
doc.Values.forEach((row, rIdx) => {
const val = row[colIdx];
const valPath = `$.Values[${rIdx}][${colIdx}]`;
this._applyConstraints(val, rule.constraints!, valPath, rule.name, errors);
});
}
}
return { valid: errors.length === 0, errors, warnings };
}
private _applyConstraints(
val: any,
c: FieldConstraint,
path: string,
fieldName: string,
errors: ValidationError[]
): void {
if (val === null || val === undefined) {
if (c.required) {
errors.push({ path, message: `Field '${fieldName}' cannot be null.`, code: "NOT_NULL_VIOLATION" });
}
return;
}
if (c.min !== undefined && typeof val === "number" && val < c.min) {
errors.push({ path, message: `Value ${val} is less than minimum ${c.min}.`, code: "MIN_BOUND_EXCEEDED" });
}
if (c.max !== undefined && typeof val === "number" && val > c.max) {
errors.push({ path, message: `Value ${val} exceeds maximum ${c.max}.`, code: "MAX_BOUND_EXCEEDED" });
}
if (c.pattern && typeof val === "string") {
const regex = typeof c.pattern === "string" ? new RegExp(c.pattern) : c.pattern;
if (!regex.test(val)) {
errors.push({ path, message: `Value '${val}' fails regex pattern constraint.`, code: "PATTERN_MISMATCH" });
}
}
if (c.enumValues && !c.enumValues.includes(String(val))) {
errors.push({
path,
message: `Value '${val}' is not in allowed set [${c.enumValues.join(", ")}].`,
code: "ENUM_CONSTRAINT_VIOLATION",
});
}
if (c.customValidator) {
const customRes = c.customValidator(val);
if (typeof customRes === "string") {
errors.push({ path, message: customRes, code: "CUSTOM_VALIDATION_FAILED" });
} else if (customRes === false) {
errors.push({ path, message: `Custom validation failed for value '${val}'.`, code: "CUSTOM_VALIDATION_FAILED" });
}
}
}
}
Fluent Schema Builder Interface
To simplify creating programmatic schemas, `lib_bejson_Core_bejson_schema.ts` provides a fluent builder class:
export class SchemaBuilder {
private schema: ProgrammaticSchema;
private constructor(name: string, version: string = "1.0.0") {
this.schema = {
schemaName: name,
schemaVersion: version,
fields: [],
requiredHeaders: [],
};
}
public static create(schemaName: string, schemaVersion?: string): SchemaBuilder {
return new SchemaBuilder(schemaName, schemaVersion);
}
public addHeaderRequirement(headerName: string): this {
this.schema.requiredHeaders!.push(headerName);
return this;
}
public addField(name: string, type: string, constraints?: FieldConstraint): this {
this.schema.fields.push({ name, type, constraints });
return this;
}
public compile(): CompiledSchemaValidator {
return new CompiledSchemaValidator(this.schema);
}
}
---
Performance Optimizations and Production Ingestion Pipelines
When validating high-volume datasets containing hundreds of thousands of matrix rows, running full scalar validation checks across every individual cell can become a performance bottleneck.
Algorithmic Complexity and Throughput Optimization
Matrix validation runs in $O(N \cdot M)$ time, where $N$ represents the number of rows and $M$ represents the field column count. Core BEJSON uses several strategies to minimize validation overhead:
1. Short-Circuit Index Cache: Field locations are mapped to positional indices once before matrix iteration begins, eliminating $O(M)$ array lookups for every cell.
2. Nullable Cell Bypassing: Matrix cells containing `null` values skip format-specific regex checks immediately, reducing unnecessary processing overhead in sparse datasets.
3. Pre-Compiled Regular Expressions: Module-scoped regular expressions (e.g., `UUID_RE`, `EMAIL_RE`, `PASCAL_CASE_RE`) avoid the costly overhead of re-instantiating `RegExp` objects inside matrix loops.
Production Pipeline Example: Ingesting Datasets in API Handlers
The following example demonstrates how to set up an API ingestion pipeline that combines parsing, core schema validation, programmatic schema enforcement, and domain object hydration:
import { parse, BEJSONDocument, BEJSONCoreError } from "./Core";
import { assertValid } from "./Core/lib_bejson_Core_bejson_validators";
import { SchemaBuilder } from "./Core/lib_bejson_Core_bejson_schema";
import { FieldMapper } from "./Core/lib_bejson_Core_bejson_field_map";
// 1. Define Domain Model
class TransactionRecord {
public transactionId!: string;
public accountNum!: string;
public amount!: number;
public timestamp!: string;
}
// 2. Pre-Compile Programmatic Domain Schema Validator
const transactionValidator = SchemaBuilder.create("TransactionPayload", "1.0.0")
.addHeaderRequirement("Source_System")
.addField("tx_id", "uuid", { required: true })
.addField("account_num", "string", { pattern: /^ACC-\d{6}$/ })
.addField("amount", "number", { min: 0.01, max: 1000000.00 })
.addField("tx_timestamp", "datetime", { required: true })
.compile();
// 3. Configure Field Mapper for Domain Model Hydration
const transactionMapper = new FieldMapper(TransactionRecord, {
transactionId: "tx_id",
accountNum: "account_num",
amount: "amount",
timestamp: "tx_timestamp",
});
/**
* Production Ingestion Pipeline Endpoint
*/
export function handleTransactionIngestion(rawJsonPayload: string): TransactionRecord[] {
// Step 1: Low-level Parsing
const doc: BEJSONDocument = parse(rawJsonPayload);
// Step 2: Core Format Verification (Hard Guard Boundary)
assertValid(doc);
// Step 3: Programmatic Domain Schema Enforcement
const domainResult = transactionValidator.validate(doc);
if (!domainResult.valid) {
const errorList = domainResult.errors
.map((e) => `[${e.code}] ${e.path}: ${e.message}`)
.join(" | ");
throw new Error(`Domain Validation Failure: ${errorList}`);
}
// Step 4: Hydrate Validated Matrix Rows into Domain Classes
return transactionMapper.hydrateAll(doc);
}
// Sample Execution
try {
const samplePayload = `{
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Source_System": "PaymentGateway_v2",
"Records_Type": ["Transaction"],
"Fields": [
{ "name": "tx_id", "type": "uuid" },
{ "name": "account_num", "type": "string" },
{ "name": "amount", "type": "number" },
{ "name": "tx_timestamp", "type": "datetime" }
],
"Values": [
[
"a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11",
"ACC-123456",
250.75,
"2026-08-08T14:22:00Z"
]
]
}`;
const records = handleTransactionIngestion(samplePayload);
console.log(`Ingested ${records.length} valid transaction record(s).`);
console.log(`TX ID: ${records[0].transactionId} | Amount: $${records[0].amount}`);
} catch (err: any) {
console.error("Pipeline Failure:", err.message);
}
---
Summary
Schema validation routines in Core BEJSON ensure data integrity across parsing and processing workflows. By separating non-throwing soft inspection from guard-based hard assertions, applications can tailor error handling to their specific runtime requirements.
Format-specific rules enforce structural constraints across BEJSON `104`, primitive `104a`, and multi-entity `104db` documents, while deep scalar checking validates matrix cell values against declared types. Programmatic schema compilers extend this system with domain-level validation rules, providing a complete framework for data verification.
The next chapter moves from individual document structures to multi-file datasets, examining the Project Chunking Engine, binary byte preservation strategies, and package version tracking mechanisms.
Chapter 4: Project Chunking, Binary File Preservation, and Package Versioning
Chapter 4: Project Chunking, Binary File Preservation, and Package Versioning
Distributed software development, backup systems, and cross-platform source distribution frequently require packaging multi-file directory structures into unified, portable transport artifacts. Traditional archival solutions rely on proprietary binary formats (such as ZIP, TAR, or 7z) that obfuscate text content, hinder line-by-line diffing, and prevent native parsing within structured data processing pipelines.
Core BEJSON solves this challenge through the Chunked-104a Specification, a project chunking engine implemented in `lib_bejson_Core_bejson_chunking.ts`. By mapping file system hierarchies into a single, standardized BEJSON `104a` document matrix, the chunking subsystem enables entire project directories—including source files, binary assets, and directory structures—to be serialized, versioned, hashed, and restored across diverse language environments (TypeScript, Python, JavaScript, Shell).
This chapter presents the architecture of the Core BEJSON project chunking engine, binary preservation via Base64 encoding, cryptographic byte verification, package versioning lifecycle routines, and multi-schema detection mechanisms.
---
The Chunked-104a Specification Architecture
The Chunked-104a format standardizes project chunking by mapping filesystem hierarchies into a tabular matrix where each row represents an individual file. Built upon the BEJSON `104a` specification, the document uses primitive data types (`string`, `boolean`, `integer`, `number`) to ensure 1:1 cross-language compatibility without requiring complex type coercion.
+-------------------------------------------------------------------+
| CHUNKED-104a ARCHITECTURE |
+-------------------------------------------------------------------+
| Source Project Directory Structure |
| ├── src/ |
| │ ├── index.ts |
| │ └── logo.png |
| └── package.json |
| | |
| v bejsonCoreChunkingCreateChunked104() |
| +-------------------------------------------------------------+ |
| | ChunkedDocument (Format: "BEJSON", Format_Version: "104a") | |
| | Schema_Name: "Chunked-104a", Schema_Version: "1.0.1" | |
| | Session_Is_Mounted: false, Package_Version: "1" | |
| +-------------------------------------------------------------+ |
| | Fields: [File_Name, File_Extension, File_Content, ...] | |
| +-------------------------------------------------------------+ |
| | Values Matrix: | |
| | [ "index.ts", ".ts", "import *...", "1.0", "hash...", ...] | |
| | [ "logo.png", ".png", "iVBORw0K...", "1.0", "hash...", ...] | |
| +-------------------------------------------------------------+ |
| | |
| v bejsonCoreChunkingUnchunkChunked104() |
| Restored Target Directory Tree |
+-------------------------------------------------------------------+
The Standardized Chunking Field Set
Every Chunked-104a document uses an 8-column field schema defined by the `CHUNKED_104_FIELDS` constant. The schema standardizes positional tracking, content storage, and filesystem metadata:
export interface BejsonField {
name: string;
type: string;
}
export const CHUNKED_104_FIELDS: BejsonField[] = [
{ name: "File_Name", type: "string" },
{ name: "File_Extension", type: "string" },
{ name: "File_Content", type: "string" },
{ name: "File_Version", type: "string" },
{ name: "File_Hash", type: "string" },
{ name: "Relative_Path", type: "string" },
{ name: "Is_Binary", type: "boolean" },
{ name: "Is_Mounted", type: "boolean" },
];
| Field Name | Type | Purpose & Constraints |
| :--- | :--- | :--- |
| `File_Name` | `string` | The terminal filename including extension (e.g., `"index.ts"`). |
| `File_Extension` | `string` | Lowercase file extension including leading dot (e.g., `".ts"`). |
| `File_Content` | `string` | Raw UTF-8 text for text files; Base64 encoded string for binary files. |
| `File_Version` | `string` | User-defined or release version applied to the file entry (e.g., `"latest"` or `"1.0.0"`). |
| `File_Hash` | `string` | SHA-256 cryptographic hex digest calculated over raw source bytes. |
| `Relative_Path` | `string` | Normalized POSIX/Windows path relative to target root directory (e.g., `"src/index.ts"`). |
| `Is_Binary` | `boolean` | Binary flag: `false` indicates raw text encoding; `true` indicates Base64 payload encoding. |
| `Is_Mounted` | `boolean` | Workspace flag indicating whether the file is mounted in an active session (defaults to `false`). |
The `ChunkedDocument` Type Definition
A chunked project document wraps the tabular matrix with root metadata headers tracking format versions, package iteration counts, and mount states:
export interface ChunkedDocument {
Format: string;
Format_Version: string;
Format_Creator: string;
Schema_Name: string;
Schema_Version: string;
Schema_Description: string;
Chunk_Date: string;
Session_Is_Mounted: boolean;
Mount_Path: string;
Records_Type: string[];
Fields: BejsonField[];
Values: any[][];
Package_Version?: string;
MFDB_Version?: string;
DB_Name?: string;
Package_Format?: string;
[key: string]: any;
}
Notice the top-level `Session_Is_Mounted` header. In early iterations of the chunking standard, this top-level header was named `Is_Mounted`. This caused a field-collision with the row-level `Is_Mounted` column inside the `Fields` array.
The standard renames the top-level session state header to `Session_Is_Mounted` (typed as a boolean) while preserving `Is_Mounted` inside the row-level field matrix.
---
Binary File Preservation Engine
Early file chunking implementations faced data-loss issues when encountering binary files (such as PNG images, compiled executables, WebAssembly modules, or zip archives). Legacy chunkers assigned an empty string (`File_Content: ""`) to any row marked `Is_Binary: true` and skipped binary files during extraction, dropping non-text assets.
Core BEJSON solves this by integrating Base64 Binary Byte Preservation directly into the chunking engine.
+-------------------------------------------------------------------+
| BINARY FILE PRESERVATION LIFECYCLE |
+-------------------------------------------------------------------+
| 1. CHUNKING PHASE |
| File Path: "assets/logo.png" |
| Read Raw Bytes via Buffer --> fs.readFileSync() |
| Inspect first 1024 bytes via TextDecoder("utf-8", fatal=true) |
| | |
| +---> Valid UTF-8 --> Is_Binary = false |
| | File_Content = raw.toString("utf8")|
| | |
| +---> Invalid UTF-8 --> Is_Binary = true |
| File_Content = raw.toString("base64")
| Compute SHA-256 Hash over rawBytes |
| |
| 2. UNCHUNKING PHASE |
| Read row from ChunkedDocument.Values |
| Inspect Is_Binary cell value |
| | |
| +---> Is_Binary === false --> fs.writeFileSync(path, str) |
| | encoding: "utf-8" |
| | |
| +---> Is_Binary === true --> Buffer.from(content, "base64")
| fs.writeFileSync(path, buf) |
+-------------------------------------------------------------------+
UTF-8 Inspection & Binary Detection Mechanics
Determining whether a file contains binary data based on its extension alone is unreliable due to unknown or custom file extensions. Core BEJSON uses byte inspection on the file contents.
The function `bejsonCoreChunkingIsBinary()` opens the file, reads up to the first 1024 bytes into a memory buffer, and attempts strict UTF-8 decoding using the Node.js native `TextDecoder` API configured with `{ fatal: true }`. If strict decoding throws a byte-sequence error, the file is classified as binary.
import * as fs from "fs";
import * as path from "path";
import * as crypto from "crypto";
/**
* Inspects up to the first 1024 bytes of a file using strict UTF-8 decoding.
* Throws on invalid byte sequences, identifying binary assets reliably.
*/
export function bejsonCoreChunkingIsBinary(filePath: string): boolean {
try {
const fd = fs.openSync(filePath, "r");
const buf = Buffer.alloc(1024);
const bytesRead = fs.readSync(fd, buf, 0, 1024, 0);
fs.closeSync(fd);
const slice = buf.subarray(0, bytesRead);
// fatal: true throws on invalid UTF-8 byte sequences
new TextDecoder("utf-8", { fatal: true }).decode(slice);
return false;
} catch {
return true;
}
}
Cryptographic Byte Verification via SHA-256
To guarantee payload integrity across network boundaries and chunking/unchunking operations, every file entry includes a SHA-256 cryptographic digest in the `File_Hash` column.
The hash is calculated directly over the raw file bytes before text conversion or Base64 encoding:
/**
* Calculates a SHA-256 hex digest directly from raw file byte buffers.
*/
export function bejsonCoreChunkingHashFileBytes(rawBytes: Buffer): string {
return crypto.createHash("sha256").update(rawBytes).digest("hex");
}
By computing `File_Hash` from raw bytes, both text and binary files maintain identical cryptographic verification behavior regardless of platform end-of-line differences (`\n` vs `\r\n`) or encoding transformations.
---
Core Chunking and Unchunking Operations
The primary operations provided by `lib_bejson_Core_bejson_chunking.ts` are project serialization (`bejsonCoreChunkingCreateChunked104`) and project restoration (`bejsonCoreChunkingUnchunkChunked104`).
Default Inclusions and Exclusions
When walking directory trees, the chunking engine applies default extension inclusions and directory exclusions to ignore build artifacts, version control stores, and dependency folders.
export const DEFAULT_EXTENSIONS: string[] = [
".py", ".js", ".ts", ".html", ".css", ".md", ".json",
".sh", ".txt", ".bejson", ".tsx", ".jsx",
];
export const DEFAULT_EXCLUDES: string[] = [
".git", "__pycache__", "node_modules", "lib", "output",
".mfdb_lock", "dist", "build",
];
Directory Walking and Filtering
Directory traversal uses a stack-based algorithm (`walkDir`) to avoid stack overflows on deeply nested project trees:
function walkDir(root: string, excludeDirs: string[]): string[] {
const results: string[] = [];
const stack: string[] = [root];
while (stack.length > 0) {
const current = stack.pop() as string;
const entries = fs.readdirSync(current, { withFileTypes: true });
for (const entry of entries) {
if (entry.isDirectory()) {
if (!excludeDirs.includes(entry.name)) {
stack.push(path.join(current, entry.name));
}
} else if (entry.isFile()) {
results.push(path.join(current, entry.name));
}
}
}
return results;
}
Implementing `bejsonCoreChunkingCreateChunked104()`
The project chunking function traverses a specified directory, filters files against allowed extensions and exclusion rules, inspects binary state, computes SHA-256 hashes, encodes file content, and builds the full `ChunkedDocument` structure:
export function bejsonCoreChunkingGetTimestamp(): string {
return new Date().toISOString().replace(/\.\d{3}Z$/, "Z");
}
export function bejsonCoreChunkingCreateChunked104(
targetDir: string,
version: string = "latest",
extensions: string[] | null = null,
excludeDirs: string[] | null = null,
packageVersion: string | null = null
): ChunkedDocument {
const targetPath = path.resolve(targetDir);
const exts = extensions !== null ? extensions : DEFAULT_EXTENSIONS;
const excl = excludeDirs !== null ? excludeDirs : DEFAULT_EXCLUDES;
const values: any[][] = [];
const allFiles = walkDir(targetPath, excl);
for (const filePath of allFiles) {
const ext = path.extname(filePath).toLowerCase();
if (!exts.includes(ext)) continue;
try {
const relPath = path.relative(targetPath, filePath);
const isBin = bejsonCoreChunkingIsBinary(filePath);
const rawBytes = fs.readFileSync(filePath);
// Base64 encode if binary; UTF-8 stringify if text
const content = isBin
? rawBytes.toString("base64")
: rawBytes.toString("utf-8");
const fileHash = bejsonCoreChunkingHashFileBytes(rawBytes);
values.push([
path.basename(filePath),
path.extname(filePath),
content,
version,
fileHash,
relPath,
isBin,
false, // Is_Mounted initialized to false
]);
} catch {
// Skip locked or unreadable files gracefully
continue;
}
}
return {
Format: "BEJSON",
Format_Version: "104a",
Format_Creator: "Elton Boehnen",
Schema_Name: "Chunked-104a",
Schema_Version: "1.0.1",
Schema_Description: "Standard schema for chunking single projects.",
Chunk_Date: bejsonCoreChunkingGetTimestamp().slice(0, 10),
Session_Is_Mounted: false,
Mount_Path: "",
Package_Version: packageVersion || "1",
Records_Type: ["Chunked"],
Fields: CHUNKED_104_FIELDS,
Values: values,
};
}
Implementing `bejsonCoreChunkingUnchunkChunked104()`
Unchunking reconstructs the original directory tree on disk from an in-memory `ChunkedDocument`. It builds a dynamic column map from the document's `Fields` array to ensure safe field access even if column positions shift in custom schemas:
export function bejsonCoreChunkingUnchunkChunked104(
doc: ChunkedDocument,
outputDir: string
): number {
const fields = doc.Fields || CHUNKED_104_FIELDS;
const fm: Record<string, number> = {};
fields.forEach((f, i) => (fm[f.name] = i));
const outRoot = path.resolve(outputDir);
let count = 0;
for (const row of doc.Values || []) {
const relPath = row[fm["Relative_Path"]];
const isBinary = row[fm["Is_Binary"]];
const content = row[fm["File_Content"]];
if (!relPath || content === null || content === undefined) continue;
const targetFile = path.join(outRoot, relPath);
// Ensure missing target directory structure is created recursively
fs.mkdirSync(path.dirname(targetFile), { recursive: true });
if (isBinary) {
// Decode Base64 content back into raw binary Buffer before writing
fs.writeFileSync(targetFile, Buffer.from(content, "base64"));
} else {
// Write raw UTF-8 string text directly
fs.writeFileSync(targetFile, content, { encoding: "utf-8" });
}
count += 1;
}
return count;
}
---
Package Versioning and Lineage Tracking
Projects naturally evolve over time through multiple iterations. To distinguish structural file revisions (`File_Version`) from overall package distribution iterations (`Package_Version`), Core BEJSON maintains an explicit versioning policy.
+-------------------------------------------------------------------+
| VERSION SEPARATION ARCHITECTURE |
+-------------------------------------------------------------------+
| File_Version (Per-Row Attribute in Values Matrix) |
| --> Identifies specific software tags or file revisions |
| (e.g., "1.0.4", "v2-beta", "latest"). |
| |
| Package_Version (Top-Level Document Metadata Header) |
| --> Tracks package generation lifecycle counts. |
| Monotonically increasing integer string ("1", "2", "3"). |
+-------------------------------------------------------------------+
Monotonic Version Bumping Routine
When re-chunking an existing workspace, applications pass the prior `ChunkedDocument` into `bejsonCoreChunkingBumpPackageVersion()` to obtain the next sequential package iteration ID:
/**
* Increments numeric Package_Version strings monotonically.
* Returns "1" if prior document is missing or invalid.
*/
export function bejsonCoreChunkingBumpPackageVersion(
priorDoc: ChunkedDocument | null | undefined
): string {
if (!priorDoc) return "1";
const n = parseInt(String(priorDoc.Package_Version ?? ""), 10);
return Number.isNaN(n) ? "1" : String(n + 1);
}
This tracking enables backup, synchronization, and database migration routines to detect document revisions instantly without requiring full matrix diffing across thousands of file rows.
---
Multi-Schema Detection and Universal Unchunking Architecture
Over time, Core BEJSON has introduced multiple schemas for project packaging and database storage. `lib_bejson_Core_bejson_chunking.ts` provides a unified interface (`bejsonCoreChunkingMfdb*`) capable of inspecting, identifying, and unchunking any BEJSON packaging format.
+-------------------------------------------------------------------+
| UNIFIED SCHEMA DETECTION & DISPATCH |
+-------------------------------------------------------------------+
| Incoming Chunked Document |
| | |
| v bejsonCoreChunkingMfdbDetectSchema(doc) |
| +-------------------------------------------------------------+ |
| | Inspect Records_Type, Schema_Name, and Fields layout | |
| +-------------------------------------------------------------+ |
| | |
| +---> BEJSON_CORE_CHUNKING_MFDB_SCHEMA_CHUNKED_104A |
| | --> Single-project or MFDB-132 flat container. |
| | --> Executing bejsonCoreChunkingUnchunkChunked104() |
| | |
| +---> BEJSON_CORE_CHUNKING_MFDB_SCHEMA_MANIFEST |
| | --> MFDB multi-version rolling database manifest. |
| | --> Resolves entity file & extracts target version. |
| | |
| +---> BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY |
| | --> Multi-version entity container. |
| | --> Filters rows where version === targetVersion. |
| | |
| +---> BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY_LEGACY |
| | --> Unsupported legacy schema; rejects execution. |
| | |
| +---> BEJSON_CORE_CHUNKING_MFDB_SCHEMA_UNKNOWN |
| --> Structural validation failed. |
+-------------------------------------------------------------------+
Supported Packaging Schema Formats
export const BEJSON_CORE_CHUNKING_MFDB_SCHEMA_MANIFEST = "mfdb_manifest";
export const BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY = "mfdb_entity";
export const BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY_LEGACY = "mfdb_entity_legacy";
export const BEJSON_CORE_CHUNKING_MFDB_SCHEMA_CHUNKED_104A = "chunked_104a";
export const BEJSON_CORE_CHUNKING_MFDB_SCHEMA_UNKNOWN = "unknown";
Implementing Schema Structural Detection
`bejsonCoreChunkingMfdbDetectSchema()` inspects root header indicators (`Records_Type`, `Schema_Name`) and compares document `Fields` names against known schema signatures:
const _MFDB_ENTITY_FIELD_NAMES = new Set([
"version", "File_Name", "File_Extension", "Relative_Path",
"File_Content", "File_Hash", "Is_Binary", "Is_Mounted",
]);
const _MFDB_ENTITY_LEGACY_FIELD_NAMES = new Set([
"version", "file_path", "file_name", "content", "is_binary", "is_base64",
]);
const _CHUNKED_104A_FIELD_NAMES = new Set(
CHUNKED_104_FIELDS.map((f) => f.name)
);
function _setsEqual(a: Set<string>, b: Set<string>): boolean {
if (a.size !== b.size) return false;
for (const x of a) if (!b.has(x)) return false;
return true;
}
export function bejsonCoreChunkingMfdbDetectSchema(doc: ChunkedDocument): string {
const recordsType = doc.Records_Type;
const fieldNames = new Set((doc.Fields || []).map((f) => f.name));
// Check MFDB-132 flat package tag
if ((Array.isArray(recordsType) && recordsType.length === 1 && recordsType[0] === "MFDB-132") ||
doc.Schema_Name === "MFDB-132") {
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_CHUNKED_104A;
}
// Check standard Chunked-104a schema
if (Array.isArray(recordsType) && recordsType.length === 1 && recordsType[0] === "Chunked" &&
_setsEqual(fieldNames, _CHUNKED_104A_FIELD_NAMES)) {
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_CHUNKED_104A;
}
// Check rolling multi-version entity schema
if (_setsEqual(fieldNames, _MFDB_ENTITY_FIELD_NAMES)) {
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY;
}
// Check legacy entity schema signature
if (_setsEqual(fieldNames, _MFDB_ENTITY_LEGACY_FIELD_NAMES)) {
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY_LEGACY;
}
// Check MFDB manifest index schema
if (Array.isArray(recordsType) && recordsType.length === 1 && recordsType[0] === "mfdb" &&
fieldNames.has("entity_name") && fieldNames.has("file_path")) {
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_MANIFEST;
}
return BEJSON_CORE_CHUNKING_MFDB_SCHEMA_UNKNOWN;
}
Version Compatibility Check
`bejsonCoreChunkingMfdbCheckVersion()` warns if a document's declared `MFDB_Version` header falls outside known supported releases:
export function bejsonCoreChunkingMfdbCheckVersion(
doc: ChunkedDocument,
knownVersions: string[] = ["1.31", "1.32", "1.38"]
): string | null {
const mfdbVersion = (doc as any).MFDB_Version;
if (mfdbVersion === undefined || mfdbVersion === null) return null;
if (!knownVersions.includes(String(mfdbVersion))) {
return `MFDB_Version '${mfdbVersion}' not in known set [${knownVersions.join(", ")}] -- ` +
`proceeding on structural detection anyway, but this is worth a look.`;
}
return null;
}
Implementing the Universal Unchunking Routine
The function `bejsonCoreChunkingMfdbUnchunk()` provides a single entry point for extracting files from any chunking or database packaging schema:
export interface MfdbUnchunkResult {
ok: boolean;
message: string;
schema: string;
warning: string | null;
out_dir?: string;
file_count?: number;
}
export function bejsonCoreChunkingMfdbUnchunk(
doc: ChunkedDocument,
outputDir: string,
version: string | null = null,
manifestDir: string | null = null
): MfdbUnchunkResult {
const schema = bejsonCoreChunkingMfdbDetectSchema(doc);
const warning = bejsonCoreChunkingMfdbCheckVersion(doc);
const outRoot = path.resolve(outputDir);
// Pathway 1: Standard Chunked-104a / MFDB-132 flat archive
if (schema === BEJSON_CORE_CHUNKING_MFDB_SCHEMA_CHUNKED_104A) {
const count = bejsonCoreChunkingUnchunkChunked104(doc, outputDir);
return {
ok: true,
message: `Restored ${count} file(s) from Chunked-104a/MFDB-132 bundle.`,
schema,
warning,
out_dir: outRoot,
file_count: count,
};
}
// Pathway 2: MFDB Manifest Index
if (schema === BEJSON_CORE_CHUNKING_MFDB_SCHEMA_MANIFEST) {
if (manifestDir === null || !version) {
return {
ok: false,
message: "MFDB manifest requires manifestDir and version.",
schema,
warning,
};
}
const fm = bejsonCoreChunkingMfdbGetFieldMap(doc);
const row = (doc.Values || []).find((r: any[]) => r[fm.entity_name] === version);
if (!row) {
return {
ok: false,
message: `Version '${version}' not found in manifest.`,
schema,
warning,
};
}
const entityPath = path.join(manifestDir, row[fm.file_path]);
if (!fs.existsSync(entityPath)) {
return {
ok: false,
message: `Entity file missing: ${entityPath}`,
schema,
warning,
};
}
const entityDoc = JSON.parse(fs.readFileSync(entityPath, "utf-8")) as ChunkedDocument;
return bejsonCoreChunkingMfdbUnchunk(entityDoc, outputDir, version, null);
}
// Pathway 3: Multi-version rolling Entity File
if (schema === BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY) {
if (!version) {
return {
ok: false,
message: "MFDB entity requires version parameter.",
schema,
warning,
};
}
const fm = bejsonCoreChunkingMfdbGetFieldMap(doc);
const rows = (doc.Values || []).filter((r: any[]) => r[fm.version] === version);
if (rows.length === 0) {
return {
ok: false,
message: `No rows for version '${version}'.`,
schema,
warning,
};
}
fs.mkdirSync(outRoot, { recursive: true });
let count = 0;
for (const row of rows) {
const relPath = row[fm.Relative_Path];
if (!relPath) continue;
const target = path.join(outRoot, relPath);
fs.mkdirSync(path.dirname(target), { recursive: true });
if (row[fm.Is_Binary]) {
fs.writeFileSync(target, Buffer.from(row[fm.File_Content] || "", "base64"));
} else {
fs.writeFileSync(target, row[fm.File_Content] || "", "utf-8");
}
count += 1;
}
return {
ok: true,
message: `Restored ${count} file(s) for version '${version}'.`,
schema,
warning,
out_dir: outRoot,
file_count: count,
};
}
// Pathway 4: Legacy Entity Schema (Unsupported)
if (schema === BEJSON_CORE_CHUNKING_MFDB_SCHEMA_ENTITY_LEGACY) {
return {
ok: false,
message: "Legacy MFDB entity schema -- no migration path by design. " +
"Re-chunk source project with current tooling first.",
schema,
warning,
};
}
return {
ok: false,
message: "Could not identify chunk schema (structural detection failed).",
schema: BEJSON_CORE_CHUNKING_MFDB_SCHEMA_UNKNOWN,
warning,
};
}
export function bejsonCoreChunkingMfdbGetFieldMap(
doc: ChunkedDocument
): Record<string, number> {
const fields = doc.Fields || [];
const map: Record<string, number> = {};
fields.forEach((f, i) => { map[f.name] = i; });
return map;
}
---
Production Workflow Example: Project Packaging and Extraction
The following end-to-end example demonstrates how to build a project packaging script using the Core BEJSON chunking engine. The script packages a workspace, increments its package version string, writes the result to disk as a JSON document, restores it to a clean workspace, and verifies binary assets against their original SHA-256 hashes.
import * as fs from "fs";
import * as path from "path";
import {
bejsonCoreChunkingCreateChunked104,
bejsonCoreChunkingUnchunkChunked104,
bejsonCoreChunkingBumpPackageVersion,
bejsonCoreChunkingHashFileBytes,
bejsonCoreChunkingMfdbUnchunk,
ChunkedDocument,
} from "./Core/lib_bejson_Core_bejson_chunking";
function runProductionChunkingPipeline(): void {
const sourceProjectDir = path.resolve("./my_project");
const exportChunkPath = path.resolve("./my_project.chunk.bejson");
const restoredDir = path.resolve("./my_project_restored");
// Step 1: Create local mock environment if missing
if (!fs.existsSync(sourceProjectDir)) {
fs.mkdirSync(path.join(sourceProjectDir, "src"), { recursive: true });
fs.mkdirSync(path.join(sourceProjectDir, "assets"), { recursive: true });
// Write source code text file
fs.writeFileSync(
path.join(sourceProjectDir, "src", "index.ts"),
"console.log('Hello Core BEJSON Chunking Engine!');\n"
);
// Write mock 4-byte binary asset file (0x89, 0x50, 0x4E, 0x47)
fs.writeFileSync(
path.join(sourceProjectDir, "assets", "icon.png"),
Buffer.from([0x89, 0x50, 0x4e, 0x47])
);
}
// Step 2: Read prior document if present to perform package version bump
let priorDoc: ChunkedDocument | null = null;
if (fs.existsSync(exportChunkPath)) {
try {
priorDoc = JSON.parse(
fs.readFileSync(exportChunkPath, "utf-8")
) as ChunkedDocument;
} catch {
priorDoc = null;
}
}
const nextPackageVersion = bejsonCoreChunkingBumpPackageVersion(priorDoc);
console.log(`[Chunker] Generating Package Version: ${nextPackageVersion}`);
// Step 3: Package directory into Chunked-104a document
const chunkDoc = bejsonCoreChunkingCreateChunked104(
sourceProjectDir,
"1.0.0", // File_Version string applied to entries
[".ts", ".js", ".png"], // Extensions filter
[".git", "node_modules"],// Directory exclusions
nextPackageVersion // Bumped package version string
);
// Step 4: Serialize ChunkedDocument to disk
fs.writeFileSync(
exportChunkPath,
JSON.stringify(chunkDoc, null, 2),
"utf-8"
);
console.log(`[Chunker] Exported ${chunkDoc.Values.length} files to ${exportChunkPath}`);
// Step 5: Execute Universal Unchunking into isolated restoration path
const unchunkResult = bejsonCoreChunkingMfdbUnchunk(chunkDoc, restoredDir);
if (!unchunkResult.ok) {
throw new Error(`[Unchunker] Failed: ${unchunkResult.message}`);
}
console.log(`[Unchunker] Status: ${unchunkResult.message}`);
// Step 6: Verify SHA-256 binary hash integrity of restored binary asset
const originalBinaryPath = path.join(sourceProjectDir, "assets", "icon.png");
const restoredBinaryPath = path.join(restoredDir, "assets", "icon.png");
const origHash = bejsonCoreChunkingHashFileBytes(
fs.readFileSync(originalBinaryPath)
);
const restHash = bejsonCoreChunkingHashFileBytes(
fs.readFileSync(restoredBinaryPath)
);
console.log(`[Verify] Original Binary Hash: ${origHash}`);
console.log(`[Verify] Restored Binary Hash: ${restHash}`);
if (origHash === restHash) {
console.log("[Verify] SUCCESS: Binary byte preservation verified perfectly!");
} else {
console.error("[Verify] ERROR: SHA-256 hash mismatch detected!");
}
}
// Execute the workflow
runProductionChunkingPipeline();
---
Summary
The Chunked-104a Specification transforms file system hierarchies into portable, transparent BEJSON documents. By representing file metadata, relative paths, and contents inside a standardized matrix, applications can serialize, transport, and extract complex projects across cross-language environments.
Base64 encoding preserves binary assets alongside source code, and strict UTF-8 inspection prevents text corruption. SHA-256 digests guarantee cryptographic payload integrity, and package versioning tracks package revisions monotonically across builds. Finally, the universal unchunker provides detection and extraction capabilities across all BEJSON chunking and database formats.
The next chapter builds upon these chunking primitives, examining the Multi-File Database (MFDB) Packaging Architecture, manifest management, entity relationship validation, and the `MFDB132Archive` session mount lifecycle.
Chapter 5: MFDB Multi-File Database Core and MFDB132Archive Session Lifecycle
Chapter 5: MFDB Multi-File Database Core and MFDB132Archive Session Lifecycle
In production data engineering, single-file documents frequently become bottlenecks when datasets grow across millions of rows or span heterogeneous domains. Storing an entire enterprise database inside a single monolith creates disk I/O contention, complicates concurrent updates, and increases memory overhead during query parsing.
Core BEJSON addresses database scalability through the Multi-File Database (MFDB) specification. An MFDB breaks monolithic data stores into modular, entity-scoped files tied together by a authoritative root manifest. Introduced in early revisions as a filesystem directory layout (and optionally packed into ZIP containers under the MFDB 1.31 specification), the format evolved in MFDB 1.32 to support single-file packaging inside a Chunked-104a document matrix without losing relational validation or entity isolation.
This chapter details the low-level mechanics of the MFDB architecture, covering manifest generation, entity registration, multi-file validation engines, the MFDB 1.32 packaging specification, and the transactional session mount lifecycle managed by `MFDB132Archive`.
---
The MFDB Multi-File Database Architecture
The Multi-File Database standard organizes database entities across discrete physical files while preserving global structural constraints, cross-file relational integrity, and version tracking.
+-------------------------------------------------------------------+
| MFDB 1.31 DIRECTORY LAYOUT |
+-------------------------------------------------------------------+
| my_database/ |
| ├── 104a.mfdb.bejson <-- Core Database Manifest |
| ├── users.entity.bejson <-- User Entity Table File |
| ├── orders.entity.bejson <-- Order Entity Table File |
| └── products.entity.bejson <-- Product Entity Table File |
+-------------------------------------------------------------------+
|
v bejsonCoreChunkingCreateMfdb132Package()
+-------------------------------------------------------------------+
| MFDB 1.32 SINGLE-FILE PACKAGE |
+-------------------------------------------------------------------+
| my_database.mfdb132.json |
| ├── Schema_Name: "MFDB-132" |
| ├── Format_Version: "104a" |
| ├── Package_Format: "MFDB-Chunked-104a" |
| └── Values Matrix (Contains 104a.mfdb.bejson + all entities) |
+-------------------------------------------------------------------+
Manifest and Entity Scope
An MFDB layout consists of two primary role types:
1. The Manifest File (`104a.mfdb.bejson`): The authoritative entry point for the database. It stores database-level metadata (`DB_Name`, `MFDB_Version`, `Schema_Version`), defines global field dictionaries, and maintains an index matrix tracking registered entity names, relative file paths, record counts, and target schemas.
2. Entity Files (`<entity_name>.entity.bejson` or relative paths): Individual BEJSON documents (`104a` or `104db`) holding the table data for specific entities. Each entity file operates independently for localized read/write performance while maintaining compliance with the master manifest definition.
MFDB Specification Versions: 1.31 vs. 1.32
* MFDB 1.31 (Directory / ZIP Layout): Entities exist as separate files inside a physical directory, optionally archived into a standard `.zip` file container. Workspace operations modify files directly on disk.
* MFDB 1.32 (Chunked-104a Package Layout): The entire directory layout—including `104a.mfdb.bejson` and all associated entity files—is chunked into a single, flat BEJSON `104a` document using the schema identifier `MFDB-132`. This provides a text-native, versioned, single-file container that can be inspected, diffed, and transported without binary compression tooling.
---
MFDB Core File Operations & Manifest Management
The core lifecycle of an MFDB involves creating the root manifest, registering entity files, unregistering entities, and synchronizing record counts. These operations are exported from `lib_bejson_Core_mfdb_core.ts`.
+-------------------------------------------------------------------+
| MANIFEST MANAGEMENT LIFECYCLE |
+-------------------------------------------------------------------+
| 1. createManifest("StoreDB", "1.32") |
| --> Writes 104a.mfdb.bejson with empty Values matrix |
| |
| 2. registerEntity(manifest, "users", "users.entity.bejson") |
| --> Inspects entity file, verifies record count & schema |
| --> Adds entity row to manifest matrix |
| |
| 3. syncRecordCount(manifest, "users", 1500) |
| --> Updates record count cell for "users" in manifest matrix |
| |
| 4. unregisterEntity(manifest, "users") |
| --> Removes entity row from manifest matrix |
+-------------------------------------------------------------------+
The Manifest Schema Standard
A valid manifest file always bears the exact filename `104a.mfdb.bejson`. Its top-level structure adheres to the BEJSON `104a` format with specific custom metadata headers:
{
"Format": "BEJSON",
"Format_Version": "104a",
"Format_Creator": "Elton Boehnen",
"Schema_Name": "MFDB_Manifest",
"Schema_Version": "1.0.0",
"DB_Name": "StoreDB",
"MFDB_Version": "1.32",
"Records_Type": ["mfdb"],
"Fields": [
{ "name": "entity_name", "type": "string" },
{ "name": "file_path", "type": "string" },
{ "name": "record_count","type": "integer" },
{ "name": "schema_name", "type": "string" }
],
"Values": [
["users", "users.entity.bejson", 1500, "UserSchema_v1"],
["orders", "orders.entity.bejson", 4200, "OrderSchema_v1"]
]
}
Creating the Database Manifest
The `createManifest()` function initializes a structural manifest object in memory or directly writes it to disk:
import * as fs from "fs";
import * as path from "path";
import { BEJSONDocument, BEJSONField } from "./lib_bejson_Core_bejson_types";
import { MFDB_MANIFEST_FILENAME } from "./lib_bejson_Core_bejson_chunking";
export interface CreateManifestOptions {
dbName: string;
mfdbVersion?: string;
schemaVersion?: string;
outputPath?: string;
}
export const MFDB_MANIFEST_FIELDS: BEJSONField[] = [
{ name: "entity_name", type: "string" },
{ name: "file_path", type: "string" },
{ name: "record_count", type: "integer" },
{ name: "schema_name", type: "string" },
];
export function createManifest(options: CreateManifestOptions): BEJSONDocument {
const {
dbName,
mfdbVersion = "1.32",
schemaVersion = "1.0.0",
outputPath,
} = options;
const manifest: BEJSONDocument = {
Format: "BEJSON",
Format_Version: "104a",
Format_Creator: "Elton Boehnen",
Schema_Name: "MFDB_Manifest",
Schema_Version: schemaVersion,
DB_Name: dbName,
MFDB_Version: mfdbVersion,
Records_Type: ["mfdb"],
Fields: MFDB_MANIFEST_FIELDS,
Values: [],
};
if (outputPath) {
const targetFile = path.resolve(outputPath);
fs.mkdirSync(path.dirname(targetFile), { recursive: true });
fs.writeFileSync(targetFile, JSON.stringify(manifest, null, 2), "utf8");
}
return manifest;
}
Registering and Unregistering Entities
Registering an entity introduces a new row into the manifest's `Values` array, binding the entity's logical table name to its relative on-disk path, current record count, and schema name:
export function registerEntity(
manifest: BEJSONDocument,
entityName: string,
filePath: string,
recordCount: number,
schemaName: string = "default"
): BEJSONDocument {
if (manifest.Schema_Name !== "MFDB_Manifest") {
throw new Error("Target document is not a valid MFDB_Manifest.");
}
// Prevent duplicate entity registration
const existingRow = manifest.Values.find((row) => row[0] === entityName);
if (existingRow) {
throw new Error(`Entity '${entityName}' is already registered in manifest.`);
}
const updatedValues = [
...manifest.Values,
[entityName, filePath, recordCount, schemaName],
];
return {
...manifest,
Values: updatedValues,
};
}
export function unregisterEntity(
manifest: BEJSONDocument,
entityName: string
): BEJSONDocument {
const updatedValues = manifest.Values.filter((row) => row[0] !== entityName);
return {
...manifest,
Values: updatedValues,
};
}
Synchronizing Record Counts
When records are inserted into or deleted from an entity file, the parent manifest's `record_count` index must stay in sync to prevent validation failures during database verification routines:
export function syncRecordCount(
manifest: BEJSONDocument,
entityName: string,
newRecordCount: number
): BEJSONDocument {
let found = false;
const updatedValues = manifest.Values.map((row) => {
if (row[0] === entityName) {
found = true;
const copy = [...row];
copy[2] = newRecordCount; // Index 2 correlates to 'record_count'
return copy;
}
return row;
});
if (!found) {
throw new Error(`Entity '${entityName}' not found in manifest.`);
}
return {
...manifest,
Values: updatedValues,
};
}
---
MFDB Validation Engine
Validation in MFDB spans structural checks on single files, relational consistency checks across the full database, and deep inspection inside chunked packages. These validation utilities are defined in `lib_bejson_Core_mfdb_validators.ts`.
+-------------------------------------------------------------------+
| MFDB VALIDATION HIERARCHY |
+-------------------------------------------------------------------+
| validateDatabase(manifestPath) |
| | |
| +---> 1. validateManifest(manifestPath) |
| | - Verifies 104a.mfdb.bejson exists |
| | - Validates fields & DB_Name presence |
| | |
| +---> 2. Loop through registered entities in manifest: |
| validateEntityFile(entityPath, expectedCount) |
| - Verifies entity file exists on disk |
| - Computes actual row count vs expectedCount |
| - Validates BEJSON field types |
+-------------------------------------------------------------------+
Role Discovery via `discoverRole()`
Before performing schema checks, the validator determines whether a target document is a Manifest, an Entity file, or a Chunked Package using `discoverRole()`:
export function discoverRole(doc: BEJSONDocument): "manifest" | "entity" | "chunked_132" | "unknown" {
if (!doc || typeof doc !== "object") return "unknown";
const recordsType = doc.Records_Type;
if (Array.isArray(recordsType) && recordsType.includes("MFDB-132")) {
return "chunked_132";
}
if (doc.Schema_Name === "MFDB_Manifest" || (Array.isArray(recordsType) && recordsType.includes("mfdb"))) {
return "manifest";
}
if (doc.Format === "BEJSON" && (doc.Format_Version === "104a" || doc.Format_Version === "104db")) {
return "entity";
}
return "unknown";
}
Entity and Database Validation Routines
Validating an entire MFDB requires verifying that the manifest file itself is valid, that every referenced entity file exists on disk, and that the physical record count inside each entity file matches the index recorded in `104a.mfdb.bejson`:
export interface EntityValidationOptions {
checkRecordCount?: boolean;
}
export interface DatabaseValidationOptions {
allowExtraFiles?: boolean;
}
export function validateManifest(manifestPath: string): boolean {
if (!fs.existsSync(manifestPath)) return false;
try {
const doc = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as BEJSONDocument;
if (discoverRole(doc) !== "manifest") return false;
if (!doc.DB_Name || !doc.MFDB_Version) return false;
return true;
} catch {
return false;
}
}
export function validateEntityFile(
entityPath: string,
expectedRecordCount?: number
): boolean {
if (!fs.existsSync(entityPath)) return false;
try {
const doc = JSON.parse(fs.readFileSync(entityPath, "utf8")) as BEJSONDocument;
if (discoverRole(doc) !== "entity") return false;
if (expectedRecordCount !== undefined && doc.Values.length !== expectedRecordCount) {
return false;
}
return true;
} catch {
return false;
}
}
export function validateDatabase(
manifestPath: string,
options: DatabaseValidationOptions = {}
): boolean {
if (!validateManifest(manifestPath)) return false;
const manifestDir = path.dirname(manifestPath);
const doc = JSON.parse(fs.readFileSync(manifestPath, "utf8")) as BEJSONDocument;
// Extract field map indices for manifest matrix
const fields = doc.Fields || [];
const nameIdx = fields.findIndex((f) => f.name === "entity_name");
const pathIdx = fields.findIndex((f) => f.name === "file_path");
const countIdx = fields.findIndex((f) => f.name === "record_count");
for (const row of doc.Values) {
const relPath = row[pathIdx] as string;
const expectedCount = row[countIdx] as number;
const fullEntityPath = path.join(manifestDir, relPath);
if (!validateEntityFile(fullEntityPath, expectedCount)) {
return false;
}
}
return true;
}
---
The MFDB 1.32 Single-File Package Extension
While the directory layout of MFDB 1.31 works well for active workspace operations, moving databases across networks or storing them in object stores benefits from a single-file packaging format. The MFDB 1.32 specification addresses this by chunking an entire MFDB directory structure into a single BEJSON `104a` document matrix.
+-------------------------------------------------------------------+
| MFDB 1.32 PACKAGE STRUCTURE |
+-------------------------------------------------------------------+
| BEJSON Chunked Document (Format: "BEJSON", Format_Version: "104a")|
| Schema_Name: "MFDB-132" |
| Package_Format: "MFDB-Chunked-104a" |
| MFDB_Version: "1.32" |
| DB_Name: "StoreDB" |
| |
| Fields: [ File_Name, File_Extension, File_Content, ... ] |
| Values Matrix: |
| [ "104a.mfdb.bejson", ".bejson", "{\n \"DB_Name\": ...}", ... ] |
| [ "users.entity.bejson", ".bejson", "{\n \"Values\": ...}", ...] |
| [ "orders.entity.bejson", ".bejson", "{\n \"Values\": ...}", ...] |
+-------------------------------------------------------------------+
Creating an MFDB 1.32 Package
`bejsonCoreChunkingCreateMfdb132Package()` reads an MFDB directory, asserts that `104a.mfdb.bejson` exists at the root, chunks all manifest and entity files into a `ChunkedDocument` matrix, and applies the MFDB 1.32 schema identity headers:
export const MFDB_CHUNK_SCHEMA_VERSION = "1.32";
export function bejsonCoreChunkingCreateMfdb132Package(
mfdbRootDir: string,
dbName: string,
extensions: string[] | null = null,
excludeDirs: string[] | null = null,
packageVersion: string | null = null,
priorPackageDoc: ChunkedDocument | null = null
): ChunkedDocument {
const rootPath = path.resolve(mfdbRootDir);
const manifestPath = path.join(rootPath, MFDB_MANIFEST_FILENAME);
if (!fs.existsSync(manifestPath) || !fs.statSync(manifestPath).isFile()) {
throw new Error(
`No ${MFDB_MANIFEST_FILENAME} found at root of ${rootPath} — cannot ` +
`package a directory that isn't a valid MFDB layout.`
);
}
const resolvedPackageVersion =
packageVersion || bejsonCoreChunkingBumpPackageVersion(priorPackageDoc);
const doc = bejsonCoreChunkingCreateChunked104(
rootPath,
MFDB_CHUNK_SCHEMA_VERSION,
extensions !== null ? extensions : DEFAULT_EXTENSIONS,
excludeDirs,
resolvedPackageVersion
);
// Apply explicit MFDB-132 schema identity headers
doc.Schema_Name = "MFDB-132";
doc.Records_Type = ["MFDB-132"];
doc.MFDB_Version = MFDB_CHUNK_SCHEMA_VERSION;
doc.DB_Name = dbName;
doc.Package_Format = "MFDB-Chunked-104a";
return doc;
}
Deep Detection and Unpackaging Verification
To verify that an incoming chunk document represents a valid MFDB 1.32 package, the validator checks both document headers and the internal manifest entry.
The function `detectMfdbInChunk()` scans the chunked document's row matrix without unchunking files to disk, locating `104a.mfdb.bejson` and verifying that all referenced entity files exist within the matrix:
export interface MfdbEntityCheck {
entityName: string;
filePath: string;
foundInChunk: boolean;
expectedRecordCount: number;
}
export interface MfdbInChunkDetection {
isMfdb: boolean;
manifestFound: boolean;
dbName: string | null;
mfdbVersion: string | null;
entities: MfdbEntityCheck[];
missingFiles: string[];
}
export function detectMfdbInChunk(doc: ChunkedDocument): MfdbInChunkDetection {
const fields = doc.Fields || [];
const pathIdx = fields.findIndex((f) => f.name === "Relative_Path");
const contentIdx = fields.findIndex((f) => f.name === "File_Content");
const result: MfdbInChunkDetection = {
isMfdb: false,
manifestFound: false,
dbName: null,
mfdbVersion: null,
entities: [],
missingFiles: [],
};
if (pathIdx === -1 || contentIdx === -1 || !Array.isArray(doc.Values)) {
return result;
}
// Find 104a.mfdb.bejson in the Values matrix
const manifestRow = doc.Values.find(
(row) => row[pathIdx] === MFDB_MANIFEST_FILENAME
);
if (!manifestRow) return result;
result.manifestFound = true;
try {
const manifestContent = manifestRow[contentIdx] as string;
const manifestDoc = JSON.parse(manifestContent);
result.dbName = manifestDoc.DB_Name || null;
result.mfdbVersion = manifestDoc.MFDB_Version || null;
// Build map of all relative paths stored in this chunk doc
const pathSet = new Set(doc.Values.map((r) => r[pathIdx] as string));
const manifestFields = manifestDoc.Fields || [];
const eNameIdx = manifestFields.findIndex((f: any) => f.name === "entity_name");
const ePathIdx = manifestFields.findIndex((f: any) => f.name === "file_path");
const eCountIdx = manifestFields.findIndex((f: any) => f.name === "record_count");
for (const eRow of manifestDoc.Values || []) {
const eName = eRow[eNameIdx] as string;
const ePath = eRow[ePathIdx] as string;
const eCount = eRow[eCountIdx] as number;
const found = pathSet.has(ePath);
result.entities.push({
entityName: eName,
filePath: ePath,
foundInChunk: found,
expectedRecordCount: eCount,
});
if (!found) {
result.missingFiles.push(ePath);
}
}
result.isMfdb = result.missingFiles.length === 0;
} catch {
result.isMfdb = false;
}
return result;
}
---
The `MFDB132Archive` Session Mount Lifecycle
When working with an MFDB 1.32 package, reading or mutating data directly inside the flattened Base64/JSON document matrix is inefficient and error-prone.
To solve this, Core BEJSON provides the `MFDB132Archive` class, implementing a Transactional Session Mount Lifecycle. The engine unchunks the package into an isolated temporary workspace directory, manages active locks, validates mutations before committing, and packs the updated directory back into a single MFDB 1.32 document upon commit.
+-------------------------------------------------------------------+
| MFDB132Archive SESSION MOUNT LIFECYCLE |
+-------------------------------------------------------------------+
| 1. MOUNT SESSION |
| MFDB132Archive.mount("db.bejson", "./workspace") |
| - Computes SHA-256 hash of original db.bejson |
| - Unchunks file matrix into ./workspace directory |
| - Writes session lock file: ./workspace/.mfdb132_lock |
| - Mutates db.bejson headers: Session_Is_Mounted=true |
| |
| 2. WORKSPACE MUTATION PERIOD |
| - Application performs CRUD directly on entity files |
| - Option: MFDB132Archive.resurrect_file("./workspace", path) |
| Restores single unmodified file from original package |
| |
| 3. COMMIT SESSION |
| MFDB132Archive.commit("./workspace") |
| - Pre-commit gate: validateDatabase("./workspace/104a...") |
| - Re-chunks workspace into temporary file: db.bejson.tmp... |
| - Atomically renames temporary file over original db.bejson |
| - Updates SHA-256 hash in lock file |
| |
| 4. UNMOUNT SESSION |
| MFDB132Archive.unmount("./workspace", cleanup=true) |
| - Releases lock file .mfdb132_lock |
| - Mutates db.bejson headers: Session_Is_Mounted=false |
| - Removes temporary ./workspace directory from disk |
+-------------------------------------------------------------------+
Lock File Specification (`.mfdb132_lock`)
Whenever a session is mounted, `MFDB132Archive` creates a lock file named `.mfdb132_lock` inside the workspace root. This lock records the process ID, mount timestamp, original package hash, chunk document path, and workspace directory path:
export const LOCK_FILE_132 = ".mfdb132_lock";
export interface LockData132 {
pid: number;
mounted_at: string;
original_hash: string;
chunk_doc_path: string;
workspace_dir: string;
}
If another process attempts to mount the same workspace directory without passing `force: true`, `MFDB132Archive.mount()` detects the lock and throws an exception, preventing concurrent process collisions and data corruption.
Header Mutation Mechanics
During `mount()`, the root package document on disk is updated with session tracking metadata:
* `Session_Is_Mounted` is set to `true`.
* `Mount_Path` is set to the absolute path of the target workspace directory.
When `unmount()` is called, `Session_Is_Mounted` is reset to `false` and `Mount_Path` is cleared (`""`).
Implementing `MFDB132Archive`
Below is the complete implementation of the `MFDB132Archive` session manager from `lib_bejson_Core_bejson_chunking.ts`:
export interface MountOptions {
force?: boolean;
sticky?: boolean;
}
function _calculateChunkHash(filePath: string): string {
const data = fs.readFileSync(filePath);
return crypto.createHash("sha256").update(data).digest("hex");
}
function _setChunkDocHeaders(
chunkDocPath: string,
isMounted: boolean,
mountPath: string
): void {
try {
const doc = JSON.parse(fs.readFileSync(chunkDocPath, "utf8")) as ChunkedDocument;
doc["Session_Is_Mounted"] = !!isMounted;
delete (doc as any)["Is_Mounted"]; // Clean up legacy schema key if present
doc["Mount_Path"] = mountPath;
fs.writeFileSync(chunkDocPath, JSON.stringify(doc, null, 2), "utf8");
} catch (e: any) {
console.warn(`[MFDB132] Could not update chunk doc headers: ${e.message}`);
}
}
export class MFDB132Archive {
/**
* Mounts an MFDB132 package to a workspace and writes a session lock.
* sticky=true reuses an existing valid workspace when the chunk doc hash matches.
* Returns the absolute path to the restored manifest.
*/
static mount(
chunkDocPath: string,
targetDir: string,
{ force = false, sticky = true }: MountOptions = {}
): string {
const chunkAbs = path.resolve(chunkDocPath);
if (!fs.existsSync(chunkAbs)) {
throw new Error(`Chunk doc not found: ${chunkDocPath}`);
}
const lockFile = path.join(targetDir, LOCK_FILE_132);
const manifestOut = path.join(targetDir, MFDB_MANIFEST_FILENAME);
const currentHash = _calculateChunkHash(chunkAbs);
// Sticky reuse optimization
if (sticky && fs.existsSync(lockFile) && fs.existsSync(manifestOut)) {
try {
const lock = JSON.parse(fs.readFileSync(lockFile, "utf8")) as LockData132;
if (lock.original_hash === currentHash) {
const { validateDatabase } = require("./lib_bejson_Core_mfdb_validators");
if (validateDatabase(manifestOut)) return path.resolve(manifestOut);
}
} catch (_) { /* Fall through to clean unchunk */ }
}
// Ownership and concurrency safety check
if (fs.existsSync(lockFile) && !force) {
const lock = JSON.parse(fs.readFileSync(lockFile, "utf8")) as LockData132;
if (lock.pid !== process.pid) {
throw new Error(
`Workspace ${targetDir} is locked by PID ${lock.pid}. Pass force=true to override.`
);
}
}
// Clear workspace and extract package files
if (fs.existsSync(targetDir)) {
fs.rmSync(targetDir, { recursive: true, force: true });
}
fs.mkdirSync(targetDir, { recursive: true });
const doc = JSON.parse(fs.readFileSync(chunkAbs, "utf8")) as ChunkedDocument;
const count = bejsonCoreChunkingUnchunkChunked104(doc, targetDir);
if (count === 0) {
fs.rmSync(targetDir, { recursive: true, force: true });
throw new Error("Unchunk produced zero files — chunk doc may be empty.");
}
if (!fs.existsSync(manifestOut)) {
fs.rmSync(targetDir, { recursive: true, force: true });
throw new Error("Invalid MFDB132 package: 104a.mfdb.bejson missing after unchunk.");
}
// Write session lock file
const lockData: LockData132 = {
pid: process.pid,
mounted_at: new Date().toISOString(),
original_hash: currentHash,
chunk_doc_path: chunkAbs,
workspace_dir: path.resolve(targetDir),
};
fs.writeFileSync(lockFile, JSON.stringify(lockData, null, 2), "utf8");
// Update session state headers in package document
_setChunkDocHeaders(chunkAbs, true, path.resolve(targetDir));
return path.resolve(manifestOut);
}
/**
* Commits workspace changes back into an MFDB132 package atomically.
* Runs full database validation as a pre-write gate before touching disk.
*/
static commit(
mountDir: string,
outputPath: string | null = null,
validate = true
): string {
const lockFile = path.join(mountDir, LOCK_FILE_132);
const manifestOut = path.join(mountDir, MFDB_MANIFEST_FILENAME);
if (!fs.existsSync(lockFile)) {
throw new Error(`No active 132 mount session in ${mountDir}`);
}
const lockData = JSON.parse(fs.readFileSync(lockFile, "utf8")) as LockData132;
// Pre-commit validation gate
if (validate) {
if (!fs.existsSync(manifestOut)) {
throw new Error("Commit rejected: manifest missing.");
}
const { validateDatabase } = require("./lib_bejson_Core_mfdb_validators");
try {
validateDatabase(manifestOut);
} catch (e: any) {
throw new Error(`Commit rejected: validation failed — ${e.message}`);
}
}
const destPath = outputPath ?? lockData.chunk_doc_path;
if (!destPath) throw new Error("Destination chunk doc path unknown.");
const manifestDoc = JSON.parse(fs.readFileSync(manifestOut, "utf8"));
const dbName: string = manifestDoc.DB_Name ?? "";
// Write to temporary file first to guarantee atomic updates
const tempChunk = `${destPath}.tmp.${Date.now()}`;
try {
const newDoc = bejsonCoreChunkingCreateMfdb132Package(mountDir, dbName);
fs.writeFileSync(tempChunk, JSON.stringify(newDoc, null, 2), "utf8");
fs.renameSync(tempChunk, destPath);
} catch (e: any) {
if (fs.existsSync(tempChunk)) fs.unlinkSync(tempChunk);
throw new Error(`Commit failed during rechunk: ${e.message}`);
}
// Refresh original hash in lock file and re-assert mounted header
const newHash = _calculateChunkHash(destPath);
lockData.original_hash = newHash;
fs.writeFileSync(lockFile, JSON.stringify(lockData, null, 2), "utf8");
_setChunkDocHeaders(destPath, true, path.resolve(mountDir));
return destPath;
}
/**
* Restores a single file from the source package back into the workspace,
* discarding any uncommitted local edits made to that file.
*/
static resurrect_file(mountDir: string, relativePath: string): boolean {
const lockFile = path.join(mountDir, LOCK_FILE_132);
if (!fs.existsSync(lockFile)) return false;
const lockData = JSON.parse(fs.readFileSync(lockFile, "utf8")) as LockData132;
const chunkDocPath = lockData.chunk_doc_path;
if (!chunkDocPath || !fs.existsSync(chunkDocPath)) return false;
try {
const doc = JSON.parse(fs.readFileSync(chunkDocPath, "utf8")) as ChunkedDocument;
const fields = doc.Fields || CHUNKED_104_FIELDS;
const fm: Record<string, number> = {};
fields.forEach((f, i) => (fm[f.name] = i));
const targetRel = path.normalize(relativePath);
for (const row of doc.Values ?? []) {
if (path.normalize(row[fm["Relative_Path"]]) === targetRel) {
const targetFile = path.join(mountDir, row[fm["Relative_Path"]]);
fs.mkdirSync(path.dirname(targetFile), { recursive: true });
const isBinary: boolean = row[fm["Is_Binary"]];
const content: string = row[fm["File_Content"]];
if (isBinary) {
fs.writeFileSync(targetFile, Buffer.from(content, "base64"));
} else {
fs.writeFileSync(targetFile, content, { encoding: "utf-8" });
}
return true;
}
}
} catch (e: any) {
console.warn(`[MFDB132] resurrect_file failed for ${relativePath}: ${e.message}`);
}
return false;
}
/**
* Releases session locks, clears mounted headers, and cleans up workspace.
*/
static unmount(mountDir: string, cleanup = true): void {
const lockFile = path.join(mountDir, LOCK_FILE_132);
if (fs.existsSync(lockFile)) {
try {
const lockData = JSON.parse(fs.readFileSync(lockFile, "utf8")) as LockData132;
const cdp = lockData.chunk_doc_path;
if (cdp && fs.existsSync(cdp)) {
_setChunkDocHeaders(cdp, false, "");
}
} catch (_) {}
fs.unlinkSync(lockFile);
}
if (cleanup && fs.existsSync(mountDir)) {
fs.rmSync(mountDir, { recursive: true, force: true });
}
}
}
---
Production Integration Guide: Lifecycle Management
The following end-to-end integration pipeline demonstrates constructing an MFDB database directory, packaging it into an MFDB 1.32 single-file document, executing a session mount via `MFDB132Archive`, inserting new records into an entity file, updating manifest indices, committing the changes with pre-validation, testing single-file resurrection, and unmounting cleanly.
import * as fs from "fs";
import * as path from "path";
import {
createManifest,
registerEntity,
syncRecordCount,
} from "./Core/lib_bejson_Core_mfdb_core";
import {
bejsonCoreChunkingCreateMfdb132Package,
MFDB132Archive,
MFDB_MANIFEST_FILENAME,
} from "./Core/lib_bejson_Core_bejson_chunking";
import { BEJSONDocument } from "./Core/lib_bejson_Core_bejson_types";
function runMfdbProductionPipeline(): void {
const stagingDir = path.resolve("./mfdb_staging");
const packageFile = path.resolve("./StoreDB.mfdb132.json");
const workspaceDir = path.resolve("./mfdb_workspace");
console.log("=== Step 1: Initialize Local MFDB Staging Directory ===");
if (fs.existsSync(stagingDir)) {
fs.rmSync(stagingDir, { recursive: true, force: true });
}
fs.mkdirSync(stagingDir, { recursive: true });
// 1a. Create Users Entity File
const usersEntityPath = path.join(stagingDir, "users.entity.bejson");
const usersDoc: BEJSONDocument = {
Format: "BEJSON",
Format_Version: "104a",
Format_Creator: "Elton Boehnen",
Records_Type: ["users"],
Fields: [
{ name: "user_id", type: "integer" },
{ name: "username", type: "string" },
{ name: "email", type: "string" },
],
Values: [
[1, "alice", "alice@example.com"],
[2, "bob", "bob@example.com"],
],
};
fs.writeFileSync(usersEntityPath, JSON.stringify(usersDoc, null, 2), "utf8");
// 1b. Create Root Manifest
const manifestPath = path.join(stagingDir, MFDB_MANIFEST_FILENAME);
let manifest = createManifest({
dbName: "StoreDB",
mfdbVersion: "1.32",
outputPath: manifestPath,
});
// Register users entity into manifest
manifest = registerEntity(
manifest,
"users",
"users.entity.bejson",
usersDoc.Values.length,
"UserSchema_v1"
);
fs.writeFileSync(manifestPath, JSON.stringify(manifest, null, 2), "utf8");
console.log(`[MFDB] Created local database with manifest and 'users' entity.`);
console.log("\n=== Step 2: Package MFDB Directory into Single-File Package ===");
const chunkPackage = bejsonCoreChunkingCreateMfdb132Package(
stagingDir,
"StoreDB"
);
fs.writeFileSync(packageFile, JSON.stringify(chunkPackage, null, 2), "utf8");
console.log(`[Package] Generated MFDB 1.32 package at ${packageFile}`);
console.log("\n=== Step 3: Mount Package via MFDB132Archive Session ===");
const restoredManifestPath = MFDB132Archive.mount(packageFile, workspaceDir, {
force: true,
});
console.log(`[Mount] Workspace initialized. Manifest path: ${restoredManifestPath}`);
console.log("\n=== Step 4: Mutate Entity & Sync Manifest in Workspace ===");
const wsUsersPath = path.join(workspaceDir, "users.entity.bejson");
const wsUsersDoc = JSON.parse(
fs.readFileSync(wsUsersPath, "utf8")
) as BEJSONDocument;
// Insert a new user record
wsUsersDoc.Values.push([3, "charlie", "charlie@example.com"]);
fs.writeFileSync(wsUsersPath, JSON.stringify(wsUsersDoc, null, 2), "utf8");
// Synchronize manifest record count
const wsManifestPath = path.join(workspaceDir, MFDB_MANIFEST_FILENAME);
let wsManifestDoc = JSON.parse(
fs.readFileSync(wsManifestPath, "utf8")
) as BEJSONDocument;
wsManifestDoc = syncRecordCount(
wsManifestDoc,
"users",
wsUsersDoc.Values.length
);
fs.writeFileSync(wsManifestPath, JSON.stringify(wsManifestDoc, null, 2), "utf8");
console.log(`[Mutation] Appended record. New count: ${wsUsersDoc.Values.length}`);
console.log("\n=== Step 5: Test Single-File Resurrection ===");
// Accidentally overwrite users file with invalid data
fs.writeFileSync(wsUsersPath, "CORRUPTED CONTENT", "utf8");
console.log("[Test] Corrupted workspace entity file.");
// Resurrect the file from original package to undo local uncommitted edits
const resurrected = MFDB132Archive.resurrect_file(workspaceDir, "users.entity.bejson");
console.log(`[Resurrect] File restored from original chunk doc: ${resurrected}`);
// Re-apply local valid modification after resurrection
const reRestoredDoc = JSON.parse(fs.readFileSync(wsUsersPath, "utf8")) as BEJSONDocument;
reRestoredDoc.Values.push([3, "charlie", "charlie@example.com"]);
fs.writeFileSync(wsUsersPath, JSON.stringify(reRestoredDoc, null, 2), "utf8");
console.log("\n=== Step 6: Commit Workspace back to Package ===");
const committedPackagePath = MFDB132Archive.commit(workspaceDir);
console.log(`[Commit] Validated and re-chunked workspace into ${committedPackagePath}`);
console.log("\n=== Step 7: Unmount Session and Clean Up ===");
MFDB132Archive.unmount(workspaceDir, true);
console.log("[Unmount] Session closed, workspace directory cleaned up.");
// Clean up temporary staging files
fs.rmSync(stagingDir, { recursive: true, force: true });
fs.unlinkSync(packageFile);
console.log("=== Pipeline Execution Complete Successfully ===");
}
// Run the integration test
runMfdbProductionPipeline();
---
Summary
The Multi-File Database (MFDB) specification extends Core BEJSON into a production-grade database system. By dividing large schemas into modular entity files governed by a central manifest (`104a.mfdb.bejson`), MFDB achieves scalable I/O performance without sacrificing relational integrity.
The MFDB 1.32 Specification unifies directory-based databases with single-file chunking standards, wrapping complete database structures into transparent BEJSON documents. Finally, `MFDB132Archive` provides a robust, transactional session mount engine featuring concurrency locks, pre-commit validation gates, atomic file writes, and single-file resurrection utilities.
