BEJSON 105 & MFDB 1.40: The Identity Era & Cryptographic Integrity Masterclass

Chapter 1: The Identity Revolution — Why BEJSON 104 Needed UUID Armor

Listen up skids. In the previous guide, I showed you how BEJSON Format 104 stomped corporate JSON into the dirt using positional integrity. And for homogeneous logs and sensor dumps, 104 is still blazing fast. But what happens when you build real distributed architectures with microservices, asynchronous workers, and schema evolutions? You run head-first into the Identity Crisis.

Consider what happens in 104 when someone renames a column from "auth_token" to "session_token", or reorders two fields during a refactor. If your microservice was reading by field name, its parser breaks. If your service was reading raw row indices, it starts interpreting tokens as IP addresses! Even worse: how do you reference a specific row across database boundaries? In 104, foreign keys are just arbitrary string conventions (like user_id_fk). If someone modifies or duplicates a primary key, your referential integrity collapses like a house of cards. Total fail. Lmao.

That is why Elton Boehnen engineered BEJSON Format 105 (The Integrity Era). BEJSON 105 preserves 100% of the raw positional streaming speed of 104, but fuses it with immutable cryptographic identity.

The Two Pillars of 105 Identity

1. field_uuid on Every Column: Every single column object in the Fields array is permanently stamped with a unique UUIDv4:

   {"name": "reading_c", "type": "number", "field_uuid": "a1b2c3d4-0003-4000-8000-000000000003"}

   Even if a developer renames the column or moves it around, the field_uuid remains immutable. Code that queries by field_uuid never breaks, ever.
2. record_uuid in the Reserved Leading Slot: Every single row in Values now carries an immutable UUIDv4 in a reserved leading column. Notice that Fields does NOT describe this column! It is an engine-level invariant. Every record on the planet has a globally unique primary fingerprint.

Direct Cell Addressing by Coordinate Pair

Because rows have record_uuid and columns have field_uuid, you can now write directly to any arbitrary cell coordinate in (1)$ without scanning: bejson_core_write_cell(doc, record_uuid, field_uuid, value). This eliminates race conditions during concurrent worker updates.

Chapter 2: The Critical +1 Offset Arithmetic — Avoiding Buffer Off-By-One Traps

Pay close attention, because if you mess up this one architectural rule, every parser you write will crash with buffer off-by-one errors. In the 104-series, the number of items in a row exactly equals len(Fields). Column 0 is field 0, column 1 is field 1. Simple, right?

In BEJSON 105, that naive assumption gets you pwned.

The 105 Row Layout Matrix

Because the engine injects record_uuid directly into the physical row without declaring it in Fields, every 105 row contains exactly ONE MORE ELEMENT than the Fields array:

Notice that carefully: in 105 and 105a, record_uuid takes column 0, and every declared field shifts to the right by +1 index! In 105db, column 0 is retained for the Record_Type_Parent discriminator, and record_uuid is inserted at column 1.

Never Do Index Math By Hand

Only total amateurs write row[idx + 1] by hand. If your code gets fed a 104 file tomorrow, your manual offset math will corrupt data. The Core library handles this automatically via bejson_core_get_field_map(doc):

# On a 104 document:
field_map = bejson_core_get_field_map(doc_104)
# Returns: {'sensor_id': 0, 'temperature': 1}

# On a 105 document:
field_map = bejson_core_get_field_map(doc_105)
# Returns: {'sensor_id': 1, 'temperature': 2}  <-- ALREADY OFFSET!
temp = row[field_map['temperature']]  # ALWAYS CORRECT, REGARDLESS OF FORMAT!

Resolving by Name or UUID

Want to write bulletproof code? Use bejson_core_resolve_field_index(doc, target). It allows you to pass a field name string, a field_uuid string, or a composite dict {"name": "...", "field_uuid": "..."}. If someone maliciously modifies the field name but the UUID doesn't match, the engine aborts with E_FIELD_INTEGRITY_MISMATCH (Code 80).

Chapter 3: The 105 Format Trinity — 105, 105a, and 105db Dissected

Just like the 104-era, the 105-series splits into three distinct operational formats depending on your architectural target. But in 105, every format carries cryptographic UUID armor. Here is the breakdown so you don't use the wrong format like a total noob.

1. BEJSON 105 (Single Entity, Complex Types)

The standard format for raw data tables, high-throughput logs, and MFDB entity files:

• Records_Type: Exactly one string (e.g. ["SensorReading"]).
• Custom Headers: Forbidden. Only Parent_Hierarchy is permitted (to link entity files back to their manifest).
• Types: Supports all 6 types, including complex array and object cells.
• Row Layout: [record_uuid, v0, v1, ...] (length = len(Fields) + 1).

2. BEJSON 105a (Configuration & Metadata)

Used for server configuration, daemon state, and environment parameters:

• Records_Type: Exactly one string.
• Custom Headers: Allowed. Must be file-level metadata using PascalCase (e.g. Cluster_ID, Retention_Days).
• Types: Primitives ONLY (string, integer, number, boolean). If you put an array or object in a 105a field, the validator rejects the file instantly!
• CRITICAL INVARIANT: MFDB manifests must ALWAYS remain 104a! If you create an MFDB manifest and set Format_Version: "105a", the MFDB validator will reject it with Code 32. Manifests do not carry row UUIDs—they index entity files.

3. BEJSON 105db (Multi-Entity Relational Database)

The ultimate flat-file multi-entity relational format:

• Records_Type: Two or more unique entity strings (e.g. ["Department", "Employee"]).
• Custom Headers: Forbidden.
• Discriminator + UUID: Column 0 is Record_Type_Parent. Column 1 is record_uuid.
• Field Ownership: Every field (except field 0) carries "Record_Type_Parent": "<Entity>" and its own field_uuid. Unowned fields are null-padded.

{
  "Format": "BEJSON",
  "Format_Version": "105db",
  "Format_Creator": "Elton Boehnen",
  "Records_Type": ["Vulnerability", "Exploit"],
  "Fields": [
    {"name": "Record_Type_Parent", "type": "string", "field_uuid": "d0000000-0000-4000-8000-000000000000"},
    {"name": "cve_id",             "type": "string", "Record_Type_Parent": "Vulnerability", "field_uuid": "d0000000-0000-4000-8000-000000000001"},
    {"name": "cvss_score",         "type": "number", "Record_Type_Parent": "Vulnerability", "field_uuid": "d0000000-0000-4000-8000-000000000002"},
    {"name": "exploit_name",       "type": "string", "Record_Type_Parent": "Exploit",       "field_uuid": "d0000000-0000-4000-8000-000000000003"},
    {"name": "target_cve_uuid_fk",  "type": "string", "Record_Type_Parent": "Exploit",       "field_uuid": "d0000000-0000-4000-8000-000000000004"}
  ],
  "Values": [
    ["Vulnerability", "e1111111-0000-4000-8000-000000000001", "CVE-2026-9999", 9.8, null, null],
    ["Exploit",       "e1111111-0000-4000-8000-000000000002", null, null, "ZeroDay_Pwn", "e1111111-0000-4000-8000-000000000001"]
  ]
}

Master Comparison Cheat Sheet

Chapter 4: The 104 → 105 Upgrade Pipeline & Atomic Rollback Protocol

So you have gigabytes of legacy 104-series data and you want to bring it into the modern Integrity Era. How do you do it without corrupting production or losing byte alignment? You don't write custom shell regexes like a reckless skid. You use the official, battle-hardened BEJSON Upgrade Engine (lib_bejson_Core_bejson_upgrade.py).

The Upgrade Engine Protocol

from lib_bejson_Core_bejson_upgrade import (
    bejson_core_upgrade_dry_run,
    bejson_core_upgrade_to_105
)

# 1. Always execute a dry-run first! Zero disk writes.
dry_report = bejson_core_upgrade_dry_run(doc, "sensors.bejson")
if not dry_report['validation_pass']:
    print("Source data is corrupted! Errors:", dry_report['errors'])
    sys.exit(1)

# 2. Execute live atomic upgrade with automatic backup snapshotting:
new_doc = bejson_core_upgrade_to_105(doc, "sensors.bejson")
print("Upgraded to 105! New target version:", new_doc['Format_Version'])

The 4 Strict Safety Guarantees

1. Deterministic Format Mapping:
   • 104 → 105 (Single entity, full types)
   • 104a → 105a (Preserves all custom PascalCase metadata headers!)
   • 104db → 105db (Preserves discriminator at column 0, inserts record_uuid at column 1)
2. The Strict Pre-Validation Gate: The upgrade engine runs a full validate_bejson check on the source document BEFORE touching disk. If the source 104 file has a short row, a type mismatch, or an invalid creator, the upgrade immediately halts with E_UPGRADE_VALIDATION_FAILED (Code 83). A corrupted file cannot be upgraded!
3. Automatic Byte-for-Byte Rollback Snapshot: Before overwriting the target, the original file is copied byte-for-byte to Backups/<filename>.104.bak next to the source. If anything fails, rollback is a simple file copy.
4. Post-Upgrade Validation & Atomic Swap: The freshly stamped 105 document is validated with strict 105 integrity checks. Only when it passes 100% is it flushed to disk using an atomic os.replace swap.

Chapter 5: MFDB 1.40 & Declarative Referential Integrity — Cascades, Restricts & Cycles

Here is where SQL database administrators start sweating. For years, the SQL lobby claimed flat-file architectures were toys because "you don't have real referential integrity." If you delete a customer in flat files, all their orders become orphaned ghosts. In MFDB v1.31, foreign keys were indeed advisory conventions (like user_id_fk).

That era is over. MFDB v1.40 introduces full, native, flat-file Declarative Referential Integrity.

How Referential Integrity Works in MFDB 1.40

In MFDB 1.40, a foreign key is declared directly on the column definition in a 105 entity file:

{
  "name": "customer_uuid_fk",
  "type": "string",
  "field_uuid": "b0000000-0002-4000-8000-000000000002",
  "fk_target_entity": "Customer",
  "fk_on_delete": "cascade"
}

• The Target is record_uuid: An FK value in a record MUST be the target entity's record_uuid. A literal null means "no relationship" and is always valid.
• The Ref_Integrity Switch: By default, referential enforcement is off to preserve backward compatibility. You activate it per database by calling mfdb_core_enable_ref_integrity(manifest) (which stamps "Ref_Integrity": "true" as a custom header in 104a.mfdb.bejson).
• The Three Delete Actions (fk_on_delete):
  • "restrict": If child records point to this parent, the delete is blocked and raises Code 44 (E_MFDB_FK_RESTRICT_VIOLATION).
  • "cascade": Deleting the parent automatically deletes all dependent child records across other files!
  • "null": Deleting the parent sets the child records' foreign key field to null.

The Plan-Then-Commit Atomic Deletion Architecture

What happens if a cascade delete has to touch 5 different entity files across disk, and the power gets cut midway through? Unlike naive scripts that delete files one by one, MFDB 1.40 uses Plan-Then-Commit:

1. The engine traces the entire dependency tree in memory and builds an execution plan.
2. If a circular dependency cycle is detected, it aborts with Code 45 (E_MFDB_FK_CASCADE_CYCLE).
3. Every entity file scheduled for mutation is snapshotted to a memory buffer.
4. The mutations are written atomically. If any write fails, all snapshotted files are restored instantly! Zero corrupted database states.

The Real Error Codes You Need to Know

• Code 39 (E_MFDB_FK_UNRESOLVED): You tried to insert a child record with an FK pointing to a ghost UUID that doesn't exist in the target entity. Rejected!
• Code 43 (E_MFDB_FK_TARGET_ENTITY_UNKNOWN): The fk_target_entity doesn't exist in the manifest, or points to a legacy 104 entity without record UUIDs.
• Code 44 (E_MFDB_FK_RESTRICT_VIOLATION): Blocked parent deletion due to existing children.
• Code 47 (E_MFDB_FK_INVALID_ON_DELETE): Invalid on_delete mode (e.g. an explicit null is illegal).

Chapter 6: The Parity Patch & Reality Audit — Exposing Core Bugs & Hardening Rules

Every software vendor lies about their test coverage. They write fancy marketing whitepapers claiming "uncompromising cryptographic validation" while their parser is skipping basic bounds checks behind the scenes. But in the BEJSON ecosystem, we don't do marketing hype. We do adversarial reality audits.

When Elton Boehnen and the engineering team audited the initial release of libraries_105 against actual test suites, they uncovered shocking parity gaps between 104 and 105. This led directly to the historic 105↔104 Parity Patch (Validator 3.4.0, Core 3.6.0, Upgrade 1.1.0, MFDB Core 3.1.0, MFDB Validator 2.4.0). Here is the unvarnished truth of what went wrong, what got fixed, and what you must know to run secure production systems.

The 5 Critical Flaws Caught by the Audit

1. The Skipped Row-Length Bug: Believe it or not, original 105 code allowed short rows, extra rows, and uuid-less rows to validate! The patch enforced offset-aware row length checks (len(Fields) + 1) with Code 9 (E_RECORD_LENGTH_MISMATCH).
2. The Type Mismatch Blindspot: You could insert a string into an integer column in 105 and it passed without error! The patch closed this hole across validate_bejson, add_record_105, write_cell, and mfdb_core_add_entity_record_105 with Code 8 (E_TYPE_MISMATCH).
3. The 105a Primitive Violation: 105a was accepting arrays and objects, violating the primitive-only invariant. The patch locked down 105a to scalar primitives only.
4. The 105db Discriminator Vacuum: 105db was not enforcing the Record_Type_Parent discriminator at column 0 or per-field entity ownership. The patch restored all 104db rules to 105db.
5. The Nightmare self_heal Data Destruction Bug: This was the biggest catch of the century. In older versions, mfdb_core_self_heal() was not offset-aware. When run on a perfectly healthy 105 entity, it assumed the row was one cell too long and deleted the last column of every row in the database! The patch fixed this permanently: self_heal now refuses to blindly modify 105 entities and routes them to remaining_errors for manual review.

The Phantom Error Codes (Busting Myth E90–E95)

Multi-Language Conformance (PY, JS, TS, Shell)

BEJSON is not a Python-only playground. The ecosystem maintains an automated conformance test harness (dev/conformance/) that runs identical test corpuses across four language runtimes:

• 82 single-document validation suites.
• 98 MFDB multi-file database suites (spanning both 104 and 105 trees).
• 11 declarative foreign-key audit suites.

The result? Zero divergence across Python, TypeScript, and Shell. The same validation rules, the same error codes ([30]–[41]), and the same path traversal protections apply everywhere.

Your Production Deployment Checklist

• Always use bejson_core_add_record_105 on 105 files; never the legacy 104 mutator.
• Always use bejson_core_atomic_write (which strips runtime memory keys like _bejson_field_map).
• When creating an MFDB 1.40 database, explicitly pass mfdb_version="1.40" (the helper defaults to 1.31 if omitted!).
• Enable referential integrity with mfdb_core_enable_ref_integrity(manifest) and verify regularly with validate_mfdb_database(manifest, strict_fk=True).