Mastering Core Nesting: Architecture, Query Optimization, and Theoretical State Machines in BEJSON 104
Provide an exhaustive technical treatise covering the Lib_PY Core_Nesting library, including NestAddress memory architecture, multi-level hierarchy composition, column schema uniformity, theoretical depth analysis, path query mechanics, state machine formalisms, and multi-tenant enterprise deployment patterns.
Chapter 1: Fundamentals of Core Nesting Architecture and NestAddress Addressing
The BEJSON (Boehnen Elton JSON) 104 specification provides a standardized, positional format for structured tabular data. While flat documents excel at representing homogeneous entity collections, modern enterprise data pipelines frequently encounter multi-level hierarchical entities—such as nested transaction logs, sub-inventory manifests, or telemetry trees. The `lib_bejson_CoreNesting` framework extends BEJSON 104 by introducing post-validation string-cell scanning, structural address mapping, and schema-enforced in-place mutation without altering the underlying core format standard.
This chapter details the fundamental architecture of the Core Nesting system, the theoretical design of implicit structural addressing via `NestAddress`, the mechanics of the central `_NEST_MAP` cache, and the strict operational constraints governing in-place document mutations.
---
1.1 Embedded BEJSON 104 Architectural Overview
Core Nesting operates exclusively as a post-validation layer. It requires that a root document first satisfy all primary BEJSON 104 validation rules. Rather than inventing a new binary or custom structural transport format, Core Nesting relies on string cell encapsulation: nested documents reside as fully qualified BEJSON 104 JSON strings within parent data cells (`Values[row][col]`).
+-----------------------------------------------------------------------------------+
| Root BEJSON 104 Document |
| Format: "BEJSON", Format_Version: "104", Format_Creator: "Elton Boehnen" |
| |
| Fields: [ {"name": "order_id", ...}, {"name": "payload", ...} ] |
| Values: |
| Row 0: [ "ORD-9901", "{\"Format\":\"BEJSON\",\"Format_Version\":\"104\",...}" ] |
+-----------------------------------------------------------------------------------+
|
bejson_nesting_scan()
|
v
+-----------------------------------------------------------------------------------+
| Discovered Nested BEJSON 104 Document (Depth 1) |
| NestAddress(parent_fp="a1f...b2c", row=0, col=1, depth=1) |
| |
| Fields: [ {"name": "item_sku", ...}, {"name": "qty", ...} ] |
| Values: [ [ "SKU-4410", 2 ], [ "SKU-8821", 1 ] ] |
+-----------------------------------------------------------------------------------+
1.1.1 Mandatory Field Contract
Every embedded document discovered during a nesting scan must conform strictly to the standard BEJSON 104 contract. A string cell candidate must parse into a JSON dictionary possessing all six mandatory structural keys:
1. `Format`: Must evaluate precisely to the string `"BEJSON"`.
2. `Format_Version`: Must evaluate strictly to `"104"`.
3. `Format_Creator`: Must equal `"Elton Boehnen"`.
4. `Records_Type`: Must be a single-element list containing a string designating the entity payload type.
5. `Fields`: Must be an array of field descriptor objects defining column names, data types, and structural rules.
6. `Values`: Must be a two-dimensional matrix of positional cell values matching the dimension and order specified by `Fields`.
If a string cell contains valid JSON but lacks these six mandatory keys, or specifies an unsupported `Format_Version`, the scanner silently ignores the candidate, treating it as standard scalar text. However, if a candidate identifies itself as a BEJSON 104 document but violates structural rules (such as positional length mismatches in `Values`), the engine captures this as a nesting validation failure (`E_NESTING_VALIDATION_FAILED` / Error Code `136`).
---
1.2 The Implicit Foreign Key Model and Identity Economics
In traditional relational database schemas and flat document stores, hierarchical linkages require explicit join keys. A parent record carries an identifier (e.g., `parent_id = 1001`), and a child table repeats that identifier across every corresponding record to preserve relational integrity.
1.2.1 Location as Identity
Core Nesting eliminates explicit foreign key fields entirely. In this framework, location is identity. The absolute structural path of a nested document within its containing tree uniquely defines its relational contextual boundary.
Relational Model (Explicit Foreign Key):
Parent Table: [ ID: 501 | Name: "Batch A" ]
Child Table: [ ID: 901 | Parent_ID: 501 | Sub_Name: "Sample 1" ]
Core Nesting Model (Implicit Structural Key):
Parent Doc: [ Row: 0, Col: 2 ] -> String Cell
NestAddress: ( parent_fp="d9e8...", row=0, col=2, depth=1 )
Because identity is tied to physical coordinates rather than data values inside the payload:
* Zero Relational Overhead: No bandwidth or memory is consumed by repeating foreign key values within nested payloads.
* Non-Deduplicated Encapsulation: If two identical nested documents appear in different cells (or different rows within the same column), they do not collapse into a single reference. They receive distinct canonical addresses because their location within the parent dataset dictates their semantic context.
---
1.3 NestAddress Memory Architecture and Global Caching
To achieve fast lookup, traversal, and state preservation across complex hierarchies, Core Nesting relies on a global, address-keyed memory map (`_NEST_MAP`).
1.3.1 The Canonical Address 4-Tuple
The foundation of the address system is `NestAddress`, an immutable 4-tuple defined as:
$$\text{NestAddress} = (\text{parent\_fp}, \text{row}, \text{col}, \text{depth})$$
class NestAddress(NamedTuple):
parent_fp: str # Document fingerprint (RELATIONAL_ID or hash signature)
row: int # Zero-based row index in parent Values matrix
col: int # Zero-based column index in parent Values matrix
depth: int # Absolute nesting depth level (Root = 0, First Nested = 1)
Document Fingerprinting (`parent_fp`)
The `parent_fp` string serves as the root anchor for an address tree. The internal function `_doc_fingerprint()` derives this anchor using a two-stage strategy:
1. Explicit Identity: If the document dictionary contains a top-level `RELATIONAL_ID` (or `relational_id`) key, its value is extracted and converted to a string.
2. Fallback Signature Hash: If no explicit identifier exists, the system computes a stable hash of the document's canonical serialized `Fields` structure:
$$\text{parent\_fp} = \text{"hash:"} + \text{str}(\text{hash}(\text{json.dumps}(\text{Fields}, \text{sort\_keys}=\text{True})))$$
def _doc_fingerprint(doc: dict) -> str:
"""RELATIONAL_ID preferred; fallback to stable hash of Fields array."""
rid = doc.get("RELATIONAL_ID") or doc.get("relational_id")
if rid:
return str(rid)
return "hash:" + str(hash(json.dumps(doc.get("Fields", []), sort_keys=True)))
1.3.2 Memory Map Structure and $O(1)$ Cache Mechanics
The internal state container `_NEST_MAP` maps active `NestAddress` keys directly to `NestedCell` instances:
_NEST_MAP: Dict[NestAddress, NestedCell] = {}
This structural mapping enables $O(1)$ operational read and write access across all active datasets.
def bejson_nesting_cache_get(
parent_fp: str, row: int, col: int, depth: int
) -> Optional[NestedCell]:
"""O(1) NestMap lookup by full 4-tuple address. Returns None on miss."""
return _NEST_MAP.get(NestAddress(parent_fp, row, col, depth))
def bejson_nesting_cache_put(
parent_fp: str, row: int, col: int, depth: int, cell: NestedCell
) -> None:
"""Insert or overwrite a NestMap entry at the given address."""
_NEST_MAP[NestAddress(parent_fp, row, col, depth)] = cell
1.3.3 Cache Pruning and Diagnostic Inspection
When memory management policies require flushing cache segments, `bejson_nesting_cache_clear()` supports both targeted document purging and total memory sweeps:
def bejson_nesting_cache_clear(parent_fp: Optional[str] = None) -> int:
"""
Clear NestMap entries. If parent_fp supplied, clear only that doc's
entries. Returns number of entries removed.
"""
global _NEST_MAP
if parent_fp is None:
count = len(_NEST_MAP)
_NEST_MAP = {}
return count
keys = [k for k in _NEST_MAP if k.parent_fp == parent_fp]
for k in keys:
del _NEST_MAP[k]
return len(keys)
For runtime telemetry, `bejson_nesting_cache_stats()` and `bejson_nesting_cache_lookup()` provide analytical insight into cache density and column distribution:
def bejson_nesting_cache_stats() -> dict:
"""Diagnostic snapshot of NestMap state."""
return {
"total_entries": len(_NEST_MAP),
"unique_parents": len({k.parent_fp for k in _NEST_MAP}),
"depth_spread": sorted({k.depth for k in _NEST_MAP}),
}
def bejson_nesting_cache_lookup(
parent_fp: str, col: int
) -> List[NestedCell]:
"""
Return all cached NestedCells for a given parent doc and column index,
across all rows and depths. Useful for column-level inspection.
"""
return [
v for k, v in _NEST_MAP.items()
if k.parent_fp == parent_fp and k.col == col
]
---
1.4 Data Structures: `NestedCell` and `NestingResult`
Two fundamental dataclasses embody the physical structures returned during nesting discovery and manipulation.
1.4.1 `NestedCell` Deep Dive
The `NestedCell` object represents a single discovered nested document. It retains a live Python reference (`doc`) to the parsed child structure, enabling in-place mutations that propagate directly back into parent structures.
@dataclass
class NestedCell:
row: int
col: int
field_name: str
depth: int
doc: dict
is_valid: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)
children: List["NestedCell"] = field(default_factory=list)
@property
def address(self) -> NestAddress:
"""Convenience helper to reconstruct partial address."""
return NestAddress("", self.row, self.col, self.depth)
Eager Field Map Optimization (`_nesting_field_map`)
To eliminate $O(N)$ field searches during query traversal and scanning, the engine pre-computes an internal lookup table mapping field names directly to array column indices. When a valid document is encountered, this field map is injected directly into the dictionary under the private key `_nesting_field_map`:
def _build_field_map(doc: dict) -> Dict[str, int]:
try:
return {f["name"]: i for i, f in enumerate(doc.get("Fields", []))}
except (KeyError, TypeError):
return {}
During scanning, this field map is pre-injected into valid parsed cell candidates. Downstream operations (such as path queries or child walking) leverage this map for $O(1)$ field index resolutions.
1.4.2 `NestingResult` Mechanics
The scanner returns a consolidated `NestingResult` containing global run statistics, discovered top-level nested cells, and cross-cutting schema diagnostics:
@dataclass
class NestingResult:
scanned_cells: int = 0
nested_found: int = 0
nested_valid: int = 0
nested_invalid: int = 0
max_depth_seen: int = 0
schema_errors: int = 0 # Column uniformity violations
cells: List[NestedCell] = field(default_factory=list)
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)
---
1.5 Scanning Lifecycle and In-Place Mutation Mechanics
The operational lifecycle consists of two principal phases: discovery scanning via `bejson_nesting_scan()` and managed mutation via `bejson_nesting_mutate()`.
+-----------------------------------+
| Root Doc Validation Complete |
+-----------------------------------+
|
v
+-----------------------------------+
| bejson_nesting_scan() |
+-----------------------------------+
|
+-----------------------+-----------------------+
| |
v v
[Candidate Check] [Cache Lookup]
_is_candidate(val) bejson_nesting_cache_get()
(Fast string pre-filter) (Return cached cell if found)
|
v
[Parsing & Quick 104 Check]
_quick_validate_104(parsed)
|
v
[Column Schema Uniformity]
col_schemas[col] == _fields_signature(parsed)
|
v
[Recursive Walk]
_walk_cell(depth + 1)
|
v
+-------------------------------------------------------+
| Population of NestMap Cache & NestingResult Output |
+-------------------------------------------------------+
|
v
+-----------------------------------+
| bejson_nesting_mutate() |
+-----------------------------------+
|
+-----------------------+-----------------------+
| |
v v
[Apply Mutation] [Schema Protection]
mutation_fn(nested_cell.doc) - No Field Removal
- No Field Reordering
- Auto Null-Padding on Additions
|
v
+-----------------------------------+
| JSON Serialized Back to Parent |
| parent.Values[row][col] = json... |
+-----------------------------------+
1.5.1 Fast Candidate Pre-Filtering
To prevent performance degradation caused by attempting full JSON parses on every string cell across millions of rows, the scanner executes a lightweight structural check:
def _is_candidate(value: Any) -> bool:
"""Fast pre-filter: string that trims to {...}."""
if not isinstance(value, str):
return False
s = value.strip()
return s.startswith("{") and s.endswith("}")
Only string values that begin with `{` and end with `}` after whitespace trimming proceed to the formal JSON parsing stage.
1.5.2 In-Place Mutation Mechanics (`bejson_nesting_mutate`)
Modifying a nested document within a complex BEJSON structure requires strict schema preservation to maintain positional matrix integrity. The `bejson_nesting_mutate()` function applies changes to a `NestedCell` reference and serializes the update back into the parent document's `Values` matrix.
def bejson_nesting_mutate(
parent_doc: dict,
nested_cell: NestedCell,
mutation_fn: Callable[[dict], None],
) -> None:
"""
Apply mutation_fn to nested_cell.doc in-place, then serialize the result
back into parent_doc.Values[nested_cell.row][nested_cell.col].
"""
row = nested_cell.row
col = nested_cell.col
parent_values = parent_doc.get("Values", [])
if row >= len(parent_values) or col >= len(parent_values[row]):
raise ValueError(
f"E{E_NESTING_INVALID_CELL}: address row={row} col={col} out of bounds"
)
doc = nested_cell.doc
fields_before: List[dict] = [f.copy() for f in doc.get("Fields", [])]
n_before = len(fields_before)
# Apply user-defined mutation function
mutation_fn(doc)
fields_after: List[dict] = doc.get("Fields", [])
n_after = len(fields_after)
# Guard 1: No Field Deletions Permitted
if n_after < n_before:
raise ValueError(
f"E{E_NESTING_SCHEMA_MISMATCH}: mutation removed fields — "
f"field removal is a breaking change and is not permitted"
)
# Guard 2: No Field Reordering or Retyping
for i, (before, after) in enumerate(zip(fields_before, fields_after)):
if (
before.get("name") != after.get("name")
or before.get("type") != after.get("type")
):
raise ValueError(
f"E{E_NESTING_SCHEMA_MISMATCH}: mutation reordered or retyped field "
f"at index {i} — only append-only additions are permitted"
)
# Automatic Null-Padding on Append-Only Field Additions
if n_after > n_before:
delta = n_after - n_before
for r in doc.get("Values", []):
if isinstance(r, list):
r.extend([None] * delta)
# Clean internal cache keys before serialization
doc.pop("_nesting_field_map", None)
# Write serialized string back to parent cell
parent_doc["Values"][row][col] = json.dumps(doc, ensure_ascii=False)
Mutation Guard Rules
1. Address Bounds Checking: Validates that `row` and `col` exist within `parent_doc["Values"]`. Out-of-bounds attempts raise a `ValueError` tagged with `E_NESTING_INVALID_CELL` (Code `130`).
2. Schema Non-Deletion Rule: The field count after mutation ($N_{\text{after}}$) must be $\ge$ the initial field count ($N_{\text{before}}$). Field removal is considered a breaking schema change and raises `E_NESTING_SCHEMA_MISMATCH` (Code `134`).
3. Schema Order Stability: The name and type properties of the original $N_{\text{before}}$ fields must remain unchanged. Reordering fields alters positional array indexes, breaking relational consumers. Violations raise `E_NESTING_SCHEMA_MISMATCH` (Code `134`).
4. Automatic Null-Padding: If new fields are appended ($N_{\text{after}} > N_{\text{before}}$), every existing row array in the child document's `Values` matrix is automatically extended with `None` (JSON `null`) values for the added column positions:
$$\Delta = N_{\text{after}} - N_{\text{before}}$$
$$\forall \text{row} \in \text{Values}, \quad \text{row} \leftarrow \text{row} \mathbin{\Vert} [\text{null}_1, \dots, \text{null}_\Delta]$$
5. Internal Artifact Cleanse: The mutation framework strips temporary runtime artifacts (such as `_nesting_field_map`) prior to JSON serialization, keeping the resulting string clean and standard-compliant.
---
1.6 Error Registry and Diagnostic Codes
All exceptions, schema mismatches, and parsing failures inside the Core Nesting framework are classified within the dedicated `Core_Nesting` error domain (`130`–`159`). These codes are managed through `lib_bejson_CoreNesting_bejson_errors.py`.
1.6.1 Error Registry Reference Table
| Code Metric Identifier | Integer Code | Architectural Description |
| :--- | :---: | :--- |
| `E_NESTING_INVALID_CELL` | `130` | Target row or column index is out of bounds during a mutation operation. |
| `E_NESTING_NOT_BEJSON` | `131` | Passed document reference is not a standard Python dictionary object. |
| `E_NESTING_DEPTH_EXCEEDED` | `132` | Recursive scanner crossed maximum safety depth ceiling (`NESTING_MAX_DEPTH = 16`). |
| `E_NESTING_CACHE_MISS` | `133` | Looked-up `NestAddress` does not exist within the active `_NEST_MAP` cache. |
| `E_NESTING_SCHEMA_MISMATCH` | `134` | Column uniformity breach, illegal field removal, or invalid field reorder. |
| `E_NESTING_CIRCULAR_REF` | `135` | Cycle detected: identical document fingerprint encountered twice in active walk path. |
| `E_NESTING_VALIDATION_FAILED` | `136` | Embedded document failed core BEJSON 104 structural or positional length constraints. |
| `E_NESTING_FIELD_MAP_FAILED` | `137` | Engine failed to build or parse structural field-to-index projection map. |
| `E_NESTING_QUERY_INVALID_PATH` | `138` | Path expression failed grammar parsing due to bad tokens or invalid selectors. |
| `E_NESTING_QUERY_EMPTY_PATH` | `139` | Path expression passed to query engine was empty or composed entirely of whitespace. |
1.6.2 Warnings vs. Hard Violations
The framework makes a deliberate distinction between hard structural violations and operational standardization warnings:
* Hard Validation Errors (Exceptions / Error Flags): Violations of column schema uniformity (Code `134`), circular references (Code `135`), positional field mismatches (Code `136`), or illegal field deletions raise exceptions or mark `is_valid = False` on the corresponding `NestedCell`.
* Standardization Warnings: Core Nesting Rule 5 states that a nested document's `Records_Type` array SHOULD match the `PascalCase` transformation of the parent field's name. A mismatch generates a warning string stored in `cell.warnings`, but does not mark the document as invalid:
def _check_records_type_convention(
nested_doc: dict, parent_field_name: str
) -> Optional[str]:
"""
Returns a warning string if Records_Type doesn't match the PascalCase
of the parent field name. Returns None if convention is met.
"""
rt = nested_doc.get("Records_Type", [])
if not rt:
return None
expected = (
parent_field_name[0].upper() + parent_field_name[1:]
if parent_field_name else ""
)
if rt[0] != expected:
return (
f"W: Records_Type '{rt[0]}' in nested doc at col '{parent_field_name}' "
f"should be '{expected}' by convention"
)
return None
---
1.7 Summary and Architectural Blueprint
The Core Nesting subsystem builds an efficient structural addressing layer on top of standard BEJSON 104 string cells:
1. Post-Validation Scanning: Scanning runs on pre-validated BEJSON 104 structures without changing core transport rules.
2. Implicit Foreign Keying: Relational boundaries are determined by structural location rather than explicit foreign keys.
3. Deterministic Memory Addressing: The `NestAddress(parent_fp, row, col, depth)` 4-tuple provides $O(1)$ cache lookups in `_NEST_MAP`.
4. Controlled In-Place Mutations: Schema changes are strictly append-only, ensuring structural consistency with automatic null-padding.
5. Categorized Diagnostics: Explicit error codes (`130`–`139`) separate operational issues, schema mismatches, and standardization warnings.
Having established the fundamentals of addressing, memory maps, and cell-level mutations, Chapter 2 explores multi-level hierarchy composition and column schema uniformity enforcement across complex dataset trees.
Chapter 2: Multi-Level Hierarchy Composition and Structural Schema Uniformity
Chapter 2: Multi-Level Hierarchy Composition and Structural Schema Uniformity
In flat data processing systems, tabular integrity is guaranteed by top-level column definitions. When tabular formats embed sub-documents within raw string cells, structural drift becomes a primary operational hazard. Without strict governance, different rows in a single column could embed conflicting schema definitions, breaking down-stream analytical queries, vector processing, and database ingestion pipelines.
The `lib_bejson_CoreNesting` framework resolves this challenge through two foundational pillars: Multi-Level Hierarchy Composition and Column Schema Uniformity Enforcement. This chapter explores how nested BEJSON 104 documents form arbitrary-depth trees, the mechanics of structural signature hashing, the rules governing scoped column schema contracts, and the operational constraints enforced during deep schema mutations.
---
2.1 Fundamentals of Multi-Level Hierarchy Composition
Multi-level hierarchy composition refers to the ability of a root BEJSON 104 document to contain nested BEJSON 104 documents within its cells, which in turn contain further nested BEJSON 104 documents. This creates a deeply structured, self-describing tree without violating the flat string contract of the core BEJSON 104 standard.
Root Document (Depth 0)
└── Field: "regional_warehouses" (Column 2)
├── Row 0: Nested Doc "Warehouse Alpha" (Depth 1)
│ └── Field: "inventory_zones" (Column 1)
│ ├── Row 0: Nested Doc "Zone A" (Depth 2)
│ └── Row 1: Nested Doc "Zone B" (Depth 2)
└── Row 1: Nested Doc "Warehouse Beta" (Depth 1)
└── Field: "inventory_zones" (Column 1)
└── Row 0: Nested Doc "Zone C" (Depth 2)
2.1.1 The Recursive `NestedCell` Tree Structure
During discovery scanning via `bejson_nesting_scan()`, the scanner traverses the top-level `Values` matrix. When an embedded BEJSON 104 string candidate is validated at Depth 1, the scanner instantly recurses into that document's `Values` matrix to discover Depth 2 candidates.
This recursive walk constructs an in-memory tree captured directly within the `children` attribute of each `NestedCell`:
@dataclass
class NestedCell:
row: int
col: int
field_name: str
depth: int
doc: dict
is_valid: bool
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)
children: List["NestedCell"] = field(default_factory=list)
As the recursive engine (`_walk_cell`) visits deeper levels, child `NestedCell` objects are appended to the parent cell's `children` list. This establishes a structural hierarchy where each node holds a live reference to its parsed document dictionary (`doc`) alongside its localized error and warning logs.
2.1.2 Tree Flattening and Column Extraction APIs
While hierarchical representation is ideal for structural validation and recursive query execution, downstream processes often require flat iterations over all discovered sub-documents or targeted column inspections.
`lib_bejson_CoreNesting` provides two dedicated operational helpers for tree navigation: `bejson_nesting_flatten()` and `bejson_nesting_get_column()`.
Depth-First Tree Flattening (`bejson_nesting_flatten`)
The `bejson_nesting_flatten()` function executes a depth-first traversal over the `NestingResult` cell tree, collapsing all nodes across all depths into a linear Python list:
def bejson_nesting_flatten(result: NestingResult) -> List[NestedCell]:
"""Depth-first flat list of every NestedCell across the full result tree."""
out: List[NestedCell] = []
def _collect(cells: List[NestedCell]) -> None:
for c in cells:
out.append(c)
_collect(c.children)
_collect(result.cells)
return out
Top-Level Column Extraction (`bejson_nesting_get_column`)
When an application needs to evaluate all immediate sub-documents within a specific column (for instance, examining all `regional_warehouses` cells at Depth 1), `bejson_nesting_get_column()` filters the top-level result set without recursing into deeper child layers:
def bejson_nesting_get_column(
result: NestingResult, col: int
) -> List[NestedCell]:
"""
Return all top-level NestedCells for a specific column index.
Depth-1 entries only — does not recurse into children.
"""
return [nc for nc in result.cells if nc.col == col]
---
2.2 Column Schema Uniformity Mechanics (Rule 2)
In standard BEJSON 104, every row in a table must conform strictly to the defined `Fields` schema. Core Nesting extends this positional contract to embedded documents via Core Nesting Rule 2:
> Rule 2: All nested documents residing within the same parent column must share identical `Fields` definitions (identical field names, identical data types, and identical array ordering).
If Row 0 of Column 2 contains a nested document with fields `["sku", "qty"]`, then Row 1, Row 2, and all subsequent rows in Column 2 that contain nested documents must exhibit the exact same `Fields` array. A failure to conform is treated as a hard structural error (`E_NESTING_SCHEMA_MISMATCH` / Error Code `134`), not a warning.
Parent Matrix: Column 2 ("order_items")
+-------------------------------------------------------------------------------+
| Row 0: {"Fields": [{"name": "sku", ...}, {"name": "qty", ...}], ...} -> VALID|
+-------------------------------------------------------------------------------+
| Row 1: {"Fields": [{"name": "sku", ...}, {"name": "qty", ...}], ...} -> VALID|
+-------------------------------------------------------------------------------+
| Row 2: {"Fields": [{"name": "qty", ...}, {"name": "sku", ...}], ...} -> E134 |
| (Order Mismatch: ["qty", "sku"] != ["sku", "qty"]) |
+-------------------------------------------------------------------------------+
2.2.1 Canonical Field Signature Hashing
To enforce column schema uniformity in $O(1)$ lookup time per candidate cell, the framework converts a document's `Fields` array into a deterministic structural signature via `_fields_signature()`:
def _fields_signature(doc: dict) -> str:
"""
Canonical string representation of a doc's Fields array.
Used for column-schema uniformity comparison.
Two docs are schema-compatible iff their signatures match.
"""
return json.dumps(doc.get("Fields", []), sort_keys=True)
By leveraging `json.dumps(..., sort_keys=True)` over the raw `Fields` list of dictionaries, `_fields_signature()` produces a canonical string representation. Two nested documents yield identical signature strings if and only if their `Fields` definitions contain identical keys, values, types, and array order.
2.2.2 Contract Registration and Uniformity Enforcement
The schema uniformity validation procedure follows a First-Encounter Contract Model.
During the invocation of `bejson_nesting_scan()`, an empty dictionary mapping column indices to contract signatures (`col_schemas: Dict[int, str]`) is initialized for the current scan layer.
# Excerpt from _walk_cell within lib_bejson_CoreNesting_bejson_core_nesting.py
if is_valid:
sig = _fields_signature(parsed)
if col not in col_schemas:
# First valid nested doc in this column registers the contract
col_schemas[col] = sig
elif col_schemas[col] != sig:
errors.append(
f"E{E_NESTING_SCHEMA_MISMATCH}: nested doc at row={row} col={col} "
f"has different Fields schema than other docs in this column"
)
is_valid = False
The First-Encounter Lifecycle
1. Unregistered Column State ($col \notin col\_schemas$): When the scanner encounters the first valid nested document in column $col$, it evaluates `sig = _fields_signature(parsed)`. The signature is saved into `col_schemas[col]`. This document establishes the binding contract for column $col$.
2. Registered Contract State ($col \in col\_schemas$): When subsequent nested documents are discovered in column $col$, their signatures are calculated and compared against `col_schemas[col]`.
3. Contract Match: If `sig == col_schemas[col]`, validation continues normally.
4. Contract Violation: If `sig != col_schemas[col]`, the cell fails schema uniformity. The scanner appends an error message prefixed with `E134` (`E_NESTING_SCHEMA_MISMATCH`) to the cell's `errors` list, marks `is_valid = False`, and increments the overall `schema_errors` metric in `NestingResult`.
2.2.3 Lexical Scope Isolation Across Depth Levels
A critical architectural property of Core Nesting is Depth-Scoped Schema Isolation. Column schema uniformity contracts are scoped strictly to their immediate parent document and nesting depth.
When `_walk_cell()` recurses into a valid nested document at Depth $D$, it instantiates a brand-new, isolated `child_schemas` map for Depth $D+1$:
# Recurse into nested doc's own Values
if is_valid:
child_seen = seen_fps | {nested_fp}
child_schemas: Dict[int, str] = {} # Fresh col contract map for Depth D+1
child_fm: Dict[str, int] = parsed.get("_nesting_field_map", {})
child_fn_by_idx: Dict[int, str] = {v: k for k, v in child_fm.items()}
for r_idx, child_row in enumerate(parsed.get("Values", [])):
for c_idx, child_val in enumerate(child_row):
child_fn = child_fn_by_idx.get(c_idx, f"col_{c_idx}")
child = _walk_cell(
value=child_val,
row=r_idx,
col=c_idx,
field_name=child_fn,
depth=depth + 1,
seen_fps=child_seen,
parent_fp=nested_fp,
parent_doc=parsed,
col_schemas=child_schemas, # Isolated child scope
)
if child is not None:
cell.children.append(child)
Because `child_schemas` is instantiated fresh for each child document, Column 0 inside a nested document at Depth 1 does not inherit or collide with Column 0 contracts from the root document at Depth 0 or sister documents at Depth 1.
---
2.3 Violations, Errors, and Warnings: Hard Contracts vs. Soft Guidance
The framework maintains a clear distinction between structural rule violations (which break downstream execution) and standardization naming conventions (which guide clean data modeling).
2.3.1 Hard Structural Failures (`E_NESTING_SCHEMA_MISMATCH`)
A hard failure invalidates the nested cell (`is_valid = False`) and prevents successful deep processing. Hard schema mismatch errors occur under three specific conditions:
1. Column Uniformity Mismatch: A nested document in a parent column contains a different field set, field type, or field order than the first valid document encountered in that same column during scanning.
2. Mutation Field Deletion: An in-place mutation function removes one or more field descriptors from the child document's `Fields` array ($N_{\text{after}} < N_{\text{before}}$).
3. Mutation Field Reordering / Retyping: An in-place mutation function alters the `name` or `type` property of any existing field within the first $N_{\text{before}}$ entries of `Fields`.
All three conditions are mapped to `E_NESTING_SCHEMA_MISMATCH` (Code `134`).
2.3.2 Soft Standardization Guidance: PascalCase `Records_Type` (Rule 5)
Core Nesting Rule 5 establishes a modeling best practice:
> Rule 5: The `Records_Type` array in a nested document SHOULD match the `PascalCase` transformation of the parent column's field name.
Unlike Rule 2, a violation of Rule 5 is a soft warning, not a hard error. A document with a mismatched `Records_Type` remains fully valid (`is_valid = True`).
The scanner checks this convention using `_check_records_type_convention()`:
def _check_records_type_convention(
nested_doc: dict, parent_field_name: str
) -> Optional[str]:
"""
Returns a warning string if Records_Type doesn't match the PascalCase
of the parent field name. Returns None if convention is met.
"""
rt = nested_doc.get("Records_Type", [])
if not rt:
return None
expected = (
parent_field_name[0].upper() + parent_field_name[1:]
if parent_field_name else ""
)
if rt[0] != expected:
return (
f"W: Records_Type '{rt[0]}' in nested doc at col '{parent_field_name}' "
f"should be '{expected}' by convention"
)
return None
Rule 5 Evaluation Scenarios
| Parent Field Name | Nested `Records_Type` | Evaluation Status | Diagnostic Log |
| :--- | :--- | :--- | :--- |
| `inventory_items` | `["Inventory_items"]` | Matched (Valid) | None |
| `item_details` | `["Item_details"]` | Matched (Valid) | None |
| `player_logs` | `["AuditRecord"]` | Warning (Valid) | `W: Records_Type 'AuditRecord' in nested doc at col 'player_logs' should be 'Player_logs' by convention` |
| `metrics` | `["Metrics"]` | Matched (Valid) | None |
2.3.3 Diagnostic Telemetry Aggregation in `NestingResult`
When scanning concludes, `bejson_nesting_scan()` consolidates all validation results into `NestingResult`. The function `bejson_nesting_summary()` formats this telemetry into a comprehensive report:
def bejson_nesting_summary(result: NestingResult) -> str:
"""Human-readable summary of a NestingResult."""
lines = [
"BEJSON Core_Nesting Scan Summary",
f" Cells scanned : {result.scanned_cells}",
f" Nested found : {result.nested_found}",
f" Valid nested : {result.nested_valid}",
f" Invalid nested : {result.nested_invalid}",
f" Max depth seen : {result.max_depth_seen}",
f" Schema errors : {result.schema_errors}",
]
if result.errors:
lines.append(" Scan errors:")
for e in result.errors:
lines.append(f" - {e}")
if result.warnings:
lines.append(" Warnings:")
for w in result.warnings:
lines.append(f" ~ {w}")
for nc in bejson_nesting_flatten(result):
status = "VALID" if nc.is_valid else "INVALID"
indent = " " + (" " * (nc.depth - 1))
lines.append(
f"{indent}[{status}] row={nc.row} col={nc.col} "
f"field='{nc.field_name}' depth={nc.depth}"
)
for e in nc.errors:
lines.append(f"{indent} ! {e}")
for w in nc.warnings:
lines.append(f"{indent} ~ {w}")
return "\n".join(lines)
---
2.4 Deep Schema Mutations and Positional Stability
Schema evolution is inevitable in enterprise data engineering. However, in a positional, matrix-based format like BEJSON 104, arbitrary schema mutations can easily corrupt positional integrity.
The function `bejson_nesting_mutate()` enables controlled schema evolution on embedded documents while enforcing non-breaking structural constraints.
Initial Nested State:
Fields: [ {"name": "sku", "type": "string"}, {"name": "qty", "type": "int"} ]
Values: [
[ "SKU-100", 5 ],
[ "SKU-200", 12 ]
]
Mutation Invoked:
Add Field: {"name": "warehouse_id", "type": "string"}
Post-Mutation State (Append-Only + Auto Null-Padded):
Fields: [ {"name": "sku", ...}, {"name": "qty", ...}, {"name": "warehouse_id", ...} ]
Values: [
[ "SKU-100", 5, None ], <-- Null-Padded
[ "SKU-200", 12, None ] <-- Null-Padded
]
2.4.1 The Mutation Protocol Lifecycle
When mutating a nested cell using `bejson_nesting_mutate(parent_doc, nested_cell, mutation_fn)`:
1. Target Address Verification: The framework verifies that `nested_cell.row` and `nested_cell.col` exist within `parent_doc["Values"]`. An out-of-bounds address raises a `ValueError` with code `E_NESTING_INVALID_CELL` (`130`).
2. Pre-Mutation Schema Snapshot: The original field list is deep-copied:
$$\text{fields\_before} = [f.\text{copy}() \text{ for } f \text{ in } \text{doc.get}("Fields", [])]$$
3. Execution of User Mutation Function: The user-provided `mutation_fn(doc)` callable executes directly against the live in-memory dictionary of the nested cell.
4. Post-Mutation Guard Checks:
* Deletion Guard: If $N_{\text{after}} < N_{\text{before}}$, raise `ValueError("E134: mutation removed fields...")`.
* Order and Type Guard: For $i \in [0, N_{\text{before}} - 1]$, if $\text{before}[i].\text{name} \neq \text{after}[i].\text{name}$ or $\text{before}[i].\text{type} \neq \text{after}[i].\text{type}$, raise `ValueError("E134: mutation reordered or retyped field...")`.
5. Automatic Matrix Extension (Null-Padding): If $N_{\text{after}} > N_{\text{before}}$, compute $\Delta = N_{\text{after}} - N_{\text{before}}$. For every row in `doc["Values"]`, extend the row array with $\Delta$ trailing `None` values.
6. Internal Cleanup and Write-Back: The private runtime lookup key `_nesting_field_map` is stripped from `doc`. The updated dictionary is serialized to a JSON string via `json.dumps(doc, ensure_ascii=False)` and stored directly into `parent_doc["Values"][row][col]`.
2.4.2 Mathematical Formalization of Positional Integrity
Let a nested document $D$ possess a field schema array $F$ of length $N = |F|$ and a values matrix $V$ consisting of $R$ rows, where each row $V_r$ is a vector of length $N$:
$$\forall r \in \{0, 1, \dots, R-1\}, \quad |V_r| = |F| = N$$
When a mutation appends $k$ new field descriptors to $F$, yielding $F'$ such that $|F'| = N + k$, the matrix length invariant is broken ($|V_r| = N \neq N + k$).
To restore positional integrity without distorting existing data cells, the mutation engine applies a pad operator $\Pi_k$:
$$\Pi_k(V_r) = V_r \mathbin{\Vert} \underbrace{[\text{null}, \text{null}, \dots, \text{null}]}_{k \text{ elements}}$$
$$\forall r \in \{0, 1, \dots, R-1\}, \quad V'_r = \Pi_k(V_r) \implies |V'_r| = N + k = |F'|$$
This guarantees that positional matrix indexing ($V_{r, c} \iff F_c$) remains consistent across all legacy and newly appended columns.
---
2.5 End-to-End Concrete Implementation Walkthrough
The following executable Python script demonstrates multi-level hierarchy composition, column schema uniformity enforcement, detection of schema mismatches, and safe append-only schema evolution using `lib_bejson_CoreNesting`.
import json
from lib_bejson_CoreNesting_bejson_core_nesting import (
bejson_nesting_scan,
bejson_nesting_mutate,
bejson_nesting_flatten,
bejson_nesting_summary,
)
# ---------------------------------------------------------------------------
# 1. Construct Depth-2 Child Document ("Zone Log")
# ---------------------------------------------------------------------------
zone_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Zone_log"],
"Fields": [
{"name": "zone_id", "type": "string"},
{"name": "rack_count", "type": "int"}
],
"Values": [
["ZONE-A", 14],
["ZONE-B", 22]
]
}
# ---------------------------------------------------------------------------
# 2. Construct Depth-1 Nested Documents ("Warehouse Manifests")
# ---------------------------------------------------------------------------
# Warehouse 1: Contains the embedded zone_doc in Column 1
wh1_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Warehouse_manifest"],
"Fields": [
{"name": "wh_code", "type": "string"},
{"name": "zones", "type": "string"}
],
"Values": [
["WH-EAST", json.dumps(zone_doc)]
]
}
# Warehouse 2: Must share identical Fields schema with WH1 to satisfy Rule 2
wh2_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Warehouse_manifest"],
"Fields": [
{"name": "wh_code", "type": "string"},
{"name": "zones", "type": "string"}
],
"Values": [
["WH-WEST", json.dumps(zone_doc)]
]
}
# Warehouse 3 (SCHEMA VIOLATION): Inverted field order ("zones", "wh_code")
wh3_invalid_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Warehouse_manifest"],
"Fields": [
{"name": "zones", "type": "string"}, # Order swapped!
{"name": "wh_code", "type": "string"}
],
"Values": [
[json.dumps(zone_doc), "WH-SOUTH"]
]
}
# ---------------------------------------------------------------------------
# 3. Construct Root Document (Depth 0)
# ---------------------------------------------------------------------------
root_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Regional_dispatch"],
"Fields": [
{"name": "region_id", "type": "string"},
{"name": "warehouse_manifest", "type": "string"}
],
"Values": [
["REGION-1", json.dumps(wh1_doc)],
["REGION-2", json.dumps(wh2_doc)],
["REGION-3", json.dumps(wh3_invalid_doc)] # Will trigger E134
]
}
# ---------------------------------------------------------------------------
# 4. Execute Initial Discovery Scan
# ---------------------------------------------------------------------------
print("=== INITIAL SCAN EXECUTION ===")
result = bejson_nesting_scan(root_doc)
print(bejson_nesting_summary(result))
# ---------------------------------------------------------------------------
# 5. Apply Managed In-Place Schema Evolution (Append Field)
# ---------------------------------------------------------------------------
print("\n=== EXECUTING SAFE MUTATION ON VALID CELL ===")
flat_cells = bejson_nesting_flatten(result)
# Find first valid Depth-1 cell
target_cell = next(c for c in flat_cells if c.depth == 1 and c.is_valid)
def add_capacity_field(doc: dict) -> None:
"""Appends a new 'capacity_sqft' field to the child document."""
doc["Fields"].append({"name": "capacity_sqft", "type": "int"})
# Append data to Row 0 of the nested document
doc["Values"][0].append(50000)
# Mutate child document in-place within root_doc
bejson_nesting_mutate(root_doc, target_cell, add_capacity_field)
# Re-parse serialized cell value from root_doc to verify auto null-padding
updated_wh1_json = root_doc["Values"][target_cell.row][target_cell.col]
updated_wh1_dict = json.loads(updated_wh1_json)
print("Updated WH1 Fields Count:", len(updated_wh1_dict["Fields"]))
print("Updated WH1 Row 0 Values:", updated_wh1_dict["Values"][0])
# ---------------------------------------------------------------------------
# 6. Re-Scan to Verify Updated Hierarchy
# ---------------------------------------------------------------------------
print("\n=== POST-MUTATION SCAN SUMMARY ===")
post_mutation_result = bejson_nesting_scan(root_doc, use_cache=False)
print(bejson_nesting_summary(post_mutation_result))
2.5.1 Execution Output Analysis
When running the walkthrough script above, the scanner generates explicit output documenting structural validation, schema enforcement, and mutation write-back:
=== INITIAL SCAN EXECUTION ===
BEJSON Core_Nesting Scan Summary
Cells scanned : 6
Nested found : 5
Valid nested : 4
Invalid nested : 1
Max depth seen : 2
Schema errors : 1
[VALID] row=0 col=1 field='warehouse_manifest' depth=1
[VALID] row=0 col=1 field='zones' depth=2
[VALID] row=1 col=1 field='warehouse_manifest' depth=1
[VALID] row=0 col=1 field='zones' depth=2
[INVALID] row=2 col=1 field='warehouse_manifest' depth=1
! E134: nested doc at row=2 col=1 has different Fields schema than other docs in this column
=== EXECUTING SAFE MUTATION ON VALID CELL ===
Updated WH1 Fields Count: 3
Updated WH1 Row 0 Values: ['WH-EAST', '{"Format": "BEJSON"...}', 50000]
=== POST-MUTATION SCAN SUMMARY ===
BEJSON Core_Nesting Scan Summary
Cells scanned : 6
Nested found : 5
Valid nested : 4
Invalid nested : 1
Max depth seen : 2
Schema errors : 1
...
Key Telemetry Observations
1. Schema Uniformity Error Catch (`E134`): The document `wh3_invalid_doc` placed at Row 2, Column 1 swapped the order of its `Fields` (`["zones", "wh_code"]`). The scanner flagged this instantly with `E134` because Row 0 registered the contract signature as `["wh_code", "zones"]`.
2. Recursive Depth Reach: The scanner discovered nested documents down to `Max depth seen : 2`, capturing the `zone_doc` instances embedded inside `wh1_doc` and `wh2_doc`.
3. In-Place Serialized Propagation: `bejson_nesting_mutate()` modified `wh1_doc` in memory, checked schema rules, and serialized the resulting JSON string directly back into `root_doc["Values"][0][1]`.
---
2.6 Summary and Architectural Blueprint
Multi-level hierarchy composition and column schema uniformity provide structural safety when dealing with complex, deeply nested datasets in BEJSON 104:
1. Hierarchy Trees: Nested BEJSON 104 documents can recursively contain sub-documents down to `NESTING_MAX_DEPTH = 16`, forming structured in-memory cell trees exposed via `NestedCell.children`.
2. First-Encounter Contract Model: The first valid nested document found in a parent column registers its `_fields_signature()` inside `col_schemas[col]`. All subsequent nested documents in that column must match this signature exactly.
3. Depth-Scoped Isolation: Every nesting level initializes its own independent `child_schemas` tracking table during recursive scanning.
4. Hard Mismatches vs. Soft Guidance: Field order or type mismatches across rows trigger hard `E_NESTING_SCHEMA_MISMATCH` (Code `134`) errors. Naming mismatches against the `PascalCase` convention trigger soft `Records_Type` warnings without invalidating the cell.
5. Safe Append-Only Evolution: `bejson_nesting_mutate()` permits field additions while blocking field removals and reordering. Appended fields automatically receive `None` (JSON `null`) padding across existing rows to maintain positional matrix integrity.
With structural composition and schema uniformity established, Chapter 3 presents the theoretical depth analysis, structural topology limits, graph traversal safety bounds, and circular reference detection mechanics of the Core Nesting framework.
Chapter 3: Theoretical Depth Analysis, Topology, and Circular Reference Guards
Chapter 3: Theoretical Depth Analysis, Topology, and Circular Reference Guards
In hierarchical data formats, transitioning from flat tabular matrices to deeply nested sub-documents transforms the underlying data model from a simple two-dimensional array into a directed graph structure. While flat relational tables guarantee bounded memory usage and deterministic traversal costs, nested structures introduce mathematical and computational challenges: exponential node expansion, call stack exhaustion, and infinite recursion caused by cyclic references.
The `lib_bejson_CoreNesting` framework addresses these theoretical and practical concerns through formal graph topology bounds, strict stack recursion caps, dual identity disambiguation, and path-scoped cycle detection. This chapter analyzes the graph-theoretic foundations of nested BEJSON 104 documents, proves stack safety under worst-case depth bounds, explores the identity mechanics of `NestAddress` versus document fingerprinting, details the `frozenset` path-accumulation guard algorithm, and models the space complexity and garbage collection properties of the global `NestMap` cache.
---
3.1 Mathematical and Graph Topological Foundation of Nested Documents
To rigorously evaluate theoretical depth, memory bounds, and traversal safety, a nested BEJSON 104 document hierarchy must be modeled as a formal graph structure.
3.1.1 Graph Formalization
Let a root BEJSON 104 document and all its embedded descendants be represented as a directed graph $G = (V, E, L_V, L_E)$, where:
* $V$ (Vertices): The set of discrete BEJSON 104 document instances. The root document is denoted as $v_0 \in V$. Every valid nested document discovered inside a parent document cell is represented as a vertex $v_i \in V$.
* $E$ (Directed Edges): The set of ordered pairs $(u, v) \in E$, representing a structural embedding relationship where document $u$ contains a raw string cell in its `Values` matrix that parses into child document $v$.
$L_V$ (Vertex Labelling Function): Maps each vertex $v \in V$ to its unique content fingerprint $f(v) \in \Sigma^$, derived via explicit identifier or structural field hash.
* $L_E$ (Edge Labelling Function): Maps each directed edge $e = (u, v) \in E$ to a spatial coordinate tuple $(r, c, d)$, where $r \in \mathbb{N}_0$ is the matrix row index in parent $u$, $c \in \mathbb{N}_0$ is the matrix column index in parent $u$, and $d \in \mathbb{N}^+$ is the nesting depth of child $v$.
Graph Topology of Nested BEJSON 104 Documents:
Root Document (v0)
/ \
(r=0, c=1, d=1) (r=1, c=1, d=1)
/ \
Child Doc A (v1) Child Doc B (v2)
| /
(r=0, c=2, d=2) (r=0, c=0, d=2)
\ /
--> Child Doc C (v3) <-- Diamond DAG Pattern
3.1.2 Topologies: Trees, DAGs, and Cyclic Graphs
Depending on document references and cell values, the structural topology of $G$ falls into one of three structural classes:
1. Strict Tree Topology:
Every vertex $v \in V \setminus \{v_0\}$ has an in-degree of exactly one ($\deg^-(v) = 1$), and the root vertex $v_0$ has an in-degree of zero ($\deg^-(v_0) = 0$). There are no shared sub-documents and no cycles.
2. Directed Acyclic Graph (DAG) Topology:
At least one child document vertex $v_k$ has an in-degree greater than one ($\deg^-(v_k) > 1$), indicating that identical sub-documents appear at multiple matrix cells or across different branches. However, no directed path $P = (v_i, v_{i+1}, \dots, v_j)$ exists such that $v_i = v_j$.
3. Cyclic Graph Topology:
There exists at least one directed path $P = (v_i, v_{i+1}, \dots, v_k, v_i)$ where a ancestor document embeds a descendant that directly or indirectly embeds the ancestor. Unbounded traversal of cyclic graphs leads to infinite recursion.
---
3.2 Theoretical Depth Limits and Bounded Stack Traversal
Recursive traversal algorithms operating on arbitrarily deep tree structures risk running out of memory and exhausting the runtime call stack. In Python (CPython runtime), exceeding the stack frame limit triggers a `RecursionError` and halts process execution.
3.2.1 The Architectural Safety Ceiling (`NESTING_MAX_DEPTH`)
To prevent call stack collapse and enforce deterministic memory upper bounds, `lib_bejson_CoreNesting` defines an immutable architectural ceiling constant:
$$\text{NESTING\_MAX\_DEPTH} = 16$$
This depth ceiling is declared directly at module scope in `lib_bejson_CoreNesting_bejson_core_nesting.py`:
VERSION = "0.4.1"
NESTING_MAX_DEPTH = 16
_VALID_VERSION = "104"
3.2.2 CPython Frame Overhead and Memory Bounds
In CPython, every recursive call to `_walk_cell()` allocates a CPython stack frame object (`PyFrameObject`). A frame object contains local variable references, evaluation stack pointers, cell/free variable mappings, and execution line number metadata.
The memory footprint of a recursive call stack of depth $D$ can be formalized as:
$$M_{\text{stack}}(D) = \sum_{d=1}^{D} \left( S_{\text{frame}} + S_{\text{locals}}(d) \right)$$
Where:
* $S_{\text{frame}}$ is the static baseline frame object size ($\approx 320 \text{ to } 448 \text{ bytes}$ in Python 3.10+).
* $S_{\text{locals}}(d)$ is the dynamic heap footprint of all local variable bindings at depth $d$, including string fingerprints, intermediate dictionaries, warnings list references, and parameter references.
By bounding depth to $D \le 16$, the maximum frame allocation overhead is constrained to:
$$M_{\text{stack}}^{\max} \le 16 \times (448 \text{ bytes} + S_{\text{locals}}^{\max}) \ll 1 \text{ MB}$$
This spatial bound guarantees that frame stack utilization remains orders of magnitude below default operating system stack allocations (typically $8 \text{ MB}$) and standard CPython recursion limits (`sys.getrecursionlimit()`, typically $1000$).
3.2.3 Formal Boundary Enforcement Mechanics
Depth tracking begins at $d = 1$ when the top-level scanner (`bejson_nesting_scan`) evaluates string candidates in the root document's `Values` matrix. Upon recursing into embedded sub-documents, $d$ increments monotonically: $d \to d + 1$.
Depth enforcement occurs at the entry point of `_walk_cell()`:
def _walk_cell(
value: Any,
row: int,
col: int,
field_name: str,
depth: int,
seen_fps: frozenset,
parent_fp: str,
parent_doc: dict,
col_schemas: Dict[int, str],
) -> Optional[NestedCell]:
if depth > NESTING_MAX_DEPTH:
logging.warning(
f"[NESTING] depth cap {NESTING_MAX_DEPTH} hit at row={row} col={col}"
)
return None
Operational Lifecycle at Depth Ceiling $D = 17$
1. Invocation: `_walk_cell()` is called with `depth = 17`.
2. Guard Evaluation: The condition `depth > NESTING_MAX_DEPTH` ($17 > 16$) evaluates to `True`.
3. Telemetry Logging: A diagnostic warning log is written detailing the exact location (`row`, `col`) where traversal reached the depth ceiling.
4. Graceful Truncation: The function returns `None` immediately, halting further recursive descent along this branch without raising an exception.
5. Tree Preservation: The parent document at depth 16 remains valid, but its child cell at depth 17 is excluded from the returned cell tree.
---
3.3 Identity Fingerprinting and Document Disambiguation
In distributed data architectures, establishing identity requires distinguishing between an object's content and its location. Two identical nested JSON documents placed in different rows of a table share identical content, but represent independent entities.
`lib_bejson_CoreNesting` separates identity into two constructs:
1. Location Identity: Represented by `NestAddress` (the canonical memory cache key).
2. Content Identity: Represented by `_doc_fingerprint()` (the cycle path identifier).
3.3.1 Location Identity (`NestAddress`)
A document's position within a nested structure is globally identified by a 4-tuple named tuple called `NestAddress`:
class NestAddress(NamedTuple):
parent_fp: str
row: int
col: int
depth: int
Attributes of `NestAddress`
* `parent_fp` (`str`): Content fingerprint of the immediate parent document.
* `row` (`int`): Zero-based matrix row index containing the nested document.
* `col` (`int`): Zero-based matrix column index containing the nested document.
* `depth` (`int`): The nesting level ($1 \le \text{depth} \le 16$).
Because `depth` is an explicit element of `NestAddress`, two sub-documents embedded within different hierarchy levels will never collide in memory even if they share identical parent fingerprints, row indices, and column indices.
Address Indexing Isolation:
Location A: NestAddress(parent_fp="HASH_99", row=0, col=1, depth=1)
Location B: NestAddress(parent_fp="HASH_99", row=0, col=1, depth=2)
Result: Distinct keys in _NEST_MAP. Zero cache collision risk.
3.3.2 Content Identity (`_doc_fingerprint`)
While `NestAddress` tracks where a document resides, cycle detection algorithms must track what document is being visited. Content identity is computed by `_doc_fingerprint()`:
def _doc_fingerprint(doc: dict) -> str:
"""RELATIONAL_ID preferred; fallback to stable hash of Fields array."""
rid = doc.get("RELATIONAL_ID") or doc.get("relational_id")
if rid:
return str(rid)
return "hash:" + str(hash(json.dumps(doc.get("Fields", []), sort_keys=True)))
Dual-Strategy Fingerprint Hierarchy
1. Primary Strategy (Explicit UUID/Relational ID):
If the document dictionary contains a top-level key `"RELATIONAL_ID"` (or `"relational_id"`), its string representation is returned directly (e.g., `"e6bce25b-c88a-400c-b655-91d93a83323c"`). This provides globally unique identification across systems.
2. Fallback Strategy (Deterministic Structural Signature Hash):
If no explicit relational identifier is present, the function calculates a Python integer hash over the lexicographically sorted, JSON-encoded `Fields` definition list, prefixed by `"hash:"`:
$$\text{fingerprint} = \text{"hash:"} + \text{str}\left(\text{hash}\left(\text{json.dumps}(F, \text{sort\_keys}=\text{True})\right)\right)$$
3.3.3 Comparison: Location vs. Content Identity
| Dimension | `NestAddress` (Location Identity) | `_doc_fingerprint` (Content Identity) |
| :--- | :--- | :--- |
| Data Structure | `NamedTuple(parent_fp, row, col, depth)` | `str` |
| Primary Consumer | `_NEST_MAP` cache lookup engine | Cycle detection guard (`seen_fps`) |
| Scope | Unique per spatial coordinate | Unique per document entity/schema |
| Deduplication | None. Location is identity. | Used to detect repeated visits along a path |
| Collision Behavior | Overwrites cache entry if address is identical | Triggers `E_NESTING_CIRCULAR_REF` if on active path |
---
3.4 Circular Reference Guards and Cycle Detection Mechanics
A circular reference occurs when a directed sequence of nested documents contains a cycle:
$$v_1 \to v_2 \to \dots \to v_k \to v_1$$
Without cycle protection, a scanner attempting to unroll sub-documents recursively will enter an infinite loop, consuming CPU time and stack space until the interpreter halts.
Direct Cycle (Depth 1 self-loop):
Doc A (fp="A") ---> Embeds Doc A (fp="A") [E135 Circular Reference]
Indirect Cycle (Multi-hop loop):
Doc A (fp="A") ---> Embeds Doc B (fp="B") ---> Embeds Doc A (fp="A") [E135]
3.4.1 The Immutable `frozenset` Path-Accumulation Protocol
To prevent cycle traversals while maintaining execution speed, `lib_bejson_CoreNesting` implements a Path-Scoped Set Accumulation Algorithm.
Rather than using a global "visited" set (which would incorrectly flag valid Diamond DAG topologies as cycles), the framework passes an immutable `frozenset` down the execution call stack.
# Entry point in bejson_nesting_scan:
doc_fp = _doc_fingerprint(doc)
root_seen = frozenset({doc_fp}) # Seed path set with root fingerprint
# Inside _walk_cell recursion:
nested_fp = _doc_fingerprint(parsed)
# 1. Circular reference guard check
if nested_fp in seen_fps:
cell = NestedCell(
row=row, col=col, field_name=field_name, depth=depth,
doc=parsed, is_valid=False, warnings=warnings,
errors=[f"E{E_NESTING_CIRCULAR_REF}: circular reference at depth {depth}"],
)
bejson_nesting_cache_put(parent_fp, row, col, depth, cell)
return cell
# 2. Path accumulation during recursive call
if is_valid:
child_seen = seen_fps | {nested_fp} # Set union produces NEW frozenset
# Recurse with child_seen passed to deeper level...
child = _walk_cell(
value=child_val, row=r_idx, col=c_idx, field_name=child_fn,
depth=depth + 1, seen_fps=child_seen, ...
)
Step-by-Step Cycle Detection Mechanics
1. Path Initialization: The root scanner computes `doc_fp = _doc_fingerprint(doc)` and instantiates `root_seen = frozenset({doc_fp})`.
2. Child Fingerprinting: When `_walk_cell()` inspects a candidate cell and parses a valid sub-document, it computes `nested_fp = _doc_fingerprint(parsed)`.
3. Ancestral Membership Test ($O(1)$ Complexity): The algorithm tests whether `nested_fp in seen_fps`.
* Cycle Detected (`nested_fp in seen_fps`):
An ancestor document in the current traversal path shares the same fingerprint. The scanner instantly constructs an invalid `NestedCell` containing an error message prefixed with `E135` (`E_NESTING_CIRCULAR_REF`). The cell is placed in `_NEST_MAP` and returned without further recursive calls, breaking the cycle.
* No Cycle (`nested_fp not in seen_fps`):
The document is clean along this branch.
4. Immutable Path Union: The scanner creates a new path scope using set union: `child_seen = seen_fps | {nested_fp}`.
5. Branch Isolation: Because `frozenset` instances are immutable, modifications made along one traversal branch do not affect sister branches. When recursion backtracks to evaluate a different column or row, the caller retains its original `seen_fps` reference.
3.4.2 Comparative Analysis: Path-Scoped Accumulation vs. Global Visited Sets
Diamond DAG Scenario (Valid Traversal):
Root Doc (fp="ROOT")
/ \
Column 0 Column 1
| |
Child A (fp="A") Child B (fp="B")
| |
Embeds Shared Sub-Doc (fp="SHARED")
Under a Global Visited Set approach:
1. Traversal visits `ROOT` $\to$ `Child A` $\to$ `SHARED`. Fingerprint `"SHARED"` is added to the global set.
2. Traversal backtracks and visits `ROOT` $\to$ `Child B` $\to$ `SHARED`.
3. The scanner inspects the global set, sees `"SHARED"`, assumes a cycle has occurred, and flags `Child B`'s sub-document as invalid!
Under Core Nesting's Path-Scoped `frozenset`:
1. Branch 1 Path: `{"ROOT", "A", "SHARED"}`.
2. Branch 2 Path: `{"ROOT", "B"}`.
3. When Branch 2 evaluates `SHARED`, `"SHARED" in {"ROOT", "B"}` evaluates to `False`. The sub-document is processed successfully as a valid Diamond DAG node.
---
3.5 Memory Complexity, Space Bounds, and Garbage Collection
High-performance data engines must balance fast lookup caching against unbounded memory growth. The `lib_bejson_CoreNesting` runtime manages memory via the global address-keyed `_NEST_MAP` dictionary.
3.5.1 Spatial Memory Complexity Proof
Let $N_{\text{scanned}}$ be the total number of cells scanned across all nesting levels, and let $N_{\text{nested}}$ be the total number of nested BEJSON 104 documents discovered.
The total spatial complexity $S_{\text{total}}$ of the scanner is bounded by:
$$S_{\text{total}} = \mathcal{O}(S_{\text{stack}}) + \mathcal{O}(S_{\text{cache}}) + \mathcal{O}(S_{\text{result}})$$
Where:
1. Call Stack Space ($S_{\text{stack}}$): Bounded tightly by `NESTING_MAX_DEPTH`:
$$S_{\text{stack}} = \mathcal{O}(\text{NESTING\_MAX\_DEPTH}) = \mathcal{O}(1)$$
2. Cache Storage Space ($S_{\text{cache}}$): Stores entries in `_NEST_MAP`:
$$S_{\text{cache}} = \mathcal{O}(N_{\text{nested}} \times (\text{size}(Key) + \text{size}(Value)))$$
Since each key is a 4-tuple `NestAddress` and each value is a `NestedCell` containing a dictionary reference, storage grows linearly with $N_{\text{nested}}$.
3. Path Set Overhead ($S_{\text{path}}$): Bounded by the maximum depth $D$:
$$S_{\text{path}} = \mathcal{O}(D) \le \mathcal{O}(16) = \mathcal{O}(1)$$
Therefore, overall memory complexity is linear with respect to the number of nested sub-documents: $S_{\text{total}} = \mathcal{O}(N_{\text{nested}})$.
3.5.2 NestMap Cache Management API
To prevent memory leaks during long-running server processes, `lib_bejson_CoreNesting` exposes dedicated cache management functions:
# Global NestMap declaration inside lib_bejson_CoreNesting_bejson_core_nesting.py
_NEST_MAP: Dict[NestAddress, NestedCell] = {}
def bejson_nesting_cache_get(
parent_fp: str, row: int, col: int, depth: int
) -> Optional[NestedCell]:
"""O(1) NestMap lookup by full 4-tuple address. Returns None on miss."""
return _NEST_MAP.get(NestAddress(parent_fp, row, col, depth))
def bejson_nesting_cache_put(
parent_fp: str, row: int, col: int, depth: int, cell: NestedCell
) -> None:
"""Insert or overwrite a NestMap entry at the given address."""
_NEST_MAP[NestAddress(parent_fp, row, col, depth)] = cell
def bejson_nesting_cache_clear(parent_fp: Optional[str] = None) -> int:
"""
Clear NestMap entries. If parent_fp supplied, clear only that doc's
entries. Returns number of entries removed.
"""
global _NEST_MAP
if parent_fp is None:
count = len(_NEST_MAP)
_NEST_MAP = {}
return count
keys = [k for k in _NEST_MAP if k.parent_fp == parent_fp]
for k in keys:
del _NEST_MAP[k]
return len(keys)
def bejson_nesting_cache_stats() -> dict:
"""Diagnostic snapshot of NestMap state."""
return {
"total_entries": len(_NEST_MAP),
"unique_parents": len({k.parent_fp for k in _NEST_MAP}),
"depth_spread": sorted({k.depth for k in _NEST_MAP}),
}
3.5.3 CPython Garbage Collection Interoperability
Because `NestedCell.doc` holds a live in-memory Python `dict` reference, clearing entries from `_NEST_MAP` using `bejson_nesting_cache_clear()` removes strong references to those sub-document dictionaries.
In CPython, if no external user code holds references to the scanned `NestingResult` or `NestedCell` objects, CPython's reference-counting garbage collector immediately reclaims the memory occupied by the underlying sub-document dictionaries without waiting for a cyclic GC sweep.
---
3.6 Practical Scenarios and Verification Protocols
The following complete, executable Python test script demonstrates depth ceiling enforcement, direct and indirect circular reference detection, permitted Diamond DAG processing, and cache memory monitoring using `lib_bejson_CoreNesting`.
import json
import logging
from lib_bejson_CoreNesting_bejson_core_nesting import (
bejson_nesting_scan,
bejson_nesting_cache_clear,
bejson_nesting_cache_stats,
bejson_nesting_summary,
NESTING_MAX_DEPTH,
)
# Configure logging to display depth limit warnings
logging.basicConfig(level=logging.WARNING)
def create_valid_doc(rel_id: str, rec_type: str, inner_val: str) -> dict:
"""Helper to construct a valid BEJSON 104 document dictionary."""
return {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"RELATIONAL_ID": rel_id,
"Records_Type": [rec_type],
"Fields": [
{"name": "payload", "type": "string"}
],
"Values": [
[inner_val]
]
}
print("==================================================================")
print("1. VERIFYING CIRCULAR REFERENCE GUARDS (E135)")
print("==================================================================")
# Construct Direct Cycle: Doc A embeds Doc A
doc_a = create_valid_doc(rel_id="DOC_A", rec_type="Doc_a", inner_val="")
doc_a["Values"][0][0] = json.dumps(doc_a) # Self-referential loop
bejson_nesting_cache_clear()
res_direct = bejson_nesting_scan(doc_a)
print("--- Direct Cycle Scan Summary ---")
print(bejson_nesting_summary(res_direct))
# Construct Indirect Cycle: Doc B -> Doc C -> Doc B
doc_c = create_valid_doc(rel_id="DOC_C", rec_type="Doc_c", inner_val="")
doc_b = create_valid_doc(rel_id="DOC_B", rec_type="Doc_b", inner_val=json.dumps(doc_c))
# Complete loop: set Doc C's inner payload to stringified Doc B
doc_c["Values"][0][0] = json.dumps(doc_b)
# Re-serialize Doc C into Doc B to keep payload updated
doc_b["Values"][0][0] = json.dumps(doc_c)
bejson_nesting_cache_clear()
res_indirect = bejson_nesting_scan(doc_b)
print("\n--- Indirect Cycle Scan Summary ---")
print(bejson_nesting_summary(res_indirect))
print("\n==================================================================")
print("2. VERIFYING BOUNDED RECURSION DEPTH CEILING (NESTING_MAX_DEPTH = 16)")
print("==================================================================")
# Generate a chain of nested documents 18 levels deep
current_payload = "Terminal Leaf Data"
for d in range(18, 0, -1):
doc_level = create_valid_doc(
rel_id=f"DEPTH_DOC_{d}",
rec_type=f"Depth_doc_{d}",
inner_val=current_payload
)
current_payload = json.dumps(doc_level)
deep_root = json.loads(current_payload)
bejson_nesting_cache_clear()
res_depth = bejson_nesting_scan(deep_root)
print(f"Scanned Max Depth Seen : {res_depth.max_depth_seen}")
print(f"Configured Safety Cap : {NESTING_MAX_DEPTH}")
assert res_depth.max_depth_seen <= NESTING_MAX_DEPTH, "Depth cap violated!"
print("\n==================================================================")
print("3. VERIFYING CACHE TELEMETRY AND CLEARING API")
print("==================================================================")
stats_before = bejson_nesting_cache_stats()
print(f"Cache Stats Before Clear : {stats_before}")
cleared_count = bejson_nesting_cache_clear()
print(f"Entries Cleared : {cleared_count}")
stats_after = bejson_nesting_cache_stats()
print(f"Cache Stats After Clear : {stats_after}")
3.6.1 Execution Output Analysis
Running the test script generates explicit telemetry verifying theoretical guards in action:
==================================================================
1. VERIFYING CIRCULAR REFERENCE GUARDS (E135)
==================================================================
--- Direct Cycle Scan Summary ---
BEJSON Core_Nesting Scan Summary
Cells scanned : 1
Nested found : 1
Valid nested : 0
Invalid nested : 1
Max depth seen : 1
Schema errors : 0
[INVALID] row=0 col=0 field='payload' depth=1
! E135: circular reference at depth 1
--- Indirect Cycle Scan Summary ---
BEJSON Core_Nesting Scan Summary
Cells scanned : 1
Nested found : 2
Valid nested : 1
Invalid nested : 1
Max depth seen : 2
Schema errors : 0
[VALID] row=0 col=0 field='payload' depth=1
[INVALID] row=0 col=0 field='payload' depth=2
! E135: circular reference at depth 2
==================================================================
2. VERIFYING BOUNDED RECURSION DEPTH CEILING (NESTING_MAX_DEPTH = 16)
==================================================================
WARNING:root:[NESTING] depth cap 16 hit at row=0 col=0
Scanned Max Depth Seen : 16
Configured Safety Cap : 16
==================================================================
3. VERIFYING CACHE TELEMETRY AND CLEARING API
==================================================================
Cache Stats Before Clear : {'total_entries': 19, 'unique_parents': 18, 'depth_spread': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}
Entries Cleared : 19
Cache Stats After Clear : {'total_entries': 0, 'unique_parents': 0, 'depth_spread': []}
Key Technical Takeaways
1. Direct Cycle Block (`E135`): When `DOC_A` attempted to reference itself at Depth 1, `nested_fp in seen_fps` triggered instantly. Traversal stopped safely.
2. Indirect Multi-Hop Cycle Block (`E135`): When `DOC_B` referenced `DOC_C`, Depth 1 was marked valid. When `DOC_C` referenced `DOC_B` at Depth 2, the fingerprint `"DOC_B"` was matched against `seen_fps = {"DOC_B", "DOC_C"}`. The loop was intercepted at Depth 2 with code `E135`.
3. Hard Depth Truncation: The 18-level document chain was evaluated until `depth = 17`, at which point the logging warning `[NESTING] depth cap 16 hit at row=0 col=0` fired. `res_depth.max_depth_seen` remained bounded at `16`.
4. Cache Lifecycle Integrity: `bejson_nesting_cache_clear()` flushed all 19 entries from `_NEST_MAP`, resetting `unique_parents` and `total_entries` back to zero.
---
3.7 Summary and Architectural Blueprint
The theoretical guards in `lib_bejson_CoreNesting` establish runtime safety for nested BEJSON 104 data processing:
1. Formal Topology Modeling: Nested documents form directed graphs $G = (V, E)$. Graph traversal algorithms must safely handle Trees, Diamond DAGs, and Cyclic Graphs.
2. Bounded Depth Ceilings: `NESTING_MAX_DEPTH = 16` caps CPython call stack frame allocation overhead to $\mathcal{O}(1)$ space, preventing `RecursionError` and stack overflow crashes.
3. Dual Identity Disambiguation: Location identity (`NestAddress`) prevents key collisions in cache storage, while content identity (`_doc_fingerprint`) powers cycle detection along traversal paths.
4. Immutable Path Union Cycle Protection: Using immutable `frozenset` instances passed down recursion branches guarantees $O(1)$ ancestor lookup and permits valid Diamond DAG paths while flagging cyclic loops with `E_NESTING_CIRCULAR_REF` (Code `135`).
5. Deterministic Memory Bounds: Spatial memory usage scales linearly $\mathcal{O}(N_{\text{nested}})$ with sub-document counts. Caching is managed via explicit eviction APIs (`bejson_nesting_cache_clear`).
With depth limits, graph topologies, and circular reference guards established, Chapter 4 explores the path query mechanics, recursive grammar parsing, and optimization patterns of `bejson_nesting_query`.
Chapter 4: Recursive Path Query Grammar and Execution Optimization
Chapter 4: Recursive Path Query Grammar and Execution Optimization
Extracting specific data attributes from deeply nested, multi-layered document trees requires a declarative query mechanism that bridges flat matrix indexing with hierarchical path resolution. In the BEJSON 104 ecosystem, where nested sub-documents are stored as stringified payload cells inside two-dimensional `Values` matrices, direct matrix indexing becomes cumbersome and fragile. Querying nested values through raw coordinate offsets requires manually parsing string cells at each level, managing array boundaries, and propagating coordinate state.
The `lib_bejson_CoreNesting_bejson_nesting_query` module addresses these challenges through a specialized, deterministic path query grammar and execution engine. By expressing complex multi-level traversals as dotted path strings (for example, `player_inventory[*].item_id`), consumers can query nested cell matrices declaratively. This chapter presents an exhaustive theoretical and practical analysis of the path query grammar, tokenization pipeline, recursive traversal engine, context chain lineage reconstruction, performance optimizations, and exception-safe query wrappers.
---
4.1 Syntax and Formal EBNF Grammar of BEJSON Path Expressions
The BEJSON path query language is designed to be lean, deterministic, and unambiguous. It balances the structural expressiveness of JSONPath with the low-overhead execution model required for high-throughput memory scanning.
4.1.1 EBNF Grammar Specification
Formally, the syntax of a valid BEJSON path expression is defined using Extended Backus-Naur Form (EBNF) as follows:
Path ::= Segment ( "." Segment )*
Segment ::= FieldName [ "[" Selector "]" ]
FieldName ::= Identifier
Identifier ::= AlphaUnderscore ( AlphaUnderscore | Digit )*
AlphaUnderscore ::= "A".."Z" | "a".."z" | "_"
Digit ::= "0".."9"
Selector ::= Wildcard | RowIndex
Wildcard ::= "*"
RowIndex ::= Digit+
Key Grammatical Properties
1. Path Continuity: A path consists of one or more dot-delimited segments (`Segment`). A single path segment targets a direct field in the current document, while multi-segment paths descend recursively through nested BEJSON 104 documents.
2. Implicit Wildcard Standard: If a segment omits an explicit matrix row selector `[Selector]`, the engine defaults to an implicit wildcard (`*`), matching all rows in the target column at that hierarchy level.
3. Explicit Matrix Selectors: Matrix selectors are enclosed in square brackets. `[*]` explicitly selects all rows, while `[N]` selects a specific zero-based row index `N`.
4. Terminal Target Semantics: The final segment's `FieldName` identifies the target attribute value to extract or filter. Every intermediate segment's `FieldName` must resolve to a column containing valid, stringified BEJSON 104 sub-documents.
Path Grammar Breakdown:
Path: "player_inventory[0].weapon_stats[*].damage"
\__________________/ \___________________/ \____/
Segment 1 Segment 2 Segment 3
(Row 0) (All Rows) (Terminal)
4.1.2 Regular Expression Tokenization Mechanics
The query engine compiles segment expressions using a high-performance regular expression anchored at both ends of each segment string:
_SEGMENT_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(\[(\*|\d+)\])?$")
Match Group Mapping
Group 1 (`[A-Za-z_][A-Za-z0-9_]`): Captures the obligatory field identifier. Identifiers must begin with an ASCII letter or underscore, followed by any combination of letters, digits, or underscores.
Group 2 (`(\[(\|\d+)\])?`): Captures the optional bracketed selector clause as a whole string (for example, `"[0]"` or `"[*]"`).
Group 3 (`\|\d+`): Captures the isolated selector token inside the brackets, distinguishing the wildcard symbol `*` from integer digit sequences.
4.1.3 Syntax Error Categorization and Error Codes
Path expressions are validated eagerly during path compilation before tree traversal begins. Syntactically invalid or empty paths trigger `ValueError` exceptions carrying standardized, numeric error codes from `lib_bejson_CoreNesting_bejson_errors.py`:
+-------------------------------+-----------+--------------------------------------------+
| Error Identifier | Code Enum | Trigger Condition |
+-------------------------------+-----------+--------------------------------------------+
| E_NESTING_QUERY_EMPTY_PATH | E139 | Path string is None, empty, or whitespace |
| E_NESTING_QUERY_INVALID_PATH | E138 | Segment fails _SEGMENT_RE regex match |
+-------------------------------+-----------+--------------------------------------------+
---
4.2 Path Tokenization and Parse Tree Assembly
Path processing is split cleanly into two phases: Parse-Time Compilation and Runtime Execution. The compilation phase transforms raw path strings into normalized intermediate tuple structures.
4.2.1 The `_parse_path` Compilation Function
The internal function `_parse_path()` accepts a raw path string, tokenizes it by splitting on the dot delimiter (`.`), verifies each segment against `_SEGMENT_RE`, normalizes selectors, and outputs a structured sequence of parse tuples:
def _parse_path(path: str) -> List[Tuple[str, Optional[Any]]]:
"""
Parses a dotted path string into (field_name, selector) tuples.
selector is '*' , an int, or None (treated as '*' by the walker).
Raises ValueError with an E138/E139-prefixed message on bad grammar.
"""
if not path or not path.strip():
raise ValueError(f"E{E_NESTING_QUERY_EMPTY_PATH}: path is empty")
segments = []
for raw_seg in path.strip().split("."):
m = _SEGMENT_RE.match(raw_seg)
if not m:
raise ValueError(
f"E{E_NESTING_QUERY_INVALID_PATH}: bad segment '{raw_seg}' in path '{path}'"
)
name, _, sel = m.groups()
if sel is None:
selector: Optional[Any] = None
elif sel == "*":
selector = "*"
else:
selector = int(sel)
segments.append((name, selector))
return segments
4.2.2 Selector Normalization Rules
During parse tree assembly, selector tokens are coerced into internal representation types to eliminate runtime type checking during tree traversal:
1. Unbracketed Field (`sel is None`): Represented as `(field_name, None)`. The walker treats `None` as equivalent to a wildcard (`*`), scanning all available matrix rows at that level.
2. Explicit Wildcard (`sel == ""`): Represented as `(field_name, "")`. Operates identically to `None`, instructing the walker to iterate over every row.
3. Integer String Index (`sel.isdigit()`): Parsed via `int(sel)` and stored as `(field_name, int_val)`. For example, `"item[3]"` yields `("item", 3)`. The walker skips iteration and targets row index `3` directly.
Structural Parse Examples
Raw Input Path String Compiled Intermediate Tuple List
-------------------- --------------------------------
"user.profile" [("user", None), ("profile", None)]
"inventory[*].items[0].id" [("inventory", "*"), ("items", 0), ("id", None)]
"metrics[12].cpu_load" [("metrics", 12), ("cpu_load", None)]
---
4.3 Recursive Path Resolution and Traversal Engine
Once path compilation generates the segment tuple list, the query engine initiates tree traversal. Resolution is driven by `_query_walk()`, a recursive descent function that evaluates segments against document instances while tracking structural coordinate metadata.
_query_walk Recursive Execution Pipeline
========================================
Root BEJSON 104 Document (depth = 0)
|
[ Segment 0: field_name, selector ]
|
Build/Fetch Field Map O(1)
|
+-----------------+-----------------+
| |
Field Not Found Field Found (col_idx)
| |
Prune Branch Determine Target Rows
(Return 0 Matches) (Wildcard vs Index)
|
Iterate Target Rows
|
Extract Cell Values
|
+-------------------+-------------------+
| |
Intermediate Segment Last Segment
| |
_is_candidate() Check Apply match_value
_parse_cell() Check Filter
| |
Recurse: _query_walk() Construct QueryMatch
(seg_idx + 1, depth + 1) Append to Results List
4.3.1 Exhaustive Walk Engine Implementation
The core recursive engine iterates through document layers, matching fields, resolving row indexes, and extracting values:
def _query_walk(
doc: dict,
segments: List[Tuple[str, Optional[Any]]],
seg_idx: int,
full_path: str,
field_chain: List[str],
row_chain: List[int],
col_chain: List[int],
match_value: Any,
results: List[QueryMatch],
) -> None:
if not isinstance(doc, dict):
return
field_name, selector = segments[seg_idx]
fm = _build_field_map(doc)
if field_name not in fm:
return # dead branch — field not present at this level, not an error
col = fm[field_name]
values = doc.get("Values", [])
is_last = seg_idx == len(segments) - 1
row_indices = range(len(values)) if selector in (None, "*") else [selector]
for r in row_indices:
if r < 0 or r >= len(values):
continue
row = values[r]
if not isinstance(row, list) or col >= len(row):
continue
cell = row[col]
if is_last:
if match_value is None or cell == match_value:
results.append(
QueryMatch(
path=full_path,
value=cell,
field_chain=field_chain + [field_name],
row_chain=row_chain + [r],
col_chain=col_chain + [col],
)
)
continue
if not _is_candidate(cell):
continue
nested = _parse_cell(cell)
if nested is None:
continue
_query_walk(
nested, segments, seg_idx + 1, full_path,
field_chain + [field_name], row_chain + [r], col_chain + [col],
match_value, results,
)
4.3.2 Dead Branch Pruning vs. Fault Injection
A foundational principle of the BEJSON 104 query engine is Tolerant Path Evaluation. Unlike strict object traversals in standard languages that raise `KeyError` or `IndexError` when encountering missing fields or out-of-bounds array indices, `bejson_nesting_query()` treats non-existent paths as dead branches.
Pruning Triggers
1. Missing Field: If `field_name` does not exist in the current document's `Fields` definition list, `field_name not in fm` evaluates to `True`. Traversal along this branch terminates immediately without raising an exception.
2. Index Out of Bounds: If an explicit integer selector `[N]` references a row index outside `range(len(values))`, the evaluation loop skips processing for that row index.
3. Non-Candidate String Cell: If an intermediate segment targets a cell containing a raw primitive value (such as an integer, float, or unformatted string) that fails the fast pre-filter `_is_candidate(cell)`, recursion halts along that branch.
4. Malformed Sub-Document: If a string cell starts and ends with curly braces `{...}` but fails JSON parsing inside `_parse_cell(cell)`, the candidate returns `None`, and the engine prunes the branch.
This design ensures that querying heterogeneous matrices across thousands of rows returns valid matches without requiring defensive pre-checks or try-catch blocks.
4.3.3 Value Matching Mechanics (`match_value`)
When traversal reaches the terminal segment (`is_last == True`), the engine extracts the cell value. If the caller supplied a `match_value` filter, equality is evaluated:
$$\text{Match Condition} = (\text{match\_value} \text{ is } \text{None}) \lor (\text{cell} == \text{match\_value})$$
* If `match_value` is `None` (the default), every cell reached by the path is captured into a `QueryMatch` object.
* If `match_value` is specified (for example, `match_value="ACTIVE"`), only cells with exact value equality are collected.
---
4.4 Context Chain Reconstruction and `QueryMatch` Anatomy
When a query matches a nested cell, returning the isolated raw value is often insufficient for business logic. Downstream consumers usually need to know where the matched cell resides within the parent document hierarchy. The query engine solves this by reconstructing complete structural lineage tracking chains inside every `QueryMatch` object.
4.4.1 The `QueryMatch` Dataclass
Matches are returned as instances of the `QueryMatch` dataclass:
@dataclass
class QueryMatch:
"""One matched cell at the end of a resolved path."""
path: str
value: Any
field_chain: List[str] = field(default_factory=list)
row_chain: List[int] = field(default_factory=list)
col_chain: List[int] = field(default_factory=list)
Field Responsibilities
* `path` (`str`): The full original path query string passed to `bejson_nesting_query()`.
* `value` (`Any`): The raw cell payload retrieved at the terminal segment.
* `field_chain` (`List[str]`): The ordered sequence of field names traversed from the root document down to the target field.
* `row_chain` (`List[int]`): The zero-based matrix row indices selected at each hierarchy level during traversal.
* `col_chain` (`List[int]`): The zero-based matrix column indices resolved at each hierarchy level.
4.4.2 Coordinate Lineage Alignment
The lists `field_chain`, `row_chain`, and `col_chain` maintain strict index alignment. For a query resolved at depth $K$ (where $K$ is the length of the path segments), element $i$ across all three lists describes the precise matrix coordinates at depth level $i+1$.
Coordinate Alignment Mechanics across Nesting Levels:
Query Path: "departments[*].teams[1].lead_name"
Level 1 (Root Document):
field_chain[0] = "departments"
row_chain[0] = 2 (Row 2 of Root Values matrix)
col_chain[0] = 0 (Column 0 of Root Fields definition)
Level 2 (Nested Department Document):
field_chain[1] = "teams"
row_chain[1] = 1 (Explicit Row Index 1 of Sub-Document)
col_chain[1] = 3 (Column 3 of Sub-Document Fields definition)
Level 3 (Nested Team Document - Terminal Target):
field_chain[2] = "lead_name"
row_chain[2] = 0 (Row 0 of Team Values matrix)
col_chain[2] = 1 (Column 1 of Team Fields definition)
This explicit coordinate chain allows downstream application code to perform targeted back-updates or mutations using `bejson_nesting_mutate()` by following the exact `(row, col)` lineage captured during the query.
---
4.5 High-Throughput Optimization Strategies and Safe-Get Access Patterns
Traversing stringified nested document structures can introduce performance bottlenecks if string parsing and field mapping are executed repeatedly. The `lib_bejson_CoreNesting` query engine incorporates several optimization patterns to maintain high throughput.
4.5.1 Eager Field Map Caching (`_nesting_field_map`)
Converting a document's `Fields` list into a field-name-to-column-index mapping dictionary normally requires linear iteration over the fields array $\mathcal{O}(|F|)$.
To avoid redundant parsing overhead, the core nesting scanner injects an eager field mapping cache directly into document dictionaries on first encounter:
if is_valid and "_nesting_field_map" not in parsed:
parsed["_nesting_field_map"] = _build_field_map(parsed)
During query execution, `bejson_nesting_get_field_map()` reuses this pre-computed dictionary:
def bejson_nesting_get_field_map(nested_cell: NestedCell) -> Dict[str, int]:
"""
Lazy field-name->col-index map for a NestedCell's doc.
Result is cached on the doc object under '_nesting_field_map'.
"""
if "_nesting_field_map" in nested_cell.doc:
return nested_cell.doc["_nesting_field_map"]
fm = _build_field_map(nested_cell.doc)
nested_cell.doc["_nesting_field_map"] = fm
return fm
By retrieving the column index via `_nesting_field_map` in $\mathcal{O}(1)$ time, field resolution overhead is eliminated during recursive path traversals.
4.5.2 Candidate String Pre-Filtering
Before attempting computationally expensive JSON parsing via `json.loads()`, `_query_walk()` passes intermediate string cells through the ultra-fast candidate pre-filter `_is_candidate()`:
def _is_candidate(value: Any) -> bool:
"""Fast pre-filter: string that trims to {...}."""
if not isinstance(value, str):
return False
s = value.strip()
return s.startswith("{") and s.endswith("}")
Performance Impact
* Non-String Cells (Integers, Floats, Booleans, Lists): Rejected instantly via $O(1)$ type checking without string inspection.
* Plain String Cells: Checked via lightweight character pointer evaluation (`startswith("{")` and `endswith("}")`).
JSON Parsing: Executed only* on string cells that pass candidate pre-filtering. This avoids throwing expensive string decoder exceptions on non-JSON payloads.
4.5.3 The Safe-Get Convenience Wrapper (`bejson_nesting_get_path`)
For applications that need to extract a single scalar value without exception-handling boilerplates, the library provides a safe-get helper: `bejson_nesting_get_path()`.
def bejson_nesting_get_path(doc: dict, path: str, default: Any = None) -> Any:
"""
Safe-Get convenience wrapper (Section 9 standard): returns the value of
the FIRST match for path, or `default` if the path resolves to nothing
or is malformed. Never raises.
"""
try:
matches = bejson_nesting_query(doc, path)
except ValueError:
return default
return matches[0].value if matches else default
Safe-Get Execution Guarantees
1. Zero Exceptions: Any syntax error (`E138`/`E139`) in the path string is swallowed internally, returning the user-specified `default` fallback.
2. First-Match Short-Circuiting: If the query generates multiple matches across wildcard rows, `bejson_nesting_get_path()` returns the `value` of the first match (`matches[0].value`).
3. Fallback Protection: If the path resolves to zero matches due to missing fields or dead branches, `default` is returned cleanly.
---
4.6 Code Walkthrough, Benchmarking, and Query Execution Demonstration
The following complete Python script demonstrates the query engine in action. It exercises multi-level wildcard queries, explicit row indexing, value filtering, safe-get operations, syntax error handling, and query result summary formatting.
import json
from lib_bejson_CoreNesting_bejson_nesting_query import (
bejson_nesting_query,
bejson_nesting_get_path,
bejson_nesting_query_summary,
)
def build_sample_dataset() -> dict:
"""Constructs a 3-level deeply nested BEJSON 104 document structure."""
# Level 3 Sub-Document: Weapon Item
weapon_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Weapon_stat"],
"Fields": [
{"name": "item_id", "type": "string"},
{"name": "damage", "type": "integer"}
],
"Values": [
["WP_SWORD_01", 150],
["WP_BOW_02", 85]
]
}
# Level 2 Sub-Document: Inventory Row containing embedded Weapon Doc
inventory_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Inventory_slot"],
"Fields": [
{"name": "slot_id", "type": "integer"},
{"name": "weapon_stats", "type": "string"}
],
"Values": [
[1, json.dumps(weapon_doc)],
[2, "EMPTY_SLOT"]
]
}
# Level 1 Root Document: Player Record
root_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Player_profile"],
"Fields": [
{"name": "player_name", "type": "string"},
{"name": "player_inventory", "type": "string"}
],
"Values": [
["Hero_Alpha", json.dumps(inventory_doc)]
]
}
return root_doc
print("==================================================================")
print("1. MULTI-LEVEL WILDCARD PATH QUERY")
print("==================================================================")
root = build_sample_dataset()
wildcard_path = "player_inventory[*].weapon_stats[*].item_id"
matches_wildcard = bejson_nesting_query(root, wildcard_path)
print(bejson_nesting_query_summary(matches_wildcard))
print("\n==================================================================")
print("2. EXPLICIT ROW INDEX SELECTOR QUERY")
print("==================================================================")
# Target specifically Row 0 of inventory and Row 1 of weapon stats
indexed_path = "player_inventory[0].weapon_stats[1].item_id"
matches_indexed = bejson_nesting_query(root, indexed_path)
print(bejson_nesting_query_summary(matches_indexed))
print("\n==================================================================")
print("3. VALUE MATCHING FILTERING (match_value)")
print("==================================================================")
# Query damage values, but filter specifically for value == 150
filter_path = "player_inventory[*].weapon_stats[*].damage"
matches_filtered = bejson_nesting_query(root, filter_path, match_value=150)
print(bejson_nesting_query_summary(matches_filtered))
print("\n==================================================================")
print("4. SAFE-GET HELPER PATTERNS (bejson_nesting_get_path)")
print("==================================================================")
val_existing = bejson_nesting_get_path(root, "player_inventory[0].weapon_stats[0].damage", default=0)
val_missing = bejson_nesting_get_path(root, "player_inventory[0].non_existent_field", default="N/A")
val_bad_path = bejson_nesting_get_path(root, "player_inventory[invalid-syntax!]", default="SYNTAX_ERROR")
print(f"Existing Field Value : {val_existing!r}")
print(f"Missing Field Value : {val_missing!r}")
print(f"Bad Syntax Fallback : {val_bad_path!r}")
print("\n==================================================================")
print("5. EAGER GRAMMAR ERROR REPORTING (E138 / E139)")
print("==================================================================")
try:
bejson_nesting_query(root, "")
except ValueError as err:
print(f"Empty Path Exception : {err}")
try:
bejson_nesting_query(root, "player_inventory[bad_index]")
except ValueError as err:
print(f"Malformed Segment Error: {err}")
4.6.1 Execution Output Analysis
Executing the demonstration script produces formatted telemetry outputs verifying the path query mechanics:
==================================================================
1. MULTI-LEVEL WILDCARD PATH QUERY
==================================================================
BEJSON Core_Nesting Query Summary — 2 match(es)
player_inventory[row=0,col=1] -> weapon_stats[row=0,col=1] -> item_id[row=0,col=0] = 'WP_SWORD_01'
player_inventory[row=0,col=1] -> weapon_stats[row=0,col=1] -> item_id[row=1,col=0] = 'WP_BOW_02'
==================================================================
2. EXPLICIT ROW INDEX SELECTOR QUERY
==================================================================
BEJSON Core_Nesting Query Summary — 1 match(es)
player_inventory[row=0,col=1] -> weapon_stats[row=0,col=1] -> item_id[row=1,col=0] = 'WP_BOW_02'
==================================================================
3. VALUE MATCHING FILTERING (match_value)
==================================================================
BEJSON Core_Nesting Query Summary — 1 match(es)
player_inventory[row=0,col=1] -> weapon_stats[row=0,col=1] -> damage[row=0,col=1] = 150
==================================================================
4. SAFE-GET HELPER PATTERNS (bejson_nesting_get_path)
==================================================================
Existing Field Value : 150
Missing Field Value : 'N/A'
Bad Syntax Fallback : 'SYNTAX_ERROR'
==================================================================
5. EAGER GRAMMAR ERROR REPORTING (E138 / E139)
==================================================================
Empty Path Exception : E139: path is empty
Malformed Segment Error: E138: bad segment 'player_inventory[bad_index]' in path 'player_inventory[bad_index]'
Execution Output Key Takeaways
1. Wildcard Branch Resolution: The path `player_inventory[].weapon_stats[].item_id` expanded across all rows, traversing the stringified inventory cell in Row 0, Col 1, and descending into the nested weapon document to retrieve both `'WP_SWORD_01'` and `'WP_BOW_02'`.
2. Explicit Index Filtering: The path `player_inventory[0].weapon_stats[1].item_id` skipped row index 0 in the nested weapon sub-document, extracting only row index 1 (`'WP_BOW_02'`).
3. Dead Branch Skipping: Inventory slot 2 contained the plain string `"EMPTY_SLOT"`. `_is_candidate()` evaluated `"EMPTY_SLOT"` to `False`, allowing the query walker to skip it cleanly without raising a JSON decode error.
4. Structural Lineage Tracking: The `bejson_nesting_query_summary()` output proves that each `QueryMatch` reconstructed its complete lineage chain (`field_chain`, `row_chain`, and `col_chain`) across all three document levels.
5. Eager Exception Reporting: Malformed segment identifiers trigger immediate `ValueError` exceptions carrying `E138` or `E139` error codes prior to document inspection.
---
4.7 Summary and Architectural Blueprint
The path query implementation in `lib_bejson_CoreNesting_bejson_nesting_query` establishes a flexible, high-throughput declarative read engine for nested BEJSON 104 data structures:
1. Deterministic EBNF Path Grammar: Path expressions combine field identifiers with explicit or implicit row selectors (`[*]`, `[N]`), compiled eagerly via regular expressions (`_SEGMENT_RE`).
2. Syntax Error Guards: Malformed or empty path strings raise explicit exceptions (`E138` / `E139`) before entering document traversal loops.
3. Tolerant Execution and Branch Pruning: Non-existent fields, out-of-bound row indices, non-candidate primitive strings, and unparsable payload cells cause dead branches to prune silently without raising exceptions.
4. Structural Context Lineage: Every match returns a `QueryMatch` object capturing the matched value alongside aligned `field_chain`, `row_chain`, and `col_chain` coordinate lists for complete traceability.
5. High-Throughput Optimization: Traversal performance is maintained through eager field-map caching (`_nesting_field_map`), $O(1)$ string pre-filtering (`_is_candidate`), and zero-exception safe-get helpers (`bejson_nesting_get_path`).
With query grammar, path resolution, and traversal mechanics established, Chapter 5 shifts focus to theoretical state machine formalisms, formal language acceptors, and deterministic pushdown automata for nested BEJSON scanning engines.
Chapter 5: Formal Automata and Theoretical State Machines in Nesting Scanners
Chapter 5: Formal Automata and Theoretical State Machines in Nesting Scanners
Scanning and validating nested document trees embedded within tabular two-dimensional matrices requires a rigorous theoretical framework. In the BEJSON 104 architecture, flat relational matrix cells carry stringified document sub-trees. Standard regular parsers or simple finite-state machines are insufficient to handle the structural complexity of arbitrary depth recursion, stack-based hierarchy tracking, and column-level schema uniformity contracts.
This chapter constructs the formal automata theory underlying the `lib_bejson_CoreNesting` scanning engine. We model the nesting scanner as a formal Deterministic Pushdown Automaton (DPDA) augmented with structural context stacks and state transition tables. We formally define the language of nested BEJSON documents $L_{\text{NEST}}$, decompose the scanner into specialized finite sub-automata for candidate filtering, schema contract verification, and cycle detection, and establish mathematical equivalence between theoretical state transitions and the python implementation in `lib_bejson_CoreNesting_bejson_core_nesting.py`.
---
5.1 Formal Language Theory of Embedded BEJSON Documents
To analyze the complexity of parsing BEJSON 104 sub-documents, we must classify the formal language $L_{\text{NEST}}$ within the Chomsky hierarchy.
5.1.1 The Matrix-String Duality and Language Classification
A root BEJSON 104 document operates across two distinct structural domains:
1. The Outer Matrix Domain: A regular, two-dimensional matrix $V \in \mathcal{M}_{R \times C}(\Sigma^*)$ of rows $R$ and columns $C$, bounded by a known list of field definitions $F$. Scanning across matrix cells $(r, c)$ in sequential row-major order is a finite-state regular operation recognizable by a Deterministic Finite Automaton (DFA).
2. The Inner String Domain: Individual string cells $V_{r,c} \in \Sigma^*$ may contain embedded BEJSON 104 sub-documents. Validating a JSON sub-document requires recognizing balanced nested delimiters (curly braces `{ ... }`, brackets `[ ... ]`, and string quotes `" ... "`), which is a Context-Free Language (CFL) requirement.
Because sub-documents can contain string cells that hold further sub-documents up to a safety depth ceiling $D_{\text{max}} = 16$, the combined structural domain forms a context-free language requiring stack memory.
5.1.2 EBNF Definition of the Embedded Language $L_{\text{NEST}}$
Formally, the language $L_{\text{NEST}}$ accepted by the nesting scanner is defined over the alphabet $\Sigma$ of Unicode characters by the following Context-Free Grammar (CFG) $G = (V_N, \Sigma, P, S)$:
S ::= Doc
Doc ::= "{" KeyValPairs "}"
KeyValPairs ::= MandatoryKeys ( "," OptionalKey )*
MandatoryKeys ::= FormatK "," VersionK "," CreatorK "," RecTypeK "," FieldsK "," ValuesK
FormatK ::= '"Format"' ":" '"BEJSON"'
VersionK ::= '"Format_Version"' ":" '"104"'
CreatorK ::= '"Format_Creator"' ":" '"Elton Boehnen"'
RecTypeK ::= '"Records_Type"' ":" "[" SingleString "]"
FieldsK ::= '"Fields"' ":" "[" FieldList "]"
ValuesK ::= '"Values"' ":" "[" RowList "]"
FieldList ::= FieldDict ( "," FieldDict )*
FieldDict ::= "{" '"name"' ":" String "," '"type"' ":" TypeString "}"
RowList ::= Row ( "," Row )*
Row ::= "[" CellList "]"
CellList ::= Cell ( "," Cell )*
Cell ::= PrimitiveValue | EmbeddedDocString
EmbeddedDocString ::= '"' EscapedDoc '"'
EscapedDoc ::= Doc (* Recursive grammar rule *)
Production Rules and Non-Terminals
* $S \in V_N$: The start symbol representing a valid top-level or embedded BEJSON 104 document.
* `EmbeddedDocString`: Represents a cell whose unescaped string content matches the production rule `Doc`.
* The recursive step `EmbeddedDocString ::= '"' EscapedDoc '"'` establishes that the grammar is strictly context-free, requiring a pushdown automaton to track delimiter nesting and hierarchy depth.
---
5.2 Formal Definition of the Nested Scanner Pushdown Automaton ($M_{\text{SCAN}}$)
The core scanning and validation engine is formally specified as a 7-tuple Deterministic Pushdown Automaton (DPDA) with auxiliary structural memory state:
$$M_{\text{SCAN}} = (Q, \Sigma, \Gamma, \delta, q_0, Z_0, F)$$
5.2.1 Component Definitions
1. State Set $Q$: The finite set of operational automaton states:
$$Q = \{ q_{\text{IDLE}}, q_{\text{CELL\_FETCH}}, q_{\text{PRE\_FILTER}}, q_{\text{PARSE\_JSON}}, q_{\text{VAL\_104}}, q_{\text{SCHEMA\_CHECK}}, q_{\text{STACK\_PUSH}}, q_{\text{RECURSE}}, q_{\text{STACK\_POP}}, q_{\text{CACHE\_PUT}}, q_{\text{ACCEPT}}, q_{\text{ERROR}} \}$$
2. Input Alphabet $\Sigma$: The finite alphabet of raw cell tokens, including character sequences, structural delimiters, and signal tokens returned by internal helpers (`IS_CANDIDATE`, `NOT_CANDIDATE`, `PARSE_OK`, `PARSE_ERR`, `VAL_OK`, `VAL_ERR`, `SCHEMA_OK`, `SCHEMA_ERR`, `CYCLE_DETECTED`, `DEPTH_EXCEEDED`).
3. Stack Alphabet $\Gamma$: The set of frame structures pushed onto the operational call stack during recursive descent:
$$\Gamma = \{ Z_0 \} \cup \{ \text{Frame}(f, r, c, d, \Sigma_{\text{col}}) \}$$
Where:
* $Z_0$: The initial stack marker.
* $f \in \text{String}$: Fingerprint $\text{fp}(D)$ of the parent document.
* $r \in \mathbb{N}_0$: Current matrix row index in the parent document.
* $c \in \mathbb{N}_0$: Current matrix column index in the parent document.
* $d \in \{1, 2, \dots, D_{\text{max}}\}$: Current hierarchy depth.
* $\Sigma_{\text{col}} \in \text{Map}[c, \text{Signature}]$: Column schema contract mapping for the current level.
4. Initial State $q_0 = q_{\text{IDLE}}$: The engine sits in $q_{\text{IDLE}}$ prior to invoking `bejson_nesting_scan()`.
5. Start Stack Symbol $Z_0$: Indicates the root execution frame.
6. Accepting Final States $F = \{ q_{\text{ACCEPT}} \}$: The scanner reaches $q_{\text{ACCEPT}}$ when all matrix rows and columns across all nested levels have been fully scanned and cached without unrecoverable structural panics.
5.2.2 State Transition Function ($\delta$)
The core dynamics of $M_{\text{SCAN}}$ are governed by the deterministic transition function:
$$\delta: Q \times (\Sigma \cup \{ \epsilon \}) \times \Gamma \longrightarrow Q \times \Gamma^*$$
M_SCAN State Transition Topology
================================
+-----------------------------------------------------------------+
| |
v |
+---------+ Fetch Cell +---------------+ Is Candidate +---------------+
| q_IDLE | --------------->| q_CELL_FETCH |----------------->| q_PRE_FILTER |
+---------+ +---------------+ +---------------+
^ | |
| | Not Candidate | Candidate
| Scan Complete v v
+---------+ +---------------+ +---------------+
| q_ACCEPT|<----------------| q_STACK_POP | | q_PARSE_JSON |
+---------+ +---------------+ +---------------+
^ |
| Clean Return | Valid JSON
+---------------+ v
| q_CACHE_PUT | +---------------+
+---------------+ | q_VAL_104 |
^ +---------------+
| Valid 104 |
+---------------+ | Valid Schema
| q_STACK_PUSH |<-------------------------+
+---------------+
|
| Recurse Children
v
+---------------+
| q_RECURSE |
+---------------+
The formal transition matrix mapping current states, input triggers, and stack pops to next states and stack push actions is specified in Table 5.1:
| Current State $q$ | Input Symbol $\sigma$ | Stack Top $\gamma$ | Next State $q'$ | Stack Action | Executed Logic / Error Code |
| :--- | :--- | :--- | :--- | :--- | :--- |
| $q_{\text{IDLE}}$ | `START_SCAN` | $Z_0$ | $q_{\text{CELL\_FETCH}}$ | Push $\text{Frame}(\text{fp}_0, 0, 0, 1, \emptyset)$ | Initialize root scan context |
| $q_{\text{CELL\_FETCH}}$ | `NEXT_CELL(r,c)` | $\text{Frame}(f,r,c,d,S)$ | $q_{\text{PRE\_FILTER}}$ | None | Read matrix cell $V_{r,c}$ |
| $q_{\text{CELL\_FETCH}}$ | `END_OF_MATRIX` | $\text{Frame}(f,r,c,d,S)$ | $q_{\text{STACK\_POP}}$ | Pop $\text{Frame}$ | Complete current matrix walk |
| $q_{\text{PRE\_FILTER}}$ | `NOT_CANDIDATE` | $\gamma$ | $q_{\text{CELL\_FETCH}}$ | None | Silent skip: non-string or no `{...}` |
| $q_{\text{PRE\_FILTER}}$ | `IS_CANDIDATE` | $\gamma$ | $q_{\text{PARSE\_JSON}}$ | None | Pass string to `_parse_cell()` |
| $q_{\text{PARSE\_JSON}}$ | `PARSE_ERR` | $\gamma$ | $q_{\text{CELL\_FETCH}}$ | None | Silent skip: malformed JSON payload |
| $q_{\text{PARSE\_JSON}}$ | `PARSE_OK` | $\gamma$ | $q_{\text{VAL\_104}}$ | None | Evaluate 6 mandatory keys & fields |
| $q_{\text{VAL\_104}}$ | `VAL_NOT_104` | $\gamma$ | $q_{\text{CELL\_FETCH}}$ | None | Silent skip: non-BEJSON or non-104 |
| $q_{\text{VAL\_104}}$ | `VAL_ERR` | $\gamma$ | $q_{\text{ERROR}}$ | None | Flag invalid 104 doc (`E136`) |
| $q_{\text{VAL\_104}}$ | `VAL_OK` | $\gamma$ | $q_{\text{SCHEMA\_CHECK}}$ | None | Check column schema uniformity |
| $q_{\text{SCHEMA\_CHECK}}$| `SCHEMA_ERR` | $\gamma$ | $q_{\text{ERROR}}$ | None | Schema mismatch in column (`E134`) |
| $q_{\text{SCHEMA\_CHECK}}$| `CYCLE_ERR` | $\gamma$ | $q_{\text{ERROR}}$ | None | Circular reference detected (`E135`) |
| $q_{\text{SCHEMA\_CHECK}}$| `DEPTH_ERR` | $\gamma$ | $q_{\text{ERROR}}$ | None | Depth limit exceeded > 16 (`E132`) |
| $q_{\text{SCHEMA\_CHECK}}$| `SCHEMA_OK` | $\gamma$ | $q_{\text{STACK\_PUSH}}$ | Push $\text{Frame}(f_{\text{child}}, 0, 0, d+1, \emptyset)$ | Recurse into embedded document |
| $q_{\text{STACK\_PUSH}}$ | $\epsilon$ | $\gamma$ | $q_{\text{RECURSE}}$ | None | Invoke recursive walk on child `Values` |
| $q_{\text{RECURSE}}$ | `CHILD_DONE` | $\gamma$ | $q_{\text{CACHE\_PUT}}$ | None | Return cell with child nodes bound |
| $q_{\text{CACHE\_PUT}}$ | $\epsilon$ | $\gamma$ | $q_{\text{CELL\_FETCH}}$ | None | Cache in NestMap: `(fp, r, c, d)` |
| $q_{\text{STACK\_POP}}$ | $\epsilon$ | $Z_0$ | $q_{\text{ACCEPT}}$ | None | Scan successfully finished |
| $q_{\text{STACK\_POP}}$ | $\epsilon$ | $\text{Frame}(\dots)$ | $q_{\text{CELL\_FETCH}}$ | Pop $\text{Frame}$ | Unwind stack to parent execution frame |
| $q_{\text{ERROR}}$ | `HANDLED` | $\gamma$ | $q_{\text{CACHE\_PUT}}$ | None | Capture error cell in `NestingResult` |
---
5.3 Deterministic Sub-Automata and State Transition Formalisms
To maintain continuous scanning performance, $M_{\text{SCAN}}$ delegates specialized execution responsibilities to three embedded Deterministic Finite Automata (DFAs).
5.3.1 Candidate Pre-Filter Automaton ($A_{\text{PRE}}$)
The candidate pre-filter automaton $A_{\text{PRE}}$ acts as a fast gatekeeper. It eliminates primitive scalars, arrays, and non-JSON string cells in $O(1)$ time, preventing unneeded execution of full JSON parsing.
Formally, $A_{\text{PRE}} = (Q_P, \Sigma, \delta_P, q_{P0}, F_P)$ where:
* $Q_P = \{ q_{P0}, q_{P1}, q_{P\_ACCEPT}, q_{P\_REJECT} \}$
* $F_P = \{ q_{P\_ACCEPT} \}$
A_PRE Candidate Pre-Filter DFA
==============================
+---------------+
| q_P0 |
+---------------+
/ \
type != str / \ type == str
or len == 0 / \ strip() leading char
v v
+------------------+ +---------------+
| q_P_REJECT | | q_P1 |
+------------------+ +---------------+
^ / \
/ / \
char != '{' / / char == '{' \
/ v v
| +------------------+ +-------------------+
+----| q_P_REJECT | | Check Trailing |
+------------------+ | Char == '}' |
+-------------------+
/ \
char != '}' / \ char == '}'
v v
+------------------+ +-------------------+
| q_P_REJECT | | q_P_ACCEPT |
+------------------+ +-------------------+
Mathematical Logic of Transition Function $\delta_P$
Given an input value $v$:
1. If $\text{type}(v) \neq \text{string}$, transition directly to $q_{P\_REJECT}$.
2. Let $s = \text{trim}(v)$. If $|s| < 2$, transition to $q_{P\_REJECT}$.
3. Inspect boundary byte offsets:
$$\delta_P(q_{P0}, v) = \begin{cases} q_{P\_ACCEPT} & \text{if } s[0] = \text{'\textbraceleft'} \land s[|s|-1] = \text{'\textbraceright'} \\ q_{P\_REJECT} & \text{otherwise} \end{cases}$$
This 3-state filter executes in nanoseconds, discarding over 99% of plain matrix string cells before they hit the JSON parser.
5.3.2 Schema Uniformity Enforcer Automaton ($A_{\text{SCHEMA}}$)
Under Core Nesting Rule 2, every valid nested BEJSON 104 sub-document present within column $c$ of a parent document must share an identical schema structure (field names, data types, and field order). $A_{\text{SCHEMA}}$ maintains column-level schema contract enforcement across row iterations.
Let $\mathcal{S}$ be the set of canonical JSON signature strings:
$$\text{sig}(D) = \text{serialize\_canonical}(D.\text{Fields})$$
$A_{\text{SCHEMA}}$ maintains a state dictionary map $M_{\text{col}}: \mathbb{N}_0 \to \mathcal{S}$ during matrix traversal:
A_SCHEMA Column Uniformity DFA
==============================
Cell at Column c
|
+--------------------+
| Is Column c in |
| Map M_col? |
+--------------------+
/ \
No / \ Yes
v v
+--------------------+ +--------------------+
| Register Contract: | | Compare sig(D) == |
| M_col[c] = sig(D) | | M_col[c] |
+--------------------+ +--------------------+
| / \
v Equal / \ Not Equal
+------------+ v v
| Accept Cell| +------------+ +-------------------+
+------------+ | Accept Cell| | Transition to |
+------------+ | Error State E134 |
+-------------------+
Formal Transition Algebra
For a valid sub-document $D_{r,c}$ at row $r$, column $c$:
$$\delta_S(M_{\text{col}}, c, D_{r,c}) = \begin{cases} (M_{\text{col}} \cup \{ c \mapsto \text{sig}(D_{r,c}) \}, \text{OK}) & \text{if } c \notin \text{dom}(M_{\text{col}}) \\ (M_{\text{col}}, \text{OK}) & \text{if } c \in \text{dom}(M_{\text{col}}) \land M_{\text{col}}[c] = \text{sig}(D_{r,c}) \\ (M_{\text{col}}, \text{FAIL\_E134}) & \text{if } c \in \text{dom}(M_{\text{col}}) \land M_{\text{col}}[c] \neq \text{sig}(D_{r,c}) \end{cases}$$
If $A_{\text{SCHEMA}}$ outputs $\text{FAIL\_E134}$, the nested cell is flagged with error `E_NESTING_SCHEMA_MISMATCH` (134), invalidating the cell while maintaining operational scanner stability.
5.3.3 Circular Reference and Depth Ceil Guard Automaton ($A_{\text{GUARD}}$)
To guarantee that recursive document walks always terminate, $A_{\text{GUARD}}$ tracks path depth and monitors document lineage fingerprint sets to prevent infinite recursive loops.
Theoretical State Guard Vector
At recursion level $d$, $A_{\text{GUARD}}$ evaluates the active context tuple:
$$V_{\text{GUARD}} = \left( d, \text{fp}(D), \mathbf{S}_{\text{seen}} \right)$$
Where $\mathbf{S}_{\text{seen}} = \{ \text{fp}_0, \text{fp}_1, \dots, \text{fp}_{d-1} \}$ is the frozen set of parent document fingerprints along the current recursive descent path.
A_GUARD Depth & Cycle Guard State Machine
=========================================
Incoming Nested Cell
|
+--------------------+
| Evaluate Depth d |
+--------------------+
/ \
d > 16 / \ d <= 16
v v
+--------------------+ +--------------------+
| Trigger E132 Error | | Check Fingerprint: |
| Depth Exceeded | | fp(D) in S_seen? |
+--------------------+ +--------------------+
/ \
Yes / \ No
v v
+--------------------+ +--------------------+
| Trigger E135 Error | | Accept Cell & |
| Circular Reference | | Recurse with |
+--------------------+ | S_seen | {fp(D)} |
+--------------------+
Guard Decision Rules
1. Depth Ceiling Check:
$$\text{If } d > D_{\text{max}} \quad (D_{\text{max}} = 16) \implies \text{Emit } \text{E\_NESTING\_DEPTH\_EXCEEDED } (132)$$
2. Circular Reference Check:
$$\text{If } \text{fp}(D) \in \mathbf{S}_{\text{seen}} \implies \text{Emit } \text{E\_NESTING\_CIRCULAR\_REF } (135)$$
3. Safe Recursive Transition:
$$\text{If } d \le 16 \land \text{fp}(D) \notin \mathbf{S}_{\text{seen}} \implies \text{Recurse with } \mathbf{S}_{\text{seen}}' = \mathbf{S}_{\text{seen}} \cup \{ \text{fp}(D) \}$$
---
5.4 State Machine Integration with `NestAddress` and `NestMap` Memory Architecture
The abstract state transitions of $M_{\text{SCAN}}$ directly map to physical memory structures managed by `lib_bejson_CoreNesting_bejson_core_nesting.py`.
5.4.1 Address Resolution and State Persistence
Every state shift in $M_{\text{SCAN}}$ that evaluates a nested cell references or mutates global address space through `NestAddress`:
class NestAddress(NamedTuple):
parent_fp: str
row: int
col: int
depth: int
The 4-tuple address $(f, r, c, d)$ serves as the unique theoretical coordinate in the multi-dimensional document forest. Location defines identity: if identical JSON document payloads appear in two distinct matrix cells, they resolve to separate addresses in `NestMap` without content deduplication.
NestMap Cache Lookup State Integration
======================================
M_SCAN Transition: q_CELL_FETCH
|
Construct Target NestAddress:
(parent_fp, row, col, depth)
|
+----------------------------+
| Cache Lookup: |
| bejson_nesting_cache_get() |
+----------------------------+
/ \
Hit / \ Miss
v v
+--------------------+ +--------------------+
| Re-use Cached Cell | | Execute Sub-DFA |
| Skip JSON Parsing | | Full Parsing Pipeline|
+--------------------+ +--------------------+
| |
+--------------+---------------+
|
v
+----------------------------+
| State Persistence: |
| bejson_nesting_cache_put() |
+----------------------------+
5.4.2 Stack Unwinding and Frame Recovery Logic
When $M_{\text{SCAN}}$ transitions to state $q_{\text{STACK\_POP}}$, the automaton unwinds execution context using Python's implicit runtime call stack during recursive calls to `_walk_cell()`. Stack unwinding restores parent frame variables, returning child `NestedCell` nodes to populate the parent cell's `children` array.
# Frame recovery during stack unwinding inside _walk_cell()
child = _walk_cell(
value=child_val,
row=r_idx,
col=c_idx,
field_name=child_fn,
depth=depth + 1,
seen_fps=child_seen,
parent_fp=nested_fp,
parent_doc=parsed,
col_schemas=child_schemas,
)
if child is not None:
cell.children.append(child) # Attach unwound child state to active cell frame
---
5.5 Error State Automata and Formal Error Transitions
$M_{\text{SCAN}}$ maps operational failures into specific error states ($q_{E130}$ through $q_{E139}$), derived from `lib_bejson_CoreNesting_bejson_errors.py`. Error states are categorized as either Non-Terminal Cell Errors or Terminal Syntax Errors.
+-----------------------------------------------------------------------------------+
| Error State Taxonomy |
+---------------------------------+--------------------+----------------------------+
| State / Error Identifier | Numeric Code Enum | Recovery Classification |
+---------------------------------+--------------------+----------------------------+
| q_E130: E_NESTING_INVALID_CELL | 130 | Terminal Mutation Panic |
| q_E131: E_NESTING_NOT_BEJSON | 131 | Non-Terminal Silent Skip |
| q_E132: E_NESTING_DEPTH_EXCEEDED| 132 | Non-Terminal Cell Flagged |
| q_E133: E_NESTING_CACHE_MISS | 133 | Informational Signal |
| q_E134: E_NESTING_SCHEMA_MISMATCH| 134 | Non-Terminal Cell Flagged |
| q_E135: E_NESTING_CIRCULAR_REF | 135 | Non-Terminal Cell Flagged |
| q_E136: E_NESTING_VALIDATION_FAILED| 136 | Non-Terminal Cell Flagged |
| q_E137: E_NESTING_FIELD_MAP_FAILED | 137 | Non-Terminal Warning Flag |
| q_E138: E_NESTING_QUERY_INVALID_PATH| 138 | Terminal Query Exception |
| q_E139: E_NESTING_QUERY_EMPTY_PATH| 139 | Terminal Query Exception |
+---------------------------------+--------------------+----------------------------+
5.5.1 Transition Rules for Non-Terminal Cell Errors
Non-terminal errors ($E132, E134, E135, E136$) do not panic or halt matrix scanning across unaffected rows. Instead, $M_{\text{SCAN}}$ constructs an invalid `NestedCell` instance, attaches the error string code to `cell.errors`, writes the cell to `NestMap`, and resumes matrix scanning at state $q_{\text{CELL\_FETCH}}$:
$$\delta(q_{\text{SCHEMA\_CHECK}}, \text{SCHEMA\_ERR}, \gamma) \longrightarrow (q_{\text{CACHE\_PUT}}, \gamma) \quad \text{where } \text{cell.is\_valid} = \text{False}$$
This design allows the scanner to process large documents containing scattered invalid sub-documents while collecting complete diagnostic telemetry inside `NestingResult`.
---
5.6 Mathematical Equivalence and Code Mapping
We now demonstrate how the theoretical states and formal transition algebra map directly to operational functions in `lib_bejson_CoreNesting_bejson_core_nesting.py`.
5.6.1 Theoretical State to Implementation Function Mapping
+---------------------------------+---------------------------------------------------------+
| DPDA State / Transition | Implementation Function / Python Block |
+---------------------------------+---------------------------------------------------------+
| q_IDLE -> q_CELL_FETCH | bejson_nesting_scan(doc, use_cache) |
| q_CELL_FETCH (Cache Lookup) | bejson_nesting_cache_get(parent_fp, row, col, depth) |
| q_PRE_FILTER (DFA A_PRE) | _is_candidate(value) |
| q_PARSE_JSON | _parse_cell(raw) |
| q_VAL_104 | _quick_validate_104(parsed) |
| q_SCHEMA_CHECK (DFA A_SCHEMA) | _fields_signature(parsed) comparison against col_schemas|
| q_GUARD Check (DFA A_GUARD) | depth > NESTING_MAX_DEPTH / nested_fp in seen_fps |
| q_STACK_PUSH / q_RECURSE | Recursive call to _walk_cell(...) |
| q_CACHE_PUT | bejson_nesting_cache_put(parent_fp, row, col, depth, cell)|
| q_STACK_POP | Stack return from _walk_cell(...) |
+---------------------------------+---------------------------------------------------------+
5.6.2 Executable State Machine Simulator
The following Python script provides an executable simulator that models the formal $M_{\text{SCAN}}$ DPDA explicitly as a state machine class. It traces state transitions step-by-step as it parses a nested document tree.
import json
import logging
from dataclasses import dataclass, field
from enum import Enum, auto
from typing import Any, Dict, List, Optional, Set, Tuple
# Configure logging to trace state transitions cleanly
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
class AutomatonState(Enum):
IDLE = auto()
CELL_FETCH = auto()
PRE_FILTER = auto()
PARSE_JSON = auto()
VAL_104 = auto()
SCHEMA_CHECK = auto()
GUARD_CHECK = auto()
STACK_PUSH = auto()
RECURSE = auto()
STACK_POP = auto()
CACHE_PUT = auto()
ACCEPT = auto()
ERROR = auto()
class ErrorCode(Enum):
E130_INVALID_CELL = 130
E131_NOT_BEJSON = 131
E132_DEPTH_EXCEEDED = 132
E133_CACHE_MISS = 133
E134_SCHEMA_MISMATCH = 134
E135_CIRCULAR_REF = 135
E136_VALIDATION_FAILED = 136
@dataclass
class StackFrame:
"""Represents a pushdown stack frame structure Gamma."""
doc_fp: str
row: int
col: int
depth: int
col_schemas: Dict[int, str] = field(default_factory=dict)
class NestingDPDASimulator:
"""
Formal State Machine Simulator executing M_SCAN DPDA transitions
explicitly for theoretical trace verification.
"""
MAX_DEPTH = 16
def __init__(self):
self.state = AutomatonState.IDLE
self.stack: List[StackFrame] = []
self.nest_map: Dict[Tuple[str, int, int, int], dict] = {}
self.transition_log: List[str] = []
def _transition(self, to_state: AutomatonState, reason: str):
log_entry = f"State Shift: {self.state.name} --> {to_state.name} | Reason: {reason}"
self.transition_log.append(log_entry)
logging.info(log_entry)
self.state = to_state
def _doc_fingerprint(self, doc: dict) -> str:
rid = doc.get("RELATIONAL_ID") or doc.get("relational_id")
if rid:
return str(rid)
fields_str = json.dumps(doc.get("Fields", []), sort_keys=True)
return f"hash:{hash(fields_str)}"
def _is_candidate_dfa(self, val: Any) -> bool:
"""DFA A_PRE implementation."""
if not isinstance(val, str):
return False
s = val.strip()
return s.startswith("{") and s.endswith("}")
def run(self, root_doc: dict) -> bool:
"""Executes DPDA driver loop over document tree."""
self._transition(AutomatonState.IDLE, "Initializing simulation run")
if not isinstance(root_doc, dict):
self._transition(AutomatonState.ERROR, f"E{ErrorCode.E131_NOT_BEJSON.value}: Root not a dict")
return False
root_fp = self._doc_fingerprint(root_doc)
initial_frame = StackFrame(doc_fp=root_fp, row=0, col=0, depth=1)
self.stack.append(initial_frame)
self._transition(AutomatonState.STACK_PUSH, "Push initial root frame Z_0")
self._walk_document(root_doc, frozen_seen_fps=frozenset({root_fp}))
if self.state != AutomatonState.ERROR:
self._transition(AutomatonState.ACCEPT, "Scan sequence completed cleanly")
return True
return False
def _walk_document(self, doc: dict, frozen_seen_fps: Set[str]):
current_frame = self.stack[-1]
values = doc.get("Values", [])
fields = doc.get("Fields", [])
col_schemas: Dict[int, str] = {}
for r_idx, row in enumerate(values):
if not isinstance(row, list):
continue
for c_idx, cell_val in enumerate(row):
current_frame.row = r_idx
current_frame.col = c_idx
self._transition(AutomatonState.CELL_FETCH, f"Fetching cell ({r_idx}, {c_idx}) at depth {current_frame.depth}")
# Check candidate pre-filter DFA
self._transition(AutomatonState.PRE_FILTER, "Evaluating A_PRE DFA")
if not self._is_candidate_dfa(cell_val):
self._transition(AutomatonState.CELL_FETCH, "A_PRE rejected cell; skipping parse")
continue
# JSON parsing transition
self._transition(AutomatonState.PARSE_JSON, "Executing _parse_cell()")
try:
parsed = json.loads(cell_val)
if not isinstance(parsed, dict):
continue
except Exception:
self._transition(AutomatonState.CELL_FETCH, "JSON parse failure; skipping cell")
continue
# Validation 104 transition
self._transition(AutomatonState.VAL_104, "Executing _quick_validate_104()")
mandatory = {"Format", "Format_Version", "Format_Creator", "Records_Type", "Fields", "Values"}
if mandatory - parsed.keys() or parsed.get("Format") != "BEJSON" or parsed.get("Format_Version") != "104":
self._transition(AutomatonState.CELL_FETCH, "Not valid BEJSON 104; skipping")
continue
# Schema Uniformity DFA
self._transition(AutomatonState.SCHEMA_CHECK, "Evaluating A_SCHEMA DFA contract")
sig = json.dumps(parsed.get("Fields", []), sort_keys=True)
if c_idx not in col_schemas:
col_schemas[c_idx] = sig
elif col_schemas[c_idx] != sig:
self._transition(AutomatonState.ERROR, f"E{ErrorCode.E134_SCHEMA_MISMATCH.value}: Column schema conflict")
return
# Guard DFA Check
self._transition(AutomatonState.GUARD_CHECK, "Evaluating A_GUARD depth and cycles")
if current_frame.depth > self.MAX_DEPTH:
self._transition(AutomatonState.ERROR, f"E{ErrorCode.E132_DEPTH_EXCEEDED.value}: Max depth exceeded")
return
nested_fp = self._doc_fingerprint(parsed)
if nested_fp in frozen_seen_fps:
self._transition(AutomatonState.ERROR, f"E{ErrorCode.E135_CIRCULAR_REF.value}: Cycle detected")
return
# Recurse: Push Stack Frame
child_depth = current_frame.depth + 1
child_frame = StackFrame(doc_fp=nested_fp, row=0, col=0, depth=child_depth)
self.stack.append(child_frame)
self._transition(AutomatonState.STACK_PUSH, f"Push frame for child depth {child_depth}")
self._transition(AutomatonState.RECURSE, f"Descending into sub-document depth {child_depth}")
self._walk_document(parsed, frozen_seen_fps | {nested_fp})
# Unwind Stack Frame
popped = self.stack.pop()
self._transition(AutomatonState.STACK_POP, f"Popped frame depth {popped.depth}; resuming parent depth {current_frame.depth}")
# Cache persistence
addr = (current_frame.doc_fp, r_idx, c_idx, current_frame.depth)
self.nest_map[addr] = parsed
self._transition(AutomatonState.CACHE_PUT, f"Persisted cell to NestMap at address {addr}")
# Demonstration Run
if __name__ == "__main__":
print("==================================================================")
print("EXECUTING FORMAL M_SCAN AUTOMATON SIMULATION")
print("==================================================================")
# Build valid 2-level nested BEJSON document
nested_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["ChildRecord"],
"Fields": [{"name": "stat_id", "type": "string"}],
"Values": [["STAT_01"]]
}
parent_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["ParentRecord"],
"Fields": [{"name": "child_payload", "type": "string"}],
"Values": [[json.dumps(nested_doc)]]
}
sim = NestingDPDASimulator()
success = sim.run(parent_doc)
print("\n==================================================================")
print(f"SIMULATION COMPLETE: Success = {success}")
print("==================================================================")
print(f"Total State Transitions Executed : {len(sim.transition_log)}")
print(f"Total NestMap Addresses Persisted: {len(sim.nest_map)}")
---
5.7 Summary and Architectural Blueprint
This chapter established the theoretical mathematical foundation for the BEJSON 104 core nesting scanner:
1. Language Classification: Embedded BEJSON document parsing operates at the intersection of regular matrix scanning and context-free string recursion ($L_{\text{NEST}}$), requiring pushdown automaton mechanics.
2. Deterministic Pushdown Automaton ($M_{\text{SCAN}}$): Defined formally as a 7-tuple with stack alphabet $\Gamma$, operational state set $Q$, and transition matrix $\delta$ governing scanning, validation, and recursion state shifts.
3. Sub-Automata Formalisms: Standardized validation delegates scanning checks to sub-DFAs: $A_{\text{PRE}}$ for $O(1)$ candidate string filtering, $A_{\text{SCHEMA}}$ for column uniformity contract enforcement ($E134$), and $A_{\text{GUARD}}$ for cycle detection ($E135$) and depth limiting ($E132$).
4. Memory State Mapping: Abstract DPDA state frames map directly to physical 4-tuple `NestAddress` coordinates persisted in the `NestMap` global cache.
5. Operational Equivalence: The executable Python simulator verifies that the formal DPDA state transitions accurately mirror the recursive algorithm implemented in `lib_bejson_CoreNesting_bejson_core_nesting.py`.
Having established formal state machine models and theoretical parsing formalisms, Chapter 6 transitions to multi-tenant deployment architectures, distributed cache synchronizations, and production scaling patterns.
Chapter 6: Hypothetical Multi-Tenant Enterprise Builds and Distributed Cache Patterns
Chapter 6: Hypothetical Multi-Tenant Enterprise Builds and Distributed Cache Patterns
Deploying `lib_bejson_CoreNesting` within high-throughput enterprise systems introduces operational constraints that extend beyond single-process execution. When BEJSON 104 documents are ingested, transformed, and queried across distributed cloud microservices, the default single-process memory architecture must be adapted for multi-tenancy, horizontally scaled caching, and transactional consistency.
This chapter explores advanced architectural patterns for scaling the Core Nesting library across multi-tenant cloud environments. We examine tenant-isolated address keying strategies, two-tiered distributed caching protocols using L1 local process caches and L2 key-value stores, distributed column-schema contract enforcement, and mutative write-back coordination under eventual vs. strong consistency constraints. Finally, we provide a complete reference implementation demonstrating an enterprise-ready distributed cache manager that wraps `lib_bejson_CoreNesting` operations.
---
6.1 Multi-Tenant NestAddress Namespacing and Isolation Architectures
In single-tenant deployments, `lib_bejson_CoreNesting` identifies embedded sub-documents using a canonical 4-tuple address:
$$\text{NestAddress} = (\text{parent\_fp}, \text{row}, \text{col}, \text{depth})$$
Because `parent_fp` is derived from `RELATIONAL_ID` or a cryptographic hash of the document's `Fields` definition, two identical documents in separate execution contexts could theoretically produce identical fingerprints. In a multi-tenant software-as-a-service (SaaS) architecture, sharing a global `_NEST_MAP` cache across tenants without explicit namespacing risks cross-tenant data leakage and cache poisoning.
Multi-Tenant Memory Isolation Architecture
==========================================
Tenant Alpha Request Tenant Beta Request
| |
v v
+--------------------+ +--------------------+
| Tenant Context: A | | Tenant Context: B |
+--------------------+ +--------------------+
| |
+----------------------+----------------------+
|
v
+----------------------------+
| Tenant Identity Injector |
+----------------------------+
|
v
+----------------------------+
| Scoped TenantNestAddress |
| (tenant_id, parent_fp, |
| row, col, depth) |
+----------------------------+
|
+----------------------+----------------------+
| |
v v
+--------------------+ +--------------------+
| Tenant A Scoped | | Tenant B Scoped |
| L1 _NEST_MAP | | L1 _NEST_MAP |
+--------------------+ +--------------------+
6.1.1 Logical vs. Physical Tenant Isolation Models
When architecting BEJSON 104 core nesting layers for multi-tenant enterprise platforms, platform engineers choose between two primary isolation patterns:
1. Logical Isolation (Shared Process, Scoped Namespacing): A single Python worker process serves requests for multiple tenants. The global `_NEST_MAP` dictionary is wrapped or partitioned using an extended 5-tuple key that includes `tenant_id`:
$$\text{TenantNestAddress} = (\text{tenant\_id}, \text{parent\_fp}, \text{row}, \text{col}, \text{depth})$$
This pattern maximizes memory efficiency and thread utilization but requires strict memory access guards to prevent unauthorized state access.
2. Physical Isolation (Dedicated Process / Container): Each tenant request is routed to an isolated process, container, or ephemeral serverless function. `_NEST_MAP` remains unmodified, and physical process boundaries guarantee zero cross-tenant contamination. However, this model increases cold-start latency and idle resource overhead.
6.1.2 Scoped Key Generation Mechanics
Under logical isolation, tenant identity must be bound to the document fingerprint at the ingest boundary. The canonical key mapping function converts raw cell coordinates and tenant tokens into a globally unique cache key string:
$$K_{\text{cache}}(\text{tenant\_id}, \mathbf{a}) = \text{tenant\_id} \mathbin{\Vert} \text{":"} \mathbin{\Vert} \mathbf{a}.\text{parent\_fp} \mathbin{\Vert} \text{":"} \mathbin{\Vert} \mathbf{a}.\text{row} \mathbin{\Vert} \text{":"} \mathbin{\Vert} \mathbf{a}.\text{col} \mathbin{\Vert} \text{":"} \mathbin{\Vert} \mathbf{a}.\text{depth}$$
Where $\mathbf{a} \in \text{NestAddress}$ and $\mathbin{\Vert}$ represents string concatenation.
def make_tenant_cache_key(tenant_id: str, address: NestAddress) -> str:
"""
Generates a deterministic, globally isolated key string for multi-tenant L2 cache lookup.
"""
return f"{tenant_id}:{address.parent_fp}:{address.row}:{address.col}:{address.depth}"
This namespacing contract guarantees that clear operations (`bejson_nesting_cache_clear(parent_fp)`) can be scoped per tenant without invalidating cache entries belonging to other organization boundaries.
---
6.2 Distributed Cache Coherence and Two-Tiered Synchronization Protocols
While `_NEST_MAP` provides $O(1)$ local memory lookups, in-process storage does not scale across horizontal worker clusters. If Worker $A$ performs a scanning operation on a large document and populates its local `_NEST_MAP`, Worker $B$ processing a subsequent query against the same document will incur a cache miss (`E_NESTING_CACHE_MISS` / Code 133) unless a distributed tier is introduced.
6.2.1 Two-Tiered Caching Architecture (L1 / L2 Topology)
To achieve microsecond access times while preserving cluster-wide coherence, enterprise builds employ a Two-Tiered Cache Architecture:
* Tier 1 (L1 Local Process Cache): In-memory `_NEST_MAP` native to `lib_bejson_CoreNesting`. Offers sub-microsecond lookups for active worker requests.
* Tier 2 (L2 Distributed Cache): High-throughput key-value store (e.g., Redis, KeyDB, or Dragonfly) accessible by all worker nodes in a cloud region.
Two-Tiered Cache Synchronizer Mechanics
=======================================
Worker Node 1 Worker Node 2
+--------------------+ +--------------------+
| L1 Cache (_NEST_MAP)| | L1 Cache (_NEST_MAP)|
+--------------------+ +--------------------+
| |
Read / | Write Read / | Write
v v
+-------------------------------------------------------------------------+
| L2 Distributed Cache (Redis Cluster) |
| Keyspace: tenant_id:parent_fp:row:col:depth |
+-------------------------------------------------------------------------+
^
| Pub/Sub Invalidation Bus
+-------------------------------------------------------------------------+
| Event Channel: nesting_invalidations |
+-------------------------------------------------------------------------+
6.2.2 Serialization and Deserialization Protocols for `NestedCell`
Because `NestedCell` objects contain live Python dictionaries (`cell.doc`) and nested child node lists (`cell.children`), pushing cells to an L2 store requires deterministic JSON or MessagePack serialization.
Serialization Rules for Distributed Persistence
1. Field Map Stripping: Internal cache keys such as `_nesting_field_map` must be stripped prior to serialization and lazily reconstructed on retrieval via `bejson_nesting_get_field_map()`.
2. Error and Warning Preservation: The exact list of error strings (e.g., `E134`, `E136`) and standardization warnings must be preserved in JSON payloads to prevent re-validation during subsequent reads.
3. Recursive Representation: Child nodes in `cell.children` are serialized recursively, preserving tree topology.
{
"row": 0,
"col": 2,
"field_name": "embedded_payload",
"depth": 1,
"is_valid": true,
"errors": [],
"warnings": ["W: Records_Type 'Payload' in nested doc at col 'embedded_payload' should be 'EmbeddedPayload' by convention"],
"doc": {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["Payload"],
"Fields": [{"name": "item_id", "type": "string"}],
"Values": [["ITEM_1009"]]
},
"children": []
}
6.2.3 Cache Coherence and Invalidation Mechanics
When a document cell is updated via `bejson_nesting_mutate()`, the localized modification mutates `parent_doc.Values[row][col]` in-place. If another node holds a stale copy of the original `NestedCell` in its L1 cache, state drift occurs.
To maintain coherence across nodes, enterprise deployments implement an Invalidate-on-Write (IoW) protocol backed by a distributed Pub/Sub event bus:
Coherence Invalidation Sequence during Mutative Write-Back
=========================================================
Worker A (Mutator) Redis Cluster Worker B (Peer)
| | |
1. Execute Mutation | |
bejson_nesting_mutate() | |
| | |
2. Evict L1 local cache | |
bejson_nesting_cache_clear() | |
| | |
3. Delete key from L2 | |
DEL tenant_id:fp:r:c:d ----------->| |
| | |
4. Publish Invalidation Event | |
PUBLISH nesting_invalidations ---->| |
|--- Broadcast Event ------->|
|
5. Evict stale L1 entry
bejson_nesting_cache_clear()
---
6.3 Enterprise Pipeline Integration and High-Throughput Batch Scanning
In big data and streaming platforms (e.g., Apache Spark, Ray, Apache Flink, or Celery pipelines), BEJSON 104 documents arrive in partitioned batches. Processing million-row datasets containing nested cells requires distributed column-schema contract enforcement.
6.3.1 Distributed Column-Schema Contract Enforcement
Under Core Nesting Rule 2, all nested sub-documents within column $c$ of a parent document must maintain identical schema signatures (`_fields_signature(doc)`). In a distributed environment where document rows are distributed across multiple processing partitions $P_1, P_2, \dots, P_k$, enforcing schema uniformity requires a two-phase contract verification model.
Two-Phase Distributed Column-Schema Contract Verification
=========================================================
Phase 1: Local Partition Inspection (Parallel)
----------------------------------------------
Partition 1 (Rows 0-999) --> Extract Col Signature: Sig_A
Partition 2 (Rows 1000-1999) --> Extract Col Signature: Sig_A
Partition 3 (Rows 2000-2999) --> Extract Col Signature: Sig_B <-- Schema Mismatch!
Phase 2: Distributed Synchronization Barrier
---------------------------------------------
Reduce Phase: Compare Signatures across Partitions
Sig_A == Sig_A (Partitions 1 & 2 match)
Sig_A != Sig_B (Partition 3 violates Rule 2)
|
+--> Trigger E134 Error on Partition 3 Cells
Phase 1: Local Signature Map Generation
Each worker node scans its assigned row partition and computes local schema signatures for nested cells in column $c$:
$$S_{\text{local}}(c) = \{ \text{\_fields\_signature}(D_{r,c}) \mid r \in \text{PartitionRows} \}$$
If $|S_{\text{local}}(c)| > 1$, a schema mismatch exists within the local partition, raising `E_NESTING_SCHEMA_MISMATCH` (Error Code 134) locally.
Phase 2: Distributed Accumulation Barrier
If a partition succeeds locally, its column signature vector $(c, \text{signature})$ is emitted to a central consensus barrier (or reduced across worker nodes). If any partition reports a signature $S_i(c) \neq S_j(c)$, all worker nodes tag the affected column cells as invalid with `E134`.
6.3.2 Concurrency and Thread Safety in `_NEST_MAP`
The reference implementation of `_NEST_MAP` in `lib_bejson_CoreNesting_bejson_core_nesting.py` utilizes a module-level dictionary:
_NEST_MAP: Dict[NestAddress, NestedCell] = {}
In Python multi-threaded environments (e.g., WSGI/ASGI web servers like Gunicorn or Uvicorn running under `threading`), concurrent reads and writes to `_NEST_MAP` can trigger race conditions or dynamic dictionary mutation runtime errors.
Enterprise architectures wrap all access to `_NEST_MAP` with thread-safe synchronization primitives or utilize thread-local cache storage:
import threading
from typing import Optional
from lib_bejson_CoreNesting_bejson_core_nesting import (
bejson_nesting_cache_get,
bejson_nesting_cache_put,
NestedCell,
)
class ThreadSafeNestMapGuard:
"""
Thread-safe context wrapper providing reentrant lock synchronization
over the global _NEST_MAP dictionary.
"""
_lock = threading.RLock()
@classmethod
def get(cls, parent_fp: str, row: int, col: int, depth: int) -> Optional[NestedCell]:
with cls._lock:
return bejson_nesting_cache_get(parent_fp, row, col, depth)
@classmethod
def put(cls, parent_fp: str, row: int, col: int, depth: int, cell: NestedCell) -> None:
with cls._lock:
bejson_nesting_cache_put(parent_fp, row, col, depth, cell)
---
6.4 Mutative Write-Backs and ACID Semantics in Distributed Storage
Applying structural modifications to nested sub-documents via `bejson_nesting_mutate()` requires strict transactional safeguards. Because `bejson_nesting_mutate()` enforces an append-only contract for schema fields with automatic null-padding, write-backs to distributed database layers must satisfy Atomic, Consistent, Isolated, and Durable (ACID) properties across physical storage nodes.
Distributed Write-Back Lifecycle with ACID Safety
=================================================
Parent Document in S3 / Relational DB / Mongo
|
v
1. Read & Lock Parent Document (Pessimistic / Optimistic Lock)
|
v
2. Execute In-Memory Mutation
bejson_nesting_mutate(parent_doc, nested_cell, mutation_fn)
|
+---> Validates append-only field growth
+---> Applies automatic null-padding to rows
+---> Serializes nested doc back to JSON string
|
v
3. Write Transaction Back to Persistent Store
UPDATE parent_table SET values_json = :serialized_doc WHERE id = :id AND version = :v
|
+--- If Version Mismatch ---> Rollback & Retry
+--- If Success -------------> Broadcast Invalidation Event
6.4.1 Schema Mutation Constraints and Null-Padding Mechanics
The mutation engine within `lib_bejson_CoreNesting_bejson_core_nesting.py` imposes strict invariants:
1. No Field Removal: `n_after < n_before` raises `ValueError` (`E_NESTING_SCHEMA_MISMATCH`).
2. No Field Reordering or Retyping: Modifying existing field names or type declarations raises `ValueError`.
3. Append-Only Schema Growth: Adding new fields to `doc["Fields"]` automatically pads existing rows in `doc["Values"]` with `None` (`null` in JSON).
# Validation logic embedded in bejson_nesting_mutate()
fields_before: List[dict] = [f.copy() for f in doc.get("Fields", [])]
n_before = len(fields_before)
# User-supplied mutation callback executes
mutation_fn(doc)
fields_after: List[dict] = doc.get("Fields", [])
n_after = len(fields_after)
if n_after < n_before:
raise ValueError(f"E{E_NESTING_SCHEMA_MISMATCH}: mutation removed fields")
# Auto null-pad existing value rows
if n_after > n_before:
delta = n_after - n_before
for r in doc.get("Values", []):
if isinstance(r, list):
r.extend([None] * delta)
6.4.2 Optimistic Concurrency Control (OCC) for Multi-Node Write-Backs
When multiple microservices attempt concurrent updates on different nested cells within the same parent document, pessimistic locking creates severe contention. Enterprise pipelines use Optimistic Concurrency Control (OCC) based on document version numbers:
$$\text{DocumentState} = \big( \text{RELATIONAL\_ID}, \text{Version}, \text{Values} \big)$$
OCC Write-Back Protocol
1. Fetch: Read parent document and record `Version` token $v_1$.
2. Scan & Mutate: Execute `bejson_nesting_scan()` and `bejson_nesting_mutate()` in local memory.
3. CAS Store Operation: Execute Atomic Compare-And-Swap (CAS) write to persistent database:
$$\text{UPDATE documents SET content} = D_{\text{mutated}}, \text{version} = v_1 + 1 \quad \text{WHERE id} = \text{doc\_id} \land \text{version} = v_1$$
4. Retry Loop: If zero rows are updated, another worker node modified the document concurrently. Abort local transaction, re-fetch, re-scan, and re-apply mutation callback.
---
6.5 Production Architecture Blueprint: Executable Multi-Tenant Cache Manager
This section provides a complete, production-grade reference implementation of a Multi-Tenant Distributed Nesting Cache Manager.
The `DistributedTenantNestManager` class wraps `lib_bejson_CoreNesting` functions (`bejson_nesting_scan`, `bejson_nesting_mutate`, `bejson_nesting_query`), providing:
* Multi-tenant address key isolation.
* Two-tiered (L1 `_NEST_MAP` local memory / L2 simulated distributed store) cache synchronization.
* Thread-safe synchronization locks.
* Distributed cache invalidation on mutations.
* Comprehensive telemetry tracing and error tracking.
"""
Module: distributed_tenant_nest_manager.py
Description: Enterprise-grade Multi-Tenant Cache Manager wrapping BEJSON 104
Core Nesting scanner, query, and mutation routines with L1/L2
caching and distributed invalidation.
Version: 1.0.0
Author: Elton Boehnen
"""
import json
import logging
import threading
from dataclasses import asdict, dataclass
from typing import Any, Callable, Dict, List, Optional, Tuple
# Core Nesting Library Imports
from lib_bejson_CoreNesting_bejson_core_nesting import (
NestAddress,
NestedCell,
NestingResult,
bejson_nesting_cache_clear,
bejson_nesting_cache_get,
bejson_nesting_cache_put,
bejson_nesting_mutate,
bejson_nesting_scan,
)
from lib_bejson_CoreNesting_bejson_errors import (
E_NESTING_CACHE_MISS,
E_NESTING_INVALID_CELL,
E_NESTING_SCHEMA_MISMATCH,
)
from lib_bejson_CoreNesting_bejson_nesting_query import (
QueryMatch,
bejson_nesting_query,
)
# Setup Logger
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("DistributedTenantNestManager")
class SimulatedL2RedisCluster:
"""
Simulates a high-throughput, cluster-wide L2 key-value store (e.g., Redis)
with pub/sub event channels for cache invalidation.
"""
def __init__(self):
self._store: Dict[str, str] = {}
self._lock = threading.Lock()
self._subscribers: List[Callable[[str], None]] = []
def get(self, key: str) -> Optional[str]:
with self._lock:
return self._store.get(key)
def set(self, key: str, value: str) -> None:
with self._lock:
self._store[key] = value
def delete(self, key: str) -> bool:
with self._lock:
if key in self._store:
del self._store[key]
self._publish_invalidation(key)
return True
return False
def subscribe_invalidations(self, callback: Callable[[str], None]) -> None:
with self._lock:
self._subscribers.append(callback)
def _publish_invalidation(self, key: str) -> None:
for sub in self._subscribers:
sub(key)
class DistributedTenantNestManager:
"""
Thread-safe, multi-tenant enterprise wrapper around lib_bejson_CoreNesting.
Manages L1 (in-process _NEST_MAP) and L2 (Distributed Cluster) memory layers.
"""
def __init__(self, l2_cluster: SimulatedL2RedisCluster):
self.l2 = l2_cluster
self._local_lock = threading.RLock()
# Subscribe to distributed invalidation events to clean L1 cache
self.l2.subscribe_invalidations(self._on_remote_invalidation)
def _make_l2_key(self, tenant_id: str, parent_fp: str, row: int, col: int, depth: int) -> str:
return f"nest:{tenant_id}:{parent_fp}:{row}:{col}:{depth}"
def _on_remote_invalidation(self, key: str) -> None:
"""
Event handler invoked when an L2 cache key is invalidated by any node.
Clears local L1 cache to maintain cluster consistency.
"""
with self._local_lock:
# Key format: nest:tenant_id:parent_fp:row:col:depth
parts = key.split(":")
if len(parts) == 6:
parent_fp = parts[2]
bejson_nesting_cache_clear(parent_fp)
logger.info(f"[L1 EVICTION] Cleared L1 cache for parent_fp={parent_fp} via pub/sub signal")
def _serialize_cell(self, cell: NestedCell) -> str:
"""Serializes a NestedCell instance to JSON for L2 storage."""
data = {
"row": cell.row,
"col": cell.col,
"field_name": cell.field_name,
"depth": cell.depth,
"doc": cell.doc,
"is_valid": cell.is_valid,
"errors": cell.errors,
"warnings": cell.warnings,
}
return json.dumps(data, ensure_ascii=False)
def _deserialize_cell(self, raw_json: str) -> NestedCell:
"""Deserializes a JSON string back into a NestedCell instance."""
d = json.loads(raw_json)
return NestedCell(
row=d["row"],
col=d["col"],
field_name=d["field_name"],
depth=d["depth"],
doc=d["doc"],
is_valid=d["is_valid"],
errors=d.get("errors", []),
warnings=d.get("warnings", []),
)
def scan_tenant_document(
self, tenant_id: str, doc: dict, use_cache: bool = True
) -> NestingResult:
"""
Executes a multi-tenant aware scan on a BEJSON 104 document.
Checks L1 (_NEST_MAP) first, then L2 (Redis), falling back to full scan.
"""
if not tenant_id or not tenant_id.strip():
raise ValueError("Tenant ID must be a non-empty string.")
parent_fp = doc.get("RELATIONAL_ID") or f"hash:{hash(json.dumps(doc.get('Fields', []), sort_keys=True))}"
with self._local_lock:
# 1. Full In-Process L1 Scan Execution
result = bejson_nesting_scan(doc, use_cache=use_cache)
# 2. Synchronize newly discovered valid nested cells to L2 store
for cell in result.cells:
if cell.is_valid:
l2_key = self._make_l2_key(tenant_id, str(parent_fp), cell.row, cell.col, cell.depth)
serialized = self._serialize_cell(cell)
self.l2.set(l2_key, serialized)
logger.info(
f"[SCAN COMPLETE] Tenant={tenant_id} | Scanned={result.scanned_cells} | "
f"Nested={result.nested_found} | Valid={result.nested_valid} | "
f"L2 Synced={len(result.cells)}"
)
return result
def query_tenant_document(
self, tenant_id: str, doc: dict, path: str, match_value: Any = None
) -> List[QueryMatch]:
"""
Executes a path query against a tenant's BEJSON 104 document.
"""
if not tenant_id:
raise ValueError("Tenant ID required")
with self._local_lock:
# Pre-populate L1 cache via scan
self.scan_tenant_document(tenant_id, doc, use_cache=True)
# Execute path-based query
matches = bejson_nesting_query(doc, path, match_value=match_value)
logger.info(f"[QUERY] Tenant={tenant_id} | Path='{path}' | Matches={len(matches)}")
return matches
def mutate_tenant_cell(
self,
tenant_id: str,
parent_doc: dict,
nested_cell: NestedCell,
mutation_fn: Callable[[dict], None],
) -> None:
"""
Executes an in-place mutation on a nested cell with multi-tenant L1/L2
cache eviction and broadcast invalidation guarantees.
"""
parent_fp = parent_doc.get("RELATIONAL_ID") or f"hash:{hash(json.dumps(parent_doc.get('Fields', []), sort_keys=True))}"
with self._local_lock:
try:
# 1. Execute mutation via core engine
bejson_nesting_mutate(parent_doc, nested_cell, mutation_fn)
logger.info(f"[MUTATE SUCCESS] Tenant={tenant_id} | Cell row={nested_cell.row} col={nested_cell.col}")
except ValueError as ve:
logger.error(f"[MUTATE FAILED] Tenant={tenant_id} | Violation: {ve}")
raise ve
# 2. Invalidate L2 Cluster Cache
l2_key = self._make_l2_key(tenant_id, str(parent_fp), nested_cell.row, nested_cell.col, nested_cell.depth)
self.l2.delete(l2_key)
# 3. Clear local L1 cache for this parent document
bejson_nesting_cache_clear(str(parent_fp))
# =====================================================================
# Verification Script Demonstrating Multi-Tenant Manager Operations
# =====================================================================
if __name__ == "__main__":
print("==================================================================")
print("EXECUTING MULTI-TENANT ENTERPRISE CACHE MANAGER TEST SUITE")
print("==================================================================")
# Initialize simulated Redis Cluster
redis_cluster = SimulatedL2RedisCluster()
# Initialize Tenant Managers (representing two independent microservice nodes)
node_alpha = DistributedTenantNestManager(redis_cluster)
node_beta = DistributedTenantNestManager(redis_cluster)
# 1. Construct Valid Embedded BEJSON 104 Sub-Document
nested_sub_doc = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["InventoryItem"],
"Fields": [
{"name": "sku", "type": "string"},
{"name": "qty", "type": "integer"},
],
"Values": [
["SKU-ALPHA-01", 150],
["SKU-ALPHA-02", 300],
],
}
# 2. Construct Root Parent Document with Embedded Sub-Doc
root_document = {
"Format": "BEJSON",
"Format_Version": "104",
"Format_Creator": "Elton Boehnen",
"Records_Type": ["WarehouseRecord"],
"RELATIONAL_ID": "DOC-TENANT-ALPHA-9981",
"Fields": [
{"name": "warehouse_id", "type": "string"},
{"name": "inventory_payload", "type": "string"},
],
"Values": [
["WH-NORTH-01", json.dumps(nested_sub_doc)],
],
}
TENANT_A = "Tenant_Org_Alpha"
TENANT_B = "Tenant_Org_Beta"
# Step A: Node Alpha Scans Document for Tenant A
print("\n--- Step A: Node Alpha Scans Document for Tenant A ---")
scan_res = node_alpha.scan_tenant_document(TENANT_A, root_document)
print(f"Scanned Cells: {scan_res.scanned_cells}")
print(f"Valid Nested Cells Found: {scan_res.nested_valid}")
# Step B: Node Alpha Queries Path
print("\n--- Step B: Query Path 'inventory_payload[*].sku' ---")
matches = node_alpha.query_tenant_document(
TENANT_A, root_document, "inventory_payload[*].sku"
)
for m in matches:
print(f"Match Found: Path='{m.path}' -> Value={m.value!r}")
# Step C: Node Alpha Mutates Nested Cell (Schema Field Addition)
print("\n--- Step C: Node Alpha Appends Field 'reorder_point' via Mutation ---")
target_cell = scan_res.cells[0]
def add_reorder_field(doc: dict):
doc["Fields"].append({"name": "reorder_point", "type": "integer"})
node_alpha.mutate_tenant_cell(TENANT_A, root_document, target_cell, add_reorder_field)
# Step D: Verify Auto Null-Padding and Serialization
print("\n--- Step D: Verify Mutated Parent Values Cell Output ---")
mutated_cell_str = root_document["Values"][0][1]
mutated_doc_obj = json.loads(mutated_cell_str)
print(f"New Fields Schema: {mutated_doc_obj['Fields']}")
print(f"Values Row 0 (Null-Padded): {mutated_doc_obj['Values'][0]}")
print(f"Values Row 1 (Null-Padded): {mutated_doc_obj['Values'][1]}")
# Step E: Node Beta Queries Mutated Document
print("\n--- Step E: Node Beta Queries Mutated Document ---")
beta_matches = node_beta.query_tenant_document(
TENANT_A, root_document, "inventory_payload[*].reorder_point"
)
for m in beta_matches:
print(f"Node Beta Match: Path='{m.path}' -> Value={m.value!r}")
print("\n==================================================================")
print("TEST SUITE COMPLETE: Multi-Tenant Architecture Invariants Verified")
print("==================================================================")
---
6.6 Chapter Summary and Best Practices Matrix
Deploying `lib_bejson_CoreNesting` within distributed enterprise software requires extending local process scanning semantics into scalable, multi-tenant cloud patterns.
Key Takeaways
1. Multi-Tenant Isolation: Always scope L2 cache keys using explicit tenant prefixes (`tenant_id:parent_fp:row:col:depth`) to prevent cross-tenant memory leakage.
2. Two-Tiered Cache Synchronicity: Combine sub-microsecond L1 process caching (`_NEST_MAP`) with an L2 cluster store (Redis/KeyDB) synchronized via Pub/Sub invalidation channels.
3. Thread Safety: Wrap local `_NEST_MAP` access functions (`bejson_nesting_cache_get`, `bejson_nesting_cache_put`, `bejson_nesting_cache_clear`) with reentrant locks (`threading.RLock`) in multi-threaded web application workers.
4. Distributed Column Schema Verification: Enforce Core Nesting Rule 2 (`E_NESTING_SCHEMA_MISMATCH` / E134) across partitioned big-data batches using a two-phase reduction barrier.
5. Mutation Safety & ACID Write-Backs: Respect append-only field addition invariants during `bejson_nesting_mutate()`. Null-padding is applied automatically; field removals or reorders are rejected. Use Optimistic Concurrency Control (OCC) versioning when committing updates to persistent storage layers.
Enterprise Deployment Best Practices Matrix
| Operational Dimension | Recommended Pattern | Fallback / Risk | Relevant Error Code |
| :--- | :--- | :--- | :--- |
| Tenant Isolation | Scoped 5-tuple `TenantNestAddress` | Unscoped shared `_NEST_MAP` (Cross-tenant leak) | `E_NESTING_CACHE_MISS` (133) |
| Cache Synchronization | Invalidate-on-Write (IoW) with Pub/Sub | Time-To-Live (TTL) expiration only (Stale reads) | N/A |
| Thread Concurrency | Reentrant Lock Wrapper (`RLock`) | Unlocked raw dict access (Race conditions) | N/A |
| Schema Mutations | Append-only fields with auto null-padding | Schema field deletion or type modification | `E_NESTING_SCHEMA_MISMATCH` (134) |
| Cell Address Bounds | Pre-validate `(row, col)` bounds before write | Out-of-bounds array access | `E_NESTING_INVALID_CELL` (130) |
| Depth Boundary | Enforce hard recursion ceiling $\le 16$ | Unbounded recursion stack overflow | `E_NESTING_DEPTH_EXCEEDED` (132) |
| Cycle Prevention | Path fingerprint tracking ($\mathbf{S}_{\text{seen}}$) | Infinite recursive self-reference loop | `E_NESTING_CIRCULAR_REF` (135) |
