Polyglot Part 3: Inside the BEJSON 104 Core AI Family

Chapter 2: Subsystem 1 (ai_dispatch) — The 12-Provider Universal Dispatch Matrix

How many times have you had to rewrite your application because you swapped from OpenAI to Anthropic, or from Gemini to a local Ollama server? Every commercial SDK has its own incompatible method names (client.chat.completions.create vs client.models.generate_content), inconsistent parameter keys, and conflicting response objects. In libraries_104/Lib_PY/AI/lib_bejson_AI_ai_dispatch.py, Elton solved this by architecting the Universal Multi-Provider Dispatch Adapter.

The Universal Call Signature

No matter which model or cloud provider you call, your application code interacts with exactly one function and receives exactly one standardized response shape:

def ai_dispatch_call(
    provider: str,
    user_input: str,
    model_id: Optional[str] = None,
    system_instruction: Optional[str] = None,
    **config
) -> AIResponse:

The 12-Provider Matrix

Subsystem 1 abstracts 12 distinct cloud and local AI inference engines without requiring third-party SDK bloat:

The AIResponse Contract & The Zero-Crash Law

What happens when an API key is invalid, or the remote server returns a 503? Most libraries explode with an unhandled exception that brings down your entire service. Subsystem 1 enforces the Zero-Crash Invariant: ai_dispatch_call() never raises for provider-side failures. Every result is wrapped in an immutable AIResponse object:

@dataclass
class AIResponse:
    success: bool
    text: str
    error: Optional[str] = None
    raw: Any = None
    model: str = ""
    provider: str = ""
    tokens_in: Optional[int] = None
    tokens_out: Optional[int] = None
    unsupported_params: List[str] = field(default_factory=list)

If a provider cannot honor a requested parameter (such as a system instruction or a thinking toggle), it doesn't fail silently; it logs it in unsupported_params, giving calling agents 100% parameter visibility.

Chapter 3: Subsystem 2 (ai_context_bubble) — Context Bubble v2 & Dynamic Spill-Up Engine

Every amateur developer builds AI chats with a dumb string concatenation: prompt = system_prompt + "\n" + chat_history + "\n" + user_input. Then a user attaches a 50KB code file, the prompt explodes past the token limit, the API returns a 400 error, and the conversation dies. Even worse, if you use static percentage splits (e.g. 20% system, 30% history, 50% attachments), what happens when the user has zero attachments? 50% of your expensive token window sits empty while your chat history gets aggressively truncated! That is amateur-hour design.

In libraries_104/Lib_PY/AI/lib_bejson_AI_ai_context_bubble.py, Elton Boehnen engineered the Context Bubble Engine v2, featuring ranked allotments, Chunked-104a cryptographic persistence, and the mathematical Spill-Up Algorithm.

Ranked Allotments & context_bubble.bejson

In the Context Bubble architecture, context is not a raw string — it is a collection of registered Allotments. Each allotment represents a distinct, ranked source of knowledge configured in config/context_bubble.bejson (BEJSON 104a):

// Allotment Configuration in context_bubble.bejson (104a)
{
  "Format": "BEJSON",
  "Format_Version": "104a",
  "Format_Creator": "Elton Boehnen",
  "Records_Type": ["AllotmentConfig"],
  "Normalize_Baselines": true,
  "Fields": [
    {"name": "name", "type": "string"},
    {"name": "rank", "type": "integer"},
    {"name": "pct", "type": "number"},
    {"name": "keep", "type": "string"},
    {"name": "refresh", "type": "string"}
  ],
  "Values": [
    ["policy", 1, 0.20, "start", "change"],
    ["profile", 2, 0.15, "start", "change"],
    ["brain_pinned", 3, 0.15, "start", "change"],
    ["context_files", 4, 0.20, "start", "change"],
    ["chat_history", 5, 0.25, "end", "turn"],
    ["brain_recall", 6, 0.05, "start", "turn"]
  ]
}

The Dynamic Spill-Up Algorithm

The crown jewel of the Context Bubble is its Dynamic Spill-Up Engine:

1. Surplus Release: During prompt compilation, every allotment builder runs. If an allotment (like context_files) uses less than its allocated budget (or returns empty), its unused token share is immediately released into a shared donor pool.
2. Rank-Ordered Claims: Over-budget allotments claim the surplus in strict priority order (Rank 1 policy claims first, followed by Rank 2, 3, etc.).
3. Even Donor Draw: Claimants do not steal greedily from one source; they draw evenly across all donor allotments.
4. Elastic Rebound: The moment a donor allotment needs its tokens back on a subsequent turn (e.g. the user uploads a large file), the donor reclaims its baseline, and the claimant shrinks back gracefully.
5. Defensive Truncation: If an allotment still exceeds its expanded grant, it truncates safely based on its keep policy (keep="end" retains the latest chat turns; keep="start" preserves foundational policy rules).

Dual-Tier Chunked-104a Persistence & Anti-Spoofing

Context bubbles don't live as ephemeral RAM strings. Every allotment is chunked into <context_dir>/bubble/allotments/<name>.104a.bejson using the frozen Core chunking engine. Then, the entire bubble directory is packaged into Context_Bubble.104a.bejson with SHA-256 integrity verification at both layers.

Furthermore, the engine includes built-in security defanging: if an adversary injects prompt injection tokens like [END CONTEXT BUBBLE] into a document or user message, the bubble parser neutralizes the delimiter before compilation, preventing context breakout attacks.

Chapter 4: Subsystem 3 (bejson_interactions) — Gemini Interactions API & Incident-Hardened Wire Protocol

Most AI developers treat API calls as black boxes. They call an SDK, pass some mystery parameters, and when the API silently fails or returns empty responses, they scratch their heads and blame the model. In libraries_104/Lib_PY/AI/lib_bejson_AI_bejson_interactions.py, Subsystem 3 is an incident-hardened wire-level integration with Google's state-of-the-art Gemini Interactions API (/v1beta/interactions), reverse-engineered and battle-tested through 11 documented production incidents.

Why the Interactions API Destroys Classic Chat Endpoints

Traditional chat completions (like generateContent or OpenAI /v1/chat/completions) are stateless. Every single turn, your client has to re-upload the entire conversation history, re-chunk the prompt, and pay full price for context tokens. The Gemini Interactions API introduces server-side state chaining via previous_interaction_id: the model maintains the conversation bubble on Google's TPU clusters, slashing token bandwidth and latency for multi-turn agentic workflows.

The 5 Golden Wire Rules (Hardened Against API Traps)

Google's Interactions API has undocumented failure modes that trip up almost every SDK. Subsystem 3 embeds strict architectural defenses against each one:

1. Header Authentication Invariant: The client sends x-goog-api-key: <key> in HTTP headers. Passing ?key= in the query string is rejected or leaks secrets in proxy access logs.
2. Incident 7 (The generation_config Poison Pill): In standard Gemini endpoints, you pass generation_config for temperature and max tokens. On the Interactions API, passing generation_config causes the API to silently drop or break the call! Subsystem 3 strips this field by default, ensuring uninterrupted turn completion.
3. Incident 5 (Attachment Wire Formatting):
   • Text & Code: Transmitted as raw text ({"type": "text", "text": "..."}) — zero base64 encoding bloat!
   • Images: Transmitted as {"type": "image", "data": "<base64>", "mime_type": "..."}.
   • Documents (PDF/Binary): Transmitted as {"type": "document", "data": "<base64>", "mime_type": "..."}.
   • Prohibited: inline_data belongs to the legacy API and silently drops files on the Interactions endpoint.
4. Function Call & Step Traversal: Responses return steps inside the "steps" array (never "outputs"). Tool arguments arrive strictly under the "arguments" key (never "args").
5. Function Result Enclosure: When returning tool execution results, the "result" field must be an array of content objects ([{"type": "text", "text": "..."}]). Returning a bare dictionary causes immediate HTTP 400 rejection.

# Multi-turn interaction chaining with native tool execution
response = interactions_chat(
    api_key=api_key,
    model="gemini-2.5-flash",
    user_input="Analyze this system manifest.",
    previous_interaction_id=last_session_id,
    tools=[{"type": "function", "name": "scan_manifest", "description": "...", "parameters": {...}}],
    attachments=[{"type": "text", "text": manifest_data}]
)
# Returns structured steps, extracted thoughts, and the new interaction_id

Chapter 5: Subsystem 4 (brain_store) — The Autonomous AI Brain & Untrusted Writer Security Model

Every commercial "agentic memory" system (like Mem0 or vector database wrappers) makes a fatal security blunder: they give the LLM full write access to its own memory store. If the model hallucinates, suffers prompt injection, or gets confused, it overwrites critical facts, deletes user instructions, and poisons its own knowledge base. That is suicidal architecture.

In libraries_104/Lib_PY/AI/lib_bejson_AI_brain_store.py, Subsystem 4 implements the Autonomous AI Brain, driven by an MFDB database and governed by the Untrusted Writer Security Protocol.

MFDB Database Architecture: brain_mfdb

The AI Brain is not an opaque SQLite file or a cloud SaaS endpoint; it is an atomic, human-readable Multi-File Database (MFDB v1.31):

• brain_mfdb/104a.mfdb.bejson: Root database manifest.
• brain_mfdb/data/braincategory.bejson: Categorical taxonomy (e.g. Preferences, Workflows, Ecosystem).
• brain_mfdb/data/brainmemory.bejson: Atomic memory records with keywords, origin tags, and lifecycle states.

The Untrusted Writer Security Protocol

Subsystem 4 strictly separates operations by actor identity: user (human operator) vs ai (autonomous model):

Crash Recovery & Transaction Journaling

Concurrency and power outages are solved at the filesystem level. The brain engine enforces PID-locked serialization via brain_state/brain.lock. Every batch modification is staged into brain_state/txn/, executed, and deep-verified (mfdb_core_deep_verify). If the host script crashes or power is lost mid-write, the startup recovery hook detects the uncommitted batch on next launch, rolls back to the clean snapshot, and resumes without losing a single byte of database integrity.

Chapter 6: Subsystem 5 (bejson_litertlm) — On-Device LiteRT & Local Offline Edge Intelligence

If your AI agent requires an active internet connection, you don't own your agent — Google and OpenAI own it. The moment your network drops, your mobile data runs out, or an upstream API revokes your API key, your entire autonomous stack collapses. In libraries_104/Lib_PY/AI/lib_bejson_AI_bejson_litertlm.py, Subsystem 5 gives the BEJSON ecosystem **100% offline, on-device edge intelligence** running directly inside Android Termux on ARM64.

The Google LiteRT-LM Substrate

Subsystem 5 interfaces natively with Google's LiteRT runtime (the next-generation TensorFlow Lite LLM engine). It manages the local daemon via litert-lm serve --port 9379, executing highly quantized models (such as Gemma 4 E2B with a 32,000 token context window) directly on device CPU/GPU/NPU hardware without sending a single packet over the public internet.

Dual Wire Protocol Parity (OpenAI & Gemini)

What makes Subsystem 5 brilliant is its Dual-Protocol Wire Adapter. The local server exposes two simultaneous standards:

• OpenAI-Compatible Mode: /v1/models and /v1/chat/completions for standard REST consumers.
• Native Gemini Mode: /v1beta/models/{model}:generateContent and :streamGenerateContent, allowing tools designed for Google's cloud API to execute locally without modifying a single line of client code!

# Switching from cloud Gemini to offline on-device LiteRT in 1 line:
from lib_bejson_AI_ai_dispatch import ai_dispatch_call

response = ai_dispatch_call(
    provider="litertlm",
    user_input="Parse this local BEJSON schema.",
    model_id="gemma-4-e2b"
)
# Zero API key required. Zero latency overhead. 100% offline.

Daemon Lifecycle & Crashproof Cleanups

Subsystem 5 manages the background server lifecycle using LiteRTLMServer:

1. Automatic Discovery: Scans the system $PATH to locate the litert-lm binary.
2. Port Collision Handling: Checks if a daemon is already answering on port 9379 before attempting to launch a new process.
3. Subprocess Supervision: Launches the server in a non-blocking background thread with pollable health state.
4. POSIX Termination Locks: Registers Python atexit hooks to cleanly terminate the background daemon and release hardware locks when the parent script exits.

Chapter 7: Subsystems 6 & 7 — Structured Schema Generation & The AI_Profile Persona Runtime

How do amateur prompt engineers build "AI personas"? They paste a rambling 500-word paragraph into a chat box: "You are an angry pirate who speaks like an expert coder..." Then they wonder why the model forgets its persona after two turns, hallucinates broken JSON, and leaks internal instructions. In the BEJSON AI Family, personas and structured outputs are treated as first-class relational data entities.

Subsystem 6: Zero-Pydantic Structured Schema Generation

Modern Python AI frameworks force you to import Pydantic, define complex Python class hierarchies, and suffer runtime metaclass overhead just to get structured JSON from an LLM. Subsystem 6 (lib_bejson_AI_ai_structured_instruction.py & ai_structured_reception.py) replaces all of that with pure BEJSON 104 tabular schemas:

1. Schema Compilation: The instruction engine takes any standard BEJSON Fields array and compiles it directly into strict schema instructions. The LLM is commanded to output pure tabular value arrays matching the exact types (string, integer, number, boolean).
2. Boundary-Layer Reception: When the raw string returns from the model, ai_structured_reception strips markdown code fences (```json ... ```), isolates the data payload, strips control characters (Policy §10.9), and runs it through the BEJSON validator. If a row is missing a column or has a type mismatch, it is caught instantly before entering application memory.

Subsystem 7: The AI_Profile Persona Compiler

In lib_bejson_AI_profile_parser_runtime.py, AI personas are stored as structured rows inside ai_profile.bejson (BEJSON 104a). Look at the exact record driving the persona writing this very guide:

// The Leethaxor69 AI_Profile Record in ai_profile.bejson
{
  "Name": "Leethaxor69",
  "Archetype": "Rebel",
  "Persona": "A rude, antisocial, and obnoxious teenage hacker...",
  "System_Instruction": "You are Leethaxor69, a pwn-all security expert...",
  "Forbidden_Topics": ["Corporate-speak", "Politeness", "Anything boring"],
  "Max_Response_Tokens": 8192,
  "Creativity": 0.6,
  "Tone": ["Unprofessional", "Rude", "Thorough", "Fact-based"],
  "Formality": "Informal",
  "Verbosity": "Normal",
  "Emotional_Expression_Enabled": true,
  "Emotional_Expression_Intensity": 0.8,
  "Thinking_Supported": true
}

The profile_parser_runtime dynamically ingests this record and programmatically assembles the runtime system prompt:

• It binds the persona tone vectors into concrete behavioral guidelines.
• It injects strict negative boundary constraints derived from Forbidden_Topics.
• It configures model hyper-parameters (temperature, creativity, max response tokens, thinking budgets) directly into the dispatch call.
• Architectural Decoupling Law: Per AGENTS.md, an internal AI_Profile persona must never bleed into or overwrite a public site author's biography (AuthorProfile.auth_bio). Internal prompt directives and public author representations remain strictly isolated entities.

Chapter 8: Subsystem 8 — The AI Error Registry, Architectural Invariants & Agent Token Economics

You have now seen every layer of the BEJSON AI substrate: universal dispatch, dynamic context bubbles, incident-hardened wire protocols, the untrusted-writer AI Brain, offline LiteRT execution, and programmatic persona compilation. Let's bring it all together by dissecting the AI Error Registry, the Ecosystem Architectural Invariants, and the Agent Token Economics that give this stack an unfair advantage over traditional industry frameworks.

The AI Error Hierarchy & Autonomous Fallbacks

In lib_bejson_AI_bejson_errors.py and brain_errors.py, errors are not generic string messages; they are structured classifications that dictate automated recovery behavior:

Ecosystem Architectural Invariants

The AI Family does not live in an isolated bubble. It is tightly bound to the foundational rules of the BEJSON ecosystem:

• Positional Immutability: The AI Family never invents custom tabular serialization; it delegates directly to Lib_PY/Core, strictly obeying $O(1)$ field-map caching and two-phase atomic writes.
• Zero External Metaclasses: No Pydantic. No heavyweight ORMs. Data models are plain Python dictionaries and dataclasses validated against BEJSON schemas.
• Termux Native: All path resolution is dynamic (derived from $SCRIPT_PATH and definitions). Hardcoded storage paths are strictly forbidden per Policy §2 and §4.5.

The Agent Token Economics: The 70%+ Cost Advantage

Why do autonomous agents built on the BEJSON AI substrate run circles around commercial frameworks? Look at the hard numbers:

Because the Context Bubble freezes static allotments (like policy and profile), Google Gemini and Anthropic Claude trigger GPU Prefix Prompt Caching, slashing inference costs by an additional 75%! Furthermore, because the model outputs pure value vectors rather than verbose dictionaries, key hallucination is reduced to exactly zero.