Learn · 05

The architecture
of the system.

MLabChain is two programs: a Python scientific client and a C++ consensus core. They are separated by a subprocess boundary that is small on purpose. This page explains what belongs where, why, and what the boundary guarantees.

System shape

MLabChain deliberately splits into two programs, each with a narrow responsibility and no overlap. The split exists because the two halves answer to different requirements:

  • The scientific layer must be easy to change. New datasets, new verifiers, new training loops, new metrics are all normal evolution of a research tool. It should be written in the language most scientists already use: Python.
  • The consensus layer must be stable. Every node has to agree on the exact same state transition for the same block. It should be written in a language with explicit control over integer arithmetic, memory layout, and error handling: C++.

Nothing in the C++ core knows what a "model" is. Nothing in the Python layer knows what a Merkle tree is. That is the whole design.

The Python layer

mlabchain.py is a single-file command-line tool. It owns everything that touches scientific artifacts.

Training
Deterministic linear regression with a fixed SGD order. Seeded dataset generation, deterministic train/test split. run_training() · train_linear_regression()
Challenges
Manifest creation, canonical JSON serialisation, manifest hashing, dataset commitment. The manifest format is versioned and refused if the recorded hash does not match. Challenge dataclass · compute_hash()
Artifacts
Writes model.pkl, model.txt, and proof.json to disk. Streams SHA-256 over any file in 1 MiB chunks. save_training_artifacts() · hash_file()
Wallet
Ed25519 keypair generation, Scrypt + AES-GCM encryption, checksum address derivation, key rotation and multisig signing helpers. Wallet class · wallet_password()
Transactions
Builds the exact canonical signing messages that the C++ core expects, signs them, and calls the core through a subprocess. ml_message() · transfer_message()
Verification
The verify-ml command re-executes the recorded training, recomputes metrics, and compares them against the transaction. This is the audit path, not a consensus rule. command_verify_ml()

The C++ core

mera_core is a single executable. It is the authoritative implementation of every state transition on the chain.

Authentication
Ed25519 signature verification via OpenSSL. Address derivation from public keys with checksum. Multisig threshold checking. verify_ed25519() · address_from_pubkey()
State replay
Loads the canonical chain from SQLite, replays every transaction from genesis, and derives the current state from scratch. The database is a materialised view; the chain is the source of truth. Replay · current_replay()
Mempool
Per-sender nonce tracking with pending awareness, balance reservation, fee-priority selection, and byte/count limits. submit() · select_for_block()
Reward arithmetic
Integer-only fixed-point computation of quality_ppm, delta_q, and the final reward. Uses boost::multiprecision::uint128_t for intermediate products. quality_ppm() · work_reward_for_quality()
Fork choice
Cumulative chain work comparison, canonical chain replay, reorg with pending-transaction re-insertion. accept_chain() · mine()
P2P
Boost.Asio TCP server, line-based message protocol, block and transaction relay, GETCHAIN for full chain sync. P2P class
Governance
Timelocked parameter changes from a genesis-defined multisig. Only fee parameters are modifiable. GOV_PARAM handler

The boundary

The two programs communicate through a subprocess. The Python layer invokes mera_core with command-line arguments, and parses a simple KEY=VALUE response.

# Python calls the core
result = subprocess.run(
    [core_path, "submit-ml",
     "--sender", wallet.address,
     "--challenge-id", challenge_id,
     "--manifest-sha256", manifest_hash,
     "--model-sha256", model_hash,
     # ... 20+ more arguments ...
     "--signature", signature],
    text=True, capture_output=True, timeout=60.0,
)

# C++ replies in a simple key=value format
TXID=6a8b3c...
STATUS=accepted

Why a subprocess

  • No FFI layer. A shared library would need a C ABI, symbol versioning, and careful memory management across the boundary. A subprocess needs none of that.
  • Crash isolation. If the C++ core segfaults, the Python session survives with a clear error. If they were linked in-process, a bug in either side would take down both.
  • Auditable. Every state transition is expressible as a shell command. Any reader can inspect exactly what the core was asked to do.

Costs of the subprocess

  • Process startup. Each invocation spawns a new process and opens the SQLite database. For a single command this is a few milliseconds. For a tight loop it would be too slow — which is why the tool is not designed for tight loops.
  • Argument limits. Very large transactions must be passed through temporary files or environment variables if they exceed the OS argument size limit. In practice the current transaction sizes are small enough that this has not been an issue.
  • Error propagation. The C++ core returns non-zero on failure and writes the error message to stderr. The Python side raises RuntimeError with the same message.

Storage model

State lives in a single SQLite database at mera_data/mera.sqlite. SQLite is configured with WAL mode, foreign keys enabled, and synchronous FULL.

Tables

TableContents
metaNetwork name, genesis hash, schema version, supply, parameters, governance config
accountsAddress, balance, nonce, current authorization public key
blocksBlock headers with producer signature and cumulative chain work
txsConfirmed transactions, indexed by height and order
pendingMempool, ordered by fee priority
challengesRegistered challenge state including best quality and budget remaining
work_claimsReserved (challenge, dataset, model) tuples
gov_updatesScheduled parameter changes with effective heights

Indexes

Five indexes are maintained for the hot paths:

  • pending(sender, nonce) — for pending-aware nonce checks
  • pending(fee DESC, seq ASC) — for fee-priority selection
  • txs(height, ord) — for block replay
  • txs(sender, nonce) — for account history queries
  • blocks(height) — for chain traversal

Atomicity

Block commitment is a single SQLite transaction. If any part fails — replay, insertion, state update — the whole transaction rolls back and the block is not committed. This is what makes the database always consistent with the canonical chain.

Migration from v0.2

v0.3 does not silently interpret a v0.2 database. When the core opens a database whose schema_version does not match PROTOCOL_VERSION, it refuses with a clear error.

The migration path is a separate tool:

python tools/migrate_v2.py \
  --input ./old/mera_data/mera.sqlite \
  --output ./migration/v2_snapshot.json

The output is an explicit snapshot containing:

  • Source schema version and network name.
  • Old genesis hash and tip hash.
  • Block and transaction counts.
  • Per-account balance and nonce.
  • Total balance.
  • A snapshot SHA-256 over the canonicalised JSON.
Why an explicit snapshot

Silently replaying v0.2 history under changed consensus rules would be less safe than an explicit migration boundary. The snapshot documents exactly what was in the old chain, so a production migration can commit to it in a new genesis and import balances under human review. The old chain remains valid as an archival record.

Build system

The C++ core uses CMake. The Python side uses standard pyproject.toml. On Linux and macOS the build is driven by build.sh; on Windows by build.ps1.

Dependencies

  • OpenSSL — Ed25519 and SHA-256
  • SQLite3 — state storage
  • Boost — Asio for TCP, cpp_int for 128-bit arithmetic
  • cryptography (Python) — Ed25519, Scrypt, AES-GCM

The build produces a single binary: build/mera_core on Unix, build/Release/mera_core.exe on Windows.

CMake presets

A CMakePresets.json provides a release preset:

cmake --preset release
cmake --build --preset release
ctest --preset release

Container build

A Dockerfile is provided for reproducible builds and CI. The container image includes the compiler toolchain, all libraries, and the Python runtime.

Trust boundaries

Two programs means two sets of assumptions. The architecture makes both explicit.

ComponentTrustsIs trusted by
Python client The C++ core to enforce consensus Nobody — it produces transactions that the core validates
C++ core OpenSSL, SQLite, Boost The Python client, and every other node on the network
Network peers Nothing — all messages are untrusted input Nothing — peers are anonymous
Verifier class The node binary itself The whole monetary system
Challenge author Nothing beyond what the registry enforces Nothing — a challenge is just a record

The most important row is the verifier class. The entire monetary mechanism depends on the correctness of MLabChain-linear-v4, which is compiled into the node binary. A bug there would allow invalid rewards. This is why the core is small, and why the reward arithmetic uses integer arithmetic only.

What is not in the trust boundary

The Python client's local reward estimate is not authoritative. It is a preview. The core recomputes and either accepts or rejects. If they disagree, the core wins — the Python side prints the core's error message and exits.

Take-away

Two programs, one boundary, no overlap. Python owns science. C++ owns state. The subprocess boundary keeps them honest with each other and keeps a bug in one from silently corrupting the other.