The reproducibility
problem.

01The reproducibility problem

Science is supposed to be reproducible. A result that cannot be independently obtained is not a result — it is an anecdote, or a bug. And yet a large fraction of published computational work cannot be reproduced by anyone, including the authors, six months after it was done.

This is not a moral failing. It is a structural problem. The computational record of an analysis is spread across dozens of files in dozens of places: the source code in one repository, the data in another, the configuration in a YAML file someone edited locally, the model weights in a checkpoint whose training script no longer matches, the exact library versions in an environment that has since been upgraded, the exact random seed in a shell history that was never committed.

When a paper is written, the authors know what they did. Two years later, they *think* they know. Five years later, they know they used to know. The record has drifted.

This is a specific and tractable problem. It is not the same as fraud, and it is not the same as sloppy work. It is the simple, relentless entropy of computational artifacts over time, and it affects careful people as much as careless ones.

02What "reproducible" actually means

The word is used loosely in practice, and this causes as much confusion as the underlying problem. There are at least four different things that a claim of reproducibility can mean.

1. Replicable. A different team, with their own data and their own methods, arrives at a consistent conclusion. This is the strongest claim in experimental science and it is what the word usually means to a physicist or a biologist.
2. Reproducible by others. Another team takes the authors' code, data, and configuration and produces the same result on their own hardware. This is the standard claim in computational work.
3. Reproducible by the authors. The original team re-runs their own pipeline, using their own artifacts, and gets the same result. Surprisingly often this is the claim that fails first.
4. Recorded. The authors can prove *what they did* — the exact inputs, the exact configuration, the exact artifacts — even if nobody has yet re-run it. This is the weakest claim, and it is the only one that a chain can make directly.

MLabChain addresses (4) and helps with (3). It does not, by itself, produce (1) or (2). A chain does not reproduce an experiment; it records the claim that one was performed. Whether the claim is true is a separate question, and — for the narrow class of work MLabChain pays for — a separately checkable one.

03The determinism ladder

Whether an analysis is reproducible at all depends on how much of the pipeline is deterministic. Most computational work sits somewhere on a ladder from "fully deterministic" to "irreproducible by construction", and the position on that ladder determines what kind of verification is even possible.

Almost all modern machine learning lives at Tier 2 or below. Random initialisation, dropout, data shuffling, GPU non-determinism, and floating-point accumulation order that depends on the number of threads — none of these are compatible with byte-exact replay. The metrics agree in distribution, but the model weights do not.

MLabChain's native verifier is deliberately placed at Tier 4. It is linear regression with a fixed seed, a fixed order, and no parallelism. Every node can replay it and get the same weights. That is what makes consensus-paid verification possible for this class of work and not for others.

04The verifiability boundary

Once you accept that reproducibility is a spectrum, the design space for a scientific ledger becomes clearer. There are two very different things a chain can do:

MLabChain does both, for a narrow class of work. Transactions commit the record for anything. Rewards are paid only for work that can be verified. The chain does not pretend that Tier 2 work is Tier 4, and it does not pay for what it cannot check.

05The physics lineage

MLabChain was not written by a machine-learning researcher. It grew out of a project called HEPLabChain, which was a provenance ledger for high-energy-physics analyses, and the design choices reflect that origin.

HEP analyses have a layered structure that most other fields do not. There is a central bookkeeping system that records which production produced which file. There is a software release system that pins exact application versions. There is a conditions database that pins the calibration and alignment of the detector at the time of any given data-taking run. All of this exists because HEP is a collaboration of thousands of people working on shared data over decades, and reconstruction of any single result would be impossible without it.

What HEP does not track — what no central system tracks — is the user's own last mile. The options file that a specific analyst wrote, the exact selection cuts they applied, the tuple they derived from an official sample, the fit configuration they used, the results they published. This is the part that disappears first, and it is the part that MLabChain was originally designed to record.

MLabChain generalises that idea. The chain still records the last mile. The last mile is just machine learning instead of an LHCb fit.

06Why this specific class of ML work

Given the full space of machine-learning tasks, only a narrow slice can be consensus-verified. The slice is defined by three requirements:

1. Deterministic. The same inputs must produce the same outputs on any node.
2. Bounded. The verification cost must be knowable in advance, so that a reward can be priced.
3. Arithmetic-simple. The operation count must be definable symbolically, without reference to hardware or timing.

Linear regression with a fixed seed and a fixed update order satisfies all three. Its op count is epochs × n_train × (3 × n_features + 4) — arithmetic on integers, computable by any node. Its training is deterministic. Its verification is bounded.

Almost nothing else does. A transformer with dropout fails (1). A stochastic gradient descent with random shuffling fails (1). A model trained on a GPU whose kernel order depends on the driver fails (1). A model whose operation count depends on dynamic sparsity patterns fails (2) and (3). Every one of these is scientifically valuable, and none of them can be paid for by consensus.

07What "verified" means here

When MLabChain says a submission has been verified, it means the following things and only the following things:

• The transaction is signed by the account that submitted it.
• The challenge it references exists and is active.
• The operation count matches the formula for the declared dimensions.
• The reward matches the formula for the marginal improvement over the challenge's current best.
• The challenge's budget has not been exceeded.
• The global supply cap has not been exceeded.
• The claim (challenge, dataset, model) has not already been paid.

It does not mean the model was actually trained. It does not mean the metrics are true. It does not mean the dataset was read. A malicious submitter who knows the challenge parameters can compute the fixed-point metrics by hand and craft a transaction that passes every check in the list above.

What prevents this is the separate verify-ml command, which re-trains the model from the recorded config and compares the metrics. That command runs in Python, is not part of consensus, and is not run automatically by every node. It is an audit tool, not a consensus rule.

08The LSWU audit score

Every transaction carries a secondary score called the LabChain Scientific Work Unit, or LSWU. It is computed from the operation count, the wall time, and the quality of the model relative to the challenge baseline:

LSWU = D(N)^0.15 · T(t)^0.30 · Q(I)^1.00

D(N) = ln(1 + N / N₀)      N₀ = 100,000      # dataset size
T(t) = ln(1 + t / t₀)      t₀ = 60 s         # wall time
Q(I) = I^η / (1 + I^η)      η  = 2           # quality
I    = NMSE_baseline / NMSE_model

LSWU is not used for minting. The consensus reward uses only the symbolic operation count and the marginal quality delta. LSWU exists as a broader scientific score, and its role in v0.3 is documentary. It is stored so that an auditor looking at a chain of transactions can see not only what was paid but how the work compares on a broader measure of scientific effort.

The three terms are logarithmic on purpose. If they were linear, a miner could inflate the score by reporting a slower clock or a larger dataset without doing more work. Log scale gives diminishing returns, and the quality term is bounded between zero and one, so the score cannot explode from an unusually small error.

The exponents do not sum to one. Summing to one produces a geometric mean, which compresses the score's dynamic range to about five. The version above spans roughly twelve and is dominated by quality, which matches the intent.

09What MLabChain does not solve

This is the section that matters most, because it is the one a reader is most likely to skip, and the one the project's credibility depends on.

It does not verify arbitrary ML training

A model hash is a commitment to a file. It is not a proof that any particular computation produced that file. The only way to verify a training run in general is to re-run it, and re-running is expensive and often non-deterministic. This is a research problem, and MLabChain does not claim to have solved it.

It does not make verification cheap

Verification re-executes the training. That is the design. There is no succinct proof of the computation, no sampling shortcut, no probabilistic check that could replace re-execution at scale. A chain whose verification is cheap would not be paying for useful work; it would be paying for something else.

It does not decide what science is worth doing

Challenges are registered by whoever chooses to register them. The chain pays against the pinned baseline the challenge defines. It does not evaluate whether the challenge is scientifically interesting, whether the baseline is a meaningful reference, or whether the resulting model is useful in the world. Those judgements are made by humans, and the chain records them.

It does not solve Sybil resistance

Account creation is free. A challenge bond exists and a per-proposer active-challenge limit exists, but an attacker with patience can register many challenges across many accounts. What stops the attacker is the marginal reward structure: a challenge whose best quality has already been paid out yields nothing further until somebody beats that quality, and the operation cap bounds the total work any single submission can claim. These are mitigations, not solutions.

It does not replace CERN Analysis Preservation, DIRAC, or CVMFS

MLabChain is a personal, local ledger. It records what one analyst did on one machine. It is designed to complement, not replace, the institutional systems that HEP already has. The Bookkeeping knows what files exist; CVMFS knows what software was released; DIRAC knows what jobs ran; MLabChain knows what a specific person did with all of that when they sat down to produce a result.

10Further reading

The reproducibility literature is large, but a few papers frame the problem better than the rest.

• Estimating the reproducibility of psychological science — Open Science Collaboration, 2015. The paper that made the problem public outside computer science.
• 1,500 scientists lift the lid on reproducibility — Baker, 2016, Nature. A survey of researchers across fields.
• Reproducibility of computational workflows is automated using continuous analysis — Beaulieu-Jones & Greene, 2017. A computational approach to the same problem.
• The Turing Way — a community handbook on reproducible data science. Freely available online.
• SoK: Research Perspectives and Challenges for Bitcoin and Cryptocurrencies — Bonneau et al., 2015. The best single survey of what blockchains can and cannot do.
• Machine Learning Reproducibility Checklist — Joelle Pineau, McGill. A practical checklist for ML papers.

Next: the protocol in detail →