Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 5 of 8

5. Models as content-addressed things

If code is named by its hash, a model should be named by the hashes of everything it is made from.

What a model is made from

Chapter 1 split a model into a recipe and weights. To run it you also need a tokenizer (chapter 2) and a set of operations to run it on. Each of those can be pinned by content:

Model code: its Clay hash Tensor + Tokenizer ability hashes weights file: sha256 tokenizer files: sha256 source repo + revision, licence one pinned modelsame inputs every time
Figure 5.1. What identifies a model run, per docs/ml-m2-gpt2.md ("Tokenizer: how it is hashed and pinned") and the pin file packages/tensor-host/models/gpt2.json.

Pinning to a Hugging Face revision

The Hugging Face hub stores each model as a git repository, so a revision is already a hash. The pin file for GPT-2 records the repository (openai-community/gpt2), the revision 607a30d783dfa663caf39e06633721c8d4cfcd7e, the licence (MIT), and for each of the five files its size and its sha256: the 548,105,171-byte weights file, config.json, vocab.json, merges.txt and tokenizer.json. A download script fetches them and refuses on a mismatch; the test re-verifies them. Today the check runs at download and in tests. The host does not check a digest when it runs; making a runner refuse unpinned files is part of rung M3 (wishlist W33).

The identity of "GPT-2 as run" is therefore not "a name", but: this code hash, these ability hashes, these file hashes. A job that names them all runs the same inputs wherever and whenever it runs. This is the same "carving in granite" rule as for code, applied to data.

The registry

Clay already has a signed registry for third-party code it wraps (docs/registry.md). The plan is to let it sign a model-plus-code manifest the same way, so that a runner can refuse an unsigned model (docs/ml-workloads.md section 6). open Nothing of the model side is built: the registry's five seed entries are wrapped third-party code, and its trust root and signer policy are themselves undecided.

Reproducible does not mean bit-identical

Pinning fixes the inputs. It does not fix the output bits. Computers add decimal numbers in a particular order, and with finite precision (a 32-bit float keeps about seven significant digits) (a + b) + c and a + (b + c) can differ in the last digit. A matrix multiply adds hundreds of products; a different machine, a different number of threads, or a different library adds them in a different order and gets slightly different last digits. A GPU differs from a CPU in the same way.

So we never promise that a result's bytes are the same everywhere. What we promise is the check in chapter 6: logits within a stated tolerance, and greedy tokens identical. The measured size of the effect on GPT-2: the Clay port's logits differ from the independent oracle by at most 2.7e-4 over all 20 prompts at any position, against logit values of order tens (order of magnitude inferred; not stated in the doc). Bit-level determinism across machines or thread counts was not tested (docs/ml-m2-gpt2.md, "What was NOT verified"). The practical consequence for Clay Cloud: a result's hash is not a promise, and a replay should return the recorded answer rather than recompute and compare bits (docs/ml-workloads.md section 6).

Licences

Models carry their own terms. The three candidates read in M0 are tagged MIT (GPT-2) and Apache-2.0 (the others) on the hub; a model's training data can carry separate terms that M0 did not read (risk R9). A pin file records the licence, and hosting a model for third parties needs a per-model check: open (docs/ml-workloads.md section 6). The Rust tokenizer crate and its 69 added dependencies were checked: all MIT or Apache-2.0, nothing copyleft-only (docs/ml-m2-gpt2.md, "Downloads").

To remember. Pin everything a run depends on by hash; expect numbers to differ in the last digits across machines; accept by tolerance, not by equality; record the licence next to the hash.