Machine learning in Clay Cloud / chapter 5 of 8
5. Models as content-addressed things
If code is named by its hash, a model should be named by the hashes of everything it is made from.
What a model is made from
Chapter 1 split a model into a recipe and weights. To run it you also need a tokenizer (chapter 2) and a set of operations to run it on. Each of those can be pinned by content:
Pinning to a Hugging Face revision
The Hugging Face hub stores each model as a git repository, so a revision is already a hash. The pin file for GPT-2 records the repository (openai-community/gpt2), the revision 607a30d783dfa663caf39e06633721c8d4cfcd7e, the licence (MIT), and for each of the five files its size and its sha256: the 548,105,171-byte weights file, config.json, vocab.json, merges.txt and tokenizer.json. A download script fetches them and refuses on a mismatch; the test re-verifies them. Today the check runs at download and in tests. The host does not check a digest when it runs; making a runner refuse unpinned files is part of rung M3 (wishlist W33).
The identity of "GPT-2 as run" is therefore not "a name", but: this code hash, these ability hashes, these file hashes. A job that names them all runs the same inputs wherever and whenever it runs. This is the same "carving in granite" rule as for code, applied to data.
The registry
Clay already has a signed registry for third-party code it wraps (docs/registry.md). The plan is to let it sign a model-plus-code manifest the same way, so that a runner can refuse an unsigned model (docs/ml-workloads.md section 6). open Nothing of the model side is built: the registry's five seed entries are wrapped third-party code, and its trust root and signer policy are themselves undecided.
Reproducible does not mean bit-identical
Pinning fixes the inputs. It does not fix the output bits. Computers add decimal numbers in a particular order, and with finite precision (a 32-bit float keeps about seven significant digits) (a + b) + c and a + (b + c) can differ in the last digit. A matrix multiply adds hundreds of products; a different machine, a different number of threads, or a different library adds them in a different order and gets slightly different last digits. A GPU differs from a CPU in the same way.
So we never promise that a result's bytes are the same everywhere. What we promise is the check in chapter 6: logits within a stated tolerance, and greedy tokens identical. The measured size of the effect on GPT-2: the Clay port's logits differ from the independent oracle by at most 2.7e-4 over all 20 prompts at any position, against logit values of order tens (order of magnitude inferred; not stated in the doc). Bit-level determinism across machines or thread counts was not tested (docs/ml-m2-gpt2.md, "What was NOT verified"). The practical consequence for Clay Cloud: a result's hash is not a promise, and a replay should return the recorded answer rather than recompute and compare bits (docs/ml-workloads.md section 6).
Licences
Models carry their own terms. The three candidates read in M0 are tagged MIT (GPT-2) and Apache-2.0 (the others) on the hub; a model's training data can carry separate terms that M0 did not read (risk R9). A pin file records the licence, and hosting a model for third parties needs a per-model check: open (docs/ml-workloads.md section 6). The Rust tokenizer crate and its 69 added dependencies were checked: all MIT or Apache-2.0, nothing copyleft-only (docs/ml-m2-gpt2.md, "Downloads").
To remember. Pin everything a run depends on by hash; expect numbers to differ in the last digits across machines; accept by tolerance, not by equality; record the licence next to the hash.