Machine learning in Clay Cloud / chapter 7 of 8
7. Decisions, trade-offs and open questions
What is settled, what is only recommended, and what is still yours to decide.
Status words: decided the project owner agreed or it was built and kept; recommended proposed in a design note, no decision recorded; open not decided. Each row names the document that holds the reasoning. This page is a summary and the documents win if they disagree.
| Question | Status | Choice and alternatives | Source |
|---|---|---|---|
| Where does the Tensor host run? | decided 2026-11-11 | A separate native process started by the runner, behind the existing job contract; the whole job goes to it, so there is no per-call hop. Alternative: a native addon inside the Node runner. A separate process isolates crashes and is where a GPU would later live. | ml-workloads.md sections 12 and 14 |
| What is the sandbox unit per job? | decided in principle; open in product | The launcher wraps the runner and the host together as one unit per job. Isolation is a launch requirement because the code comes from strangers. Candidates: a Firecracker microVM (Linux with KVM), gVisor, Kata, a wasm runtime; "chosen later". Not yet built: only a local runner exists. | topology-ambiguity.md "Isolation is a launch requirement"; runner-contracts.md |
| GPU hosting | open | Which provider and budget is undecided; the current server has no GPU. A multi-tenant GPU would need isolation (MIG or MPS, defined in the glossary) and a rule against running arbitrary kernels (signed registry kernels only). Backends considered: wgpu first (reaches WebGPU in a browser), CUDA second (rung M5). No GPU result exists. | ml-workloads.md sections 9, 10, 12 |
| Tokenizer: native or written in Clay? | decided for now | Native: the Rust tokenizers crate behind a Tokenizer ability (same library transformers wraps, so its ids are the reference ids). Alternative: a Clay tokenizer, blocked on a persistent map: today a dict is a sorted list and a 50,000-entry lookup would cost about 5 to 40 ms inferred. Also unverified for wasm: the regex backend would need to change. | language-wishlist.md W16; ml-m2-gpt2.md |
| How are shapes written? | decided and built | list<int> in the ability's operation signatures, e.g. [12, 64]; the host also accepts the text form "12,64". Alternative: text only (the first workaround). Needed a language change, built 2026-11-10. A shape in the type is a long-term idea, not planned. | language-wishlist.md W13, W19 |
| How does the operation set evolve? | decided 2026-11-11 | An operation is added only at the request of an accepted port; each addition is a new ability version (a new ability hash), the host registers both, and the port's record names the host release. Alternative: edit the ability in place (rejected: it would change every model's hash). | ml-workloads.md section 14; ml-m0-scope.md decision 6 |
| Are weights shared between jobs? | decided as a goal; open in the host | Shared read-only from a content-addressed cache (memory-mapped), safe because every blob is verified against its hash. Today the host reads the whole 548 MB file and copies each tensor to f32, peaking at about 1.1 GB (W32). | ml-workloads.md section 14; language-wishlist.md W32 |
| Memory on the 4 GB server | open | The planned Hetzner server is 2 vCPU / 4 GB. GPT-2 peaks at 1.02 GB when run alone measured (on the laptop). Whether it fits beside the other services, and how many concurrent jobs it could hold if each copies the weights (about 1 GB each, so about 3 at most before anything else runs inferred), has not been tested. Sharing weights (previous row) is the lever. | server-hosting.md; ml-m2-gpt2.md |
| Scope of the first public claim | recommended | Inference only; leave training out until there is demand. Training and automatic differentiation are a much larger project and are not scoped. | ml-workloads.md sections 9, 12 |
| What accepts a port? | decided | An independent pure-Python oracle anchored by published values. Deferred: a PyTorch + transformers environment (large download; needs a newer Python than the machine has, or old pinned wheels) until a disagreement or M4 needs a second opinion. | ml-m0-scope.md section 4 |
| Are models signed in the registry? | open | Planned: a model-plus-code manifest signed like wrapped third-party code, refused if unsigned. Not built; the registry's trust root and signer policy are themselves undecided. | registry.md; ml-workloads.md section 6 |
| Threads for the matrix-vector kernel | open | One thread today; threads are opt-in because spawning per call made it slower. A persistent pool is the next step. | ml-m2-gpt2.md "Speed" |
| Browser or phone | open | Not verified: the host is native only. 548 MB of f32 weights do not fit a browser tab comfortably; the browser story would be a MiniLM-sized model or quantisation. (wgpu reaches WebGPU, but that is unbuilt.) | ml-m0-scope.md risk R7 |
| tinygrad as a port source | open | Candidate, not evaluated; see chapter 6. | ml-workloads.md section 14 |
The fuller list of Clay language wishes that ML work exposed (W26 to W34: string concatenation only on text, field reads on pattern-bound variables, no namespaces across files, building an id tensor, and others) is in docs/language-wishlist.md; the choices made without asking during M2 are in docs/judgement-calls.md J142.