Skip to content
CLAY CLOUD DOCSMachine learningConceptsStatusDashboard

Machine learning in Clay Cloud / chapter 4 of 8

4. How Python does it, and how Clay does it

Both are glue over native kernels. The difference is where the seam between them is drawn.

Python's arrangement

Almost no machine-learning arithmetic runs in Python. A PyTorch program is Python code that calls a library (PyTorch, NumPy); each call, such as "matrix multiply these two tensors", jumps into compiled C++ or CUDA code that does the arithmetic and returns a tensor handle. Python owns the control flow: the loop over tokens, the cache, batching, which model to load. The kernels, the part that needs to be fast, are native and heavily tuned over years. That is what we mean by "Python is glue".

The weight of the ecosystem is the real asset: thousands of operators, GPU kernels tuned for every card, hundreds of architectures on the Hugging Face hub. docs/ml-workloads.md section 9 says so plainly: "The ecosystem is the moat".

Clay's arrangement

Clay plays Python's role with one difference: the seam is an ability. Recall from the language docs that an ability is a set of operations a function may perform, which some enclosing handler answers. We declare one ability, Tensor, with 22 operations (matmul, add, softmax, layerNorm, gather, reshape, ...), plus 3 for arena memory (push, pop, keep) (docs/ml-m0-scope.md). A tensor in Clay code is a tiny opaque handle {id: nat}; the actual numbers stay in the host's memory. The model is then an ordinary Clay program that performs Tensor.* and has no idea who answers.

5 Execution handlerseager CPU, GPU, graph-recording, counting, simulated 4 Models and hub(code hash, weights hash) pinned to a revision 3 Loaderssafetensors, config, tokenizer 2 Neural-network librarylayers as records and functions, in Clay 1 Tensor ability22 operations on opaque handles Only layer 1 and the byte-level builtins are native.
Figure 4.1. The layers, after docs/ml-workloads.md section 2 (a design; today layers 1 and 2 exist for GPT-2, layer 3 is partly native, layer 4 is the pinning manifest packages/tensor-host/models/gpt2.json, layer 5 has CPU, counting and simulated handlers). Highlighted: the ability and the handlers, which Figure 4.2 animates.

Handlers are the backends

This is the new thing relative to Python. In PyTorch, "run on GPU instead" means moving tensors to a device and relying on the library to dispatch. In Clay it means wrapping the same program in a different handler. Try it: pick a handler and step through one Tensor.matmul call.

Figure 4.2. One call, three real handlers from packages/tensor-host/clay/tensor.clay (the CPU host is the Rust process packages/tensor-host; countTensor and simulateTensor are Clay). The FLOP figure is computed: 2 x 1 x 768 x 2,304 for GPT-2's first attention projection. The two greyed buttons are designed (docs/ml-workloads.md section 4) and not built.

Here are the two real clauses that differ, copied from tensor.clay. The counting handler forwards the call to whatever answers Tensor outside it and bumps a counter; the simulated one answers without any host.

// countTensor (line 104)
Tensor.matmul(a, b, resume) => {
  let r = Tensor.matmul(a, b);
  resume(r, n + 1)
};

// simulateTensor (line 188)
Tensor.matmul(a, b, resume) => resume(mkTensor(n), n + 1n);

What the model code looks like

This is the real code of one GPT-2 block, from packages/tensor-host/clay/gpt2.clay. It reads like NumPy: calls on tensors, ordinary records, ordinary functions. The <Tensor> after the name is the declared ability requirement.

let affine<Tensor> = (l: gLinear, x: tensor): tensor => Tensor.add(Tensor.matmul(x, l.w), l.b)

let layerNormed<Tensor> = (n: gNorm, x: tensor): tensor => Tensor.layerNorm(x, n.g, n.b, 0.00001)

let block<Tensor> = (b: gBlock, x: tensor, past: option<gKv>, t: int, offset: int): (tensor, gKv) => {
  let (h, kv) = attend(b, layerNormed(b.ln1, x), past, t, offset);
  let x1 = Tensor.add(x, h);
  let m = affine(b.out, Tensor.act(affine(b.fc, layerNormed(b.ln2, x1)), "gelu_new"));
  (Tensor.add(x1, m), kv)
}

Because the program is an ordinary Clay definition it has a content hash like any other, so "the GPT-2 forward pass" can be named, reviewed and pinned exactly. The same file runs unchanged under the counting and simulated handlers (docs/ml-m2-gpt2.md, "The model in Clay").

An honest comparison of overheads

Python + PyTorchClay + tensor host
Who does the arithmeticnative kernels (CPU libraries, CUDA)a native Rust host (matrixmultiply crate and a custom matrix-vector kernel); CPU only today
Cost of one tensor operation callnot measured: no PyTorch exists in this repository (docs/ml-m0-scope.md chose a dependency-free oracle instead)7.3 microseconds end to end measured (suspend, JSON, host, resume)
Share of a GPT-2 token spent outside the kernelsnot measuredabout 27% (engine 3.9 ms + glue 1.0 ms of 16 to 18.5) measured
Interpreter speedCPython is the baselineabout 1.4 to 4 times slower than CPython 3.9 on the twelve benchmarks of docs/engine-performance.md (docs/ml-workloads.md section 8 rounds this to "1.4 to 5x"); Clay interpretation proper is well under 1 ms per token
Switching backendmove tensors to a devicewrap in another handler; the model is unchanged
Identity of the programa file in a repositorya content hash; the effects it may perform are in its type
Ecosystem, GPU, trainingmaturenone of GPU or training exist; operators we have: 22

One measured comparison we do have, with a caveat. In M0 a pure-Python dot product took 22.7 ns per multiply-add measured; the host's matrix-vector kernel does a token's 124 M multiply-adds in about 13 ms, roughly 0.1 ns each computed, over 200 times faster. That compares native code with pure Python, which nobody uses for this; it explains why the independent oracle needs 18 minutes for a 313-token prompt that the host does in 1.7 s, and it says nothing about PyTorch.

To remember. The seam in Clay is a language feature, not a library convention. The price today is about a quarter of a token's time at batch one, on small kernels (engine boundary, chapter 3); the gains are interchangeable backends, a hashed, effect-typed model, and a single description that runs in a test, a counter or the real thing.