Machine learning in Clay Cloud / chapter 3 of 8
3. Where the time goes
Doing the arithmetic is cheap. Bringing the weights to the arithmetic is not.
All numbers in this chapter are from one machine: an 8-core Apple Silicon laptop, native release build, one host thread, CPU only (docs/ml-m2-gpt2.md, "Speed and where the time goes"). They are measured unless labelled otherwise. Nothing here was measured on a GPU, in a browser, or on the 4 GB server.
Two ways to be slow
A processor can be limited by two different things.
- Compute-bound: it cannot multiply and add fast enough. The numbers are already close by; the arithmetic units are the bottleneck.
- Memory-bound: the arithmetic units sit idle waiting for numbers to arrive from main memory. The bottleneck is memory bandwidth, bytes per second.
Which one applies depends on arithmetic intensity: how many multiply-adds you do for each byte you fetch. If you fetch a weight and use it once, intensity is low and memory sets the speed. If you fetch it once and use it many times, the arithmetic becomes the limit.
Why one token is memory-bound
Generating one token multiplies a single row (the one new token's vector) by each weight matrix. Each weight is fetched, used for exactly one multiply-add, and not needed again until the next token. A token therefore reads all of the weights, about 500 MB of them, to do 0.249 GFLOP of work. In numbers:
| Quantity | Value | Label | Source |
|---|---|---|---|
| Work per token | 0.249 GFLOP (124 M multiply-adds) | measured | ml-m2-gpt2.md |
| Weights read per token | about 500 MB (f32) | measured | ml-m2-gpt2.md |
| Intensity at one token | 0.249 GFLOP / 0.5 GB = 0.5 FLOP per byte | computed | from the two rows above |
| Host kernel time per token | 12 to 14 ms (the 73 matmul calls: 11.6 to 13.6 ms) | measured | ml-m2-gpt2.md |
| Memory speed achieved | 500 MB in about 12 ms is about 40 GB/s on one core | measured | ml-m2-gpt2.md |
| Best matrix-multiply speed seen | about 100 GFLOP/s (M1, large multiplies) | measured | ml-workloads.md rung M1 |
| Speed we got at one token | 0.249 GFLOP / 12 to 14 ms = 18 to 21 GFLOP/s | computed | from the rows above |
So at one token the machine runs at about a fifth of its demonstrated arithmetic speed, and exactly as fast as memory lets it. That is memory-bound.
Why GPUs help, and what batching is
inferred / background knowledge, not measured by us. A graphics card is a processor with thousands of simple arithmetic units and memory several times (often an order of magnitude) faster than a laptop's. Both help a memory-bound job: more bytes per second per token, and more arithmetic when the job turns compute-bound. A GPU is much less helpful if you feed it one token at a time and waste its arithmetic.
Batching is the standard remedy: process several sequences (or several prompt tokens) in one go, so that one fetch of a weight serves many multiply-adds. That moves a job to the right on Figure 3.1. The prompt pass already does this for free: all the prompt tokens go through together. A measured hint: the 313-token prompt plus 16 generated tokens took 1.7 s in total; the 1,000-token prompt plus 8 tokens, 4.4 s. Serving many users in one batch is a scheduler question for Clay Cloud (docs/cloud-workloads.md) and has not been built or measured for models.
The time split of one token
If the arithmetic were all, a token would cost only the host-kernel slice. The rest is the cost of driving the host from Clay: each tensor operation is an ability call that the engine suspends at, packages as JSON, hands to the host, and resumes from.
Tensor.shape call costs 7.3 microseconds end to end, so 433 calls is about 3.2 ms computed.What was tried, and what it taught
- First version: 58 ms per token (17 tokens/s). Two causes found by measuring:
gathercopied the whole 150 MB embedding table on each call, and the general matrix-multiply routine is slow for a single row. - A dedicated streaming matrix-vector kernel took it to 16 to 18 ms alone: 54 to 61 tokens/s.
- More threads made it slower (1 thread 61 tokens/s, 2: 46, 4: 38, 8: 31; measured with six busy background processes, so only the ordering is firm) because threads were started per call. They are opt-in until a persistent pool exists.
- Peak memory 1.02 GB: the host holds the 548 MB file and the 500 MB of converted tensors (wishlist W32 would halve it).
To remember. At one token per step the job is limited by how fast 500 MB can be read, not by arithmetic. Faster memory, sharing a read between tokens, or smaller weights (quantisation, chapter 8) help; a faster multiplier alone would not.