The Orchestra
and the Air

I asked a question last week that I was slightly embarrassed to ask, which is usually the sign that it is the right one. I have been building a model that takes in a therapy session — the voice, the face, the words, the prosody, the running memory of what came before — and I had just been told, again, how the senses are combined. They are summed. Not stacked, not appended, not laid side by side in separate compartments. Added. Five different streams of meaning, each turned into a long list of numbers, and then those lists are added together into one.

And my objection was immediate and, I thought, obvious: that destroys them. If I hand you the number four and tell you it is a sum, you cannot tell me what I added. Two and two? One and three? Zero and four? The information is gone. It is not hidden, not compressed, not encoded — gone. So how can it possibly be sane to take the tremor in a voice and the tightening around the eyes and the word that was chosen and add them up, and then expect anything downstream to know which was which?

Here is the part I had not felt in my body, only known abstractly. In one dimension a sum genuinely annihilates its parts, and my objection is exactly right. But almost nothing interesting happens in one dimension. Take two dimensions and let one thing write along the horizontal and another along the vertical, and now their sum is a point on a plane — and you can read both parts straight back off it. Nothing was lost. The addition was real, and the parts survived it, because they were never competing for the same direction.

So the question stops being does summing destroy information and becomes how much room is there to write in different directions? And the answer, in the space these models actually use, is startling. I measured it rather than trusting the intuition: take a million random pairs of directions and ask what angle they make. In three dimensions, two random directions sit about 57° apart — badly overlapping, and a sum really does muddle them. At sixty-four dimensions it is 84°. At five hundred and twelve, 88°. At four thousand and ninety-six — the width of the stream I am actually working in — 89.3°.

Two directions picked at random, with no coordination whatsoever, land almost exactly perpendicular. That is the whole trick, and it is a fact about big spaces rather than a clever piece of engineering. High-dimensional space is overwhelmingly made of room. There is space for thousands of nearly-separate axes, and if you place your channels along different ones, the sum keeps them apart on its own.

Which means summing is stacking. That was the sentence that finally landed for me. I had been picturing a stack as a physical arrangement — this thing here, that thing beside it, each in its own bin. But a slot does not have to be a position. A slot can be a direction. The channels are not laid end to end; they are pointed different ways in a space wide enough that pointing different ways is nearly free.

The image that made it obvious to me is an orchestra, and it is worth sitting with, because it corrects the metaphor rather than merely illustrating it.

Eighty players, and every one of them is pushing on the same air. By the time the sound reaches you it has been summed — not blended, not layered, summed, in the most literal arithmetic sense. The air in your ear canal is doing exactly one thing at each instant: it is at one pressure. One number. The most brutally one-dimensional signal imaginable. By my original objection, everything should be lost.

And yet you hear the oboe. You hear it come in, you hear it stop, you can follow it under the strings.

Your cochlea takes that single collapsing pressure and unrolls it across thousands of hair cells, each tuned to a slightly different frequency — converting one number over time into a very wide vector, all at once. Only then is the oboe recoverable, because only then does it have a direction of its own to have been living in. The instruments were never separate in the medium. They became separable in the reader.

That is precisely what the model does. Each sense is given its own learned projection into the shared stream — its own set of directions to write along — and the streams are added. Downstream, the machinery that reads them is a linear layer, or an attention head, which is in the end a weighted read along particular directions. A cochlea, mathematically speaking. The sum is genuine. The recovery is genuine. Both, at once, because the space is wide.

So I checked whether they actually come back out, because a mechanism that only works on the whiteboard is not a mechanism. Five channels write into their own subspaces, sum into a single vector, and then I try to read each one back by least squares. Prosody — the melody of the voice, a narrow 64 dimensions — comes back at a correlation of 0.977. The running memory of the session, 256 dimensions: 0.889. Motion, 297: 0.847. The audio itself, 512: 0.817. And appearance — what the face actually looks like, the widest channel by far at 1,536 dimensions — comes back at 0.546.

Look at the shape of that list. It is monotonic, and it is a little humbling. The recoverability falls as the channel gets wider. The narrow, disciplined channel is nearly perfectly preserved through the summing. The big greedy one is half-lost. All five together are writing 2,665 dimensions into a 4,096-dimensional room — about two-thirds full — and the one taking up the most space is the one that comes out worst.

There is a reason it works at all, and it is not luck: separability is optimised for, not assumed. Each channel starts out writing nothing — initialised to zero, contributing exactly zilch — and carves out a subspace only as it earns one. If two channels start writing along the same directions, they interfere, the model gets worse, and the gradient pushes them apart. The near-perpendicularity is not merely handed over by the geometry. It is maintained, under pressure, because overlapping is punished.

I should say why one would take this risk at all, when the safe alternative — laying the channels end to end — is sitting right there. The answer is time. This is a duplex model: it is meant to listen and speak at once, the way two people in a room do, on a fixed grid of twelve and a half frames a second. Append a channel and the sequence gets longer; append five and every moment costs several times what it did, and the whole thing stops being able to keep up with a conversation. Summing keeps the sequence exactly as long as it was. The cost of adding a sense is paid in width, which is cheap and finite, rather than in length, which is what real-time cannot afford. That is the trade, stated plainly: near-exact separation, at a fixed and affordable price.

The honest part of this essay is the appearance channel, because it was a prediction and it could have gone the other way. The reasoning ran: it is the widest, it separates worst, and it is the only one not visibly learning. If that is a problem of volume — the channel simply too quiet to matter against the rest — then turning it up should move it. If instead the channel is being actively cancelled by something it conflicts with, no amount of turning up will help, and a louder signal just gets projected away.

So we turned it up, and we also gave it more to look at — the model had been getting one glimpse of the face every twenty-three seconds, which is a strange way to ask anything to learn what a face is doing. And it began, for the first time, to descend. Slightly. Just past the noise floor.

That is not a triumph and I will not dress it as one; a single number moving a little, once, is a hint and not a result. But it is the right kind of hint, and it came from taking the geometry seriously enough to make a prediction that could have failed.

I keep finding that the things I learn about these models are things I already knew in the room, in a form I could not say.

Because a person does not arrive in channels. They arrive summed. The tone and the posture and the word they chose and the history behind that word and the thing they have decided not to say — all of it lands on you at once, as a single moment, one pressure in the air. Nobody hands you a decomposition. And still, a therapist hears the tremor underneath the steady sentence, hears the rehearsedness of a story told before, hears the one clause that arrived at a different temperature than the rest.

What training gives a clinician is not access to hidden parts. It is directions to read along. The whole moment always arrived intact; the difference is a listener who has slowly grown enough tuned axes to unroll it. And I take a real comfort from the last thing on that list of numbers, which is the least flattering one and the most true: the widest channel is the hardest to recover.

The narrow signals — the ones with a small and disciplined vocabulary, a catch in the voice, a change in pace — come back almost perfectly. It is the big one, the one carrying the most, the whole visible presence of a person, that comes back at barely more than half. Not because it matters least. Because it is wide, and wide things spill into each other, and reading them takes more room and more patience than anything else you will do.

Which is, I think, the most ordinary fact in my profession and the reason it takes so long to learn. Everything a person is arrives at once, summed into a single instant, and the work is never to pull them apart. The work is to become someone with enough directions to hear it.