Inside one machine, the exchange does not even have to send a bid request to the bidder. It can write the request into memory that both programs see, and the bidder reads it where it lies. Trading systems have worked like this for years.
I experimented with moving the call between exchange and bidder from the network to a Unix socket, a file on the machine that both programs connect to. The request still travels, though, because every message between two programs goes through the kernel, the core of the operating system. Think of two offices in one building where every letter goes through the front desk. The exchange hands the letter to the desk, the desk makes a copy, carries it to the bidder’s office and knocks to say it has arrived. The answer goes back the same way. Between two data centers there is no way around this. On one machine there is.
Shared memory is a table that both offices can reach. The kernel sets it up once, and after that the exchange puts the request on the table and the bidder reads it from there, with no copy and no trip through the desk.
Trading firms built a simple way to use that table. It is cut into a fixed number of slots, for example 1,024. The exchange writes each new request into the next slot, and after the last slot it goes back to the first one and writes over a request the bidder has already read. This is called a ring buffer, and the same memory is reused for every request. After each write the exchange adds one to a counter, so the bidder only has to watch that number to know a new request is waiting. The bidder reads the request straight from the slot and writes its answer into a second ring, where only the bidder writes. Each ring has one writer, so the two programs never wait for each other to finish.
There are two ways to watch it. The bidder can sleep and ask the desk to knock when the counter changes, or it can keep checking the counter from its own core, which is called spinning. On a cloud machine with real bid requests, a request and its answer took about 18 µs over a Unix socket. With shared memory and a spinning bidder they took about 1 µs. With a sleeping bidder it was 18 µs again, because the knock from the desk costs about as much as the whole delivery.
This topic goes deep, so I will write a few posts about it. Speed is only part of the story, because at high traffic shared memory can also noticeably lower the CPU each request costs. The 1 µs has a price, though: a spinning bidder keeps one core busy even when no requests come.
A bidder in a separate data center always pays for the full trip through the desk. A bidder inside the exchange’s cluster gets new options that were never possible over the internet, and shared memory is one of them
