Lower latency on the bid path means fewer resources on both sides, the exchange and the bidder. It sounds backwards, since going faster usually costs more. For HTTP/1.1 it follows from a simple rule called Little’s law.
Little’s law says how many requests are open at the same moment: requests per second times the time each one takes. If 10 requests arrive every second and each takes one second, 10 are open at any moment. If each takes a tenth of a second, only one is open. Same traffic, less time, fewer open requests.
HTTP/1.1 carries one request at a time per connection, so every open request holds a connection on both ends. At 100,000 requests per second, an 80 ms round trip keeps about 8,000 connections busy. At 5 ms, with the bidder in the same cluster, it is about 500. Newer protocols put many requests on one connection, but switching has a cost, so HTTP/1.1 is still common on the bid path.
An idle connection is cheap. A busy one also holds buffers with the data and, on the exchange side, the request itself until the answer comes back. Latency also jumps: a resent packet or a short slowdown, and the same traffic suddenly needs more connections. If every connection is busy, the request waits and loses part of its deadline, or a new connection opens and pays a handshake at the worst moment.
I tested it. Two containers, an exchange and a bidder, 5,000 HTTP/1.1 requests per second, with added delay from 0 to 80 ms. Busy connections followed Little’s law: 15 at 0 ms, 405 at 80 ms. Memory grew with them, 2.7x on the exchange and 2.1x on the bidder, while CPU stayed flat.
Then I capped the pool at 480 connections and slowed the bidder by 15 ms for one second in every ten, still inside the 100 ms deadline. Up to 50 ms, the pool absorbed it. At 80 ms, timeouts closed connections faster than new ones could replace them. After the first slowdown, not a single request was answered in time for the rest of the run. A real exchange would likely throttle the bidder first, which also means fewer bids.
A containerized bidder running inside the exchange’s cluster removes most of the distance from the bid path. The exchange needs a smaller pool for the same traffic, and a short slowdown no longer turns into lost requests. The bidder holds fewer open connections and less memory, and does not spend CPU on a wave of new handshakes.
