Agent Latency: Where WebSockets Earn Their Complexity

A single model call can absorb connection setup without anyone noticing. An agent that alternates between reasoning and tools forty times cannot. Small per-turn costs — handshake, negotiation, serialization — stop being rounding errors and start being the reason a user is watching a spinner.

That is the real argument for persistent transport, and it is narrower than the enthusiasm around it suggests.

Break the loop into stages you can name

Before optimizing anything, split observed latency into five buckets: model inference, API and transport overhead, tool execution, your own application processing, and interface rendering. Almost every team that has not done this is wrong about where their time goes.

The point of the exercise is that a persistent connection only touches one bucket. It will not help a database query missing an index, a tool returning ten thousand lines of log output, or a frontier model doing work a mid-tier model finishes in a third of the time.

• Instrument time to first useful event, not only time to final answer
• Return compact structured tool results — never raw logs the model has to parse
• Parallelize independent reads; serialize only genuinely dependent actions
• Stream progress that reflects completed work, not a fake animation
• Cancel abandoned tasks, because latency you do not stop becomes spend you cannot recover

When persistent transport is worth it

The clear case is a long-lived agent session making many sequential requests where connection state can be reused across turns. The overhead amortizes and the accumulated savings are visible to the user. Voice and interactive copilots sit squarely here.

The unclear case is a one-shot summarizer or a batch job running overnight. There, plain request-response is simpler to debug, simpler to load balance, and gives up nothing that matters. Persistent connections carry real operational weight: reconnection logic, backpressure, message ordering, and a deploy story that has to handle open connections gracefully. Take that on when the loop shape justifies it.

Correctness first, then the slowest measured stage

Make the agent finish reliably and emit a trace you can read. Only then optimize, and only the stage the trace says is slowest. Latency work done without a trace reliably produces more architecture and the same wall-clock time — you moved the complexity, not the bottleneck.