When an agent listens, sees and acts, latency becomes capability

Multimodality does not simply add an input type: it redesigns the whole perception, decision and action loop.

A real-time system’s capability is the joint result of understanding, response timing and action safety.

01 / VOICEChoose the voice architecture first

A cascaded stack separates recognition, language model and synthesis: it is controllable but latency accumulates. An end-to-end multimodal model can sound more natural. A full-duplex system can listen while speaking and handle interruption, but brings the highest engineering complexity. Choose the route by the user’s timing needs, interruption pattern, audit needs and repair path, not by a demo alone.

02 / FAST & SLOWCoordinate fast response with slow reasoning

A real-time system may use a quick path to acknowledge, clarify or maintain the conversation while a slower path builds the complete answer. The interface need not carry text alone: tone, intent and uncertainty may matter. A text agent can wait seconds; a real-time agent must decide while the world continues to change, so understanding quality and response timing must be designed together.

03 / COMPUTER USEUse the computer only with observation and verification

Computer use needs an action space, visual location and failure recovery. Animation, pop-ups and device layouts change the environment, so observe again after every action. Prefer a stable API and operate a GUI only where no interface exists. Verify click, entry and submission separately. Keep preview and confirmation for high-risk actions, and record screenshots plus action trajectories for review.

04 / EMBODIED AGENTLayer planning and control for the embodied world

A high-level model handles goals and long-horizon plans; a lower-level controller delivers stable, real-time action. Simulation also faces a Sim2Real gap, and safety boundaries in a real environment exceed those on a screen. First exercise: choose a phone or desktop task, record observation, location, action and verification at every step, then measure total latency and recovery count.

Sources

bojieli/ai-agent-book.
Chapter 9 — Multimodality and real-time interaction.
This is an applied reorganization, not a replacement for the source book; use the source text and code for exact claims.

Next: when multi-agent collaboration actually helps.

Continue to collaboration

When an agent listens, sees and acts, latency becomes capability.