Today on The Gateway Signal: Disaggregated inference is crossing from experimental clusters into managed cloud environments, anchored by AWS SageMaker's latest updates. In parallel, Microsoft is pushing identity governance directly into the enterprise gateway layer to rein in multi-agent routing loops.
Following up on the general availability of Azure AI Gateway we tracked yesterday, Microsoft has expanded its Model Router within Foundry Models to support 45 models. The update introduces a single-line 'Auto' deployment option, fine-grained request controls, and adaptive session affinity to keep prompt caches warm during agentic loops.
Why it matters
Native model routing inside an enterprise-grade cloud gateway allows engineering teams to implement multi-model fallbacks and dynamic cost optimization without custom sidecar code. The addition of session affinity specifically targets agent execution loops where cache evictions during multi-turn tool calling cause expensive prefill recomputations.
New architectural benchmarks evaluate the Go-based Bifrost gateway against LiteLLM, Cloudflare, and Bedrock AgentCore specifically for Model Context Protocol (MCP) and agent-to-agent (A2A) traffic. While confirming the 11-microsecond proxy overhead at 5,000 RPS we noted previously, the tests highlight Bifrost's Code Mode feature, which reduced token spend on a 508-tool run from $377 to $29.
Why it matters
As applications transition from single-turn chat to multi-agent architectures, traditional proxy overhead and broad tool schema injection become major latency and cost bottlenecks. Filtering tool definitions directly within the proxy data plane—as shown by the Code Mode savings—significantly slashes token consumption on complex agent runs.
AWS introduced managed integration of NVIDIA Dynamo into Amazon SageMaker HyperPod Inference on Wednesday, November 18, 2026. The setup automates Kubernetes AI scheduling, EFA networking, and node health checks to split prompt prefill and token decode phases across GPU instances, demonstrating a 2.05x output token throughput gain for Qwen3-235B on two p5en.48xlarge instances.
Why it matters
Multi-node inference for ultra-large models has historically required complex custom cluster orchestration to prevent decode phases from stalling behind long prefill prompts. Abstracting NVIDIA Dynamo within HyperPod simplifies disaggregated serving for production engineering teams running open-weight flagships. This move highlights how cloud providers are embedding specialized KV-transfer runtimes into standard managed infrastructure.
Following yesterday's release of OpenAI's GPT-6.2 Astra and Sol, unified gateways including LiteLLM and Evolink immediately updated their catalogs. Capitalizing on the models' tiered pricing—Astra at $8.00 per million input tokens and Sol at $2.00—Evolink published benchmark guidance advising developers to route complex cross-file faults to Astra while pinning routine PR reviews to the cheaper Sol.
Why it matters
As frontier labs reach price parity on top-tier models, gateway routing logic shifts from vendor selection to workload tiering within a single model family. Evolink's analysis on cost-per-accepted-fix demonstrates that using cheaper models like Sol for standard execution loops and reserving Astra for escalation paths yields significant token savings without sacrificing agent resolution rates.
Building on Tuesday's launch of KV cache-aware routing for SageMaker AI that we tracked yesterday, AWS announced the general availability of distributed KV caching via Amazon ElastiCache for Valkey on Wednesday, November 18. While SageMaker's routing mechanism achieves 80–93% cache hit rates on Llama 3.1 70B, new benchmarks note an added routing overhead of 14 to 19 milliseconds.
Why it matters
While we noted the throughput gains of cache-aware routing previously, the newly measured 14-19ms routing penalty requires platform engineers to selectively apply this feature to long-prompt agentic workloads rather than short-latency classification APIs. Combining state-aware ingress routing with an external L2 Redis/Valkey cache tier over EFA allows self-hosted fleets to maintain shared prompt memory across instance boundaries.
Salesforce's Agentforce Foundation team integrated SageMaker AI's new cross-Region inference routing across 30+ AWS regions on Wednesday, November 18, 2026. The deployment replaces custom internal failover gateways with SageMaker's native capacity-aware router to shift excess traffic during regional spikes while logging granular destination ARNs into OpenTelemetry metrics.
Why it matters
Operating high-throughput agent platforms globally creates severe regional GPU capacity bottlenecks during peak hours. By adopting native cloud cross-region routing rather than maintaining custom health-check proxies, enterprise infrastructure teams can prevent cascading regional outages without violating strict data residency perimeters. This indicates a growing preference for managed cloud traffic routing over custom in-house gateway software at hyper-scale.
Google Cloud engineers published details on Thursday, November 19, 2026, regarding Envoy proxy extensions designed for Layer 8 Model Context Protocol (MCP) and Layer 9 agent coordination. The updated stack embeds wire-format body parsing, token-per-minute (TPM) rate limiting, KV-cache prefix affinity via Kubernetes Gateway API, and external processing (ext_proc) security callouts.
Why it matters
Traditional L7 reverse proxies lack awareness of token billing units or multi-turn agent payload structures. Extending standard cloud-native proxies like Envoy to parse wire-format LLM payloads enables enterprise platform teams to apply token-aware rate limiting and cache locality directly at the ingress boundary. This avoids the necessity of deploying separate specialized sidecar proxies for Kubernetes-hosted agent fleets.
Agent infrastructure platform Linq raised a $90 million Series B round led by Andreessen Horowitz on Wednesday, November 18, 2026, with Ben Horowitz joining the board. Having processed over 1 billion messages for enterprise clients like DoorDash, Linq plans to expand its backend stack into Agentic Computer Use and HD Voice Calling protocols.
Why it matters
Substantial capital investment in lower-level agent communication and desktop automation stacks highlights a pivot away from basic text interfaces toward multimodal execution layers. As enterprise workflows integrate computer-use agents, middleware providers that abstract desktop UI interaction and voice transport will form critical components of the platform layer.
China Telecom, Guangdong Tianyima, and Huawei operationalized an Ascend 950DT compute cluster in Shantou co-located at an international submarine cable landing station. The 1 billion yuan ($150M) facility executes a 'token export' architecture where raw input data remains local while generated tokens are transmitted across borders to overseas API clients, scaling generation volume to 100 billion daily tokens.
Why it matters
This facility represents a structural commercial blueprint for Chinese cloud operators to monetize domestic silicon like Huawei's Ascend chips internationally while complying with cross-border data restrictions and mitigating hardware export sanctions. Gateway aggregators servicing global clients can tap into these coastal token hubs for cost-effective inference endpoints, provided they account for potential routing latencies across subsea links.
Advancing the transition to a pure Rust core and the Agent Skills Hub introduced in recent release candidates, LiteLLM's stable v1.110.0 and v1.111.0rc1 updates add multi-tenant organization hierarchies with budget templates. The releases also implement automated alerts for latency or spend spikes.
Why it matters
The new multi-tenant budget hierarchies directly target enterprise platform teams who need to govern team-level spending and standardize tool distributions across agent frameworks without paying commercial SaaS gateway fees.
vLLM integrated FlyDSL MonoKernels on Wednesday, November 18, 2026, combining attention, projection, and expert computations into persistent GPU kernel launches for AMD Instinct MI355X and MI325X accelerators. Performance testing on a TP2 MI355X node demonstrated an increase in concurrency-1 interactivity for DeepSeek-V4.1-Flash from 318.1 to 429.1 tokens per second per user.
Why it matters
Low-batch decode latency on non-NVIDIA hardware is frequently bottlenecked by host-side CPU kernel launch overhead rather than compute availability. Using domain-specific compilers like FlyDSL to fuse dependent operator steps into persistent launches unlocks significant per-user generation speed on AMD ROCm. This optimization enhances the viability of self-hosting open models on alternative GPU hardware.
Microsoft announced Agent Controls on Wednesday, November 18, 2026, a managed Azure service built on the open-source Agent Control Specification (ACS) and Agent Hooks primitives. The platform evaluates agent tool execution and API calls against Rego policies, LLM judges, and deterministic safety rules at runtime, allowing enterprises to enforce transfer limits and block PII leaks during live execution.
Why it matters
Autonomous agents capable of invoking external APIs present security risks that static prompt filters cannot mitigate. Externalizing policy evaluation into an out-of-band proxy layer running Rego rules ensures that business logic and compliance gates are enforced deterministically before actions take effect. This decoupled architecture allows platform teams to update security guardrails without modifying underlying agent application code.
Disaggregated Prefill and Decode serving enters managed cloud runtimes Architectures that separate prompt prefill from token decode are moving from custom DIY cluster setups into managed platforms like AWS SageMaker HyperPod via NVIDIA Dynamo and Cerebras Inference Cloud. By offloading prefill to dedicated GPU nodes or wafer-scale systems, providers are attempting to bypass memory bandwidth bottlenecks for long-context workloads.
Real-time request-path policy enforcement replaces static enterprise registers Enterprise security architecture is moving away from passive GRC compliance toward live proxy inspection. Governance platforms like Microsoft Agent Controls, Bifrost, and Envoy extensions are embedding deterministic policy checks, token budgets, and MCP tool access controls directly onto the request path before model or tool invocation occurs.
Compiled Rust gateways consolidate control planes under memory pressure Open-source control planes like LiteLLM are mandating Rust cores by default to eliminate Python runtime overhead, while specialized proxies like Bifrost showcase microsecond-level request overhead. This shift reflects growing operational friction with Python-based proxies as agentic workloads flood gateways with high-frequency tool calls.
Cross-region and KV-cache aware routing become default load balancers Hyperscalers and gateway operators are integrating native KV-cache indexing and multi-region failover directly into model routers. Platforms like AWS SageMaker AI and Microsoft Model Router are using cache-state events and session affinity to prevent redundant prefill computations and handle regional GPU capacity shortages automatically.
Token export models formalize around domestic chip clusters in Asia Operators in regional hubs like Shantou are packaging domestic silicon like Huawei's Ascend 950DT into dedicated 'token export' compute centers. By processing queries locally at international cable landing stations and returning generated tokens, infrastructure providers are establishing alternative monetization routes amid hardware trade restrictions.
What to Expect
2026-11-24—CloudBackend scheduled to launch free developer beta for its backend operations platform.
2027-01-01—Azure AI Gateway transitions from free preview allowance to paid commercial billing tier ($500/mo base).
How We Built This Briefing
Every story, researched.
Every story verified across multiple sources before publication.
🔍
Scanned
Across multiple search engines and news databases
451
📖
Read in full
Every article opened, read, and evaluated
111
⭐
Published today
Ranked by importance and verified across sources
12
— The Gateway Signal
🎙 Listen as a podcast
Subscribe in your favorite podcast app to get each new briefing delivered automatically as audio.
Apple Podcasts
Library tab → ••• menu → Follow a Show by URL → paste