🛰️ The Gateway Signal

Thursday, November 19, 2026

12 stories · Standard format

Generated with AI from public sources. Verify before relying on for decisions.

🎧 Listen to this briefing or subscribe as a podcast →

Today on The Gateway Signal: Disaggregated inference is crossing from experimental clusters into managed cloud environments, anchored by AWS SageMaker's latest updates. In parallel, Microsoft is pushing identity governance directly into the enterprise gateway layer to rein in multi-agent routing loops.

AI Gateways

Microsoft Expands Model Router and Hits General Availability for Azure AI Gateway

Following up on the general availability of Azure AI Gateway we tracked yesterday, Microsoft has expanded its Model Router within Foundry Models to support 45 models. The update introduces a single-line 'Auto' deployment option, fine-grained request controls, and adaptive session affinity to keep prompt caches warm during agentic loops.

Native model routing inside an enterprise-grade cloud gateway allows engineering teams to implement multi-model fallbacks and dynamic cost optimization without custom sidecar code. The addition of session affinity specifically targets agent execution loops where cache evictions during multi-turn tool calling cause expensive prefill recomputations.

Verified across 3 sources: Microsoft Tech Community · Ayesa365 · Darthicksomu.com

Bifrost Benchmarks Showcase 11 µs Latency Overhead for Multi-Agent MCP Routing

New architectural benchmarks evaluate the Go-based Bifrost gateway against LiteLLM, Cloudflare, and Bedrock AgentCore specifically for Model Context Protocol (MCP) and agent-to-agent (A2A) traffic. While confirming the 11-microsecond proxy overhead at 5,000 RPS we noted previously, the tests highlight Bifrost's Code Mode feature, which reduced token spend on a 508-tool run from $377 to $29.

As applications transition from single-turn chat to multi-agent architectures, traditional proxy overhead and broad tool schema injection become major latency and cost bottlenecks. Filtering tool definitions directly within the proxy data plane—as shown by the Code Mode savings—significantly slashes token consumption on complex agent runs.

Verified across 3 sources: GenAI Brief · GenAI Brief · The Frontier Wire

LLM Inference Platforms

AWS Integrates NVIDIA Dynamo into SageMaker HyperPod for Disaggregated Serving

AWS introduced managed integration of NVIDIA Dynamo into Amazon SageMaker HyperPod Inference on Wednesday, November 18, 2026. The setup automates Kubernetes AI scheduling, EFA networking, and node health checks to split prompt prefill and token decode phases across GPU instances, demonstrating a 2.05x output token throughput gain for Qwen3-235B on two p5en.48xlarge instances.

Multi-node inference for ultra-large models has historically required complex custom cluster orchestration to prevent decode phases from stalling behind long prefill prompts. Abstracting NVIDIA Dynamo within HyperPod simplifies disaggregated serving for production engineering teams running open-weight flagships. This move highlights how cloud providers are embedding specialized KV-transfer runtimes into standard managed infrastructure.

Verified across 2 sources: Inside AI News · Moka Air

Model Releases

OpenAI Launches GPT-6.2 Astra and Sol Pricing Tiers as Gateways Onboard Models

Following yesterday's release of OpenAI's GPT-6.2 Astra and Sol, unified gateways including LiteLLM and Evolink immediately updated their catalogs. Capitalizing on the models' tiered pricing—Astra at $8.00 per million input tokens and Sol at $2.00—Evolink published benchmark guidance advising developers to route complex cross-file faults to Astra while pinning routine PR reviews to the cheaper Sol.

As frontier labs reach price parity on top-tier models, gateway routing logic shifts from vendor selection to workload tiering within a single model family. Evolink's analysis on cost-per-accepted-fix demonstrates that using cheaper models like Sol for standard execution loops and reserving Astra for escalation paths yields significant token savings without sacrificing agent resolution rates.

Verified across 6 sources: Evolink.ai · Shattered · Gagadget · TechRepublic · LiteLLM · Plymouth AI Lab

AI Infrastructure

AWS Adds KV Cache-Aware Routing and ElastiCache Valkey L2 Tier to SageMaker

Building on Tuesday's launch of KV cache-aware routing for SageMaker AI that we tracked yesterday, AWS announced the general availability of distributed KV caching via Amazon ElastiCache for Valkey on Wednesday, November 18. While SageMaker's routing mechanism achieves 80–93% cache hit rates on Llama 3.1 70B, new benchmarks note an added routing overhead of 14 to 19 milliseconds.

While we noted the throughput gains of cache-aware routing previously, the newly measured 14-19ms routing penalty requires platform engineers to selectively apply this feature to long-prompt agentic workloads rather than short-latency classification APIs. Combining state-aware ingress routing with an external L2 Redis/Valkey cache tier over EFA allows self-hosted fleets to maintain shared prompt memory across instance boundaries.

Verified across 2 sources: AI In Context · AWS Database Blog

Salesforce Deploys Amazon SageMaker AI Cross-Region Routing for Agentforce

Salesforce's Agentforce Foundation team integrated SageMaker AI's new cross-Region inference routing across 30+ AWS regions on Wednesday, November 18, 2026. The deployment replaces custom internal failover gateways with SageMaker's native capacity-aware router to shift excess traffic during regional spikes while logging granular destination ARNs into OpenTelemetry metrics.

Operating high-throughput agent platforms globally creates severe regional GPU capacity bottlenecks during peak hours. By adopting native cloud cross-region routing rather than maintaining custom health-check proxies, enterprise infrastructure teams can prevent cascading regional outages without violating strict data residency perimeters. This indicates a growing preference for managed cloud traffic routing over custom in-house gateway software at hyper-scale.

Verified across 2 sources: AWS Machine Learning Blog · AWS

Google Cloud Extends Envoy Proxy Layer for Layer 8 MCP and Tokenomics Governance

Google Cloud engineers published details on Thursday, November 19, 2026, regarding Envoy proxy extensions designed for Layer 8 Model Context Protocol (MCP) and Layer 9 agent coordination. The updated stack embeds wire-format body parsing, token-per-minute (TPM) rate limiting, KV-cache prefix affinity via Kubernetes Gateway API, and external processing (ext_proc) security callouts.

Traditional L7 reverse proxies lack awareness of token billing units or multi-turn agent payload structures. Extending standard cloud-native proxies like Envoy to parse wire-format LLM payloads enables enterprise platform teams to apply token-aware rate limiting and cache locality directly at the ingress boundary. This avoids the necessity of deploying separate specialized sidecar proxies for Kubernetes-hosted agent fleets.

Verified across 1 sources: Google Cloud Blog

AI Startup Funding

Linq Secures $90M Series B Led by a16z to Build Agent Computer Use Infrastructure

Agent infrastructure platform Linq raised a $90 million Series B round led by Andreessen Horowitz on Wednesday, November 18, 2026, with Ben Horowitz joining the board. Having processed over 1 billion messages for enterprise clients like DoorDash, Linq plans to expand its backend stack into Agentic Computer Use and HD Voice Calling protocols.

Substantial capital investment in lower-level agent communication and desktop automation stacks highlights a pivot away from basic text interfaces toward multimodal execution layers. As enterprise workflows integrate computer-use agents, middleware providers that abstract desktop UI interaction and voice transport will form critical components of the platform layer.

Verified across 1 sources: Citybiz

China AI Scene

China Formalizes 'Token Export' Model with Ascend 950DT Subsea Compute Cluster

China Telecom, Guangdong Tianyima, and Huawei operationalized an Ascend 950DT compute cluster in Shantou co-located at an international submarine cable landing station. The 1 billion yuan ($150M) facility executes a 'token export' architecture where raw input data remains local while generated tokens are transmitted across borders to overseas API clients, scaling generation volume to 100 billion daily tokens.

This facility represents a structural commercial blueprint for Chinese cloud operators to monetize domestic silicon like Huawei's Ascend chips internationally while complying with cross-border data restrictions and mitigating hardware export sanctions. Gateway aggregators servicing global clients can tap into these coastal token hubs for cost-effective inference endpoints, provided they account for potential routing latencies across subsea links.

Verified across 4 sources: South China Morning Post · IT之家 · GeekPark · TechNewsReel

Open Source AI

LiteLLM v1.110.0 and v1.111.0rc1 Mandate Rust Core Engine and Add Agent Skills Hub

Advancing the transition to a pure Rust core and the Agent Skills Hub introduced in recent release candidates, LiteLLM's stable v1.110.0 and v1.111.0rc1 updates add multi-tenant organization hierarchies with budget templates. The releases also implement automated alerts for latency or spend spikes.

The new multi-tenant budget hierarchies directly target enterprise platform teams who need to govern team-level spending and standardize tool distributions across agent frameworks without paying commercial SaaS gateway fees.

Verified across 2 sources: LiteLLM Docs · LiteLLM Docs

vLLM Integrates FlyDSL MonoKernels to Boost Interactivity on AMD MI355X GPUs

vLLM integrated FlyDSL MonoKernels on Wednesday, November 18, 2026, combining attention, projection, and expert computations into persistent GPU kernel launches for AMD Instinct MI355X and MI325X accelerators. Performance testing on a TP2 MI355X node demonstrated an increase in concurrency-1 interactivity for DeepSeek-V4.1-Flash from 318.1 to 429.1 tokens per second per user.

Low-batch decode latency on non-NVIDIA hardware is frequently bottlenecked by host-side CPU kernel launch overhead rather than compute availability. Using domain-specific compilers like FlyDSL to fuse dependent operator steps into persistent launches unlocks significant per-user generation speed on AMD ROCm. This optimization enhances the viability of self-hosting open models on alternative GPU hardware.

Verified across 1 sources: vLLM

Enterprise AI Adoption

Microsoft Launches Agent Controls for Rego-Based Policy Enforcement on Azure

Microsoft announced Agent Controls on Wednesday, November 18, 2026, a managed Azure service built on the open-source Agent Control Specification (ACS) and Agent Hooks primitives. The platform evaluates agent tool execution and API calls against Rego policies, LLM judges, and deterministic safety rules at runtime, allowing enterprises to enforce transfer limits and block PII leaks during live execution.

Autonomous agents capable of invoking external APIs present security risks that static prompt filters cannot mitigate. Externalizing policy evaluation into an out-of-band proxy layer running Rego rules ensures that business logic and compliance gates are enforced deterministically before actions take effect. This decoupled architecture allows platform teams to update security guardrails without modifying underlying agent application code.

Verified across 1 sources: Microsoft Command Line


The Big Picture

Disaggregated Prefill and Decode serving enters managed cloud runtimes Architectures that separate prompt prefill from token decode are moving from custom DIY cluster setups into managed platforms like AWS SageMaker HyperPod via NVIDIA Dynamo and Cerebras Inference Cloud. By offloading prefill to dedicated GPU nodes or wafer-scale systems, providers are attempting to bypass memory bandwidth bottlenecks for long-context workloads.

Real-time request-path policy enforcement replaces static enterprise registers Enterprise security architecture is moving away from passive GRC compliance toward live proxy inspection. Governance platforms like Microsoft Agent Controls, Bifrost, and Envoy extensions are embedding deterministic policy checks, token budgets, and MCP tool access controls directly onto the request path before model or tool invocation occurs.

Compiled Rust gateways consolidate control planes under memory pressure Open-source control planes like LiteLLM are mandating Rust cores by default to eliminate Python runtime overhead, while specialized proxies like Bifrost showcase microsecond-level request overhead. This shift reflects growing operational friction with Python-based proxies as agentic workloads flood gateways with high-frequency tool calls.

Cross-region and KV-cache aware routing become default load balancers Hyperscalers and gateway operators are integrating native KV-cache indexing and multi-region failover directly into model routers. Platforms like AWS SageMaker AI and Microsoft Model Router are using cache-state events and session affinity to prevent redundant prefill computations and handle regional GPU capacity shortages automatically.

Token export models formalize around domestic chip clusters in Asia Operators in regional hubs like Shantou are packaging domestic silicon like Huawei's Ascend 950DT into dedicated 'token export' compute centers. By processing queries locally at international cable landing stations and returning generated tokens, infrastructure providers are establishing alternative monetization routes amid hardware trade restrictions.

What to Expect

2026-11-24 — CloudBackend scheduled to launch free developer beta for its backend operations platform.
2027-01-01 — Azure AI Gateway transitions from free preview allowance to paid commercial billing tier ($500/mo base).

Every story, researched.

Every story verified across multiple sources before publication.

🔍

Scanned

Across multiple search engines and news databases

451
📖

Read in full

Every article opened, read, and evaluated

111
⭐

Published today

Ranked by importance and verified across sources

12

— The Gateway Signal

🎙 Listen as a podcast

Subscribe in your favorite podcast app to get each new briefing delivered automatically as audio.

Apple Podcasts
Library tab → ••• menu → Follow a Show by URL → paste
Overcast
+ button → Add URL → paste
Pocket Casts
Search bar → paste URL
Castro, AntennaPod, Podcast Addict, Castbox, Podverse, Fountain
Look for Add by URL or paste into search

Spotify isn’t supported yet — it only lists shows from its own directory. Let us know if you need it there.