Today on The Gateway Signal: Azure AI Gateway hits general availability with native MCP tool federation, while Cerebras and AMD unveil disaggregated inference architectures splitting prefill and decode across distinct silicon.
Microsoft announced the general availability of Azure AI Gateway at Microsoft Ignite on Wednesday, November 18, transitioning from an API Management tier into a standalone service across 17 Azure regions. The platform federates Model Context Protocol (MCP) servers and OpenAPI specifications under unified model aliases spanning Azure OpenAI, Anthropic, Bedrock, and Google Vertex AI. The service includes Microsoft Entra ID authorization, prompt safety screening, and integration with Microsoft Agent 365, offering 10,000 free requests monthly with paid billing ($500/month for up to 10M requests) starting January 1, 2027.
Why it matters
Hyper-scale cloud providers are expanding managed gateways into full tool-governance control planes. By federating MCP tool servers directly at the gateway layer alongside commercial LLM endpoints, platform architects can enforce Entra ID policy and Microsoft Purview DLP rules across multi-cloud agent calls without custom sidecar proxies. This directly challenges standalone gateway providers like Portkey, LiteLLM, and Kong by bundling enterprise authorization and auditability directly into core cloud infrastructure.
DeepInfra announced a strategic partnership with Stockholm-based Opper AI on Wednesday, November 18, integrating over 100 DeepInfra-hosted open models into Opper's GDPR-compliant, EU-hosted AI gateway. The unified endpoint exposes models including DeepSeek V4.1 Flash, GLM-5.3, Kimi K3, and MiniMax M3 using OpenAI-compatible formats. The system introduces flex, standard, and priority pricing tiers paired with centralized rate limits, zero data retention (ZDR) options, and budget caps.
Why it matters
Combining US-based raw inference scaling with EU-hosted gateway governance provides a clear template for compliance-heavy enterprise deployments. By offering discounted flex tiers for asynchronous tasks alongside strict European data sovereignty controls, gateway operators allow engineering teams to run high-throughput open-weight models without violating cross-border data transfer regulations.
Expanding its unified API beyond the Gemini and MiniMax image models we've tracked, EvoLink introduced multi-route support for ByteDance's Seedance 2.0 video generation model on Wednesday, November 18. The platform exposes six routes, including Text-to-Video and Reference-to-Video, priced at $0.993 per 5-second 720p output in Standard mode, with a 25% discount for Fast 480p draft renders. EvoLink also previewed an upcoming Seedance 2.5 integration supporting multi-reference inputs and up to 30-second clips.
Why it matters
Unified API gateways are actively expanding beyond LLM text completion into generative video and multimodal workloads. By establishing standardized request schemas and draft-versus-production pricing tiers for video generation, gateways allow engineering teams to manage high-cost multimodal rendering calls with the same rate-limiting and budget controls applied to text models.
Cerebras Systems introduced commercial disaggregated inference on Wednesday, November 18, offloading prompt prefill processing to AMD Instinct GPUs while assigning autoregressive token decode to Cerebras CS-3 Wafer Scale Engines. In production tests on the Cerebras Inference Cloud, the split architecture delivered a 5x increase in token output per megawatt compared to a CS-3-only baseline without degrading single-stream generation speed. Early access partners adopting the reference stack include AWS, Fireworks AI, Gimlet Labs, and General Compute.
Why it matters
Disaggregating prompt prefill from token decoding addresses the structural compute mismatch in long-context agentic workloads. Prefill operations require compute-dense matrix multiplication best served by traditional GPUs, whereas autoregressive decoding is bound by memory bandwidth where wafer-scale processors excel. This hardware co-design provides a practical blueprint for high-throughput inference providers to bypass memory bandwidth bottlenecks and cut data center power consumption.
Amazon SageMaker AI launched KV cache-aware routing (KV_AWARE) for real-time endpoints on Tuesday, November 17. The feature reads container cache state events published by vLLM instances via the SM_VLLM_KV_EVENTS_CONFIG environment variable to map incoming requests to nodes holding pre-existing prefix caches. In multi-node Llama 3.1 70B benchmarks across ml.p5.48xlarge instances, KV-aware routing reduced p50 time to first token (TTFT) by up to 97% and boosted throughput by 234% over round-robin load balancing.
Why it matters
Static prefix hashing fails in dynamic multi-turn agent conversations where context growth varies per user. By embedding real-time KV-cache state tracking directly into endpoint routers, managed inference platforms eliminate redundant GPU prefill execution. This optimization provides self-hosted and cloud enterprise teams with significantly higher throughput on existing GPU allocations.
Following up on the GPT-6 tier rollouts we've tracked over the past month, OpenAI launched GPT-6.2 Astra and GPT-6.2 Sol on Tuesday, November 17. The models feature a 1.05 million-token context window and updated multimodal reasoning capabilities. Astra is priced at $8.00 per million uncached input tokens and $40.00 per million output tokens (a 20% drop from prior flagship pricing), with prompt cache hits discounted to $0.40 per million. The release was made available immediately across Vercel AI Gateway and OpenAI's Responses API.
Why it matters
Rapid price compression at the frontier layer forces gateway architectures to handle granular prompt caching and dynamic tier routing natively. As frontier lab input/output pricing settles into $8/$40 standard tiers with 95% cache discounts, gateway platforms like OpenRouter, Portkey, and Vercel must accurately pass through cache write/read states to prevent developers from paying unnecessary prefill surcharges on multi-turn coding sessions.
Reflecting the shift toward lightweight decision models we covered yesterday with Vercel's telemetry—which showed them capturing 26% of gateway traffic—Microsoft launched seven specialized models at Ignite on Tuesday, November 17. Led by MAI Code 2 and MAI Cyber 2 Flash, the release accompanied the general availability of Microsoft's custom Maia 200 silicon. The company claims MAI Code 2 achieves 98% of GPT-6.1 Sol coding accuracy at 40% of the cost. Simultaneously, Vercel added Microsoft's new Decision-1.1 model to its AI Gateway for structured probability and numerical scoring requests.
Why it matters
The rise of lightweight decision and task-specific models changes how AI gateways perform ingress routing. Rather than using expensive frontier LLMs to evaluate user intent or classify guardrails, gateway operators can pass requests through sub-cent decision endpoints to evaluate branch logic, drastically cutting overall application latency and token spend.
Codenotary updated its open-source AgentMon Live tool on Tuesday, November 17, adding execution monitoring for OpenAI Codex and AWS Kiro alongside existing Claude Code support. The local web interface tracks step-by-step shell command execution, file system modifications, and sub-agent task completion state to flag redundant token consumption and runaway loops.
Why it matters
Autonomous coding agents execute shell actions and file modifications with elevated local permissions, creating blind spots for engineering managers tracking token spend and code quality. Framework-agnostic, open-source monitoring tools allow teams to audit multi-agent tool execution locally across disparate CLI coding tools without sending proprietary source code traces to third-party SaaS vendors.
A research collaboration across MIT Han Lab, Georgia Tech, UIUC, UC Berkeley, NVIDIA, and OpenAI published the open-source Evolving Infra framework (MIT license) on Monday, November 17. The system decouples GPU compute scheduling from distributed KV-cache management, allowing node clusters to dynamically swap between prefill and decode modes using context compaction signals emitted by agent harnesses. Tested on 256 NVIDIA GB300 GPUs with GLM-5.2 and Kimi K2.6, the framework reduced tail TTFT by up to 14x compared to NVIDIA Dynamo.
Why it matters
Traditional static serving runtimes struggle with agentic workloads that mix long, unpredictable tool-use prefills with multi-turn generation phases. Co-designing agent context management directly with the serving cluster's KV-cache evicted state allows self-hosted infrastructure operators to achieve massive throughput gains on heterogeneous GPU clusters without purchasing extra compute capacity.
Following up on the Blackwell initialization signatures we noted over the weekend, Unemployment Labs published its formal forensic study on Tuesday, November 17. The paper reveals that the evidence stems from Philox algorithm patterns found across six distinct Chinese open-weight models released since April 2026. The findings provide technical backing for Monday's newly enacted US Bureau of Industry and Security (BIS) restrictions on overseas cloud computing.
Why it matters
Forensic weight analysis provides concrete evidence that Chinese frontier labs continue to access restricted silicon through overseas cloud compute leases. As US regulators close remote data center access loopholes, Chinese AI platforms will face stronger pressure to accelerate domestic chip cluster integration on Huawei Ascend and SMIC hardware.
Advancing the transition to a pure Rust core we tracked with last week's v1.110 release candidate, BerriAI released LiteLLM v1.108.11 (LTS) alongside v1.111.0-dev.4 on Wednesday, November 18. The latest version updates its open-source catalog to index 2,317 models across 113 inference providers. The update migrates Claude API calls on Azure AI Foundry directly to the new Rust core gateway, adds OpenSearch vector store filter support, and fixes rate-limiter memory consumption bugs during high-concurrency streaming retries.
Why it matters
LiteLLM's incremental migration of hot-path components to Rust illustrates the operational limits of pure Python proxy layers in production. As enterprise API gateways handle thousands of concurrent streaming connections, shifting authentication, rate-limiting, and header translation to compiled binaries prevents proxy memory bloat and reduces median latency overhead.
European cloud provider OVHcloud announced Olala mini 7A1B on Tuesday, November 17, its first open-source mixture-of-experts model. Built on Dragon LLM technology and trained on European public supercomputers (Jupiter, MeluXina, Jean Zay), the model features 7 billion total parameters with 1 billion active per token via a Differential-TPA and Mamba-3 hybrid architecture. Weights will be published on Hugging Face under an Apache 2.0 license.
Why it matters
European cloud vendors are building sovereign, lightweight open-weight models designed to run locally on workstation hardware with 8GB of VRAM. This provides privacy-sensitive European enterprises with a fully self-hosted, Apache 2.0-licensed alternative that bypasses external US and Asian API gateways entirely.
Disaggregated Prefill and Decode Pipelines Reach Commercial Serving Inference providers are abandoning monolithic GPU node clusters for heterogeneous hardware splits. By routing heavy prefill phases to merchant GPUs or cloud instances while pinning low-latency decode operations to specialized wafer-scale silicon or disaggregated KV-cache clusters, platforms like Cerebras and research frameworks like Evolving Infra are reshaping token economics.
Managed Gateways Consolidate Model Execution with MCP Governance Hyper-scale cloud providers and serverless hosts are moving beyond simple model proxying to bundle Model Context Protocol (MCP) tool federation directly into the request path. Azure AI Gateway, Vercel, and enterprise control planes now inspect, sanitize, and authorize both LLM completion calls and outbound agent tool interactions under unified identity layers.
Rust Migration Accelerates Across Open-Source Gateway Control Planes Open-source proxies and serving engines are aggressively rewriting hot paths in Rust to eliminate Python GIL bottlenecks. As seen in vLLM v0.34.0, LiteLLM v1.108.11, and SGLang, routing logic, rate-limiting, and memory callbacks are transitioning to compiled binaries to maintain sub-millisecond overhead under high concurrency.
Open-Weight Fine-Tuning and Specialized Model Tiers Undercut Frontier APIs Enterprise architectures are shifting away from direct reliance on closed frontier models toward domain-specific fine-tunes and specialized decision models. Provider releases like Microsoft's MAI Code 2 and Decision-1.1 demonstrate that lightweight, task-focused endpoints can deliver equivalent task accuracy at a fraction of the per-token cost.
Hardware Fingerprinting and Distillation Controls Reshape International Supply Chains Geopolitical compute tracking is expanding from physical silicon export bans to weight-level forensic audits and synthetic data restrictions. As researchers identify hardware signatures in open-weight releases and US regulators target remote cloud hosting loopholes, Chinese model developers are increasingly turning to industrial-scale distillation to sustain performance leaps.
What to Expect
2026-11-20—OVHcloud AI Summit showcasing Olala Mini 7A1B open-source MoE weights
2027-01-01—Azure AI Gateway paid billing transitions from public preview freemium tier
How We Built This Briefing
Every story, researched.
Every story verified across multiple sources before publication.
🔍
Scanned
Across multiple search engines and news databases
488
📖
Read in full
Every article opened, read, and evaluated
131
⭐
Published today
Ranked by importance and verified across sources
12
— The Gateway Signal
🎙 Listen as a podcast
Subscribe in your favorite podcast app to get each new briefing delivered automatically as audio.
Apple Podcasts
Library tab → ••• menu → Follow a Show by URL → paste