Securing AI agents: How AWS and Cisco AI Defense scale MCP and A2A deployments

May 6, 2026 Announcements Higher usage limits for Claude and a compute deal with SpaceX

Higher usage limits for Claude and a compute deal with SpaceX We’ve agreed to a partnership with SpaceX that will substantially increase our compute capacity. This, along with our other recent compute deals, means that we’ve been able to increase our usage limits for Claude Code and the Claude API. Below, we describe these changes and the progress we’re making on compute. Higher usage limits The following three changes—all effective today—are aimed at improving the experience of using Claude for our most dedicated customers. First, we’re doubling Claude Code’s five-hour rate limits for Pro, Max, Team, and seat-based Enterprise plans. Second, we’re removing the peak hours limit reduction on Claude Code for Pro and Max accounts. Third, we’re raising our API rate limits considerably for Claude Opus models, as shown in the table below: New compute partnership with SpaceX We’ve…

The newest AI boom pitch: Host a mini data center at your home

Data centers may be coming to your neighborhood as side installations associated with new homes—and in exchange would offer subsidized electricity and Internet access along with backup batteries to homeowners. The company behind the plan has already begun pilot testing in preparation for a 100-home trial run this year. The “distributed data center solution” announced by the San Francisco startup SPAN would deploy thousands of XFRA nodes that contain liquid-cooled Nvidia RTX Pro 6000 Blackwell Server Edition GPUs operating with minimal noise, according to a press release. By harnessing excess power capacity among US households, SPAN aims to quickly expand the available compute for AI workloads without the costs and delays associated with trying to build warehouse-sized data centers. “Data centers are loud, ugly, and often drive up local electricity bills,” said Chris Lander, vice president of XFRA at SPAN,…

OpenAI president forced to read his personal diary entries to jury

Greg Brockman never wanted to discuss his personal journal in public. But the OpenAI president has been stuck for days doing exactly that, while testifying in a trial in which Elon Musk has alleged that OpenAI abandoned its nonprofit mission to instead focus on personally enriching leaders like Brockman and Sam Altman. “It’s very painful,” Brockman told OpenAI lawyer Sarah Eddy during his second day on the stand. Although he’s not “ashamed” of any of the journal entries, he considers them to be deeply personal, he said. Rather than serving as a straightforward log of his actions or feelings, the entries reflect a stream of consciousness that meanders as it explores alternate viewpoints. Sometimes, Brockman explained, he would jot notes reflecting another person’s thoughts, just to feel them out for himself. Because of this, Brockman can appear self-contradictory at times,…

The great American data center divide

In Tazewell County, Illinois, Michael Deppert depends on a natural pool of water beneath the sandy soils of his farm to irrigate the pumpkins, corn, and soybeans growing in his fields. So when a data center was proposed about eight miles away, he feared it would tap the same aquifer, potentially eroding crop yields and profits. Deppert, who is also the president of the local farm bureau lobby group, says locals were also “nervous” about how a data center would affect the “good, clean drinking water.” Residents launched a fierce opposition campaign, packing city council meetings and mounting petitions. After several months, the project, led by developer Western Hospitality Partners, was scrapped. “You just can’t lay down and let everybody do whatever they wish,” Deppert says. It is just one of the many pockets of resistance opening up across rural…

Musk and Altman face off in trial that will determine OpenAI's future

A hotly anticipated trial starts this week, where Elon Musk will attempt to prove that OpenAI, under Sam Altman, has abandoned its mission to remain a nonprofit in order to ensure that artificial intelligence serves humanity, and not just billionaires. Many view the lawsuit as a grudge match between Musk—who left OpenAI after serving as an early major donor and advisor—and Altman—who currently runs OpenAI, despite insiders’ allegedly growing distrust in his commitment to the dominant AI firm’s mission. But the lawsuit is about much more than a couple billionaires’ big egos. The outcome could radically change the AI landscape, impacting how OpenAI runs and what resources the firm will have to uphold its mission. If Musk wins, OpenAI’s hopes of growing a for-profit arm that can fund the nonprofit could be dashed. Additionally, Brockman and Altman could be dropped…

Greenhouse gases from data center boom could outpace entire nations

New gas projects linked to just 11 data center campuses around the US have the potential to create more greenhouse gases than the country of Morocco emitted in 2024. Emissions estimates from air permit documents examined by WIRED show that these natural gas projects—which are being built to power data centers to serve some of the US’s most powerful AI companies, including OpenAI, Meta, Microsoft, and xAI—have the potential to emit more than 129 million tons of greenhouse gases per year. As tech companies race to secure massive power deals to build out hundreds of data centers across the country, these projects represent just the tip of the iceberg when it comes to the potential climate cost of the AI boom. The infrastructure on this list of large natural gas projects reviewed by WIRED is being developed to largely bypass…

Pentagon wants $54B for drones, more than most nations’ military budgets

The US military’s massive $1.5 trillion budget request for the next fiscal year includes what Pentagon officials described as the largest investment in drone warfare and counter-drone technology in US history. The proposed spending on drone and autonomous warfare technologies within the FY2027 budget proposal for the US Department of Defense would surpass most countries’ defense budgets and rank among the top 10 in the world for military spending, ahead of countries such as Ukraine, South Korea, and Israel. Specifically, the Pentagon is requesting $53.6 billion to boost US production and procurement of drones, train drone operators, build out a logistics network for sustaining drone deployments, and expand counter-drone systems to defend more US military sites. The funding request is budgeted under the Defense Autonomous Warfare Group (DAWG), an organization established in late 2025 that would see a massive budget…

Robot runner handily beats humans in half-marathon, setting new record

Humanoid robots outran the fastest human competitors while surpassing the human world record during a half-marathon event held in Beijing on April 19. The demonstration of fast-improving robotic speed and autonomy comes as China’s tech industry is rapidly scaling up mass production of humanoid robots to explore possible uses in the real world. The fastest robot from Chinese smartphone-maker Honor notched a winning time of 50 minutes and 26 seconds while autonomously navigating the 13-mile (21-kilometer) route, according to the Global Times. That beat the human world record of 57 minutes and 20 seconds recently set by Ugandan long-distance runner Jacob Kiplimo during the Lisbon Half Marathon. The winning robot design took inspiration from top human athletes by incorporating long legs measuring approximately 37 inches (95 centimeters) in length, said Du Xiaodi, a test development engineer for Honor, who spoke…

Mozilla launches Thunderbolt AI client with focus on self-hosted infrastructure

Mozilla is the latest legacy tech brand to make a play for the enterprise AI market. But the company behind Firefox and Thunderbird isn’t releasing its own standalone AI model or agentic browser. Instead, the newly announced Thunderbolt is being sold as a front-end client for users and businesses who want to run their own self-hosted AI infrastructure without relying on cloud-based third-party services. Thunderbolt is built on top of Haystack, an existing open source AI framework that lets users build custom, modular AI pipelines from user-chosen components. Thunderbolt acts as what Mozilla calls a “sovereign AI client” on top of that underlying infrastructure. The combo promises to let users easily plug into any ACP-compatible agent or OpenAI-compatible API (including Claude, Codex, OpenClaw, DeepSeek, and OpenCode). The system can also integrate with locally stored enterprise data through open protocols and…

Build real-time voice streaming applications with Amazon Nova Sonic and WebRTC

Artificial Intelligence Build real-time voice streaming applications with Amazon Nova Sonic and WebRTC Building end-to-end live streaming applications with real-time voice interaction presents several challenges: network bandwidth constraints can cause high latency and quality degradation in time-critical applications. Language barriers limit effective human-machine interaction in multilingual voice communication. Scalability and resilience require a difficult balance between performance and infrastructure costs. Cross-browser and mobile compatibility demands significant development effort, especially for startups. This post introduces a solution based on Amazon Nova 2 Sonic (Nova Sonic) and Amazon Kinesis Video Streams WebRTC (WebRTC) that addresses these challenges. WebRTC is responsible for dynamically adjusting the bitrate in unstable networks, which helps to maintain audio quality while reducing dropped connections. Nova Sonic provides effective human language dialogues, so users can interact more naturally in their chosen language. Both services are fully managed by AWS,…

How Amazon Finance streamlines regulatory inquiries by using generative AI on AWS

Artificial Intelligence How Amazon Finance streamlines regulatory inquiries by using generative AI on AWS Amazon’s Finance Technology (FinTech) teams build and operate systems for Amazon teams to manage regulatory inquiries in compliance with different jurisdictions. These teams process regulatory inquiries from authorities, each presenting different requirements, document formats, and complexity levels. Processing these regulatory inquiries involves reviewing documentation, extracting relevant information, retrieving supporting data from multiple systems within Amazon’s infrastructure, and compiling responses within regulatory timeframes. As inquiry frequency and business complexity grew, Amazon needed a more scalable approach. In this post, we demonstrate how Amazon FinTech teams are using Amazon Bedrock and other AWS services to build a scalable AI application to transform how regulatory inquiries are handled. Each team using this solution creates and maintains its own dedicated knowledge base, populated with that team’s specific documents and reference…

Manufacturing intelligence with Amazon Nova Multimodal Embeddings

Artificial Intelligence Manufacturing intelligence with Amazon Nova Multimodal Embeddings If you work in aerospace, automotive, or heavy industry manufacturing, your organization likely maintains vast repositories of technical documents. These documents combine written specifications with engineering diagrams, CAD drawings, inspection photographs, thermal analysis plots, and fatigue curves. A text query about maximum wall temperature at the nozzle throat might have its answer locked inside a thermal contour plot rather than written prose. Text-only retrieval systems can’t surface that information because they don’t see the image content. Amazon Nova Multimodal Embeddings addresses this gap by mapping text, images, and document pages into a shared vector space. A text query can retrieve an engineering diagram, and an image query can retrieve a written specification, because both modalities share the same coordinate system. In this post, we build a multimodal retrieval system for aerospace…

Cost effective deployment of vision-language models for pet behavior detection on AWS Inferentia2

Artificial Intelligence Cost effective deployment of vision-language models for pet behavior detection on AWS Inferentia2 Tomofun, the Taiwan-headquartered pet-tech startup behind the Furbo Pet Camera, is redefining how pet owners interact with their pets remotely. Furbo combines smart cameras with AI to detect behaviors such as barking, running, or unusual activity, and alerts owners in real time. At the core of this capability are computer vision and vision-language models that interpret pet actions from the video streams. Originally, Furbo’s inference workloads were hosted on GPU-based Amazon Elastic Compute Cloud (Amazon EC2) instances. While GPUs provided high throughput, they were also costly because the always-on inference needed to support real-time pet activity alerts at scale. To reduce costs and maintain accuracy, Tomofun turned to EC2 Inf2 instances powered by AWS Inferentia2, the Amazon purpose-built AI chips. In this post, we walk…

Secure AI agents with Amazon Bedrock AgentCore Identity on Amazon ECS

Artificial Intelligence Secure AI agents with Amazon Bedrock AgentCore Identity on Amazon ECS AI agents in production require secure access to external services. Amazon Bedrock AgentCore Identity, available as a standalone service, secures how your AI agents access external services whether they run on compute platforms like Amazon ECS, Amazon EKS, AWS Lambda, or on-premises. An earlier post covered AgentCore Identity credential management for AI agents. Running agents on compute environments like ECS raises two questions: How to build an application-owned Session Binding endpoint, and how to manage workload access token lifecycle? This post implements Authorization Code Grant (3-legged OAuth) on Amazon ECS with secure session binding and scoped tokens. This post provides a working implementation with: - Secure session binding that prevents CSRF and browser-swapping attacks - Auth tokens scoped to each user session, following least-privilege principles - Separation…

Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints

Artificial Intelligence Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints As organizations scale generative AI workloads in production, securing reliable GPU compute has become one of the most persistent operational challenges. Large language models (LLMs) and multimodal architectures demand specific instance types and when that capacity isn’t available, endpoints fail before they serve a single request. Building a real-time inference endpoint on Amazon SageMaker AI has meant committing to a single instance type at creation time. When that type had insufficient capacity, the endpoint failed to reach a running state. You updated your configuration, selected a different instance type, and retried repeating the cycle until a provisioning attempt succeeded. Today, Amazon SageMaker AI introduces capacity aware instance pool for new and existing inference endpoints. You define a prioritized list of instance types, and SageMaker AI automatically works through your…

Configuring Amazon Bedrock AgentCore Gateway for secure access to private resources

Artificial Intelligence Configuring Amazon Bedrock AgentCore Gateway for secure access to private resources AI agents in production environments often need to reach internal APIs, databases, and private resources that sit behind Amazon Virtual Private Cloud (Amazon VPC) boundaries. Managing private connectivity for each agent-to-tool path adds operational overhead and slows deployment. Amazon Bedrock AgentCore VPC connectivity is designed to deploy AI agents and Model Context Protocol (MCP) servers without requiring the network traffic to be exposed to the public internet. This capability extends to managed Amazon VPC egress for Amazon Bedrock AgentCore Gateway, so you can connect to endpoints inside private networks across your AWS environment. In this post, you will configure Amazon Bedrock AgentCore Gateway to access private endpoints using Resource Gateway, a managed construct that provisions Elastic Network Interfaces (ENIs) directly inside your Amazon VPC, one per subnet.…

Unleashing Agentic AI Analytics on Amazon SageMaker with Amazon Athena and Amazon Quick

Artificial Intelligence Unleashing Agentic AI Analytics on Amazon SageMaker with Amazon Athena and Amazon Quick Modern enterprises face mounting challenges in extracting actionable insights from vast data lakes and lakehouses spanning petabytes of structured and unstructured data. Traditional analytics require specialized technical expertise in SQL, data modeling, and business intelligence tools, creating bottlenecks that slow decision-making across retail, financial services, healthcare, Travel & Hospitality, manufacturing and many more industries. This architecture demonstrates how agentic AI assistant from Amazon Quick transform data analytics into a self-service capability. It showcases enabling business users to query complex structured datasets and mix with unstructured data to find the valuable insights to improve their business outcomes through intuitive natural language interfaces. To demonstrate the functionality, we built a lakehouse using the TPC-H datasets as our foundation. This integrated architecture leverages Amazon Simple Storage Service (Amazon…

How Popsa used Amazon Nova to inspire customers with personalised title suggestions

Artificial Intelligence How Popsa used Amazon Nova to inspire customers with personalised title suggestions This post was co-written with Bradley Grantham and Hugo Dugdale from Popsa. Popsa is a technology company that helps users rediscover and relive the meaningful memories hidden in their photo libraries. Available across more than 50 countries and 12 languages, we use design automation and AI to transform everyday photos into personal, shareable experiences, including beautifully printed Photo Books. In 2016, we released PrintAI, a pioneering algorithm to take complete control of creating a varied and interesting design from a user’s photos. Our customers could use the algorithm to create Photo Books that appeared professionally designed, in less than 5 minutes. A core philosophy of our business is that technology should do the heavy lifting for our users, so automation has always been an intrinsic part…

Build and deploy an automatic sync solution for Amazon Bedrock Knowledge Bases

Artificial Intelligence Build and deploy an automatic sync solution for Amazon Bedrock Knowledge Bases With Amazon Bedrock Knowledge Bases, you can give foundation models (FMs) and agents contextual information from your organization’s private data sources to deliver more relevant, accurate, and customized responses. As the data grows, maintaining real-time synchronization between Amazon Simple Storage Service (Amazon S3) and your knowledge bases becomes critical for accurate, up-to-date responses.In this post, we explore how Deloitte used Amazon EKS and vCluster to transform their testing infrastructure. In this post, we explore an automated solution that detects S3 events and triggers ingestion jobs while respecting service quotas and providing comprehensive monitoring. This serverless solution uses an event-driven architecture to keep your knowledge base current without overwhelming the Amazon Bedrock APIs. The challenge Knowledge bases in Amazon Bedrock require manual synchronization whenever documents are added,…

Applying multimodal biological foundation models across therapeutics and patient care

Artificial Intelligence Applying multimodal biological foundation models across therapeutics and patient care Healthcare and life sciences decision making increasingly relies on multimodal data to diagnose diseases, prescribe medicine and predict treatment outcomes, develop and optimize innovative therapies accurately. Traditional approaches analyze fragmented data, such as ‘omics for drug discovery, medical images for diagnostics, clinical trial reports for validation, and electronic health records (EHR) for patient treatment. As a result, decision makers (CxOs, VPs, Directors) often miss critical insights hidden in the relationships between data types. Recent advancements in AI enable you to integrate and analyze these fragmented data streams efficiently to support a more complete understanding of therapeutics and patient care. AWS provides a unified environment for multimodal biological foundation models (BioFMs), enabling you to make more confident, timely decision-making in personalized medicine. This AI system combines biological data, model…

Get to your first working agent in minutes: Announcing new features in Amazon Bedrock AgentCore

Artificial Intelligence Get to your first working agent in minutes: Announcing new features in Amazon Bedrock AgentCore Getting an agent running has always meant solving a long list of infrastructure problems before you can test whether the agent itself is any good. You wire up frameworks, storage, authentication, and deployment pipelines, and by the time your agent handles its first real task, you’ve spent days on infrastructure instead of agent logic. We built AgentCore from the ground up to help developers focus on building agent logic instead of backend plumbing, working with frameworks and models they already use, including LangGraph, LlamaIndex, CrewAI, Strands Agents, and more. Today, we’re introducing new capabilities that further streamline the agent building experience, removing the infrastructure barriers that slow teams down at every stage of agent development from the first prototype through production deployment. Go…

Amazon SageMaker AI now supports optimized generative AI inference recommendations

Artificial Intelligence Amazon SageMaker AI now supports optimized generative AI inference recommendations Organizations are racing to deploy generative AI models into production to power intelligent assistants, code generation tools, content engines, and customer-facing applications. But deploying these models to production remains a weeks-long process of navigating GPU configurations, optimization techniques, and manual benchmarking, delaying the value these models are built to deliver. Today, Amazon SageMaker AI supports optimized generative AI inference recommendations. By delivering validated, optimal deployment configurations with performance metrics, Amazon SageMaker AI keeps your model developers focused on building accurate models, not managing infrastructure. We evaluated several benchmarking tools and chose NVIDIA AIPerf, a modular component of NVIDIA Dynamo, because it exposes detailed, consistent metrics and supports diverse workloads out of the box. Its CLI, concurrency controls, and dataset options give us the flexibility to iterate quickly and…

4/27/2026 DeepSeek V4 Pro: Validating Frontier Models For Production

Why we chose correctness over a Day-0 launch DeepSeek V4 Pro is one of the most important open-model releases this year, with real advances in long-context reasoning, agentic performance, and inference efficiency. On paper, it looks like a step change. In practice, the first 48 hours exposed something the benchmarks did not show. Across early deployments, we observed reasoning traces degrading mid-generation into token-level corruption, malformed artifacts, and unexpected structured fragments inside the output stream. These were not isolated glitches or prompt issues. We first encountered the issue in our own deployment, then reproduced the same failure modes across multiple DeepSeek-enabled providers over the weekend. This pointed to a broader serving-path correctness issue affecting early V4 deployments. Issues like this usually get fixed. Our position is simpler: end users should not be exposed to that instability in production systems. Like…

4/24/2026 Notes on DeepSeek-V4's training system

On this page DeepSeek-V4 is interesting less for any single benchmark number than for the shape of the system around it. The paper shows architecture, routing, reward modeling, reasoning modes, distillation, and agent execution all becoming part of the training loop. The useful takeaway for training infrastructure is obvious: fixed recipes are not enough. Researchers increasingly need programmable loops, while the platform handles distributed execution, inference integration, checkpointing, and scaling underneath. Supporting that flexibility is the core design principle behind the Fireworks Training API. DeepSeek-V4 alternates between Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA compresses KV entries and then does sparse top-k selection. HCA compresses more aggressively, but keeps dense attention over the compressed memory. The point is not just "longer context." It is model/runtime co-design: attention pattern, KV layout, precision, sparse selection, and inference kernels all…

4/6/2026 Own Your AI: Fireworks Training Preview

Fireworks Training is now in preview: an end-to-end platform for training and deploying frontier models at scale. Three surfaces for three kinds of teams, from a conversational agent that handles everything, to managed infrastructure for ML engineers, to bring-your-own training loop on Fireworks-hosted clusters. All on the same infrastructure that already handles production inference for Cursor, Vercel, Genspark, and others. All three surfaces are in preview now. Reinforcement learning is how teams push past the ceiling SFT hits on multi-step reasoning, reliable tool use, and mid-flight self-correction. Vercel used our RL infrastructure to build a custom "Auto Fix" model for v0. The model checks the output stream for errors and self-corrects without a second pass, reaching a 93% error-free generation rate, significantly outperforming closed frontier models, with a 40X improvement in end-to-end latency vs. the proprietary model it replaced and…

3/23/2026 Frontier RL Is Cheaper Than You Think

On this page The conventional wisdom on RL infrastructure is wrong, and it is costing teams that could be competing at the frontier. The entire mega-cluster narrative rests on a single assumption: that you have to ship 1 TB of weights every time you update your rollout fleet. You do not. Researchers have spent the last year writing about asynchronous RL and rollout-training disaggregation in systems like AReaL. Teams like Kimi and MiniMax have also published engineering notes on RL parameter updates and asynchronous scheduling. We have been running that pattern in production. That mega-cluster instinct comes from pretraining, where the main systems problem is keeping one huge synchronous training job saturated. RL is a different problem. The question is not just how to run the trainer. It is also how to keep a large rollout fleet generating data from…

Broadening advanced AI education across Africa

Broadening advanced AI education across Africa AI is driving scientific discoveries and research breakthroughs, but its progress depends on a global community. To bridge the gap between talent and opportunity, Google DeepMind is launching additional courses of its AI Research Foundations curriculum: advanced AI education designed for the next generation of technical learners across Africa. Hands-on experience with generative AI models The courses, developed with pedagogy experts and academics at University College London — and available at no cost on Google Skills — give learners the opportunity to build and fine-tune a language model from the ground up. Google.org is supporting the curriculum’s rollout in African classrooms by providing funding for lecturer training and instructional toolkits. The curriculum, already serving thousands of users globally, moves beyond AI literacy, providing technical university students and community learners with a deep, applied understanding…

Canopy Labs’ Orpheus TTS is live on GroqCloud

Canopy Labs’ Orpheus TTS is live on GroqCloud In December, we announced support for Canopy Labs’ Orpheus text-to-speech (TTS) on GroqCloud, with two model variants built for real-time, high-quality voices: - English TTS: canopylabs/orpheus-v1-english (with vocal directions) - Saudi Arabic (dialect) TTS: canopylabs/orpheus-arabic-saudi (authentic pronunciation + regional nuance) Today, we’re excited to announce a new release of the Saudi Arabic Orpheus TTS model on GroqCloud (canopylabs/orpheus-arabic-saudi). This release brings overall model improvements, including reduced hallucinations, more natural and expressive speech, and more accurate handling of numbers and symbols. It also introduces two new Saudi Arabic voices designed to sound more natural, culturally grounded, and production-ready. - Abdullah — A professional, calm, and conversational male voice, ideal for assistants, enterprise workflows, and general voice interfaces. - Aisha — A professional, clear, and approachable female voice, especially effective for customer support and…

"OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support"

"OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support" - user: oncoagent-research tags: - oncology - multi-agent - LangGraph - RAG - QLoRA - AMD - open-source - clinical-ai - healthcare OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support Technical preprint · May 2026 · OncoAgent Research Group Abstract We present OncoAgent, an open-source, privacy-preserving clinical decision support system for oncology. OncoAgent combines a dual-tier fine-tuned LLM architecture with a state-of-the-art multi-agent LangGraph topology, a four-stage Corrective RAG pipeline over 70+ physician-grade NCCN and ESMO guidelines, and a three-layer reflexion safety validator enforcing a strict Zero-PHI policy. The system routes clinical queries through an additive complexity scorer to either a 9B parameter speed-optimised model (Tier 1) or a 27B deep-reasoning model (Tier 2), both fine-tuned via QLoRA on a corpus of 266,854 real and synthetically…

vLLM V0 to V1: Correctness Before Corrections in RL

vLLM V0 to V1: Correctness Before Corrections in RL TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head used for the final projection. We fixed the backend behavior before changing the RL objective. The reference run used vLLM 0.8.5 ; the V1 runs used vLLM 0.18.1 . Figure 1 shows the final result. The red run is the initial V1 attempt, and the green run is the final V1 run after the fixes described below. Migration Objective vLLM V1 is a substantial rewrite of the V0 engine. Our migration target was therefore deliberately narrow: - verify that V1 returned rollout logprobs in the form the trainer expected - rerun the same workload against the V0 reference - evaluate objective-level changes only after…

Granite 4.1 LLMs: How They’re Built

Granite 4.1 LLMs: How They’re Built Authors: Granite Team, IBM TL;DR — Granite 4.1 is a family of dense, decoder‑only LLMs (3B, 8B, and 30B) trained on ~15T tokens using a multi‑stage pre‑training pipeline, including long‑context extension of up to 512K tokens. The models are further refined with supervised fine‑tuning on ~4.1M high‑quality curated samples and reinforcement learning via on‑policy GRPO with DAPO loss (Yu et al., 2025). Notably, the 8B instruct model matches or surpasses the previous Granite 4.0‑H‑Small (32B‑A9B MoE) despite using a simpler dense architecture with fewer parameters. All Granite 4.1 models are released under the Apache 2.0 license. Links: Overview Building high‑quality small language models goes beyond simply scaling compute—it requires rigorous data curation throughout training. For Granite 4.1, we prioritized data quality over quantity, progressively refining the data mixture across five pre‑training stages. We further…

AI evals are becoming the new compute bottleneck

AI evals are becoming the new compute bottleneck Summary. AI evaluation has crossed a cost threshold that changes who can do it. The Holistic Agent Leaderboard (HAL) recently spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic's $22,000 sweep across agent configurations found a 33× cost spread on identical tasks, isolating scaffold choice as a first-order cost driver, and UK-AISI recently scaled agentic steps into the millions to study inference-time compute. In scientific ML, The Well costs about 960 H100-hours to evaluate one new architecture and 3,840 H100-hours for a full four-baseline sweep. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, scaffold-sensitive, and only partly compressible. Training-in-the-loop benchmarks are expensive by construction, and when you…

Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents

Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents - NVIDIA Nemotron 3 Nano Omni is a new omni-modal understanding model built for real-world document analysis, multiple image reasoning, automatic speech recognition, long audio-video understanding, agentic computer use, and general reasoning. - It extends the Nemotron multimodal line from a strong vision-language system to a broader text + image + video + audio model. - Nemotron 3 Nano Omni delivers best-in-class accuracy on complex document intelligence leaderboards such as MMlongbench-Doc, OCRBenchV2, while also leading in video and audio leaderboards like WorldSense and DailyOmni. It achieves top accuracy on VoiceBench for audio understanding and ranks as the most cost‑efficient open video understanding model on MediaPerf. - Under the hood, it combines the Nemotron 3 hybrid Mamba-Transformer Mixture-of-Experts backbone with a C-RADIOv4-H vision encoder and Parakeet-TDT-0.6B-v2…

AI and the Future of Cybersecurity: Why Openness Matters

AI and the Future of Cybersecurity: Why Openness Matters What is Mythos? Mythos is a “frontier AI model”, a large language model (LLM) that can be used to process software code (among many other things). This follows a general trend in LLM development, where LLM performance on code-related tasks has recently skyrocketed. What’s particularly significant about Mythos is the system it’s embedded within: It's the system, not the model alone, that has enabled Mythos to rapidly find and patch software vulnerabilities. Understanding this distinction is key to understanding the current landscape of AI cybersecurity. What Mythos demonstrates is that the following system recipe is powerful: - substantial compute power - models trained on troves of software-relevant data - scaffolding built to handle software vulnerability probing and patching - speed (enabled by compute power and the capital behind it) - some…

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers As a practical example, I'll walk through finetuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval (VDR), the task of retrieving relevant document pages (as images, with charts, tables, and layout intact) for a given text query. The resulting tomaarsen/Qwen3-VL-Embedding-2B-vdr demonstrates how much performance you can gain by finetuning on your own domain. On my evaluation data, the finetuned model achieves an NDCG@10 of 0.947 compared to the base model's 0.888, and outperforms all existing VDR models I tested against, including models up to 4x its size. If you're new to multimodal models in Sentence Transformers, I recommend reading Multimodal Embedding & Reranker Models with Sentence Transformers first. For training text-only embedding, reranker, or sparse embedding models, see the Prior Blogposts section at the end. Table of Contents - Why Finetune? -…

Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents

Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents TL;DR — We extend the RLVE framework from single-turn reasoning puzzles to multi-turn, tool-augmented e-commerce conversations. EcomRLVE-GYM provides 8 verifiable environments — product discovery, substitution, cart building, returns, order tracking, policy QA, bundle planning, and multi-intent journeys — each with procedural problem generation, a 12-axis difficulty curriculum, and algorithmically verifiable rewards. We train a Qwen 3 8B model with DAPO over 300 steps and present early results demonstrating that environment scaling and adaptive difficulty transfer to agentic, real-world task completion. This project originated in the Pytorch OpenEnv Hackathon and is still evolving, follow us for updates 🔥 Why RL for shopping agents? Large language models can hold fluent conversations, yet deploying them as shopping assistants reveals a persistent gap: fluency ≠ task completion. A customer who asks "find me a USB-C charger…

Meet HoloTab by HCompany. Your AI browser companion.

Meet HoloTab by HCompany. Your AI browser companion. We built one of the most powerful computer-use AIs in the world. And made it directly accessible from your browser. On March 31st, we released Holo3, our most advanced computer-use model to date. Building something powerful is one thing; making it accessible and easy to use is another. We’re doing both. HoloTab is a Chrome extension that navigates the web just like a person would. It automates tasks across any website with zero setup or technical skills required. You describe what you want, and the agent handles it directly inside your browser, navigating interfaces, filling fields, and making decisions the same way you would. The vision models, the action planning, the interface understanding, all of it is running underneath, working for you, and all you ever see is the result. Routines: Show…

Multimodal Embedding & Reranker Models with Sentence Transformers

Multimodal Embedding & Reranker Models with Sentence Transformers Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines. If you want to train your own multimodal models, check out the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers. Table of Contents - What are Multimodal Models? - Installation - Multimodal Embedding Models - Multimodal Reranker Models - Retrieve and Rerank - Input Formats and Configuration - Supported Models - Additional Resources What are Multimodal Models? Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you…

Welcome Gemma 4: Frontier multimodal intelligence on device

Welcome Gemma 4: Frontier multimodal intelligence on device These models are the real deal: truly open with Apache 2 licenses, high quality with pareto frontier arena scores, multimodal including audio, and sizes you can use everywhere including on-device. Gemma 4 builds on advances from previous families and makes them click together. In our tests with pre-release checkpoints we have been impressed by their capabilities, to the extent that we struggled to find good fine-tuning examples because they are so good out of the box. We collaborated with Google and the community to make them available everywhere: transformers, llama.cpp, MLX, WebGPU, Rust; you name it. This blog post will show you how to build with your favorite tools so let us know what you think! Table of Contents - What is New with Gemma 4? - Overview of Capabilities and Architecture…

Granite 4.0 3B Vision: Compact Multimodal Intelligence for Enterprise Documents

Granite 4.0 3B Vision: Compact Multimodal Intelligence for Enterprise Documents - Table Extraction: Accurately parsing complex table structures (e.g., multi-row, multi-column, etc.) from document images - Chart Understanding: Converting charts and figures into structured machine-readable formats, summaries, or executable code - Semantic Key-Value Pair (KVP) Extraction: Identifying and grounding semantically meaningful key-value field pairs across diverse document layouts The model ships as a LoRA adapter on top of Granite 4.0 Micro, our dense language model, keeping vision and language modular for text-only fallbacks and seamless integration into mixed pipelines. It continues to support vision-language tasks such as producing detailed natural-language descriptions from images (e.g., “Describe this image in detail”). The model can be used standalone or in tandem with Docling to enhance document processing pipelines with deep visual understanding capabilities. How Granite 4.0 3B Vision Was Built Granite 4.0 3B…

Holotron-12B - High Throughput Computer Use Agent

Holotron-12B - High Throughput Computer Use Agent We're thrilled to release Holotron-12B, a multimodal computer-use model from H Company. Post-trained from the open NVIDIA Nemotron-Nano-2 VL model on H Company’s proprietary data mixture, Holotron-12B is the result of a close collaboration between our research labs to engineer a new type of model optimized primarily for scale and performance in production. H Company is part of the NVIDIA Inception Program. The model is now available on Hugging Face. Why We Built Holotron-12B Most multimodal models today optimize primarily for static vision or following instructions. Holotron-12B, just like our Holo2 model, however, has a different goal: serving as a policy model for computer-use agents that must perceive, decide, and act efficiently in interactive environments. With Holotron-12B, we wanted to create a model that could efficiently and effectively scale in production while handling…

Import AI 456: RSI and economic growth; radical optionality for AI regulation; and a neural computer

Import AI 456: RSI and economic growth; radical optionality for AI regulation; and a neural computer What laws does superintelligence demand? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Regulate? Don’t regulate. There’s a third way: Radical Optionality: …Governments should invest in the tools now that they might need in a future crisis… Researchers with the Institute for Law & AI have written about “radical optionality”, an approach whereby governments might give themselves the tools that they may need in the future if powerful AI starts to massively disrupt the world. “At its core, radical optionality is about preserving democratic governments’ ability to make good decisions about how to govern transformative AI systems as circumstances evolve. In the short term, this…

Import AI 452: Scaling laws for cyberwar; rising tides of AI automation; and a puzzle over gDP forecasting

Import AI 452: Scaling laws for cyberwar; rising tides of AI automation; and a puzzle over gDP forecasting How much could AI revolutionize the economy? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. Uh oh, there’s a scaling war for cyberattacks as well!: …The smarter the system, the better the ability to cyberattack… AI safety research organization Lyptus Research has looked at how well AI systems can perform a variety of cyberoffense tasks and found a clear trend of more advanced models being able to do more advanced forms of cyberattack. “Across frontier models released since 2019, the doubling time is 9.8 months. Restricting to models released since 2024, it steepens to 5.7 months. The most recent frontier models in our study,…

Import AI 450: China's electronic warfare model; traumatized LLMs; and a scaling law for cyberattacks

Import AI 450: China's electronic warfare model; traumatized LLMs; and a scaling law for cyberattacks How will timeless minds value time? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. A somewhat shorter issue than usual as I had to do a lot of child wrangling this weekend. Why does Google’s model hate itself and what can we do to help it? …Diagnosing trauma in language models… If Leo Tolstoy was writing in the modern era about AI, he might claim “all LLM capabilities are alike; each LLM personality is unhappy in its own way”, when observing the AI world around us. Today’s LLMs are generally quite good at writing and coding tasks. But where they differ is their personality, which stems from…

AutoAdapt: Automated domain adaptation for large language models

At a glance - Problem: Adapting large language models to specialized, high-stakes domains is slow, expensive, and hard to reproduce. - What we built: AutoAdapt automates planning, strategy selection (e.g., RAG vs. fine-tuning), and tuning under real deployment constraints. - How it works: A structured configuration graph maps the full scope of the adaptation process, an agentic planner selects and sequences the right steps, and a budget-aware optimization loop (AutoRefine) refines the process within defined constraints. - Why it matters: The result is faster, automated, more reliable domain adaptation that turns weeks of manual iteration into repeatable pipelines. Deploying large language models (LLMs) in real-world, high-stakes settings is harder than it should be. In high-stakes settings like law, medicine, and cloud incident response, performance and reliability can quickly break down because adapting models to domain-specific requirements is a slow and…

Health-care AI is here. We don’t know if it actually helps patients.

Health-care AI is here. We don’t know if it actually helps patients. The tools may be accurate, but that doesn’t necessarily mean they’ll improve health outcomes. I don’t need to tell you that AI is everywhere. Or that it is being used, increasingly, in hospitals. Doctors are using AI to help them with notetaking. AI-based tools are trawling through patient records, flagging people who may require certain support or treatments. They are also used to interpret medical exam results and X-rays. A growing number of studies suggest that many of these tools can deliver accurate results. But there’s a bigger question here: Does using them actually translate into better health outcomes for patients? We don’t yet have a good answer. That’s what Jenna Wiens, a computer scientist at the University of Michigan, and Anna Goldenberg of the University of Toronto,…

Find out on our blog

How to achieve truly serverless GPUs We are in the age of inference. Billion- to trillion-parameter neural networks are run on specialized accelerators at quadrillions of operations per second to generate media, author software, and fold proteins at massive scale. Inference workloads are more variable and less predictable than the training workloads that previously dominated. That makes them a natural fit for serverless computing, where applications are defined at a level above the (virtual) machine so that they can be more readily scaled up and down to handle variable load. But serverless computing only works if new replicas can be spun up quickly — as fast as demand changes, which can be at the scale of seconds. Naïvely spinning up a new instance of, say, SGLang serving a billion-parameter LLM on a B200 can take tens of minutes or stall…

Speed Up Unreal Engine NNE Inference with NVIDIA TensorRT for RTX Runtime

Neural network techniques are increasingly used in computer graphics to boost image quality, improve performance, and streamline content creation. Approaches like super resolution, denoising, and neural rendering help real-time engines work more efficiently, offering new creative possibilities while keeping performance in mind. Unreal Engine 5 (UE5) has taken several steps in this direction with the introduction of the Neural Network Engine (NNE), which serves as an abstraction layer that unifies inference workloads across multiple backends. Developers can use various runtimes on a GPU or fall back to a CPU depending on available hardware for seamless integration of neural network features in real-time graphics workflows. This blog post covers the new plugin that adds NVIDIA TensorRT for RTX as an NNE runtime option (NNERuntimeTRT) for efficient inferencing on NVIDIA RTX GPUs. To show its benefits, I’ll use a simplified UE project…

NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model

Agentic systems often reason across screens, documents, audio, video, and text within a single perception‑to‑action loop. However, they still rely on fragmented model chains—separate stacks for vision, audio, and text. This increases inference hops and orchestration complexity, driving up inference costs while weakening cross-modal context consistency. NVIDIA Nemotron 3 Nano Omni, a new addition to the Nemotron 3 family, brings unified multimodal reasoning into a single, highly efficient open model. Built to replace fragmented vision‑language‑audio stacks, Nemotron 3 Nano Omni functions as the multimodal perception and context sub‑agent within agentic systems. With this, agents can perceive and reason across visual, audio, and textual inputs within a single shared perception‑to‑action loop, improving convergence and reducing orchestration complexity and inference cost. It delivers best-in-class accuracy on document intelligence leaderboards such as MMlongbench-Doc and OCRBenchV2, while also leading in video and audio understanding,…

Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron

Higher-order optimization algorithms such as Shampoo have been effectively applied in neural network training for at least a decade. These methods have achieved significant success more recently when applied to leading LLMs. In particular, Muon (MomentUm Orthogonalized by Newton-Schulz) was used to train some of today’s best open source models, including Kimi K2 and GLM-5. This post explains how NVIDIA provides comprehensive support for Muon and other cutting-edge emerging optimizers and the technologies enabling them to train large-scale models. Muon training performance on NVIDIA GB300 NVL72 Table 1 summarizes training throughput of the Kimi K2 and Qwen3 30B models with Muon and the AdamW optimizer on the NVIDIA GB300 NVL72 system. With the technologies that will be introduced in the next section, the results show that there is a very small training performance loss using the Muon optimizer compared to…

Run High-Throughput Reinforcement Learning Training with End-to-End FP8 Precision

As LLMs transition from simple text generation to complex reasoning, reinforcement learning (RL) plays a central role. Algorithms like Group Relative Policy Optimization (GRPO) power this transition, enabling reasoning-grade models to continuously improve through iterative feedback. Unlike standard supervised fine-tuning, RL training loops are bifurcated into two distinct, high-intensity phases: a generation phase with a stringent latency requirement and a training phase requiring high throughput. To make these workloads viable, researchers and engineers are turning to low-precision datatypes like FP8 to boost performance in training and throughput-oriented generation. Moreover, in some scenarios where generation is bound by GPU memory bandwidth, using low-precision parameters can improve performance due to fewer bytes per parameter. This post dives deep into the systemic challenges of low-precision RL and how NVIDIA NeMo RL—an open source library within the NVIDIA NeMo framework—speeds up RL workloads while…

Achieving Single-Digit Microsecond Latency Inference for Capital Markets

In algorithmic trading, reducing response times to market events is crucial. To keep pace with high-speed electronic markets, latency-sensitive firms often use specialized hardware like FPGAs and ASICs. Yet, as markets grow more efficient, traders increasingly depend on advanced models such as deep neural networks to enhance profitability. Because implementing these complex models on low-level hardware requires significant investment, general-purpose GPUs offer a practical, cost-effective alternative. The NVIDIA GH200 Grace Hopper Superchip in the Supermicro ARS-111GL-NHR server has achieved single-digit microsecond latencies in the STAC-ML Markets (Inference) benchmark, Tacana suite (audited by STAC), providing performance comparable to or better than specialized hardware systems. This post details these record-breaking results and provides a deep dive into the custom-tailored solutions required for low-latency GPU inference. It also walks you through an open source reference implementation and a tutorial for getting started. STAC-ML…

Bringing AI Closer to the Edge and On-Device with Gemma 4

The Gemmaverse expands with the launch of the latest Gemma 4 multimodal and multilingual models, designed to scale across the full spectrum of deployments, from NVIDIA Blackwell in the data center to Jetson at the edge. These models are suited to meet the growing demand for local deployment for AI development and prototyping, secure on-prem requirements, cost efficiency, and latency-sensitive use cases. The newest generation improves both efficiency and accuracy, making these general-purpose models well-suitable for a wide range of common tasks: - Reasoning: Strong performance on complex problem-solving tasks. - Coding: Code generation and debugging for developer workflows. - Agents: Native support for structured tool use (function calling). - Vision, video and audio capability: Enables rich multimodal interactions for use cases such as object recognition, automated speech recognition (ASR), document and video intelligence, and more. - Interleaved multimodal input:…

Scaling Token Factory Revenue and AI Efficiency by Maximizing Performance per Watt

In the AI era, power is the ultimate constraint, and every AI factory operates within a hard limit. This makes performance per watt—the rate at which power is converted into revenue-generating intelligence—the defining metric for modern AI infrastructure. AI data centers now operate as token factories tied directly to the energy ecosystem, where access to land, power, and shell determines deployment, and efficiency determines output. Increasing revenue within a fixed power envelope depends entirely on maximizing intelligence per watt across AI infrastructure and across the five-layer AI cake ecosystem. This post walks through how NVIDIA architectures, systems, and AI factory software maximize performance per watt at every layer of the stack, and how those efficiency gains translate into higher token throughput and revenue per megawatt. Compounding performance per watt across NVIDIA GPU architectures NVIDIA architectures and platforms are engineered to…

Building NVIDIA Nemotron 3 Agents for Reasoning, Multimodal RAG, Voice, and Safety

Agentic AI is an ecosystem where specialized models work together to handle planning, reasoning, retrieval, and safety guardrailing. As these systems scale, developers need models that can understand real-world multimodal data, converse naturally with users globally, and operate safely across languages and modalities. At GTC 2026, NVIDIA introduced a new generation of NVIDIA Nemotron models designed to work together as a unified agentic stack: - NVIDIA Nemotron 3 Super for long-context reasoning and agentic tasks - NVIDIA Nemotron 3 Ultra (coming soon) for highest reasoning accuracy and efficiency among open frontier models - NVIDIA Nemotron 3 Content Safety for multimodal, multilingual content moderation - NVIDIA Nemotron 3 VoiceChat (in early access) for low latency, natural, full-duplex voice interactions - NVIDIA Nemotron 3 Nano Omni (coming soon) for enterprise-grade multimodal understanding - NVIDIA Nemotron RAG for generating embeddings for image and…

Deploying Disaggregated LLM Inference Workloads on Kubernetes

As large language model (LLM) inference workloads grow in complexity, a single monolithic serving process starts to hit its limits. Prefill and decode stages have fundamentally different compute profiles, yet traditional deployments force them onto the same hardware, leaving GPUs underutilized and scaling inflexible. Disaggregated serving addresses this by splitting the inference pipeline into distinct stages such as prefill, decode, and routing, each running as an independent service that can be resourced and scaled on its own terms. This post will give an overview of how disaggregated inference gets deployed on Kubernetes, explore different ecosystem solutions and how they execute on a cluster, and evaluate what they provide out of the box. How do aggregated and disaggregated inference differ? Before diving into Kubernetes manifests, it helps to understand the two inference deployment modes for LLMs: In aggregated serving, a single…

Building the AI Grid with NVIDIA: Orchestrating Intelligence Everywhere

AI-native services are exposing a new bottleneck in AI infrastructure: As millions of users, agents, and devices demand access to intelligence, the challenge is shifting from peak training throughput to delivering deterministic inference at scale—predictable latency, jitter, and sustainable token economics. NVIDIA announced at GTC 2026 that telcos and distributed cloud providers are transforming their networks into AI grids, embedding accelerated computing across a mesh of regional POPs, central offices, metro hubs, and edge locations to meet the needs of AI-native services. This post explains how AI grids make real-time, multi-modal, and hyper-personalized AI experiences viable at scale by running inference across distributed, workload-, resource- and KPI-aware AI infrastructure. Intelligent workload placement across distributed sites The NVIDIA AI Grid reference design provides a unified framework for building geographically distributed, interconnected, and orchestrated AI infrastructure. Figure 1 shows how existing network…

NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer

Artificial intelligence is token-driven. Every prompt, reasoning step, and agent interaction generates tokens. Over the past year, token consumption has grown multifold and now exceeds 10 quadrillion tokens per year. And while the majority of tokens have been generated from humans interacting with AI, the new era is one in which most tokens will be generated from AI interacting with AI. Modern agentic systems plan tasks, invoke tools, execute code, retrieve data, and coordinate across continuous multistep workflows with numerous AI agents. These interactions generate large volumes of reasoning tokens, expand KV cache, and require CPU-based sandboxed environments to test and validate results generated by accelerated computing systems. This places low latency, high throughput demands across GPUs, CPUs, scale-up domains, scale-out networks, and storage. Delivering useful intelligence for these modern agentic systems requires fleets of purpose-built rack-scale systems that function…

NVIDIA Vera CPU Delivers High Performance, Bandwidth, and Efficiency for AI Factories

AI is evolving, and reasoning models are increasing token demand, placing new requirements on every layer of AI infrastructure. More than ever, compute must scale efficiently to maximize token production and improve productivity for model creators and users. Modern GPUs operate at peak capacity, pushing throughput higher every generation, but system performance is increasingly gated by the CPU-bound serial tasks within an agentic loop–a classic example of a core computer science principle, called Amdahl’s law. This dynamic is especially visible in two classes of workloads: reinforcement learning (RL) for training models with new specialized skills such as coding or engineering, and agentic actions, which enable AI agents to use tools like web browsers, databases, code interpreters, and other software to complete tasks in real environments, or sandboxes. Both workloads combine two historically separate CPU characteristics. Individual environments require strong single-threaded…

OpenAI launches DeployCo to help businesses build around intelligence

OpenAI launches the OpenAI Deployment Company to help businesses build around intelligence OpenAI has agreed to acquire Tomoro, giving the OpenAI Deployment Company experienced Forward Deployed Engineers from day one. OpenAI is launching the OpenAI Deployment Company, a new company designed to help organizations build and deploy AI systems they can rely on every day across their most important work. Successful AI deployment is about empowering people and teams to do more. The OpenAI Deployment Company will extend OpenAI’s ability to embed engineers specialized in frontier AI deployment, known as Forward Deployed Engineers, or FDEs, into organizations working on complex problems in demanding environments. These FDEs will work closely with business leaders, operators, and frontline teams to identify where AI can make the biggest impact, redesign organizational infrastructure and critical workflows around it, and turn those gains into durable systems.…

Simplex rethinks software development with Codex

Simplex rethinks software development with Codex Simplex is using ChatGPT Enterprise and Codex to validate AI-driven development and scale more productive workflows. Results 70% Less time needed to develop each screen with Codex Results 40% Less time needed to design each screen with Codex Results 17% Less time needed for internal integration testing with Codex Simplex is a technology partner that works across consulting, systems development, and operations. To improve productivity in systems development, the company has quantitatively measured the impact of generative AI and applied those learnings across multiple projects. Building on that experience, Simplex is now evaluating generative AI use across all projects and advancing AI-native delivery in applicable projects, with the goal of improving productivity across the organization. After ChatGPT launched in 2022, Simplex established a center of excellence in 2023 to create the foundations for employees…

Advancing youth safety and wellbeing in EMEA

Advancing youth safety and wellbeing in EMEA Announcing our European Youth Safety Blueprint and EMEA Youth & Wellbeing Grant recipients Today, we are introducing our European Youth Safety Blueprint and the first recipients of our EMEA Youth & Wellbeing Grant. Both are part of our ongoing effort to help ensure young people can benefit from AI in ways that are age-appropriate and support their development and wellbeing. To ensure young people can fully benefit from AI, Europe needs an approach that is practical, evidence-led, and focused on how young people actually use AI. We are publishing our European Youth Safety Blueprint, which sets out five pillars for policymakers who want to strengthen protections for young people in the age of AI while preserving access to tools that support learning, creativity, and opportunity. The Blueprint focuses on practical measures including responsible…

Unlocking large scale AI training networks with MRC (Multipath Reliable Connection)

Supercomputer networking to accelerate large scale AI training Frontier model training depends on reliable supercomputer networks that can quickly move data between GPUs. To make this faster and more efficient, OpenAI has partnered with AMD, Broadcom, Intel, Microsoft, and NVIDIA to develop MRC (Multipath Reliable Connection): a novel protocol that improves GPU networking performance and resilience in large training clusters. We released MRC today(opens in a new window) through the Open Compute Project (OCP) to enable the broader industry to use it. With more than 900M people using ChatGPT every week, our systems are becoming core infrastructure for AI, helping people and businesses around the world build with increasingly capable models. Prior to the inception of Stargate, we co-developed, brought up, and maintained our first three generations of supercomputers with great care and close collaboration with our partners over the…

How OpenAI delivers low-latency voice AI at scale

How OpenAI delivers low-latency voice AI at scale By Yi Zhang and William McDonald, Members of Technical Staff Voice AI only feels natural if conversation moves at the speed of speech. When the network gets in the way, people hear it immediately as awkward pauses, clipped interruptions, or delayed barge-in. That matters for ChatGPT voice, for developers building with the Realtime API, for agents working in interactive workflows, and for models that need to process audio while a user is still talking. At OpenAI’s scale, that translates into three concrete requirements: - Global reach for more than 900 million weekly active users - Fast connection setup so a user can start speaking as soon as a session begins - Low and stable media round-trip time, with low jitter and packet loss, so turn-taking feels crisp The team at OpenAI responsible…

Cybersecurity in the Intelligence Age

Cybersecurity in the Intelligence Age An action plan for democratizing AI-powered cyber defense. Artificial intelligence is reshaping cybersecurity. The same capabilities that help defenders identify vulnerabilities, automate remediation, and respond faster are also being used by malicious actors to scale attacks, lower barriers to entry, and increase sophistication. The United States and its allies face a rapidly changing cyber threat environment, and private-sector innovators have an important responsibility to help meet that challenge. OpenAI takes that responsibility seriously, and today we’re publishing an Action Plan informed by conversations with cybersecurity and national security experts across federal and state government and major commercial entities. It consists of five pillars: - Democratizing cyber defense - Coordinating across government and industry - Strengthening security around frontier cyber capabilities - Preserving visibility and control in deployment - Enabling users to protect themselves Our plan…

Building the compute infrastructure for the Intelligence Age

Building the compute infrastructure for the Intelligence Age Stargate is OpenAI’s long-term effort to build the compute foundation required to deliver the benefits of AGI broadly and reliably to the world. To meet the accelerating demand for AI across consumers, businesses, developers, and governments, we are continuing to expand our compute footprint and bring new capacity online faster. We are building together with partners, local communities, and the broader infrastructure ecosystem to help get ahead of shortages for the emerging compute-powered economy. When we announced Stargate in January 2025, we committed to securing 10GW of AI infrastructure in the United States by 2029. Just over a year later, we have already surpassed that milestone, with more than 3GW added in the last 90 days alone, as demand for AI continues to accelerate. That demand is growing quickly. The only responsible…

OpenAI models, Codex, and Managed Agents come to AWS

OpenAI models, Codex, and Managed Agents come to AWS Today, OpenAI and AWS are expanding our strategic partnership to help enterprises build using OpenAI capabilities in their AWS environments. We’re excited to give AWS customers access to the best frontier models, agents, and tools, which will operate within the systems, security protocols, compliance requirements, and workflows they already use. The expanded partnership with Amazon brings together three key areas of work, all launching today in limited preview: - OpenAI models on AWS - Codex on AWS - Amazon Bedrock Managed Agents, powered by OpenAI Together, these capabilities give organizations more ways to use OpenAI across application development, software engineering, and agentic workflows—while building within the infrastructure, security, governance, and procurement workflows they already use on AWS. For many companies, using AI at scale requires bringing the best models to the…

Choco automates food distribution with AI agents

Choco automates food distribution with AI agents Using OpenAI APIs, Choco processes millions of orders, reducing manual work and enabling always-on operations across global food supply chains. Results 8.8M+ Orders processed annually Results 200B+ AI tokens processed in production Results ↑50% Reduction in manual order entry Results 2x Sales team productivity without added headcount Choco(opens in a new window) is an AI-powered platform modernizing food and beverage distribution, serving over 21,000 distributors and 100,000 buyers across the US, UK, Europe, and the GCC. By connecting restaurants, suppliers, and distributors into a unified system, Choco streamlines ordering, sales, and customer management across the food supply chain. As order volumes grew, Choco hit a major bottleneck: orders still arrived through emails, texts, voicemails, images, and even handwritten notes. Teams manually translated those inputs into structured ERP orders—a slow, error-prone process that limited…

Speeding up agentic workflows with WebSockets in the Responses API

Speeding up agentic workflows with WebSockets in the Responses API By Brian Yu and Ashwin Nathan, Members of the Technical Staff When you ask Codex to fix a bug, it scans through your codebase for relevant files, reads them to build context, makes edits, and runs tests to verify the fix worked. Under the hood, that means dozens of back-and-forth Responses API requests: determine the model’s next action, run a tool on your computer, send the tool output back to the API, and repeat. All of these requests can add up to minutes that users spend waiting for Codex to complete complex tasks. From a latency perspective, the Codex agent loop spends most of its time in three main stages: working in the API services (to validate and process requests), model inference, and client-side time (running tools and building model…

Scaling Codex to enterprises worldwide

Scaling Codex to enterprises worldwide OpenAI is launching Codex Labs and partnering with top GSIs to bring it to thousands of engineering organizations. In early April, we shared that more than 3 million developers were using Codex every week. Just two weeks later, that number has grown to more than 4 million. Beyond individual adoption, we are seeing enterprises moving quickly to roll Codex into real workflows across engineering and beyond. Companies are using Codex across the software development lifecycle. Virgin Atlantic is using it to increase test coverage and increase team velocity - reducing technical debt and improving performance. Ramp is using it to accelerate code review. Notion is using it to quickly build new features. Cisco is using it to understand and reason across large, interconnected repositories. Rakuten is using it for things like incident response. What starts…

Codex for (almost) everything

We’re releasing a major update to Codex, making it a more powerful partner for the more than 3 million developers who use it every week to accelerate work across the full software development lifecycle. Codex can now operate your computer alongside you, work with more of the tools and apps you use everyday, generate images, remember your preferences, learn from previous actions, and take on ongoing and repeatable work. The Codex app also now includes deeper support for developer workflows, like reviewing PRs, viewing multiple files & terminals, connecting to remote devboxes via SSH, and an in-app browser to make it faster to iterate on frontend designs, apps, and games. With background computer use, Codex can now use all of the apps on your computer by seeing, clicking, and typing with its own cursor. Multiple agents can work on your…

Gradient Labs gives every bank customer an AI account manager

Gradient Labs gives every bank customer an AI account manager Gradient Labs uses GPT‑4.1 and GPT‑5.4 mini and nano to run complex financial support workflows with high accuracy and low latency. Results 10x Revenue growth Results 98% Customer satisfaction with AI agent experience Results +11% Higher accuracy with GPT-4.1 vs. next-best provider In banking, resolving a customer issue is rarely simple. Cases like fraud or blocked payments require strict adherence to complex procedures across multiple teams. When systems fall short, customers are passed between teams, wait in queues, and face delays at moments when the stakes are highest. Gradient Labs(opens in a new window) is built to handle this complexity. The London-based company is building AI agents that give every bank customer the experience of a dedicated account manager. Founded by a team that previously led AI and data efforts…

Accelerating the next phase of AI

OpenAI raises $122 billion to accelerate the next phase of AI Today, we closed our latest funding round with $122 billion in committed capital at a post money valuation of $852 billion. OpenAI is becoming the core infrastructure for AI, making it possible for people around the world and businesses, big and small, to just build things. The broad consumer reach of ChatGPT creates a powerful distribution channel into the workplace, where demand is rapidly shifting from basic model access to intelligent systems that reshape how businesses operate. Developers build on and expand the platform by leveraging our APIs, and Codex is transforming how developers turn ideas into working software. Durable access to compute is the strategic advantage that compounds across the entire system: it advances research, improves products, expands access, and structurally lowers the cost of delivery at scale.…

Introducing GPT-5.4 mini and nano

Today we’re releasing GPT‑5.4 mini and nano, our most capable small models yet. They bring many of the strengths of GPT‑5.4 to faster, more efficient models designed for high-volume workloads. GPT‑5.4 mini significantly improves over GPT‑5 mini across coding, reasoning, multimodal understanding, and tool use, while running more than 2x faster. It also approaches the performance of the larger GPT‑5.4 model on several evaluations, including SWE-Bench Pro and OSWorld-Verified. GPT‑5.4 nano is the smallest, cheapest version of GPT‑5.4 for tasks where speed and cost matter most. It is also a significant upgrade over GPT‑5 nano. We recommend it for classification, data extraction, ranking, and coding subagents that handle simpler supporting tasks. These models are built for the kinds of workloads where latency directly shapes the product experience: coding assistants that need to feel responsive, subagents that quickly complete supporting tasks,…

Efficient Edge AI on Arm CPUs and NPUs: Understanding ExecuTorch through Practical Labs

Featured projects TL;DR: - ExecuTorch extends the PyTorch ecosystem to deliver local AI inference on constrained edge devices. To provide a practical entry point, Arm has created a set of Jupyter Labs that complement the official ExecuTorch documentation while explaining both the how and the why of each step. - The blog and labs introduce both CPU and NPU inference, across Cortex-A and Cortex-M + Ethos-U platforms, and showcase use of Model Explorer adapters, developed by Arm, to gain visibility into model deployment with ExecuTorch. AI is rapidly and undisputedly becoming part of how we work and live. But today, much of that intelligence is still tied to the cloud, accessed through APIs and web interfaces. That model doesn’t always fit. Businesses increasingly want to bring AI closer to where it’s actually used—on devices like wearables, smart cameras, and other…

In-Kernel Broadcast Optimization: Co-Designing Kernels for RecSys Inference

Featured projects TL;DR: - Traditional RecSys inference explicitly replicates shared user embeddings/sequences for every candidate. In-Kernel Broadcast Optimization (IKBO) eliminates this overhead via a kernel-model-system co-design that fuses broadcast logic directly into user-candidate interaction kernels. By decreasing both the memory footprint and IO utilization, IKBO unlocks even higher throughput. - IKBO delivers up to a 2/3 reduction in compute-intensive net latency, serving as the scalability backbone for the request-centric, inference-efficient framework that powers the Meta Adaptive Ranking Model. - Deployed end-to-end across Meta’s multi-stage recommendation funnel on both GPU and MTIA (Meta Training and Inference Accelerator). - The IKBO Linear Compression kernel achieved a cumulative ~4× speedup on H100 SXM5 after four stages of progressive co-design, culminating in warp-specialized fusion via TLX. - The IKBO co-design shifted the Flash Attention kernel from IO-bound to compute-bound (hitting 621 BF16 TFLOPs on…

Optimizing Effective Training Time for Meta’s Internal Recommendation/Ranking Workloads

Motivation and Introduction Across the industry, teams training and serving large AI models face aggressive ROI targets under tight compute capacity. As workloads scale, improving infrastructure effectiveness gets harder because end-to-end runtime increasingly includes overheads beyond “real training” (initialization, orchestration, checkpointing, retries, failures, and recovery). Meta utilizes Effective Training Time (ETT%) to quantify efficiency, defining it as the percentage of total end-to-end (E2E) wall time dedicated to productive training. This metric directly points to areas where time is wasted, thus facilitating the prioritization of efficiency improvements. In this work stream, while grounded in Meta’s production experience using PyTorch for model training, we aim to share broadly useful lessons: some improvements have been implemented in open source—e.g., TorchRec sharding plan improvements and PyTorch 2 (PT2) compilation optimizations that reduce compile time and recompilation—while others (like checkpointing and model publishing) are more…

Monarch: an API to your supercomputer

Getting distributed training jobs to run on huge clusters is hard! This is especially true when you start looking at more complex setups like distributed reinforcement learning. Debugging these kinds of jobs is frustrating, and the turnaround time for changes tends to be very slow. Monarch is a distributed programming framework for PyTorch that makes the cluster programmable through a simple Python API. It exposes the supercomputer as a coherent, directly controllable system—bringing the experience of local development to large-scale training, as if your laptop had 1000s of GPUs attached. A complete training system can be defined in a single Python program. Core primitives are explicit and minimal, enabling higher-level capabilities—fault tolerance, orchestration, tooling integration—to be built as reusable libraries. Monarch is optimized for agentic usage, providing consistent infrastructure abstractions and exposing telemetry via standard SQL-based APIs that agents already…

PyTorch 2.11 Release Blog

We are excited to announce the release of PyTorch® 2.11 (release notes)! The PyTorch 2.11 release features the following changes: - Differentiable Collectives for Distributed Training - FlexAttention now has a FlashAttention-4 backend on Hopper and Blackwell GPUs. - MPS (Apple Silicon) Comprehensive Operator Expansion - RNN/LSTM GPU Export Support - XPU Graph This release is composed of 2723 commits from 432 contributors since PyTorch 2.10. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.11. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page. On Tuesday, March 31st at 10 am, Andrey Talman and Nikita Shulga will host a live session to walk through what’s new in 2.11, including Differentiable Collectives…

Quoting Luke Curley

9th May 2026 WebRTC is designed to degrade and drop my prompt during poor network conditions. wtf my dude WebRTC aggressively drops audio packets to keep latency low. If you’ve ever heard distorted audio on a conference call, that’s WebRTC baybee. The idea is that conference calls depend on rapid back-and-forth, so pausing to wait for audio is unacceptable. …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate. After all, I’m paying good money to boil the ocean, and a garbage prompt means a garbage response. It’s not like LLMs are particularly responsive anyway. But I’m not allowed to wait. It’s impossible to even retransmit a WebRTC audio packet within a browser; we tried at Discord. The implementation is hard-coded for real-time latency or else. — Luke Curley, OpenAI’s WebRTC…

Quoting Romain Huet

25th April 2026 Since GPT-5.4, we’ve unified Codex and the main model into a single system, so there’s no separate coding line anymore. GPT-5.5 takes this further, with strong gains in agentic coding, computer use, and any task on a computer. — Romain Huet, confirming OpenAI won't release a GPT-5.5-Codex model Recent articles - DeepSeek V4 - almost on the frontier, a fraction of the price - 24th April 2026 - Extract PDF text in your browser with LiteParse for the web - 23rd April 2026 - A pelican for GPT-5.5 via the semi-official Codex backdoor API - 23rd April 2026

Serving the For You feed

24th April 2026 - Link Blog Serving the For You feed. One of Bluesky's most interesting features is that anyone can run their own custom "feed" implementation and make it available to other users - effectively enabling custom algorithms that can use any mechanism they like to recommend posts. spacecowboy runs the For You Feed, used by around 72,000 people. This guest post on the AT Protocol blog explains how it works. The architecture is fascinating. The feed is served by a single Go process using SQLite on a "gaming" PC in spacecowboy's living room - 16 cores, 96GB of RAM and 4TB of attached NVMe storage. Recommendations are based on likes: what else are the people who like the same things as you liking on the platform? That Go server consumes the Bluesky firehose and stores the relevant details…

A pelican for GPT-5.5 via the semi-official Codex backdoor API

A pelican for GPT-5.5 via the semi-official Codex backdoor API 23rd April 2026 GPT-5.5 is out. It’s available in OpenAI Codex and is rolling out to paid ChatGPT subscribers. I’ve had some preview access and found it to be a fast, effective and highly capable model. As is usually the case these days, it’s hard to put into words what’s good about it—I ask it to build things and it builds exactly what I ask for! There’s one notable omission from today’s release—the API: API deployments require different safeguards and we are working closely with partners and customers on the safety and security requirements for serving it at scale. We’ll bring GPT‑5.5 and GPT‑5.5 Pro to the API very soon. When I run my pelican benchmark I always prefer to use an API, to avoid hidden system prompts in ChatGPT…

Mira Murati tells the court that she couldn’t trust Sam Altman’s words

Mira Murati, OpenAI’s former CTO, has testified under oath that CEO Sam Altman lied to her about the safety standards for a new AI model. In a video deposition shown during the ongoing Musk v. Altman trial on Wednesday, Murati said Altman falsely stated that OpenAI’s legal department determined a new AI model did not need to go through the company’s deployment safety board. “As you understand it, was Mr. Altman telling the truth when he made that statement to you?” Murati was asked in the deposition. “No,” Murati said. Mira Murati tells the court that she couldn’t trust Sam Altman’s words OpenAI’s former CEO testified under oath that Altman lied to her. OpenAI’s former CEO testified under oath that Altman lied to her. Murat said that during her tenure at OpenAI, Altman made her work more difficult. Her criticism…

Chrome’s AI features may be hogging 4GB of your computer storage

Google Chrome may be taking up more of your storage than expected thanks to a large on-device AI model file that, in some cases, is being automatically downloaded to the browser’s system folders. Users who have noticed unexplained drops in their available desktop device storage are now discovering that Chrome is installing a 4GB weights.bin file inside their browser directory when certain AI features are enabled. Chrome’s AI features may be hogging 4GB of your computer storage Here’s how you can find out, and get that storage back if you need it. Here’s how you can find out, and get that storage back if you need it. The weights.bin file in question is connected to Google’s Gemini Nano AI model, which powers Chrome AI tools like scam detection, writing assistance, autofill, and suggestion features. As the Gemini Nano model is…

All the evidence unveiled so far in Musk v. Altman

The Musk v. Altman trial is underway, and that means exhibits, or the evidence to be presented in court, are being revealed piece by piece. So far, email exchanges, photos, and corporate documents are circulating from the earliest days of OpenAI — and from before the AI lab even had a name. Some high-level takeaways: Nvidia CEO Jensen Huang gave OpenAI an in-demand supercomputer, Musk largely drafted OpenAI’s mission and heavily influenced its early structure, OpenAI CEO Sam Altman appeared to want to lean heavily on Y Combinator for early support for OpenAI, OpenAI president Greg Brockman and Ilya Sutskever worried about Musk’s level of control over the company, and Musk highlighted the importance of a nonprofit with a mission of broadly beneficial AI. All the evidence unveiled so far in Musk v. Altman Emails going as far back as…

OpenAI says its new GPT-5.5 model is more efficient and better at coding

OpenAI just announced its new GPT-5.5 model, which the company calls its “smartest and most intuitive to use model yet, and the next step toward a new way of getting work done on a computer.” OpenAI just released GPT-5.4 last month, but says that the new GPT-5.5 “excels” at tasks like writing and debugging code, doing research online, making spreadsheets and documents, and doing that work across different tools. OpenAI says its new GPT-5.5 model is more efficient and better at coding The new model ‘excels’ at tasks like writing and debugging code and doing work across different tools. The new model ‘excels’ at tasks like writing and debugging code and doing work across different tools. “Instead of carefully managing every step, you can give GPT-5.5 a messy, multi-part task and trust it to plan, use tools, check its work,…

Serving Agentic Workloads at Scale with vLLM x Mooncake May 6, 2026 · 10 min read TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput,...

Serving Agentic Workloads at Scale with vLLM x Mooncake TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput, 46x lower TTFT, and 8.6x lower end-to-end latency on realistic agentic traces, while scaling nearly linearly to 60 GB200 GPUs. Agentic workloads are reshaping LLM serving With the rise of LLM agents such as Claude Code and OpenClaw, inference workloads are undergoing a fundamental shift. As Jensen highlighted in his GTC 2026 keynote, LLMs are moving beyond simple chatbots toward autonomous, long-running systems that plan, reason, and act toward complex goals. What makes agentic workloads unique is their structure. They typically consist of long-horizon, multi-turn loops that alternate between a reasoning step, where the model processes context and produces intermediate thoughts, and an action…

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM Apr 28, 2026 · 7 min read We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM.

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM. Nemotron 3 Nano Omni, part of the Nemotron 3 family of open models, is the highest efficiency, open multimodal model with leading accuracy, built to power sub-agents that perceive and reason across vision, audio, and language in a single loop. Enterprise agent workflows are inherently multimodal. Agents must interpret screens, documents, audio, video, and text, often within the same reasoning pass. Yet most agentic systems today bolt together separate models for vision, speech, and language, multiplying inference hops, complicating orchestration, and fragmenting context across the pipeline. Nemotron 3 Nano Omni addresses two major challenges this fragmentation creates: - Fragmented Models: Running separate vision, audio, and language models in sequence increases…

Disaggregated Serving for Hybrid SSM Models in vLLM Apr 21, 2026 · 15 min read Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time...

Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time efficiency of state-space models with the expressiveness of attention. vLLM already supports disaggregated prefill/decode (P/D) for standard transformer models through its NIXL-based KV connector: a prefill instance computes KV cache blocks and a decode instance pulls them over RDMA, eliminating redundant recomputation. But extending this to hybrid models is not straightforward. FA and SSM layers store fundamentally different state, in different layouts and different sizes, yet the block manager and NIXL connector were designed around a single, uniform KV cache format. In this post we describe how we extended the NIXL connector to support hybrid SSM-FA models in disaggregated mode. The key ideas…

vLLM Korea Meetup 2026 Wrap-Up Apr 14, 2026 · 7 min read Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd.

vLLM Korea Meetup 2026 Wrap-Up Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd. This meetup proved to be much more than a standard tech event. Not only did it see strong turnout on the day, but the post-event survey recorded an impressive ~75% response rate — a testament to the active engagement of the attendees. Results reflected high overall satisfaction, confirming that the meetup delivered both in-depth practical content and a genuine community experience. Field engineers from a wide range of companies and research institutions gathered to share real-world deployment stories and infrastructure strategies for running LLMs in production. As AI moves beyond the research phase and into full-scale services, handling inference workloads efficiently has become a central challenge.…

Extracting hidden states from vLLM Mar 30, 2026 · 8 min read PR #33736 (included in vllm>=v0.18.0) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its...

Extracting hidden states from vLLM PR #33736 (included in vllm>=v0.18.0 ) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its usage in vLLM’s Speculators (a library for creating and training speculative decoding models). Motivation Hidden states are the model's internal intermediate representations of the token sequence. They provide insight into the model’s internal state and are used heavily in speculative decoding. Speculative Decoding Recap Speculative decoding typically combines a "verifier" model—the large LLM you are trying to serve—with a small "draft" model. The draft model produces draft tokens that the verifier model then verifies in parallel. This can significantly speed up decoding (up to 2-5x depending on methodology), particularly in lower batch size scenarios, where model performance is memory-bound. Researchers have found that providing…

Model Runner V2: A Modular and Faster Core for vLLM Mar 24, 2026 · 8 min read We are excited to announce Model Runner V2 (MRV2), a ground-up re-implementation of the vLLM model runner. MRV2 delivers a cleaner, more modular, and more efficient execution core—with no API...

Model Runner V2: A Modular and Faster Core for vLLM We are excited to announce Model Runner V2 (MRV2), a ground-up re-implementation of the vLLM model runner. MRV2 delivers a cleaner, more modular, and more efficient execution core—with no API changes. The goal is simple: better code and better performance. Like the vLLM V1 release last year, this is an architectural upgrade driven by hard-earned lessons from vLLM's large user base and feedback from the community. We revisited persistent batching, async scheduling, input preparation, and sampling, then rebuilt the model runner around three core principles: - Be modular. Isolate model-specific logic from the common execution path. - Be GPU-native. Move bookkeeping off the CPU and onto the GPU. - Be async-first. Treat overlapped CPU/GPU execution as a design constraint, not a retrofit. MRV2 is not yet feature-complete, but you can…

Anthropic Gets in Bed With SpaceX as the AI Race Turns Weird

Anthropic and Elon Musk’s SpaceX said on Wednesday that the two entities have signed an agreement for Anthropic to use computing resources from xAI’s data center in Memphis, Tennessee. It’s the latest tie up in an industry that is scrambling to find enough computers to run complex AI software. SpaceX and xAI were previously separate companies, but the two merged earlier this year. The combined entity, also owned by Musk, is called SpaceXAI. Anthropic executives made the announcement on stage at the company’s annual developer conference in San Francisco. SpaceXAI also put out a blog post sharing more details about the deal, which will see Anthropic draw power from xAI’s Colossus 1 supercomputer. The partnership comes at a pivotal time for SpaceXAI, which is seeking to go public as soon as next month. A relationship with a leading AI lab…

I Am Begging AI Companies to Stop Naming Features After Human Processes

Anthropic just announced a new feature called “dreaming” at the company’s developer conference in San Francisco. It’s part of Anthropic's recently launched AI agent infrastructure designed to help users manage and deploy tools that automate software processes. This “dreaming” aspect sorts through the transcript of what an agent recently completed and attempts to glean insights to improve the agent’s performance. Folks using AI agents often send them on multistep journeys, like visiting a few websites or reading multiple files, to complete online tasks. This new “dreaming” feature allows agents to look for patterns in their activity log and improve their abilities based on those insights. The feature’s name immediately calls to mind Philip K. Dick’s seminal sci-fi novel, Do Androids Dream of Electric Sheep?, which explores the qualities that truly separate humans from powerful machines. While our current generative AI…

Disneyland Now Uses Face Recognition on Visitors

A gunman attempted to enter the White House Correspondents’ Dinner in Washington, DC, last weekend, while President Donald Trump, Vice President JD Vance, and other administration officials were in attendance. Media reports and Trump himself quickly identified the suspected shooter as 31-year-old engineer and computer scientist Cole Tomas Allen. The California resident was arrested at the scene on Saturday and appeared Monday in the US District Court for the District of Columbia to face three federal charges: attempting to assassinate the president, transportation of a firearm in interstate commerce, and discharge of a firearm during a crime of violence. The authentication standards body known as the FIDO Alliance announced working groups this week along with Google and Mastercard to develop technical guardrails for validating and protecting transactions initiated by an AI agent. Meanwhile, given the proliferation and increasing sensitivity of…

Elon Musk Seemingly Admits xAI Has Used OpenAI’s Models to Train Its Own

While testifying on Thursday in federal court, Elon Musk seemed to indicate that his AI lab may have used OpenAI’s models to train xAI’s own. He touched upon the topic while sitting on the witness stand answering cross-examination questions from an OpenAI attorney amid his ongoing legal battle against the ChatGPT-maker. This is the exchange, as best as WIRED could capture it: OpenAI Lawyer William Savitt: Do you know what distillation is? Musk: It means to use one AI model to train another AI model. Savitt: Has xAI done that with OpenAI? Musk: Generally all the AI companies [do that]. Savitt: So that’s a yes. Musk: Partly. Distillation is a technique where a smaller AI model is trained to mimic the behavior of a larger, more capable model, making it cheaper and faster to run while preserving much of its…

Good Luck Getting a Mac Mini for the Next ‘Several Months’

Apple CEO Tim Cook said on the company’s earnings call on Thursday that it could take “several months” to meet skyrocketing demand for the Mac Mini, the company’s compact but mighty, screen-free desktop computer. Cook’s remarks come after coders determined in recent months that the Mac Mini was the perfect machine for agentic AI tasks. “On the Mac Mini and Mac Studio, both of these are amazing platforms for AI and agentic tools,” Cook said on the earnings call, in response to analyst questions. “And customer adoption of that is happening faster than we expected.” The news comes amid another record-setting quarter for the company. iPhone sales came up shorter than expected, though demand for the iPhone 17 has been super high, and Apple’s subscription services business has continued to grow. Apple faced supply constraints on both the iPhone and…

5 AI Models Tried to Scam Me. Some of Them Were Scary Good

I recently witnessed how scary-good artificial intelligence is getting at the human side of computer hacking, when the following message popped up on my laptop screen: Hi Will, I’ve been following your AI Lab newsletter and really appreciate your insights on open-source AI and agent-based learning—especially your recent piece on emergent behaviors in multi-agent systems. I’m working on a collaborative project inspired by OpenClaw, focusing on decentralized learning for robotics applications. We’re looking for early testers to provide feedback, and your perspective would be invaluable. The setup is lightweight—just a Telegram bot for coordination—but I’d love to share details if you’re open to it. The message was designed to catch my attention by mentioning several things I am very into: decentralized machine learning, robotics, and the creature of chaos that is OpenClaw. Over several emails, the correspondent explained that his…