awesome-papers-LMsys

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

awesome-papers-LMsys

(★ 80)

Daily Arxiv Papers on LLM Systems

awesome-papers-LMsys Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Daily Arxiv Papers (LMSys)

![Static Badge](https://img.shields.io/badge/total_papers-2991-blue?logo=gitbook)
![Static Badge](https://img.shields.io/badge/update-2026.08.17-red?logo=fireship)
[![Static Badge](https://img.shields.io/badge/arXiv-cs.DC-green)](https://arxiv.org/list/cs.DC/recent)
[![Static Badge](https://img.shields.io/badge/arXiv-cs.OS-green)](https://arxiv.org/list/cs.OS/recent)
[![Static Badge](https://img.shields.io/badge/arXiv-cs.LG-green)](https://arxiv.org/list/cs.LG/recent)

`Fetch from arxiv` → `LLM Filter` → `GitHub workflow update`

## [🔥Daily Arxiv: LLM Systems 👉 paper.tju.chat 👈](https://paper.tju.chat)

</div>

**👍Conference Papers on LMSys**: [conference.md](conference.md)

**⚠️NOTE**: Update papers up to last day every morning (8:00 UTC+8) automatically.

**🙋WANT**: Keyword subscription (email); Functional web page.

**🔖TAGS**:`serving` `training` `offline` `thinking` `RL` `MoE` `RAG` `video` `multi-modal` `sparse` `quantization` `offloading` `hardware` `storage` `kernel` `diffusion` `agentic` `edge` `networking`

---
### 2026-08-17
* `RL` `multi-modal` `training` [Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training](http://arxiv.org/abs/2608.14498v1)
  > **TL;DR**: Improves RL post-training efficiency for vision-language models by overlapping prefix computation with rollout decoding. Rollplex enables cross-phase spatial sharing via phase-aware memory management and parallelism-aware weight sharing. Achieves up to 2.24× speedup over disaggregation on 32 GPUs.
* `MoE` `serving` `kernel` [TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes](http://arxiv.org/abs/2608.13057v2)
  > **TL;DR**: Addresses load imbalance in expert-parallel MoE serving due to variable GPU latency across memory- and compute-bound regimes. Proposes TEMPO, a makespan-aware dispatcher using a max-affine cost model to optimize expert assignment. Achieves 4–6% higher throughput and ~15.6% lower p99 latency for Qwen3-235B.
* `training` `kernel` [A Barrier-Free Synchronization Algorithm for Multi-Engine AI Accelerators](http://arxiv.org/abs/2608.13757v1)
  > **TL;DR**: Addresses synchronization bottlenecks in multi-engine accelerators for loop-heavy ML kernels. Proposes a barrier-free algorithm with dynamic thresholds based on runtime loop iteration counts. Achieves 10-45% latency reduction over barrier-based baselines and 3.3x speedup on a synchronization-bound microbenchmark.
* `serving` `kernel` [CoRun: Padding is Simple and Efficient for Deterministic LLM Inference](http://arxiv.org/abs/2608.14376v1)
  > **TL;DR**: Addresses non-determinism in LLM inference due to batch-dependent GPU execution. CoRun uses isolated prefill and fixed-shape batched decode with CUDA graphs for deterministic scheduling. Achieves 15-324% higher throughput, 51.8% lower time-to-first-token, and 48.6% lower time-per-output-token versus batch-invariant methods.

### 2026-08-14
* `serving` `scaling` [OpScale: Operator-level Provisioning and Autoscaling for LLM Serving](http://arxiv.org/abs/2608.13499v1)
  > **TL;DR**: Addresses cost-efficiency and SLO violations in LLM serving via OpScale, an operator-level autoscaling framework. Leverages operator heterogeneity for fine-grained elastic scaling. Reduces GPU usage by 36.3%, power by 28%, or increases throughput by 44% under fixed cost.
* `serving` `offloading` [vToken: Token-Level Virtualization for Reclaimable KV Caches](http://arxiv.org/abs/2608.13263v1)
  > **TL;DR**: Addresses KV cache memory inefficiency in LLM serving due to intra-block fragmentation from token-level eviction policies. Introduces vToken, a token-level virtualization layer with token-table indirection and asynchronous repacking. Reduces retained KV blocks per request by 27.2%--72.3% and improves throughput by up to 1.37x.
* `serving` `MoE` [TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes](http://arxiv.org/abs/2608.13057v1)
  > **TL;DR**: Addresses load imbalance in MoE serving due to experts operating in memory- or compute-bound regimes. Introduces TEMPO, a makespan-aware dispatcher that solves a fixed-charge problem. Achieves up to 15.5% lower block time, 4-6% higher throughput, and 15.6% lower p99 latency for Qwen3-235B.
* `serving` `scaling` [InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers](http://arxiv.org/abs/2608.12915v1)
  > **TL;DR**: Addresses sustainable deployment planning for geo-distributed LLM inference. Proposes InFactPlanner, a framework combining hardware profiles, renewable models, and traces to estimate carbon, energy, water, and latency. Validates energy estimates within 10% of reference values.
* `edge` `serving` `multi-modal` [A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings](http://arxiv.org/abs/2608.12745v1)
  > **TL;DR**: Proposes a cloud-edge system for efficient medical AI in bandwidth-limited settings: edge models preprocess multimodal clinical data into compact representations, cloud LLM synthesizes outputs. Achieves bandwidth-invariant latency (25-35s) at 4-15x lower token cost versus cloud-only baselines while matching accuracy.
* `kernel` [A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family](http://arxiv.org/abs/2608.12700v1)
  > **TL;DR**: Develops a contract-grade verifier with 12 adversarial tests to detect silent errors in LLM-generated GPU kernels that pass naive validation. Flags 62.1% of 2,638 previously accepted kernels as faulty, revealing critical gaps in kernel correctness. Also implements first native Blackwell backward kernel for GLR training.
* `serving` `offloading` [vToken: Token-Level Virtualization for Reclaimable KV Caches](http://arxiv.org/abs/2608.13263v1)
  > **TL;DR**: vToken addresses intra-block fragmentation in KV caches by decoupling token liveness from physical blocks via token virtualization and asynchronous repacking. It reduces retained KV blocks by 27.2%-72.3% and improves SLA-constrained throughput by up to 1.37x in vLLM.

### 2026-08-13
* `training` `MoE` `RL` [RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning](http://arxiv.org/abs/2608.12146v1)
  > **TL;DR**: Addresses training bottlenecks in MoE RL models from imbalanced attention and expert work. Proposes RoutePack, coordinating expert rerouting and data packing to minimize EDP shard costs. Achieves up to 14.89% higher token throughput over baseline.
* `agentic` `serving` [Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control](http://arxiv.org/abs/2608.12123v1)
  > **TL;DR**: Improves LLM-agent efficiency by formalizing ready-cohort boundaries for GPU execution and avoiding host round trips. Uses exact packing and device-resident decisions, recovering 81.83% of GPU opportunity and achieving speed-ups of up to 2.39x.
* `serving` `edge` `scaling` [User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling](http://arxiv.org/abs/2608.11840v1)
  > **TL;DR**: Proposes a collaborative distributed inference system combining dedicated and user resources for efficient AI serving. Uses a generative Markov model for scheduling and QoS-aware resource allocation. Simulations show improved request completion, P99 latency, and substantial reduction in dedicated resource consumption.
* `serving` `agentic` `kernel` [Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control](http://arxiv.org/abs/2608.12123v1)
  > **TL;DR**: Investigates GPU utilization and host round-trip reduction in LLM-agent control. Formalizes ready-cohort boundaries for GPU batching and a device-resident path to avoid data transfer. Achieves 81.83% opportunity recovery in cohort packing and 1.19x-2.39x speedups in device-resident decisions.
* `serving` `offloading` `kernel` [The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks](http://arxiv.org/abs/2608.12114v1)
  > **TL;DR**: Eliminates the 'ingestion tax' of redundant weight copying during LLM inference by adopting file-backed weights via zero-copy GPU import. Proposes a contract ensuring direct access and GPU ordering. Achieves 7.14 tok/s (near parity) for Qwen2.5-72B while enabling N processes to share one copy (5.5 vs 0.08 tok/s).
* `serving` `MoE` `offloading` [Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference](http://arxiv.org/abs/2608.12103v1)
  > **TL;DR**: Proposes kernel-managed caching for trillion-parameter MoE inference expert tiering. Tests against router traces, showing kernel LRU matches an oracle frequency table (75.3% vs 74.6% hit rate) and achieves 1.09-1.10× decode speedup with token-identical outputs.

### 2026-08-12
* `serving` `RL` `offloading` [Scheduling Mixed RL Rollouts Beyond Prefix Locality](http://arxiv.org/abs/2608.11152v1)
  > **TL;DR**: Addresses scheduling for heterogeneous RL rollouts (RLHF, RLVR, agentic) in shared inference services. Proposes MISA-T, a routing-layer policy with adaptive admission and KV-capacity allocation. Improves rollout throughput by 53.3% and reduces mean iteration time by 22.8% over baselines.
* `training` `scaling` [SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training](http://arxiv.org/abs/2608.11034v1)
  > **TL;DR**: Addresses failure localization in distributed LLM pre-training where synchronization obscures root causes. Proposes SCOUT, using consensus-based outlier detection (C3) to identify rank-level discrepancies during runtime. Maintains out-of-band observability during hangs and certifies checkpoint integrity to enhance training stability.
* `serving` `kernel` [Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference](http://arxiv.org/abs/2608.08730v2)
  > **TL;DR**: Profiles WebGPU dispatch overhead as latency bottleneck for browser-based LLM inference at small batches. Introduces sequential-dispatch measurement method correcting synchronization confounds. Proves dispatch count reduction is more critical than kernel optimization for performance, guiding engine/specification improvements.
* `agentic` `networking` `scaling` [Conversational Orchestration for Organic 6G](http://arxiv.org/abs/2608.10714v1)
  > **TL;DR**: Addresses cross-domain orchestration in 6G networks with dynamic domain churn. Proposes a decentralized framework of LLM-driven agents that exchange summaries via A2A overlay and use reachability advertisements for placement. Achieves near-linear overhead scaling during domain joins and robust recovery after objective changes.
* `edge` `networking` `serving` [ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover](http://arxiv.org/abs/2608.10545v1)
  > **TL;DR**: Addresses how to maintain inference continuity in edge LLM serving during multi-user handover with limited backhaul. Proposes importance-aware KV cache ordering and a convex water-filling allocator. Achieves >93.7% accuracy within 0.5pp of full-cache in a 500ms window.
* `training` `RL` `agentic` [TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling](http://arxiv.org/abs/2608.10402v1)
  > **TL;DR**: Addresses low training goodput in agentic RL due to pauses and recomputation. Proposes TideRL with readiness-aware elastic scheduling, Continuous Task Batching, and Ref-Actor pipelining to preserve rollout state and reduce overhead. Improves RL training goodput by 5.6× and KV cache hit rate by 1.58×.
* `kernel` `quantization` [Hand-Written PTX Tensor-Core GEMM Kernels: A Multi-Precision Study on NVIDIA L4](http://arxiv.org/abs/2608.10103v1)
  > **TL;DR**: Compares hand-written PTX Tensor Core GEMM kernels with WMMA API on NVIDIA L4. Achieves 1.4x-1.8x speedup for INT8 and 2.9x-4.3x for INT4 by reducing instruction counts and avoiding software emulation, with performance driven by memory-system behavior.
* `edge` `serving` `offloading` [MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices](http://arxiv.org/abs/2608.10362v1)
  > **TL;DR**: Addresses high draft model switching overhead in adaptive speculative decoding under edge memory constraints. Proposes MemSpec, a runtime with prediction-guided draft selection and proactive working-set management. Achieves 40.7% higher throughput than state-of-the-art adaptive methods on Jetson Orin Nano.

### 2026-08-11
* `serving` [Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching](http://arxiv.org/abs/2608.09444v1)
  > **TL;DR**: Introduces continuous depth batching (CDB) for efficient adaptive-depth inference in looped LMs, scheduling loop iterations and boundary stages in separate queues with ahead exit decisions. Achieves up to 99% of theoretical speed-up, 1.5-1.9× higher offline throughput, and 45-90% lower latency under dynamic serving.
* `edge` `sparse` `quantization` [UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge](http://arxiv.org/abs/2608.09291v1)
  > **TL;DR**: Addresses inefficiencies in low-bit sparse LLM inference on edge devices caused by sparse metadata overhead. Introduces UnionSparse framework with index-efficient bitmap encoding and a shared-memory parallel SpMM kernel. Achieves up to 3.46x speedup over cuBLAS Tensor Core under W4A4 quantization.
* `kernel` `training` [SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization](http://arxiv.org/abs/2608.09160v1)
  > **TL;DR**: Mitigates communication bottlenecks in Tensor Parallelism (TP) for Query-Key Normalization (QK-Norm) during LLM training. Introduces SwiftQK, a multi-GPU RMSNorm kernel that exchanges scalars and overlaps peer-to-peer reduction with computation. Achieves 81.4-93.9% lower QK-Norm latency than All-Gather TP baseline.
* `serving` `kernel` [Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference](http://arxiv.org/abs/2608.08730v1)
  > **TL;DR**: Measures WebGPU dispatch overhead for LLM inference and identifies it as the primary bottleneck at batch size 1. Proposes reducing dispatch count via amortization for optimization. Shows overhead is data-type invariant through sequential dispatch measurements.
* `hardware` `networking` `serving` [C2C-Explorer: An Exploration Framework for Chip-to-Chip Interconnect Architectures in LLM Cloud Computing Systems](http://arxiv.org/abs/2608.08611v1)
  > **TL;DR**: Proposes C2C-Explorer, a Bayesian framework for chip-to-chip interconnect co-design in LLM systems with workload-driven traffic generation and scalable simulation. Achieves 44.1% higher goodput for DeepSeek inference.
* `MoE` `serving` `storage` [FlashBoot: Sub-Second Weight Loading for Large Models at Rack Scale](http://arxiv.org/abs/2608.08482v1)
  > **TL;DR**: Addresses slow weight loading delay in elastic MoE deployments. Designs FabricArena for contiguous memory and FlashClone for zero-setup remote replication. Achieves >270x faster concurrent rack-scale loading (0.32s vs 87s).
* `networking` [PSP: Low-Overhead Packet-Level Load Balancing for Stale-State and Bandwidth-Asymmetric Networks](http://arxiv.org/abs/2608.08425v1)
  > **TL;DR**: Addresses packet-level load balancing to mitigate congestion and tail latency in AI data center networks. Proposes probabilistic state-proportional (PSP) dispatching with band-based discrete states and local probability mapping. Outperforms JSQ and Random in loss rate and 99th-percentile buffer occupancy.
* `serving` [LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving](http://arxiv.org/abs/2608.08382v1)
  > **TL;DR**: Proposes LLMVisor, a roofline-guided latency attribution model for multi-tenant LLM serving to decompose batch latency per-request. Uses piecewise-linear modeling of FLOPs and memory I/O. Achieves up to 4.4x lower relative error at p99 for decode phases.
* `RAG` `serving` [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](http://arxiv.org/abs/2608.08340v1)
  > **TL;DR**: Proposes OpRAG, a distributed runtime for RAG workflows that models stages as operators and uses Arrow zero-copy, prefetching, batching, and overlapped execution. Achieves up to 59.62% retrieval latency reduction and 53.55% generation latency reduction in query serving with 100% recall.
* `RAG` `serving` [SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systems](http://arxiv.org/abs/2608.08237v1)
  > **TL;DR**: Addresses meeting strict SLOs on tail latency and cost in production RAG systems with variable query difficulty. Proposes SAGE, an adaptive retrieval policy using lightweight features and imitation learning to set passage count per query. Achieves 95% SLO compliance (vs 30%), 36% lower P95 latency, 51% cost reduction with 2% EM loss.
* `offloading` `serving` [OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching](http://arxiv.org/abs/2608.08097v1)
  > **TL;DR**: Addresses KV cache memory bottleneck in LLM decoding. Proposes OasisKV with lookahead sparse prefetching to offload non-critical tokens to higher-capacity memory, staged via attention pipeline. Achieves 2.1x throughput gain over dense vLLM with 6.5-9.7x less KV cache usage.
* `training` [ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling](http://arxiv.org/abs/2608.07974v1)
  > **TL;DR**: Proposes ZeroLock, a BP-free algorithm for LLM fine-tuning at the edge, using local objective construction to decouple updates. This reduces activation storage and enables concurrent training. Achieves 26.5% memory reduction and 4.9% higher throughput compared to backpropagation-based methods.
* `training` `offline` `scaling` [ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters](http://arxiv.org/abs/2608.07971v1)
  > **TL;DR**: Addresses GPU underutilization from rigid allocations in clusters running deep learning training and offline LLM inference. Introduces ElastiCo with resource shape transformation, elastic shadow pricing, and interference-aware co-location for elastic GPU sharing. Increases cluster throughput by 2.02x and average GPU utilization from 25% to 46%.

### 2026-08-10
* `serving` `offloading` `kernel` [HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management](http://arxiv.org/abs/2608.07009v1)
  > **TL;DR**: HiSparse addresses KV cache memory overflow in sparse-attention decoding with hierarchical management: full context in host memory, small GPU cache, and fused CUDA kernel for efficient selection resolution. Achieves up to 4.7x higher throughput on long-context workloads.
* `training` [StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence](http://arxiv.org/abs/2608.06838v1)
  > **TL;DR**: Addresses the challenge of efficiently parallelizing training for linear recurrent models with long sequences. Proposes StateFlow, a sequence pipeline parallelism system that partitions sequences into chunks with state propagation, profile-guided nonuniform chunking, and overlapping. Achieves up to 2.22× throughput improvement and 2.45× memory reduction compared to conventional pipeline parallelism.
* `quantization` `serving` [CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights](http://arxiv.org/abs/2608.06763v1)
  > **TL;DR**: Proposes CubicQuant, a parametric non-uniform quantization format for LLM weights, enabling flexible levels via cubic mapping while preserving dense integers for GPU efficiency. Achieves up to 28.14% lower RMSE on Laplace samples versus uniform quantization with G=128.
* `serving` `offloading` [Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving](http://arxiv.org/abs/2608.06557v1)
  > **TL;DR**: Cascade introduces a per-request latency budget to jointly optimize scheduling and KV-cache management in LLM serving. It prioritizes urgent requests and intelligently manages KV state across memory tiers. Achieves up to 2.4x higher goodput and 40% fewer SLO violations than vLLM.

### 2026-08-07
* `serving` `networking` [Routing LLM Inference to the Cleanest Grid in Real Time](http://arxiv.org/abs/2608.06188v1)
  > **TL;DR**: Investigates how to reduce carbon emissions of LLM inference via spatial routing. Proposes real-time, MOER-driven request placement across grid regions, validated in multi-region GPU testbeds. Achieves 50.9% lower operational emissions versus round-robin in modeled replay.
* `training` `networking` [ML-for-ML](http://arxiv.org/abs/2608.06046v1)
  > **TL;DR**: Addresses network resource contention in shared cloud clusters during AI training. Proposes ML-for-ML, a cross-layer co-optimization of ML and network parameters for time-to-target-loss. Achieves up to 42% faster training.
* `serving` `storage` [TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure](http://arxiv.org/abs/2608.06007v1)
  > **TL;DR**: Identifies the need for a unified tensor management layer in LLM infrastructure. Proposes TensorCast, a distributed TaaS system that decouples tensor state management from computation via programmable primitives and runtime. Reduces median TTFT by up to 93.2% in multi-turn agent workloads.
* `training` [Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening](http://arxiv.org/abs/2608.05944v1)
  > **TL;DR**: Reports operational experience for full fine-tuning a 32.76B model on two-node NVIDIA B300 using FSDP/ZeRO-3. Introduces power-draw triage for issue diagnosis and a pre-run invariant gate to prevent deadlocks. Achieves near-linear strong scaling and saves GPU-hours by converting silent failures into immediate rejections.
* `serving` `LoRA` `offloading` [PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving](http://arxiv.org/abs/2608.05483v1)
  > **TL;DR**: Addresses efficient multi-LoRA serving by introducing PLoRA, an NDP-enhanced pooled-memory system that offloads adapters and KV cache to near-data processing. Achieves 6.6x lower decode latency than S-LoRA when serving 1000 adapters on an H100.
* `RAG` `storage` [Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN](http://arxiv.org/abs/2608.05441v1)
  > **TL;DR**: Addresses filtered vector search inefficiency by composing table-format pruning and per-file IVF index in lakehouse. Embeds indices in Parquet footers and uses predicate-driven file pruning before ANN. Achieves 94x speedup on 5M embeddings with region-partitioned layout, reducing latency from 14.7s to 157ms.

### 2026-08-06
* `serving` `offloading` `networking` [RAC: Reference-Aware Activation Compression for Communication-Efficient Split LLM Inference](http://arxiv.org/abs/2608.04991v1)
  > **TL;DR**: Addresses communication bottlenecks in split LLM inference. Proposes RAC, a codec that retrieves historical spans and employs quantized alignment to compress activations. Reduces time per token by up to 2.79× with task score changes within −0.4/+2.5 points across nine model-link pairs.
* `serving` `edge` `networking` [AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks](http://arxiv.org/abs/2608.04974v1)
  > **TL;DR**: Addresses uplink constraints in cloud-edge speculative decoding. Proposes AsymSpec with asymmetric verification to compact uplink messages and confirmed-prefix pipelining for efficient scheduling. Achieves 2.82-28.03× higher throughput than baselines.
* `MoE` `serving` `disaggregation` [AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation](http://arxiv.org/abs/2608.04502v1)
  > **TL;DR**: Examines if attention-FFN disaggregation (AFD) yields higher throughput than collocated MoE deployments under constraints. Proposes AFD-Ledger, an analytical offline system for optimal hardware assignment. Reduces provisioning evaluations by 68.8%-83.5% while preserving optimal deployment selection accuracy.
* `agentic` `serving` `scaling` [Architectural Implications of Agentic AI Workflows](http://arxiv.org/abs/2608.04458v1)
  > **TL;DR**: Studies architectural inefficiencies in agentic AI workflows, exposing fragmented execution and resource mismatches. Proposes Agora, a server prototype with dynamic resource harvesting, GPU memory oversubscription, and role-based scheduling. Improves server utilization and throughput while preserving agent tail latency.

### 2026-08-05
* `training` [Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training](http://arxiv.org/abs/2608.03880v1)
  > **TL;DR**: Investigates whether Model FLOPs Utilization (MFU) predicts GPU power for LLM training. Proposes a linear MFU-based power model validated on nearly 3000 training runs. Fitting per-(GPU, dtype, batch size) reduces mean prediction error from ~10% to ~1%.
* `disaggregation` `serving` `agentic` [When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference](http://arxiv.org/abs/2608.03741v1)
  > **TL;DR**: Simulates disaggregated serving for agentic LLMs to optimize hardware per stage (prefill, decode, attention, FFN). Proposes HeteroPanacea, a cross-stack framework with quantization, scheduling, and NPU heterogeneity. Achieves up to 75% higher throughput than traditional serving.
* `MoE` `training` `networking` [TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Training](http://arxiv.org/abs/2608.03676v1)
  > **TL;DR**: Addresses load imbalance and communication overhead in MoE training. Proposes TAOT, a method using topology-aware optimal transport to schedule expert replicas and overlaps weight transfer with computation. Achieves 1.43x training speedup and up to 74% reduction in expert-communication cost.
* `kernel` `sparse` `training` [LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing](http://arxiv.org/abs/2608.01662v2)
  > **TL;DR**: Proposes LongCat Sparse Attention (LSA) to address inefficient memory access and compute overhead in sparse attention. Introduces hardware-algorithm co-design with streaming, cross-layer, and hierarchical indexing. Enables training of 1.6T-parameter models with 1M context tokens at full attention performance.
* `edge` `sparse` `networking` [Pruning-Aware Multi-Cluster Co-Inference for Large AI Models in AI-RANs](http://arxiv.org/abs/2608.03026v1)
  > **TL;DR**: Investigates the joint optimization of pruning and collaborative inference to minimize distortion in multi-cluster edge networks. Proposes a framework with pruning-aware scheduling, bandwidth allocation, and power control. Achieves superior inference accuracy and resource efficiency versus benchmarks.
* `serving` `MoE` `offloading` [AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding](http://arxiv.org/abs/2608.02989v1)
  > **TL;DR**: Introduces AcceptMoE, a speculative decoding verifier for MoE models that reduces expert activation and offloading traffic by combining router scores with commitment probabilities and cache-awareness. Achieves 2.06x throughput under offloading and reduces host-to-device traffic by up to 77.1%.
* `training` `networking` [Configurable and Hierarchical Allreduce](http://arxiv.org/abs/2608.02884v1)
  > **TL;DR**: Addresses high latency and inefficiency in hierarchical MPI_Allreduce for small-medium messages. Proposes CHIARA, a configurable hierarchical Allreduce with batch-lane topology and staged execution via rotating-root lanes and layout preservation. Achieves up to 13.48x speedup over vendor MPI_Allreduce on Fugaku.

### 2026-08-04
* `serving` `agentic` [TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving](http://arxiv.org/abs/2607.29678v2)
  > **TL;DR**: Addresses tokenization bottlenecks in agentic LLM serving by introducing TokTier, a stateful CPU+GPU tokenization service with incremental repair and GPU tokenization. Achieves 491× faster tokenization (1M chars in 0.87ms vs HF) and reduces median time to first token by 16-34%.
* `serving` [Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling](http://arxiv.org/abs/2608.02244v1)
  > **TL;DR**: Addresses inefficiency in batched LLM serving where heterogeneous requests impose max-driven costs via KV-cache imbalance. Proposes the ISJL scheduling algorithm that enforces a fairness constraint on decode progress disparity. Achieves a competitive ratio of 3/4 and improves throughput while aligning cost with token-metered revenue.
* `serving` `edge` [Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO](http://arxiv.org/abs/2608.02031v1)
  > **TL;DR**: Addresses LLM inference scheduling under soft deadlines with task dependencies in collaborative edge servers. Proposes a transformer-enhanced PPO framework for task migration to maximize completion within deadlines while minimizing extensions. Significantly outperforms baselines in task completion rate and system efficiency.
* `serving` `edge` `multi-modal` [TALSC: Timeliness-Aware Large-Small VLM Collaboration for Infrastructure-Assisted Autonomous Driving](http://arxiv.org/abs/2608.01998v1)
  > **TL;DR**: Addresses the latency-accuracy trade-off in collaborative multi-modal model inference for autonomous driving. Proposes TALSC, an online scheduling algorithm using Lyapunov optimization based on Age of Information. Achieves up to 12.6% normalized improvement in Micro-F1 score over baselines.
* `serving` `disaggregation` [Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling](http://arxiv.org/abs/2608.01891v1)
  > **TL;DR**: Investigates energy-optimal LLM serving via disaggregated Attention and FFN with operator-level DVFS. Proposes AFlex, featuring a global scheduler, local DVFS controller, and interleaved pipeline with dynamic microbatching. Achieves up to 49% lower energy per token while satisfying latency SLOs.
* `serving` `multi-modal` [HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving](http://arxiv.org/abs/2608.01785v1)
  > **TL;DR**: Proposes HorizonServe for scheduling omni-model requests with shared backbone and divergent stages under heterogeneous SLOs. Coordinates request admission and GPU allocation via profiling, slack protection, stage rotation, and SM throttling. Achieves up to 7.0× SLO attainment and reduces first-response latency by 38.4-63.7%.
* `sparse` `kernel` `training` [LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing](http://arxiv.org/abs/2608.01662v1)
  > **TL;DR**: Addresses O(L²) scoring overhead and discontinuous memory access in sparse attention. Proposes hardware-aligned streaming-aware layout, cross-layer index reuse, and hierarchical scoring. Supports training up to 1M tokens at full-attention performance for models up to 1.6T parameters.
* `serving` `offloading` [Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Caches](http://arxiv.org/abs/2608.01657v1)
  > **TL;DR**: Addresses the admission-responsibility gap in multi-tenant LLM prefix caches. Introduces PrefixShield with responsibility metering, reuse gating, and debt-based eviction, improving victim cache hit ratio by 9.39 points over LRU and restoring it to 84.87% at scale.
* `serving` `offloading` [PrefixPlace: Provable Prefix Key-Value Placement for Large Language Model Serving under Heterogeneous Compute and Transfer Costs](http://arxiv.org/abs/2608.01655v1)
  > **TL;DR**: Proposes PrefixPlace, an epoch-level planner for optimal prefix KV cache placement under heterogeneous compute and transfer costs in LLM serving. The method efficiently minimizes materialization costs via decomposition and exact assignment algorithms. Achieves 40.3% cost reduction over vLLM-APC in RAG.
* `serving` `kernel` [Bole: Efficient Tree Speculation for Hybrid-Attention Language Models](http://arxiv.org/abs/2608.01651v1)
  > **TL;DR**: Proposes Bole, a kernel-runtime co-design for efficient tree speculation in hybrid-attention LLMs by parallelizing verification and encoding speculative states. Achieves up to 4.72x offline decode throughput versus autoregressive decoding and reduces transient state memory by 82-99x.
* `training` `kernel` `edge` [Meganeura: Portable GPU Training and Inference through Vulkan and Metal](http://arxiv.org/abs/2608.01563v1)
  > **TL;DR**: Proposes Meganeura, a portable compiler and runtime for training and inference on consumer GPUs via Vulkan and Metal. Achieves 0.1-2.4s compilation (vs PyTorch's 6-96s) and three training workloads faster on an AMD GPU.
* `training` [NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training](http://arxiv.org/abs/2608.01449v1)
  > **TL;DR**: Proposes NIXT, a tool for analyzing NCCL Inspector data to improve observability of collective communication during LLM training. Enables attribution of performance variation and root cause analysis. Demonstrated for Nemotron-4 pretraining on up to 2,048 GPUs, showing communication phase changes with parallelism.
* `training` `diffusion` `video` [Zellige: Moldable Sequence Placement for Mixed Image-Video DiT Training](http://arxiv.org/abs/2608.01150v1)
  > **TL;DR**: Addresses mixed-length sequence training in Diffusion Transformers for image-video data. Introduces Zellige, a moldable placement system with hardware-profiled scheduling and coalesced attention engine. Achieves 1.12--1.54× speedup over KnapFormer on 16-32 GPU setups.
* `edge` `RAG` `serving` [From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG](http://arxiv.org/abs/2608.00922v1)
  > **TL;DR**: Proposes DEFRAG, a decentralized edge system for RAG that compresses and shares knowledge graphs and adaptively selects SLMs and RAG parameters per query. Achieves up to 98.4% cost reduction and 97.8% throughput increase over centralized services.
* `hardware` `networking` `training` [Shiftfly: Scaling the Accelerator Interconnect Past the Pod with a Shift-Routed Optical Tier](http://arxiv.org/abs/2608.00897v1)
  > **TL;DR**: Proposes Shiftfly to address interconnect scaling beyond a single pod. It replaces the global tier with a Kautz topology on an optical switch, enabling flat addressing and shift routing. At 400,000 chips, it reduces worst-case distance from 23 to 19 hops and improves spectral expansion by 2.7x.
* `serving` `networking` [NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems](http://arxiv.org/abs/2608.00867v1)
  > **TL;DR**: Mitigates non-uniform network access in multi-die GPU systems for ML inference. Introduces NUNA-aware routing and placement to optimize inter-GPU communication paths and data locality. Achieves up to 1.8x faster collectives and 28% max time per output token speedup.
* `edge` `offloading` [Collaborative Orbital Edge Intelligence: A Decentralized Paradigm for Energy-Efficient Computing in Space](http://arxiv.org/abs/2608.00487v1)
  > **TL;DR**: Proposes Collaborative Orbital Edge Intelligence (COEI) for energy-efficient satellite computation. Introduces decentralized architecture for task offloading across constellations. Case study shows maximizing task success rate while minimizing collective battery depth-of-discharge to reduce energy waste.
* `RL` `agentic` `sparse` [AReaL-DTE: Sparse Policy-Weight Transfer for Online Agentic Reinforcement Learning](http://arxiv.org/abs/2608.00455v1)
  > **TL;DR**: Addresses the systems overhead of frequent policy-weight synchronization in online agentic RL. Introduces AReaL-DTE, a snapshot-free engine that transfers sparse weight changes by reconstructing overwritten weights via AdamW inversion. Achieves up to 19.9x speedup and 41% peak GPU memory reduction.
* `training` `serving` `hardware` [HCCL: Collective Communication for Meta Training and Inference Accelerators](http://arxiv.org/abs/2608.00358v1)
  > **TL;DR**: Proposes HCCL, a collective communication library for Meta's MTIA accelerator, to reduce communication overhead in large-scale training and inference. Uses compiled dependency descriptions and topology-aware algorithms for optimized compute-communication overlap. Achieves 940 GB/s on collectives with <0.5% degradation in compute throughput.

### 2026-08-03
* `serving` `agentic` [TokTier: Exact Stateful Tokenization for Agentic LLM Serving](http://arxiv.org/abs/2607.29678v1)
  > **TL;DR**: Addresses high tokenization latency for agentic LLM serving due to repeated full-text processing. TokTier provides stateful tokenization with incremental splicing and GPU-accelerated full tokenization. Achieves up to 491x faster full tokenization on GPU compared to Hugging Face.
* `serving` `scaling` [SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving](http://arxiv.org/abs/2607.29575v1)
  > **TL;DR**: Examines throughput saturation in LLM serving under increased batch sizes, attributing it to attention kernel and DRAM bandwidth limits. Proposes SLIM, a lightweight performance model, and the Batching Configuration Advisor for optimal configuration. Identifies up to 55 GB GPU memory savings for OPT models with minimal throughput loss.
* `agentic` `serving` `scaling` [Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework](http://arxiv.org/abs/2607.29069v1)
  > **TL;DR**: Proposes Aries for experimental analysis of agentic serving systems. Identifies non-inference bottlenecks and shows context retention reduces serving capacity. The framework reveals sandbox state management costs and informs a vision for agent-native serving with trajectory metrics and elastic resources.
* `serving` `LoRA` [DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs](http://arxiv.org/abs/2607.28848v1)
  > **TL;DR**: Addresses GPU underutilization in LLM serving via co-serving inference and LoRA fine-tuning. DeltaServe uses an SLO-aware scheduler and shared execution to leverage idle compute. Achieves 2.9x higher fine-tuning throughput with full SLO compliance versus LLMStation.
* `serving` `kernel` `hardware` [Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs](http://arxiv.org/abs/2607.28824v1)
  > **TL;DR**: Addresses kernel latency degradation in multi-partition NUMA GPUs for LLM serving. Introduces memory trace analysis to categorize workgroup data access patterns. Proposes optimization strategies like pinning and co-scheduling, showing improved locality via simulation.

### 2026-08-01
* `training` `storage` [Anonymous sharing is pairwise phase-blind](http://arxiv.org/abs/2607.28377v1)
  > **TL;DR**: Examines I/O burst contention in shared storage for training jobs checkpointing. Models using integrate-and-fire oscillators, showing anonymous resources prevent pairwise phase locking and freeze firing order. Numerically, fleets avoid synchronization but concurrent writers exceed independent-phase predictions.
* `scaling` [Queue-Theoretic Admission Control for Multi-Tenant GPU Clusters](http://arxiv.org/abs/2607.28223v1)
  > **TL;DR**: Addresses unpredictable wait times for workloads in multi-tenant GPU clusters. Proposes a queueing network model with vector packing reduction to partition workloads and bound wait times. Validation shows the model correctly identifies bottlenecks and Erlang-C conservatively overestimates wait times.
* `disaggregation` `serving` `offloading` [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](http://arxiv.org/abs/2607.28150v1)
  > **TL;DR**: Addresses network bottleneck in disaggregated LLM inference caused by KV cache transfer. Designs SmartGen with three transfer paths for selective KV cache handling. Reduces time-to-second-token by up to 4.3x compared to full transfer.

### 2026-07-30
* `serving` `RAG` [InferScale: GPU-Native KV Injection for Personalized LLM Serving](http://arxiv.org/abs/2607.27090v1)
  > **TL;DR**: Reduces TTFT in LLM serving with personalized contexts by precomputing and injecting reusable KV representations. Proposes GPU-native KV storage with Chunked RoPE and Context-Window Encoding. At k=50, TTFT decreases by 72-79% and throughput increases 3.7-4.5x.
* `serving` `diffusion` `scaling` [ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform](http://arxiv.org/abs/2607.26566v1)
  > **TL;DR**: Investigates inefficiencies in text-to-image workflow serving on serverless platforms. Proposes ServerlessT2I, decomposing workflows into model functions for independent scaling and utilizing slack GPU memory to reduce overhead. Achieves up to 2× higher request rates or 3× GPU savings at fixed request rates while meeting SLOs.
* `serving` `offloading` [DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch](http://arxiv.org/abs/2607.26475v1)
  > **TL;DR**: Addresses the memory bottleneck of auxiliary states in sparse KV cache offloading for long-context LLM serving. Proposes DualDecoder with predictive KV prefetch via dual-token decoding and layer-aware scheduling to overlap transfers. Achieves up to 2.62× higher decoding throughput than state-of-the-art systems.
* `networking` `training` `serving` [StrataCL: Fabric-Native Communication Library for Production Supernodes](http://arxiv.org/abs/2607.26444v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed AI workloads across accelerators. Designs StrataCL, a fabric-native library with registration-on-allocation and NPU-driven offloading to minimize data copies. Improves LLM inference throughput by 1.9× and reduces training iteration time by 1.4× on Huawei CloudMatrix384.
* `MoE` `training` `networking` [Incast-Free MoE Rate-Based Scheduling](http://arxiv.org/abs/2607.26340v1)
  > **TL;DR**: Resolves exponential incast caused by round-robin scheduling in MoE networks via a proactive fair scheduling framework implemented in NICs. Eliminates incast, maintains near-100% link utilization, and reduces Collective Completion Time.
* `serving` `kernel` `networking` [The Fabric Is the Cluster Driver: Cross-Layer eBPF Policies for GPU-CXL Fabrics](http://arxiv.org/abs/2607.26335v1)
  > **TL;DR**: Proposes fabric_ext, an eBPF-based compiler/runtime for optimizing data movement across GPU-CXL fabrics during LLM prefill. Uses semantic movement graphs to generate device-specific eBPF programs, specializing commands for local execution. Improves efficiency in attention and FFN streaming workloads critical to inference serving.

### 2026-07-29
* `serving` `hardware` `kernel` [MDTransformer: A Hardware-Software Co-Design of Mode-Division Photonic Transformer Accelerator with Inverse-Designed Coherent Crossbar](http://arxiv.org/abs/2607.26016v1)
  > **TL;DR**: Aims to improve efficiency of photonic transformer accelerators. Proposes MDTransformer with hardware-software co-design using mode-division optics and inverse-designed photonic tensor core. Achieves 63.6% power saving and comparable latency over state-of-the-art.
* `training` `scaling` `networking` [PowerScale: Energy-Efficient Geo-Distributed Model Training with Federated Datacenter Power](http://arxiv.org/abs/2607.25650v1)
  > **TL;DR**: Reduces energy consumption in geo-distributed model training. Proposes PowerScale, a hierarchical system using cluster-based sync-async and adaptive synchronization. Achieves up to 3.9x energy reduction while matching time-to-accuracy.

### 2026-07-28
* `diffusion` `training` [Certified Parallel-in-Time Sinkhorn for Dynamic Entropic Optimal Transport](http://arxiv.org/abs/2607.24741v1)
  > **TL;DR**: Addresses the sequential bottleneck in distributed Sinkhorn for solving entropic optimal transport in Flow Matching training. Introduces TemporalSinkhorn, a parallel-in-time executor with deterministic certificates and forgetting milestones. Achieves 3.054x-3.632x speedup on Flow Matching minibatch streams with no tolerance violations.
* `serving` `agentic` `networking` [SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving](http://arxiv.org/abs/2607.23933v1)
  > **TL;DR**: Reduces latency and memory in LLM agent sandbox scheduling. Introduces SpecBox, a speculative runtime with intent-driven prewarming, prefetching, a cache, and shared-memory transport. Achieves 2.9× lower P99 latency than on-demand and 45.9% lower memory than persistent sandboxes.
* `diffusion` `serving` `kernel` [X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference](http://arxiv.org/abs/2607.23264v1)
  > **TL;DR**: Addresses unpredictable backpressure in post-issue communication during distributed DiT inference via the X-Stage model. Designs a Burst-Gap predictor and fused kernels to interleave computation and remote stores. Achieves up to 1.62x kernel speedup for MoE and 1.43x for attention fusion in long sequences.
* `training` `disaggregation` `scaling` [Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool](http://arxiv.org/abs/2607.23250v1)
  > **TL;DR**: Addresses attention workload imbalance in long-context LLM training by introducing Libra with bounded sequence pools and co-location of sequences with complementary workloads, plus tiled attention and a pipelined runtime. Achieves up to 2.54x training throughput improvement over Ulysses.
* `serving` `networking` `offloading` [Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs](http://arxiv.org/abs/2607.23115v1)
  > **TL;DR**: Enables efficient GPU sharing over LANs for AI inference via CUDA API remoting. Gleam mitigates network bottlenecks through weight caching, async execution, and a dynamic scheduler for GPU contention. Achieves 1.4-24.2× API remoting efficiency and 1.79× throughput improvements.
* `serving` `hardware` `scaling` [Application-Driven Architecture Exploration for Cross-Layer Heterogeneous Systems](http://arxiv.org/abs/2607.23042v1)
  > **TL;DR**: Explores cross-layer system designs for LLM inference workloads. Proposes CHASE, an application-driven framework using hierarchical graph representation and telemetry-guided optimization to co-design hardware and task mappings. Achieves 2.12× speedup for LLM inference while reducing cost and power compared to baselines.
* `sparse` `kernel` [$g$MAGNUS: Fast SpGEMM on GPUs for Irregular Matrices via Hierarchical Multisplit](http://arxiv.org/abs/2607.22866v1)
  > **TL;DR**: Addresses memory spilling in sparse matrix-matrix multiplication for GPUs. Proposes $g$MAGNUS algorithm using hierarchical multisplit to chunk heavy rows into local memory accumulators. Achieves 1.81-7.62x geometric-mean speedup over cuSPARSE and MKL.
* `serving` `offloading` `disaggregation` [FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon](http://arxiv.org/abs/2607.22785v1)
  > **TL;DR**: Investigates whether CPU+GPU co-execution can accelerate transformer prefill on unified memory. Proposes FusionML, a per-layer contention-aware row split with eager materialization for concurrency. Achieves up to 1.38× faster prefill and 1.25× reduced time-to-first-token.

### 2026-07-27
* `serving` `kernel` `networking` [TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters](http://arxiv.org/abs/2607.22432v1)
  > **TL;DR**: Proposes TileSight, a tile-centric analytical GPU performance model for kernels, modeling intra-core pipeline overlap, cache hierarchy, and inter-node networking to predict latency. Achieves 13.52% wMAPE in vLLM serving at 32 GPUs.
* `agentic` `serving` `offloading` [Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads](http://arxiv.org/abs/2607.22242v1)
  > **TL;DR**: Investigates scheduling for agentic AI workloads on shared CPU/GPU infrastructure. Proposes an agentic scheduler with LLM and runtime monitor for dynamic CPU/GPU assignments. Achieves brute-force optimal mapping in all 13 scenarios, outperforming baselines without offline training.
* `kernel` `sparse` `serving` [Unified Static-Dynamic Pruning for Efficient LLM Inference](http://arxiv.org/abs/2607.21985v1)
  > **TL;DR**: Addresses LLM inference inefficiency via unified static-dynamic pruning. Introduces SPDP framework with Tiled-CBC storage format and specialized CUDA/Tensor Core kernels for activation skipping. Achieves 1.24x-1.37x speedup (up to 2.51x) over SpInfer with matching perplexity.
* `serving` `agentic` [Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads](http://arxiv.org/abs/2607.19214v2)
  > **TL;DR**: Investigates prefix cache eviction in LLM agentic workloads causing repeated costly prefills. Proposes a client-side keepalive mechanism with optimized ping frequency to maintain cache residency. Reduces post-pause request cost by up to 12.5x across major providers.

### 2026-07-24
* `training` `LoRA` `sparse` [Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers](http://arxiv.org/abs/2607.21074v1)
  > **TL;DR**: Addresses aggregation errors in federated LoRA fine-tuning for vision transformers. Proposes SpecTraL with orthonormal transformations and spiked covariance analysis to derive global ranks without dense reconstruction. Reduces server computation and improves accuracy-communication trade-offs, eliminating hyperparameter search.

### 2026-07-23
* `training` `sparse` `scaling` [Odin: Primitive-Level Synchronization for Distributed Point-Based Neural Rendering](http://arxiv.org/abs/2607.19893v1)
  > **TL;DR**: Addresses synchronization bottlenecks in distributed point-based neural rendering training. Proposes Odin with primitive-level synchronization via ahead-of-time scheduler and runtime validation instead of global barriers. Achieves up to 1.89× higher throughput on 64 GPUs while preserving reconstruction quality.
* `training` `scaling` [Do Co-Located AI Training Jobs Synchronize? Load-Dependent Throttling as a Coupling Mechanism for Phase-Locking Behind a Shared Power Cap](http://arxiv.org/abs/2607.19638v1)
  > **TL;DR**: Explores synchronization in co-located AI training jobs under shared power caps. Models the system as generalized Kuramoto oscillators with load-dependent throttling coupling and proposes phase-scattering scheduling to desynchronize cycles. Prevents linear growth of aggregate power fluctuations by maintaining sub-linear scaling.
* `MoE` `training` `kernel` [Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts](http://arxiv.org/abs/2607.19539v1)
  > **TL;DR**: Addresses communication-induced latency in distributed Mixture-of-Experts execution by overlapping computation and communication via tile-level signaling and persistent kernels. Achieves up to 2.64x end-to-end speedup on a 4-GPU system.
* `kernel` `serving` `edge` [BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators](http://arxiv.org/abs/2607.19438v1)
  > **TL;DR**: Enhances on-device LLM inference throughput via hand-written Metal kernels optimized for Apple M5 Neural Accelerators, targeting compute-bound operations like GEMM and flash-attention, while retaining specialized decode kernels. Achieves up to 6.4× higher prompt processing and 1.75× higher decode throughput versus llama.cpp.

### 2026-07-22
* `serving` `agentic` [Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads](http://arxiv.org/abs/2607.19214v1)
  > **TL;DR**: Addresses cost inefficiency from KV cache eviction in agentic workloads via client-side keepalive pings that periodically replay prefixes. This reduces post-pause request cost by up to 12.5x and identifies optimal ping intervals (~4 min) to minimize cost.
* `serving` `storage` [InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata](http://arxiv.org/abs/2607.18957v1)
  > **TL;DR**: Addresses the inefficiency of LLM cold-start in inference services due to sequential initialization and fine-grained I/O. Introduces the CFA abstraction and framework for refactoring components to enable safe concurrent execution and I/O merging. Achieves up to 7.2× cold-start speedup.
* `MoE` `serving` `disaggregation` [ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels](http://arxiv.org/abs/2607.18002v2)
  > **TL;DR**: Addresses inefficiency in serving MoE LLMs due to duplicate weights and rigid resource allocation. ExpertPlex shares MoE experts across prefill-decode phases, disaggregates attention, and schedules expert computation via adaptive persistent kernels. Achieves up to 2.01× higher goodput than instance-level disaggregation.
* `MoE` `training` `serving` [Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Serving](http://arxiv.org/abs/2607.18631v1)
  > **TL;DR**: Addresses the gap between theoretically optimal and buildable plans for MoE training and serving. Proposes moefs, a realizability-aware full-space optimizer that searches parallelism, schedule, and kernels. On 8x H800 serving, achieves a 1.0304 throughput ratio vs. hand-tuned configuration.

### 2026-07-21
* `serving` `offloading` `storage` [HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory](http://arxiv.org/abs/2607.18141v1)
  > **TL;DR**: Addresses memory bottlenecks for scaling KV cache in multi-turn LLM serving with HyMCache, leveraging CXL-hybrid memory via prefetching and buffering. Achieves 3.0x performance gain over baselines under same DRAM budget.
* `serving` `MoE` `disaggregation` [ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels](http://arxiv.org/abs/2607.18002v1)
  > **TL;DR**: Addresses resource inefficiency in serving MoE LLMs by disaggregating attention and sharing experts across prefill and decode. Introduces adaptive persistent kernels and attention-initiated communication to reduce weights and communication. Achieves up to 2.01× goodput improvement over disaggregated baselines.
* `training` `edge` `offloading` [AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning](http://arxiv.org/abs/2607.17913v1)
  > **TL;DR**: Addresses communication overhead in distributed fine-tuning of foundation models on edge devices. Introduces AE-PSL, a framework that compresses activations and gradients via a lightweight autoencoder with two-stage alignment for compatibility. Aims to reduce communication without performance degradation in split learning.
* `training` `storage` [Mobius Learning: Cyclic Depth Folding in Transformers](http://arxiv.org/abs/2607.17843v1)
  > **TL;DR**: Addresses memory constraints in distributed training of Transformers. Introduces Mobius Learning, a cyclic depth folding method that stores one block group per worker. With a GPT-2 small, it achieves lower validation loss than fixed-order at more block-sequence passes.
* `training` `networking` [A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix](http://arxiv.org/abs/2607.17644v1)
  > **TL;DR**: Identifies a memory regression in training with absorbed MLA that increases activation memory by up to 34%. Proposes LAGA, which gathers the latent for communication efficiency and reconstructs per-head K/V locally. On 8x Ascend 910B, LAGA reduces communication by 1.98x and improves throughput up to 1.24x in cross-node training.
* `RL` `training` `offloading` [LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget](http://arxiv.org/abs/2607.14952v2)
  > **TL;DR**: Enables RL post-training at million-token contexts under fixed GPU budget via resident state management and response replay. Achieves 2M-token execution on Qwen-27B with 83.894 GB per rank on 8 GPUs and 2M deterministic execution on GLM-5.2 with 32 GPUs.
* `serving` `offloading` `scaling` [Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs](http://arxiv.org/abs/2607.17181v1)
  > **TL;DR**: Addresses session continuity challenges in serverless LLM serving with reusable KV prefixes. Proposes Talaria, a system with joint placement-and-admission decisions, session-prefill admission, and stable KV management. Achieves 5.3x and 2.6x speedups in p50 and p95 session completion times.
* `edge` `serving` `quantization` [LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters](http://arxiv.org/abs/2607.17175v1)
  > **TL;DR**: Addresses QoS-aware LLM orchestration on edge clusters. Introduces LMEdge, using ML predictors and a heuristic for model configuration and placement. Achieves reduced latency and increased serving ratio in a 57-device testbed.
* `MoE` `edge` `serving` [OrderMoE: An expert similarity driven distributed edge MoE inference](http://arxiv.org/abs/2607.17154v1)
  > **TL;DR**: Tackles edge deployment challenges for MoE models by leveraging expert similarity. Proposes OrderMoE with similarity-aware expert grouping and deployment, plus runtime server-expert selection. Reduces latency and cross-server traffic significantly with minor quality degradation in experiments.
* `edge` `multi-modal` `serving` [EdgeCoInfer: Hierarchical Collaborative Inference for On-Device Multimodal Large Models](http://arxiv.org/abs/2607.17143v1)
  > **TL;DR**: Addresses efficient concurrent MLLM inference on edge devices by co-optimizing inter-model sharing and intra-model partitioning via a hybrid evolutionary HRL method. Achieves 76% system cost reduction and 71.88% memory savings compared to state-of-the-art baselines.
* `MoE` `hardware` `networking` [ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts](http://arxiv.org/abs/2607.17074v1)
  > **TL;DR**: Addresses bottlenecks in MoE inference: high memory bandwidth, non-deterministic routing traffic, and tail latency. Proposes ThAME, a 3D heterogeneous chiplet architecture with FeFET/DRAM memory and optimized Network-on-Chip. Achieves 15.7x speedup and 9.8x energy efficiency over state-of-the-art.
* `kernel` `serving` [Technical Report: AI-Assisted Gated DeltaNet Optimization on NVIDIA Blackwell](http://arxiv.org/abs/2607.16831v1)
  > **TL;DR**: Optimizes Gated DeltaNet inference via end-to-end systems approach including kernel optimization and structural reformulation. Achieved 1.58× speedup with decode latency of 9.315μs and prefill latency of 239.48μs.
* `serving` `edge` [Roomie: Interference-Aware Colocation for Efficient Model Serving](http://arxiv.org/abs/2607.16784v1)
  > **TL;DR**: Proposes Roomie, an interference-aware orchestration system for colocating DNN models on GPUs. Uses offline kernel profiling and online analytical prediction to avoid interference, with a greedy placement algorithm. Reduces SLO violations by up to 3x while maintaining comparable goodput.
* `serving` `storage` `scaling` [Cold-Start Model Delivery in Kubernetes Inference Serving: An Empirical Study of OCI-Based Distribution and Its Integrity](http://arxiv.org/abs/2607.16596v1)
  > **TL;DR**: Addresses cold-start latency in model serving due to weight delivery in Kubernetes. Proposes OCI-based delivery (oci+native and oci+fetch) to cache models on nodes, reducing warm replica startup time for a 70B model to 11.7s (208x faster than object storage).
* `MoE` `networking` `edge` [AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligence](http://arxiv.org/abs/2607.16562v1)
  > **TL;DR**: Addresses bandwidth and scalability bottlenecks in wireless MoE deployment. Proposes AirMoE with statistic-augmented routing using JS divergence and over-the-air aggregation via channel-aware power control. Outperforms baseline MoE and single models in semantic segmentation tasks.
* `training` `agentic` `RL` [WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning](http://arxiv.org/abs/2607.17299v1)
  > **TL;DR**: Accelerates synchronous agentic RL training by optimizing rollout generation. WAR uses workload-dependent strategies: SuffixDecoding for low load and cache-aware scheduling for high load. Achieves up to 1.6x higher throughput in high load.
* `serving` `kernel` `edge` [Roomie: Interference-Aware Colocation for Efficient Model Serving](http://arxiv.org/abs/2607.16784v1)
  > **TL;DR**: Addresses GPU kernel-level interference during colocation in DNN serving. Proposes Roomie, which uses offline profiling to model interference, a greedy heuristic for placement, and online assignment. Reduces SLO violations by up to 3x on cloud and edge deployments compared to state-of-the-art systems.

### 2026-07-20
* `serving` `networking` `kernel` [Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives](http://arxiv.org/abs/2607.16100v1)
  > **TL;DR**: Seeks to reduce GPU collective latency critical for LLM token generation. Proposes barrier-free synchronization and symmetric memory/multicast designs via NCCL custom kernels. Achieves latency within 7% of speed-of-light, improving inter-token latency and throughput in LLM inference.
* `training` `multi-modal` `RL` [JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models](http://arxiv.org/abs/2607.16074v1)
  > **TL;DR**: JoyNexus addresses the high cost and inefficiency of exclusive resource allocation for multi-tenant VLA model post-training. It decouples services into APIs with shared base models and introduces group batching. Results show reduced aggregate GPU time and improved utilization compared to isolated execution.
* `serving` `agentic` [Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives](http://arxiv.org/abs/2607.14166v2)
  > **TL;DR**: Exposes enforcement gaps in LLM-agent control primitives (e.g., cancellation) leading to side-effect leaks. Proposes SOUNDGATE, an externally-verified gate enforcing barrier semantics. Blocks 100% of violations across six frameworks, processing writes in ~1 ms and sustaining 12k admissions/sec.
* `agentic` `serving` `RAG` [Scalable LLM Agent Tool Access in the Cloud](http://arxiv.org/abs/2607.15593v1)
  > **TL;DR**: Addresses scaling LLM agents' tool access via MCP by introducing a cloud gateway for legacy integration, protocol consolidation, and hybrid retrieval. Achieves 98% Top-15 recall, scales to 3000+ tools, reduces tool selection time by 8.9× and token usage by 23.8× with low overhead.

### 2026-07-17
* `hardware` [Don't Predict, Prioritize: Rethinking GPU Reliability Assessment](http://arxiv.org/abs/2607.15115v1)
  > **TL;DR**: Proposes HeaRank, a Learning-to-Rank framework for ranking GPU nodes by failure risk using historical patterns. Achieves 64% capture of future failures in top 5% of ranked nodes, compared to 21% by the incumbent system.
* `training` `RL` [LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget](http://arxiv.org/abs/2607.14952v1)
  > **TL;DR**: Addresses efficient long-context RL post-training under fixed GPU memory. Proposes LongStraw, an execution stack that avoids autograd for prompts, retains essential state, and replays responses sequentially. On 8 H20 GPUs, handles 2.1M tokens adding only 0.21GB peak memory per group size increase and scales to 4.46M tokens.
* `video` `edge` `disaggregation` [CODA: Algorithm-Hardware Co-design for Edge Video Diffusion via NMP-Enabled Compute-Cache Operator Disaggregation](http://arxiv.org/abs/2607.14908v1)
  > **TL;DR**: Proposes CODA to reduce latency in edge video diffusion inference by addressing cache offloading inefficiencies. Uses algorithm-hardware co-design with compute-cache operator disaggregation, near-memory processing, and execution overlap with CFG branch. Achieves up to 1.80x speedup and 1.74x higher energy efficiency.
* `serving` `edge` [HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference](http://arxiv.org/abs/2607.12839v3)
  > **TL;DR**: HeteroMosaic optimizes edge LLM inference via heterogeneity-aware scheduling and micro-batching across iGPUs and NPUs. It uses a roofline model and trace-guided co-optimization, achieving up to 1.78x speedup over NPU baseline and 45.3% energy reduction on a balanced platform.
* `diffusion` `serving` `sparse` [FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving](http://arxiv.org/abs/2607.12121v2)
  > **TL;DR**: Addresses high diffusion model serving latency via adaptive region skipping and scheduling. FlashDiff selectively executes active regions and reallocates compute slack. Achieves 30-97% latency reduction and 1.2-2.2x throughput improvement across image, video, and audio workloads.
* `networking` `training` [The Cost and Network Limits of Space-Based AI Compute](http://arxiv.org/abs/2607.14172v1)
  > **TL;DR**: Assesses cost-effectiveness of space-based AI data centers for LLM training and inference. Compares orbital and terrestrial systems using bisection bandwidth and roofline models. Concludes inference might be feasible, but training frontier-scale LLMs is uncompetitive due to network limitations.
* `quantization` `kernel` [ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM](http://arxiv.org/abs/2607.14622v1)
  > **TL;DR**: Addresses efficient low-bit GEMM for CPU-based ML inference via hardware-software codesign. Proposes ExaGEMM, a framework for co-exploring SIMD ISA support and GEMM kernels using analytical models. Achieves 13.29x lower latency over software baselines for LLM workloads.
* `quantization` `serving` `edge` [PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference](http://arxiv.org/abs/2607.14618v1)
  > **TL;DR**: Addresses efficient LLM inference on edge CPUs via quantization. Proposes PolyQ, a compiler/quantization co-design that groups channels into bit-homogeneous blocks and generates optimized kernels. Reduces activation reorder traffic by up to 70.8% and keeps energy overhead below 2% relative to a LUT-based back-end.

### 2026-07-16
* `edge` `serving` [HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference](http://arxiv.org/abs/2607.12839v2)
  > **TL;DR**: Addresses underutilization of heterogeneous resources in edge LLM inference. Proposes HeteroMosaic, a scheduling framework using a roofline model and micro-batching for cross-accelerator overlap, with co-optimization of scheduling and device allocation. Achieves 2.05× speedup and 45.3% energy reduction over llama.cpp.
* `serving` `scaling` [The Environmental Cost of Digital Sovereignty: Water, Energy, and Emissions Impacts of Sovereign AI Infrastructure in the Global South](http://arxiv.org/abs/2607.13443v1)
  > **TL;DR**: Quantifies water/energy/carbon impacts of sovereign AI deployment in water-stressed regions. Models environmental stress using GPU specs and local data across four Global South cases. Finds evaporative-cooled 1024-GPU cluster in UAE consumes >30M liters water/year, proposing frugal models and vulnerability-aware siting.
* `training` `offloading` `scaling` [Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models](http://arxiv.org/abs/2607.13332v1)
  > **TL;DR**: Addresses collective pretraining of LLMs across distributed, heterogeneous devices via bandwidth-efficient pipeline parallelism and fault-tolerant collectives. Achieved 4.2 tokens/TFLOP and 63% efficiency vs centralized baseline for an 8.6B model on 500B tokens.

### 2026-07-15
* `offloading` `edge` `serving` [Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices](http://arxiv.org/abs/2607.10183v2)
  > **TL;DR**: Addresses efficient LLM inference on resource-constrained consumer devices by introducing tensor-level offloading. ATSInfer combines static tensor placement with dynamic load-aware transfer and asynchronous CPU-GPU coordination, improving prefill throughput by up to 1.94× and decode throughput by up to 3.29× against baselines.
* `edge` `serving` [HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference](http://arxiv.org/abs/2607.12839v1)
  > **TL;DR**: Addresses inefficient resource utilization in edge LLM inference due to coarse device-level scheduling. Proposes HeteroMosaic, a framework using micro-batch decomposition and trace-guided co-optimization for heterogeneous accelerator scheduling. Achieves up to 2.05x speedup over llama.cpp and 45.3% energy reduction.
* `MoE` `serving` [Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts](http://arxiv.org/abs/2607.12696v1)
  > **TL;DR**: Addresses expert scattering in speculative decoding of MoE LLMs that increases memory traffic. Introduces EcoSpec, a cost-aware draft selection method with expert predictor and buffer to favor draft paths reusing active experts. Achieves up to 1.62× end-to-end decoding speedup on large MoE models.
* `training` `LoRA` `networking` [Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models](http://arxiv.org/abs/2607.12583v1)
  > **TL;DR**: Addresses reducing training time and communication cost for federated fine-tuning of large vision models under resource constraints and heterogeneity. Proposes FeDiSyn, combining scaling synthetic-image pre-training with diffusion and contribution-aware LoRA plus bandwidth allocation. Achieves over 52.5% less completion time and over 97.2% lower communication cost.
* `scaling` `training` [Overcoming Orchestration Bottlenecks at Exascale: A Decentralized, Policy-Driven Approach for Sim-AI Ensembles](http://arxiv.org/abs/2607.12211v1)
  > **TL;DR**: Addresses orchestration bottlenecks for large-scale simulation-AI ensembles. Proposes EnsembleLauncher, a decentralized workflow orchestrator with programmable scheduling policies. Scales to 8 million tasks on Aurora supercomputer, achieving 4× higher throughput than state-of-the-art tools.
* `diffusion` `serving` `video` [FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving](http://arxiv.org/abs/2607.12121v1)
  > **TL;DR**: Reduces diffusion serving latency via selective execution of unstable latent regions. Proposes region decomposition, activity-based skipping, and affinity-aware scheduling. Cuts latency by 30-97% and boosts throughput 1.2-2.2x across modalities.
* `edge` `offloading` [Cross-Core Inference Offload as an Operating-System Service on Dual-Core Microcontrollers](http://arxiv.org/abs/2607.12620v1)
  > **TL;DR**: Addresses efficient inference offloading on asymmetric dual-core microcontrollers. Proposes SynapticOS, a dual-core OS service using lock-free ring buffers in shared SRAM for zero-copy tensor staging. Achieves typical round trips of 15 us and worst-case 81 us, with a two-model soak test showing zero errors.
* `edge` `kernel` `offline` [SynapticOS: An Inference-First Runtime Architecture for Neural Processing Units on Resource-Constrained Microcontrollers](http://arxiv.org/abs/2607.12606v1)
  > **TL;DR**: Addresses inefficient inference execution on microcontroller NPUs. Designs SynapticOS runtime with tensor-aligned allocator (zero fragmentation, ~154 cycles/allocation), hardware abstraction layer, and model lifecycle management. Achieves constant-time tensor allocation, 67KB flash footprint, and microsecond-scale inference latency on ARM Cortex-M MCUs.

### 2026-07-14
* `serving` `quantization` `kernel` [Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs](http://arxiv.org/abs/2607.11368v1)
  > **TL;DR**: Attributes quantized serving speedups by decomposing runtime and kernel effects using a matched FP16 intermediate stack. Achieves 2.58x end-to-end speedup (two-thirds from runtime) and increases sustainable concurrent users 4x by avoiding a memory cliff.
* `kernel` `hardware` [GPU-Tile-Sim: A Tile-Centric GPU Simulation Framework for LLM Hardware-Software Co-Design](http://arxiv.org/abs/2607.11262v1)
  > **TL;DR**: Addresses accurate performance modeling for LLM GPU kernels in hardware-software co-design. Proposes GPU-Tile-Sim, a tile-graph simulation framework capturing dependency structures. Achieves 1.22%-8.71% MAPE on A100/H100 across GEMM, attention, and LLM inference workloads.
* `diffusion` `serving` `offloading` [Xema: Efficient Diffusion Serving through Fine-Grained Memory Management and Auto-Configuration](http://arxiv.org/abs/2607.11136v1)
  > **TL;DR**: Addresses high GPU memory usage in diffusion model serving. Proposes Xema, which uses trace-guided memory optimization and static memory layout for fine-grained control, with an offline planner for configuration. Achieves 3.7× higher SLO attainment and reduces planning cost from 6.3 hours to 197 seconds.
* `hardware` [Energy Calculus: A Compositional Algebra of Energy in Computational Systems](http://arxiv.org/abs/2607.11087v1)
  > **TL;DR**: Proposes energy calculus, an algebra for compositional energy modeling in computational systems. Defines operators and energy elements with uncertainty propagation, enabling joint reasoning of techniques like frequency scaling. Introduces a Reduction Theorem to simplify context dependence below uncertainty thresholds, facilitating energy optimization.
* `serving` `offline` `networking` [[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows](http://arxiv.org/abs/2607.10987v1)
  > **TL;DR**: Proposes distributed KV-cache orchestration to eliminate redundant prefill in multi-agent LLM workflows. Introduces stateful operators for zero-copy KV materialization and transfer across agents. Achieves 50.2x lower TTFT and 7.63x compute cost reduction at scale.
* `serving` `scaling` `networking` [Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Models](http://arxiv.org/abs/2607.10389v1)
  > **TL;DR**: Addresses elastic serving for stateful world models by guaranteeing bit-identical live migration without recomputation. Proposes WorldMove with incast-aware admission control and block-boundary relocation. Achieves same-node migration in 18.8 ms (101× faster than save/load) and 92.1-94.8 Gb/s transfer rate.
* `offloading` `edge` `serving` [Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices](http://arxiv.org/abs/2607.10183v1)
  > **TL;DR**: Addresses efficient LLM inference on consumer devices with limited GPU memory via tensor-granular offloading. Proposes ATSInfer, using static placement, dynamic transfer, and asynchronous CPU-GPU coordination. Achieves up to 1.94x prefill and 3.29x decode throughput improvement.
* `serving` `scaling` [Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving](http://arxiv.org/abs/2607.09992v1)
  > **TL;DR**: Addresses SLO violations in ML serving caused by untrusted learned components. Proposes a runtime guard that enforces per-request priorities and reservation-based resource guarantees. Maintains 0.0% SLO miss rate across adversarial conditions while exposing failures in baseline systems.

### 2026-07-13
* `kernel` `sparse` `edge` [STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU](http://arxiv.org/abs/2607.09385v1)
  > **TL;DR**: Proposes STEEL, a sparsity-aware fused attention kernel for NPUs to enable energy-efficient long-sequence LLM inference on edge SoCs. By optimizing dataflow and leveraging sparsity for load balancing, it reduces energy consumption by 9.17x versus CPU and achieves 9.6x latency reduction on XDNA 1.
* `training` `RL` [Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training](http://arxiv.org/abs/2607.09207v1)
  > **TL;DR**: Addresses resource idleness in asynchronous, disaggregated RL systems for LLM post-training. Proposes BiDiRL with hybrid time-space multiplexing, featuring hot-switch runtime and bidirectional scheduling to enable resource borrowing. Achieves up to 1.94x higher training throughput than state-of-the-art frameworks without convergence degradation.
* `serving` `networking` [SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference](http://arxiv.org/abs/2607.08973v1)
  > **TL;DR**: Reduces LLM inference latency by minimizing All-Reduce synchronization overhead. Proposes SiFAR with redundant pull and speculative reduction to avoid barriers during communication. Achieves 52% lower All-Reduce latency and 18.6% higher throughput for Llama-3.1-8B at TP=8.

### 2026-07-11
* `serving` `agentic` [SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling](http://arxiv.org/abs/2607.08565v1)
  > **TL;DR**: SMETRIC optimizes agentic LLM serving by scheduling session-initial requests for load balance and subsequent requests for KV reuse. Using session-turn metrics and global KV store, it achieves 10-16% higher cluster TPS and 2-34% higher prefill TPS under disaggregation.
* `hardware` `storage` `networking` [Who Needs DRAM? We Have Fiber](http://arxiv.org/abs/2607.08407v1)
  > **TL;DR**: Addresses high DRAM demands for LLM weights by proposing Fiber Memory, an optical architecture using delay lines and multi-core fibers for recirculating storage and broadcast. Reduces weight-delivery energy by >70% and redundant storage across 10,000 accelerators.
* `serving` `MoE` `hardware` [On the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend](http://arxiv.org/abs/2607.08215v1)
  > **TL;DR**: Analyzes engineering challenges of serving MoE and multimodal LLMs on non-GPU accelerators. Conducts field study on Huawei Ascend 910 with patches to CANN/vLLM-Ascend, documenting 8 platform limitations. Reports 12 source-level fixes required for reliable inference with DeepSeek models.
* `serving` `kernel` [CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems](http://arxiv.org/abs/2607.07862v1)
  > **TL;DR**: Addresses high LLM serving latency on multi-GPU systems via CTA-pipelining, a kernel-level parallelization method enabling concurrent execution of dependent kernels. Reduces GEMM layer latency by up to 31.8% vs micro-batching and 29.6% vs Tensor Parallelism.

### 2026-07-09
* `training` `quantization` [GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining](http://arxiv.org/abs/2607.07494v1)
  > **TL;DR**: Addresses gradient communication bottleneck in LLM pretraining by handling anisotropic distortion in low-precision quantization. Introduces GIFT, which transforms gradients to near-isotropic space using geometry-aware scaling and selective low-rank approximation. Achieves 7.6% reduction in end-to-end pretraining time for Llama-600M on 64 NVIDIA GH200 Superchips.
* `agentic` `serving` [Progressive Crystallization: Turning Agent Exploration into Deterministic, Lower-Cost Workflows in Production](http://arxiv.org/abs/2607.07052v1)
  > **TL;DR**: Addresses high costs of persistent LLM inference in operational AI agents by introducing progressive crystallization, a lifecycle that converts validated agent behaviors into deterministic workflows. Achieved over 70% cost reduction per incident in production.
* `edge` `serving` [Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence](http://arxiv.org/abs/2607.07046v1)
  > **TL;DR**: Proposes Voltron, a framework for elastic multi-device LLM inference at the edge to enable larger models. It dynamically utilizes available devices and adapts to edge environments, achieving up to 16.5% higher accuracy than single-device LLMs while meeting QoS.
* `serving` `MoE` `scaling` [Think Before You Grid-Search: Floor-First Triage for LLM Serving](http://arxiv.org/abs/2607.05876v2)
  > **TL;DR**: Proposes Floor-First, an analytical method to avoid profiling grid search in LLM serving by modeling decode steps into resource vectors. Case study on a 671B MoE model shows EP16+DP layout supports 644 concurrent 8K requests (10x more than TP16's 70) despite higher latency.
* `MoE` `serving` `networking` [UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods](http://arxiv.org/abs/2607.06202v2)
  > **TL;DR**: Addresses communication inefficiencies (serialization, synchronization, load imbalance) in MoE inference on superpods. Proposes UBEP, a communication library redesigning All-to-All primitives. Reduces time per output token by up to 11.1%.

### 2026-07-08
* `MoE` `serving` `networking` [UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods](http://arxiv.org/abs/2607.06202v1)
  > **TL;DR**: Addresses MoE inference bottlenecks on production superpods: strict serialization, sync overhead, and load imbalance in All-to-All communication. Proposes UBEP, a communication library restructuring All-to-All primitives. Reduces All-to-All latency by 52.4% and TPOT by 11.1%.
* `RAG` `storage` `networking` [GORIO: GPU-Centered Remote I/O for Graph ANNS over NVMe-oF](http://arxiv.org/abs/2607.04415v2)
  > **TL;DR**: Optimizes remote storage I/O for graph ANNS in RAG systems by introducing GPU-driven request generation and scheduling. GORIO offloads I/O tasks to GPU and overlaps traversal with remote page service. Achieves 1.31× and 4.89× speedups over state-of-the-art and direct page-cache paths.
* `training` `storage` `networking` [MatrixFSDP: communication-free matrix optimizers under ZeRO-3 parameter sharding](http://arxiv.org/abs/2607.05895v1)
  > **TL;DR**: Addresses communication inefficiency of matrix optimizers under ZeRO-3 sharding. Proposes MatrixFSDP, which assigns whole matrices to one rank via shard reorganization, P2P collectives, and storage optimizations. Reduces optimizer-step latency by 54.6x on 8 nodes with 2.15x end-to-end speedup.
* `serving` `MoE` [Think Before You Grid-Search: Floor-First Triage for LLM Serving](http://arxiv.org/abs/2607.05876v1)
  > **TL;DR**: Proposes Floor-First triage to avoid grid search in LLM serving by modeling decode steps as resource vectors and comparing layouts via resource walls. Case study on 671B MoE model shows EP16+DP layout increases capacity wall 10x over TP16 with ~644 concurrent requests.
* `training` `offloading` `scaling` [PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint](http://arxiv.org/abs/2607.01646v2)
  > **TL;DR**: Addresses high recovery latency and overhead in fault-tolerant LLM training. Proposes PHOENIX with zero-overhead in-memory checkpointing and hot-swapping via communicator reconstruction. Achieves zero checkpoint overhead and ≤40s recovery on 512 GPUs for 65B models.
* `storage` [Bounded-Memory Parallel Image Pulling for Large Container Images](http://arxiv.org/abs/2607.05596v1)
  > **TL;DR**: Addresses high memory consumption and OOM issues in container image pulling for AI/ML workloads. Proposes Disk-Backed Parallel Pull (DBPP) with direct-to-disk writes and parallel processing, reducing peak memory by 8.7--25.3× while avoiding OOM kills on large images.
* `diffusion` `scaling` [Design-CP: Context Parallelism for Design of Protein Nanoparticles](http://arxiv.org/abs/2607.05439v1)
  > **TL;DR**: Addresses memory constraints in diffusion models for generating large protein assemblies. Introduces Design-CP with context parallelism using 1D and 2D sharding with ring attention on multi-GPU systems. Achieves square-root scaling of feasible subunit size with GPU count and better wall-clock performance for 2D sharding.

### 2026-07-07
* `serving` `RL` `networking` [Adaptive Inference Batching using Policy Gradients](http://arxiv.org/abs/2607.05272v1)
  > **TL;DR**: Investigates whether reinforcement learning can improve adaptive batching and routing for inference serving. Develops an RL-based policy via MDP formulation and discrete-event simulation. Achieves 3.5x throughput improvement in heterogeneous multi-GPU routing over Round-Robin while reducing latency and adhering to SLAs.
* `offloading` `edge` `hardware` [Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computing](http://arxiv.org/abs/2607.05240v1)
  > **TL;DR**: Proposes an Integer Linear Programming-based framework for ML workload partitioning between CPUs and CIM accelerators to minimize inference latency under RRAM constraints. Achieves up to 30.9x speedup over CPU-only execution on edge systems.
* `MoE` `serving` `networking` [Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference](http://arxiv.org/abs/2607.05116v1)
  > **TL;DR**: Optimizes Mixture-of-Experts inference by reducing communication overhead. Proposes communication-aware expert placement and pruning to balance computation, communication, and accuracy. Achieves 1.23x-1.86x throughput improvement versus baseline methods while preserving model accuracy.
* `serving` `edge` `scaling` [When Words Predict Workload](http://arxiv.org/abs/2607.04951v1)
  > **TL;DR**: Addresses OOM crashes in edge LLM serving caused by rigid text forcing dynamic ensemble expansion. Proposes an LRF gateway with text-feature forecasting and dynamic routing to local/remote models. Reduces misroute fraction to 0.087-0.095 (vs. baseline 0.849) and maintains peak edge VRAM at 4.82 GB in a 6000-request trial.
* `training` [Direct Model State Migration for Elastic Training of Large Language Models](http://arxiv.org/abs/2607.04749v1)
  > **TL;DR**: Addresses high latency in state migration for elastic LLM training. Proposes ETC, a checkpoint-free framework using direct peer-to-peer communication and communication coalescing to bypass storage. Reduces migration overhead by up to 6.37× compared to checkpoint-based methods.
* `sparse` `networking` [Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms](http://arxiv.org/abs/2607.04676v1)
  > **TL;DR**: Investigates reducing communication overhead for sparse data in GPU collectives. Proposes adaptive algorithms using Pici bitvector format for efficient compression. Achieves up to 5.25x speedup for all-gather at 99% sparsity over NCCL.
* `storage` `RAG` [GORIO: GPU-Centered Remote I/O for Graph ANNS over NVMe-oF](http://arxiv.org/abs/2607.04415v1)
  > **TL;DR**: Optimizes GPU-centered remote I/O for vector retrieval in graph ANNS workloads over NVMe-oF. Reduces CPU involvement by offloading page-miss handling and scheduling to GPU. Achieves 1.31-3.73× speedup over existing remote-I/O paths on DiskANN-style queries.
* `serving` `diffusion` `disaggregation` [Sangam: Efficiently Serving Diffusion LLMs with the AR Stack](http://arxiv.org/abs/2607.04206v1)
  > **TL;DR**: Addresses serving efficiency for diffusion LLMs due to bidirectional attention challenges. Introduces Sangam with a deficit token-budget scheduler and hybrid prefill overflow to balance workloads. Reduces mean latency by 8-20% on various diffusion LLM workloads.
* `serving` `scaling` [CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving](http://arxiv.org/abs/2607.04181v1)
  > **TL;DR**: Addresses dynamic workload skewness in online LLM serving. Proposes CoCoScale, a layer-wise scaling mechanism that selectively parallelizes hot layers using idle resources. Reduces cold start latency by 97.9%-99.3% and average latency by 20.7%-28.1% under production traces.
* `serving` `MoE` [BrownoutMoE: Structure-Aware Expert Grouping for Efficient and Accurate LLM Web-based Services](http://arxiv.org/abs/2607.04164v1)
  > **TL;DR**: Addresses imbalanced expert activation in MoE serving by grouping experts using reinforcement learning and distillation, minimizing accuracy drop. Achieves up to 2.24x throughput improvement and reduces accuracy degradation by up to 71.4%.
* `agentic` `serving` `MoE` [SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference](http://arxiv.org/abs/2607.03333v1)
  > **TL;DR**: Addresses idle GPU time during tool calls in agentic LLM inference. SPORK uses self-speculative forking to dispatch tool calls early, overlapping with decoding, reducing GAIA P95 latency by 18% (131.9s to 108.1s) with no accuracy loss.
* `training` `multi-modal` [HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters](http://arxiv.org/abs/2607.03229v1)
  > **TL;DR**: Addresses low Model Flop Utilization (MFU) in Multimodal Large Language Model training. Introduces Mpipe, a composable algebra system deriving a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into idle pipeline regions. Achieves 2.70x speedup in small-scale NPU clusters.
* `serving` `agentic` [A Workflow-Aware Serving Layer for Agentic Applications](http://arxiv.org/abs/2607.02942v1)
  > **TL;DR**: Addresses inefficient serving for agentic applications with structured workflows. Dyserve compiles per-node model/verifier choices into an ILP, pre-solves at multiple pressure levels, and adapts via strategy shifts during execution without runtime solving. Avoids solver overhead and preserves committed work under load.
* `hardware` `networking` [HyNoC: A Hybrid Circuit-Switch/Wormhole Network-on-Chip for Distributed VLIW Computing on FPGA](http://arxiv.org/abs/2607.02729v1)
  > **TL;DR**: Proposes HyNoC, a hybrid circuit-switch/wormhole network-on-chip for FPGA-based distributed VLIW systems, enabling static routing without virtual channels. Benchmarked on LLaMA 3 8B FFN up-projection, it shows 5x throughput improvement via partitioned meshes eliminating cross-quadrant traffic.
* `offloading` `serving` [Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines](http://arxiv.org/abs/2607.02630v1)
  > **TL;DR**: Addresses fine-grained computation offload stalls in inference serving. Proposes routing offloads through existing server concurrency using deferred-response primitives, avoiding major system modifications. Recovers 1.2-5.4x in latency on stock servers like Redis.
* `serving` `sparse` [Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers](http://arxiv.org/abs/2607.02612v1)
  > **TL;DR**: Reduces computation in Vision Transformers via a unified adaptive inference framework called Fusion. Sequentially applies token merging, confidence evaluation for early exiting, and pruning using lightweight routing modules. Achieves 48% lower inference energy and 4× reduced calibration error on ImageNet-1k with DeiT-S.
* `serving` `offloading` `disaggregation` [From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving](http://arxiv.org/abs/2607.02574v1)
  > **TL;DR**: Surveys KV cache management in LLM serving, classifying over thirty systems using four axes into five architectural archetypes. Identifies seven missing KV-specific measurements, linking them to open problems including fault tolerance and isolation.
* `serving` `scaling` `edge` [Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes](http://arxiv.org/abs/2607.04668v1)
  > **TL;DR**: Proposes a kernel-level scheduler for on-device LLM inference enabling per-token core membership changes. The elastic gang uses an ACK-latched epoch protocol to safely share cores between inference and OS processes, improving general throughput by 1.75x at 25% inference duty cycle.
* `edge` `serving` [Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots](http://arxiv.org/abs/2607.02501v2)
  > **TL;DR**: Addresses deployment inefficiency for embodied AI models on heterogeneous edge robots. Proposes Embodied.cpp, a portable C++ runtime with multi-rate execution and latency-first fused inference. Reduces WAM memory footprint from 312.2 MiB to 88.1 MiB.
* `offloading` `serving` [Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines](http://arxiv.org/abs/2607.02630v1)
  > **TL;DR**: Addresses stall from microsecond-range computation offloads during serving. Proposes routing offloads to executors and leveraging server suspension/resume primitives for overlap without server rewrites. Achieves 1.2-5.4x latency recovery across off-the-shelf servers.

### 2026-07-03
* `serving` `offline` [WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs](http://arxiv.org/abs/2607.02391v1)
  > **TL;DR**: Addresses predicting inference power and latency for LLM-GPU combinations without profiling. Introduces WattGPU, predictive models using LLM metadata and GPU specifications. Reduces median absolute power error by approximately 4x in server mode for unseen LLM-GPU pairs compared to TDP baselines.
* `RL` `agentic` `training` [Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents](http://arxiv.org/abs/2607.01120v2)
  > **TL;DR**: Addresses the lack of systems for continuous agent learning in enterprise deployment. Proposes AReaL2.0, an agent-oriented online RL infrastructure with standardized data protocols, governed data proxies, and evolution control planes. Achieves automated policy updates from deployment workloads, eliminating manual fine-tuning cycles.
* `serving` `disaggregation` [Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving](http://arxiv.org/abs/2607.02043v1)
  > **TL;DR**: Examines high TTFT in disaggregated LLM serving from prefill node saturation. Proposes a proactive scheduler deflecting prefill to decode nodes via chunking without disrupting decodes. Eliminates inter-node KV transfer, reducing P95 TTFT by up to 81% and SLO attainment by up to 79%.
* `training` `MoE` [Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models](http://arxiv.org/abs/2607.01844v1)
  > **TL;DR**: Proposes Mixture-of-Parallelisms (MoP) for memory-efficient training of Mixture-of-Experts models. Combines specialized parallelism techniques and a novel optimizer strategy to enable trillion-parameter training at 1M context. Achieves 4.7x--8.2x higher per-GPU throughput than FSDP2 and sustains 1M tokens without OOM.
* `MoE` `serving` `disaggregation` [ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving](http://arxiv.org/abs/2607.00466v2)
  > **TL;DR**: Addresses high latency in MoE serving under PD disaggregation from expert activation locality. Proposes ELDR, which routes requests using expert signatures derived from prefill, with offline K-means partitioning and locality-band routing. Achieves 5.9-13.9% reduction in median time per output token.
* `disaggregation` `quantization` `offloading` [Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference](http://arxiv.org/abs/2607.01831v1)
  > **TL;DR**: Reduces KV cache transfer latency in disaggregated LLM inference using progressive quantization. System splits KV cache into Anchor (MSBs) and Residual (LSBs) streams; decoding starts with Anchor while Residual transfers. Achieves 1.43x TTFT gain over 8-bit quantization while matching BF16 accuracy.
* `training` `video` [HCMS: Head-Chunked Multi-Stream Pipeline for Communication-Computation Overlap in Long-Sequence Parallel Attention](http://arxiv.org/abs/2607.01817v1)
  > **TL;DR**: Addresses hardware underutilization in all-to-all sequence parallelism by proposing HCMS, a pipeline that partitions attention heads into chunks and uses dual CUDA streams for communication-computation overlap. Achieves 10%-17.5% speedup over Ulysses on 31K-56K token video generation sequences.
* `training` `video` `scaling` [Arachne: Orchestrating Cascades for Efficient Text-to-Video Model Training](http://arxiv.org/abs/2607.01701v1)
  > **TL;DR**: Addresses workload imbalance and under-utilization in text-to-video model training. Introduces Arachne, which decomposes training into cascades with coordinated spatial/temporal orchestration for distributed execution. Achieves up to 65% reduction in iteration time over existing frameworks while improving scalability.
* `training` `sparse` [SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication](http://arxiv.org/abs/2607.01678v1)
  > **TL;DR**: Reduces communication in distributed LLM training via an optimizer that sparsifies using first-moment statistics and overlaps communication. Achieves 43.3% lower pre-training time for Llama-500M without quality loss.
* `training` `storage` `networking` [DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint](http://arxiv.org/abs/2607.01646v1)
  > **TL;DR**: Addresses overhead and recovery latency in resilient LLM training. Proposes DeadPool, which uses in-memory checkpoints and a communicator reconstruction protocol for hot-swapping failed nodes. Achieves zero checkpoint overhead during normal execution and under 40-second recovery on 512 A100 GPUs.
* `serving` [OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters](http://arxiv.org/abs/2607.01579v1)
  > **TL;DR**: Addresses configuration selection for LLM serving in heterogeneous GPU clusters. Presents OmniPilot, an advisor with calibrated cost modeling and OOD abstention. Achieves 6.2% mean absolute error on throughput prediction and R²=0.92.
* `training` `RL` [The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning](http://arxiv.org/abs/2607.01415v1)
  > **TL;DR**: Investigates infrastructure overhead in coding-agent RL training. Compares execution substrates (containers, sandboxes, Kubernetes, VMs) for rollout efficiency. Finds up to 110× cold-start latency variation and 1.8× worker-hour spread per million trajectories, advocating substrate optimization as core training system component.
* `serving` `RAG` `offloading` [HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching](http://arxiv.org/abs/2607.01299v1)
  > **TL;DR**: Addresses inefficiency in hybrid-attention LLM serving by enabling position-independent caching via segment-cumulative operators and boundary seam recomputation. Achieves 2.45x reduction in time-to-first-token and up to 2.0x higher throughput.
* `edge` `multi-modal` `agentic` [Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots](http://arxiv.org/abs/2607.02501v1)
  > **TL;DR**: Addresses embodied AI deployment challenges on heterogeneous robots. Introduces Embodied.cpp, a portable runtime with modular multi-rate execution and latency-first fused inference. Achieves WAM memory reduction from 312.2 MiB to 88.1 MiB.

### 2026-07-02
* `RL` `agentic` [Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents](http://arxiv.org/abs/2607.01120v1)
  > **TL;DR**: Addresses the inefficiency of manual agent evolution in enterprise deployment by proposing co-designed systems for standardized trajectory protocols, governed data proxies, and autonomous evolution control planes. Demonstrates automated policy updates via AReaL2.0.
* `serving` `networking` `scaling` [ROSA: A Robotics Foundation Model Serving System for Robot Factories](http://arxiv.org/abs/2607.01088v1)
  > **TL;DR**: Addresses serving inefficiencies in robotics foundation models for factories. Proposes ROSA with shared GPU-pool networking, robotics-aware pipeline abstractions, and objective-driven scheduling to maximize SLO-qualified productivity. Achieves 12.06× higher factory throughput compared to dedicated systems.
* `diffusion` [Accelerating Discrete Diffusion Models with Parallel-In-Time Sampling](http://arxiv.org/abs/2607.00773v1)
  > **TL;DR**: Addresses sequential sampling inefficiency in discrete diffusion models. Parallelizes τ-leaping with continuous-time stochastic integrals and Picard iteration for time acceleration. Achieves up to 7-9× runtime speedup on synthetic data and 1.45-1.86× speedup on image/text tasks with 50% fewer NFE.
* `serving` `sparse` [MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression](http://arxiv.org/abs/2607.00760v1)
  > **TL;DR**: Addresses GPU memory exhaustion and low throughput in long-context LLM serving caused by large KV cache. Proposes MosaicKV, a dynamic two-D KV cache compression system exploiting segment-level importance for efficient memory and computation management. Achieves 3x memory reduction, 4.8x lower latency, and 7.3x higher throughput with <1.8% accuracy loss.
* `serving` `MoE` `disaggregation` [ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving](http://arxiv.org/abs/2607.00466v1)
  > **TL;DR**: Improves MoE serving efficiency in PD-disaggregated systems by expert-locality-aware decode routing. ELDR uses prefill activations to build expert signatures, offline balanced K-means, and online locality-band routing with a signature cache. Achieves 5.9-13.9% median TPOT reduction.
* `serving` `agentic` [SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering](http://arxiv.org/abs/2607.00151v1)
  > **TL;DR**: Addresses high TTFT in LLM agent serving caused by context engineering overhead. Proposes lookahead programming for asynchronous KV cache preparation and a scheduler to handle these operations alongside regular requests. Reduces TTFT by up to 11.9x by avoiding re-prefill delays.

### 2026-07-01
* `serving` `multi-modal` [Energy-Efficient Multimodal Inference Serving with Tri-serve](http://arxiv.org/abs/2606.29629v2)
  > **TL;DR**: Addresses power inefficiencies in multimodal inference serving. Proposes Tri-serve, a software DVFS controller for GPUs that handles dependency stalls, arithmetic-intensity effects, and thermal throttling. Achieves 22% energy savings without latency or throughput degradation.
* `edge` `serving` [COSM: A Cooperative Scheduling Framework for Concurrent PIM and CPU Execution on Mobile Devices](http://arxiv.org/abs/2606.30553v2)
  > **TL;DR**: Addresses efficient concurrent PIM and CPU execution for mobile LLM inference. Introduces COSM, a scheduling framework with low-interference PIM commands and idleness-aware scheduling. Achieves up to 2.8x PIM throughput improvement with <2.0% CPU performance loss.
* `serving` `edge` `thinking` [LASER: Load-Aware Serving with Early-Exit for Reasoning LLMs at the Edge](http://arxiv.org/abs/2606.31580v1)
  > **TL;DR**: Proposes LASER for efficient edge serving of reasoning LLMs. It uses load-aware adaptive early-exit thresholds and difficulty-based resource pre-allocation. Achieves 17-38% latency reduction and 3-6% SLO improvement at 1% accuracy cost.
* `training` `networking` [HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models](http://arxiv.org/abs/2606.30460v2)
  > **TL;DR**: Solves attention cross-contamination in sequence-parallel training with packed sequences via HSAP, a hierarchical framework using JIT-compilation for inter-device communication optimization. Outperforms state-of-the-art sequence parallelism methods in training metrics.
* `multi-modal` `diffusion` `offloading` [Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference](http://arxiv.org/abs/2606.31093v1)
  > **TL;DR**: Addresses challenges in multimodal inference orchestration, data transmission, and KV cache sharing. Proposes Omni-Flow, a framework with three-layer abstraction for workflow scheduling, distributed KV cache storage across GPU/CPU/SSD, and unified compute flow. Supports diverse multimodal and diffusion model scenarios.
* `serving` `disaggregation` `quantization` [Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving](http://arxiv.org/abs/2606.29708v2)
  > **TL;DR**: Investigates design decisions for heterogeneous LLM serving with prefill-decode disaggregation. Proposes a four-axis design space (accelerator, precision, interconnect, KV residency) and identifies three boundary decisions (compute placement, KV representation, ownership). Provides guidance to optimize efficiency in production through coordinated choices.
* `edge` `serving` `networking` [Budget-Adaptive Routing: Skipping the Weak When the Strong Answers Anyway](http://arxiv.org/abs/2606.30919v1)
  > **TL;DR**: Addresses compute waste in edge-cloud routing when offloading frames processed by weak models. Proposes budget-adaptive routing with a light weak-skipping estimator and adaptive placement, reducing latency by 30% and achieving up to +1.7pp mAP over strong models at lower compute.

### 2026-07-01
* `edge` `hardware` [COSM: A Cooperative Scheduling Framework for Concurrent PIM and CPU Execution on Mobile Devices](http://arxiv.org/abs/2606.30553v1)
  > **TL;DR**: Addresses the challenge of concurrent PIM and CPU execution for on-device LLMs. Proposes COSM, a cooperative scheduling framework that hides PIM latency and overlaps execution with data transfer by exploiting CPU idle windows. Improves PIM throughput by 2.8x with <2% CPU performance loss.
* `training` `networking` [HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models](http://arxiv.org/abs/2606.30460v1)
  > **TL;DR**: Addresses attention cross-contamination in hybrid-context packed sequences during LLM training. Proposes HSAP, a hierarchical framework with sequence-aware parallelism and JIT-optimized NCCL communication management. Outperforms state-of-the-art sequence parallelism approaches in multiple metrics.
* `serving` `scaling` `hardware` [Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs](http://arxiv.org/abs/2606.30391v1)
  > **TL;DR**: Investigates minimizing energy in serverless LLM serving on shared GPUs under TTFT/TBT SLOs. Proposes Festina, a profiling-guided scheduler with global placement, local resource adaptation, and energy-aware workload consolidation. Achieves up to 56% energy reduction while maintaining SLO attainment within a 2% margin.
* `serving` `disaggregation` `quantization` [HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators](http://arxiv.org/abs/2606.29986v1)
  > **TL;DR**: HMA-Serve enables cost-efficient LLM serving across memory-heterogeneous accelerators by disaggregating prefill (GDDR) and decode (HBM) using phase-wise quantization, compute-transfer pipelining, and deferred dequantization. Achieves up to 3.2× higher goodput and 4.8× higher goodput-per-dollar without quality loss.
* `MoE` `serving` `networking` [Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference](http://arxiv.org/abs/2606.29982v1)
  > **TL;DR**: Addresses MoE inference bottlenecks in multi-device settings via Cost-Aware Expert Execution (CAEE), which prunes low-importance experts and redistributes contributions using hardware-aware cost models. Achieves 8%-18% latency reduction across offloading and on-device deployments with under 1% accuracy drop.
* `serving` `disaggregation` `quantization` [Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving](http://arxiv.org/abs/2606.29708v1)
  > **TL;DR**: Examines design space for heterogeneous prefill-decode LLM inference serving. Organizes around accelerator, precision, interconnect, and KV residency axes. Identifies key decisions like compute placement and KV representation to enable runtime optimization, improving resource utilization based on industrial deployments.
* `serving` `multi-modal` [Energy-Efficient Multimodal Inference Serving with Tri-serve](http://arxiv.org/abs/2606.29629v1)
  > **TL;DR**: Addresses energy inefficiency in multimodal inference serving due to hardware PMU frequency decisions. Proposes Tri-serve, a software-based DVFS controller accounting for three inefficiencies: dependency stalls, arithmetic-intensity effects, and thermal throttling. Achieves 22% energy efficiency improvement without latency or throughput degradation.
* `edge` [Fog Computing and Large Language Models: A vision for the mutual beneficiaries](http://arxiv.org/abs/2606.29483v1)
  > **TL;DR**: Addresses the challenge of deploying LLMs on fog infrastructure to reduce latency and resource consumption. Surveys optimization techniques like quantization and LoRA for edge inference, enabling efficient LLM execution on edge devices for IoT applications.
* `quantization` `video` `sparse` [W4A4 Quantization for Inference on Wan2.2-I2V-A14B](http://arxiv.org/abs/2606.29337v1)
  > **TL;DR**: Addresses 4-bit weight and activation quantization for efficient video model inference. Combines MixQ-style mixed precision for sparse activation outliers with SmoothQuant-style smoothing and dual-branch GEMM. Achieves within 2-3.5% of FP16 fidelity on most quality metrics.
* `serving` `disaggregation` `agentic` [KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding](http://arxiv.org/abs/2606.29207v1)
  > **TL;DR**: Addresses inefficient scaling for long-context agentic workloads by disaggregating core-attention from dense layers. Proposes KernelFlume, with dynamic routing and pipelining, to elastically scale attention nodes. Reduces cost per million tokens by 61% on H100 versus ServerlessLLM while maintaining low latency.

### 2026-06-29
* `serving` `MoE` `disaggregation` [CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation](http://arxiv.org/abs/2606.24506v2)
  > **TL;DR**: Addresses GPU memory inefficiency in serving sparse cold MoE models by disaggregating weights and KV-cache into dedicated pools. Introduces a KV-cache planner, pipeline scheduler and persistent kernels. Reduces P99 time between tokens by up to 10.4x versus state-of-the-art.
* `training` [Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems](http://arxiv.org/abs/2606.27797v1)
  > **TL;DR**: Addresses inefficient symmetric partitioning in Knowledge Distillation (KD) on HPC systems. Proposes asymmetric teacher-student partitioning using vertical and horizontal splits and an analytical model for strategy selection. Achieves up to 67% higher throughput than TRL.

### 2026-06-26
* `training` [DMuon: Efficient Distributed Muon Training with Near-Adam Overhead](http://arxiv.org/abs/2606.27153v1)
  > **TL;DR**: Addresses high overhead of matrix-based optimization during distributed training. Introduces DMuon, a distributed Muon implementation requiring no framework changes, achieving up to 3.01x end-to-end step speedup and reducing optimizer cost to near-AdamW levels.
* `RL` `training` [RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning](http://arxiv.org/abs/2606.26997v1)
  > **TL;DR**: Addresses idle time in disaggregated RLVR training systems by overlapping rollout generation and policy training. Proposes RolloutPipe with complete-group pipelining and frontier-group dispatch to pipeline groups for training while still generating rollouts, maintaining on-policy correctness. Reduces rollout-to-train-end time by 30.7%-42.3% and trainer waiting ratio by 37%-76% compared to Slime.
* `training` `MoE` `networking` [Simulating Unified Tensor Resharding in heterogeneous AI systems](http://arxiv.org/abs/2606.26633v1)
  > **TL;DR**: Proposes Xsim, a heterogeneity-aware simulator for distributed LLM training, enabling load balancing and customized communication. It predicts training time with <5% error in heterogeneous data/tensor-parallel configurations.
* `serving` `MoE` `scaling` [Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch](http://arxiv.org/abs/2606.26607v1)
  > **TL;DR**: Addresses the challenge of dynamic workload concurrency changes in MoE model serving by seamlessly switching between tensor and expert parallelism at runtime. Introduces Moebius, which reshards weights and KV cache with fused transfers without dropping requests. Achieves 1.16-1.25x speedup on RL rollouts.
* `sparse` `serving` `kernel` [GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices](http://arxiv.org/abs/2606.26441v1)
  > **TL;DR**: Accelerates exact learned sparse retrieval inference to overcome CPU bottlenecks. Proposes GPUSparse with GPU-parallel inverted indices, batched scatter-add scoring, and fused Triton kernels. Achieves 235x speedup (1.27ms vs. 298ms per query) over CPU baseline at 8.8M documents.
* `serving` `RAG` `kernel` [TileMaxSim: IO-Aware GPU MaxSim Scoring with Dimension Tiling and Fused Product Quantization](http://arxiv.org/abs/2606.26439v1)
  > **TL;DR**: Addresses inefficient GPU MaxSim scoring for ColBERT retrieval by developing TileMaxSim, an IO-aware Triton kernel using dimension tiling and fused product quantization. Achieves 82M documents/second throughput and reduces scoring latency by 98% (1.2 ms vs 268 ms for 100K candidates).
* `training` `scaling` [Hot AI in Cold Space: Thermal-Crosstalk-Aware Scheduling for Sustainable Orbital AI Clusters](http://arxiv.org/abs/2606.26150v1)
  > **TL;DR**: Addresses thermal throttling in orbital LLM training clusters that degrades Model Flops Utilization (MFU). Proposes Thermal-Load Balancing (TLB), a scheduling framework that dynamically migrates workloads to coolest units to restore MFU and extend hardware lifespan.

### 2026-06-25
* `training` `quantization` [NEURON-Fabric: Architecture-Runtime Co-Design for Controlled Low-Bit Gradient Communication](http://arxiv.org/abs/2606.25759v1)
  > **TL;DR**: Proposes NEURON-Fabric, a profile-guided runtime system for dynamic low-bit gradient communication in distributed training. It uses operating profiles and runtime checks to decide aggregation precision, preserving near full-precision accuracy and reducing communication traffic as validated on Transformers and billion-parameter models.
* `RL` `diffusion` `disaggregation` [Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation](http://arxiv.org/abs/2606.24369v2)
  > **TL;DR**: Proposes DigenRL, a disaggregated RL framework for diffusion generative LLMs, using fine-grained pipelining (GAP, TSP) and trainer-assisted generation (TAG) to reduce execution bubbles. Achieves 1.56-2.10× throughput gain over state-of-the-art systems on multi-GPU setups.
* `kernel` [EmuGEMM: Fused Tensor Core Kernels for Precision Emulation in Matrix Multiplication](http://arxiv.org/abs/2606.25453v1)
  > **TL;DR**: Addresses poor high-precision GEMM performance on GPUs by fusing Ozaki scheme operations into efficient Tensor Core kernels to minimize data movement. EmuGEMM achieves 83% of INT8 peak (1,639 TOp/s on Hopper) and up to 1.7x higher throughput than cuBLAS TF32 on Blackwell.
* `serving` `offloading` `edge` [Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off](http://arxiv.org/abs/2606.25091v1)
  > **TL;DR**: Analyzes when distributed speculative decoding (DSD) with edge-cloud partitioning benefits LLM inference. Theoretical modeling shows DSD primarily boosts multi-tenant capacity under server saturation, sustaining (1 + γ·td/tv) times more concurrent clients at the same per-client rate.
* `hardware` `RL` `scaling` [Energy Efficient Scheduling of AI/ML Workloads on Multi Instance GPUs with Dynamic Repartitioning](http://arxiv.org/abs/2606.25082v1)
  > **TL;DR**: Addresses energy efficiency and tardiness for AI/ML workload scheduling on Multi-Instance GPUs. Introduces a dynamic repartitioning framework using reinforcement learning for automatic MIG reconfiguration. Achieves up to 68% improvement in combined energy and tardiness metrics over baseline no partitioning.
* `edge` `serving` [EnerInfer: Energy-Aware On-Device LLM Inference](http://arxiv.org/abs/2606.23001v2)
  > **TL;DR**: Addresses high energy and thermal costs in on-device LLM inference by proposing EnerInfer, a framework that uses model-structure-aware prediction and online feedback to dynamically select efficient configurations. Achieves up to 65% higher energy efficiency on phones without quality of experience violation.

### 2026-06-24
* `MoE` `serving` `disaggregation` [CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation](http://arxiv.org/abs/2606.24506v1)
  > **TL;DR**: Addresses GPU memory inefficiency in serving cold MoE models by disaggregating weights and KV-cache into separate pools. Introduces CrossPool with a planner, virtualizer, and pipeline scheduler. Reduces P99 TBT by up to 10.4x compared to state-of-the-art systems.
* `RL` `diffusion` `disaggregation` [Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation](http://arxiv.org/abs/2606.24369v1)
  > **TL;DR**: Proposes DigenRL, a disaggregated RL framework for diffusion generative models, employing diffusion-based parallelism and trainer-assisted generation to reduce execution bubbles. Achieves 1.56-2.10x throughput improvements over existing systems.
* `training` `networking` [Quantum ring all-reduce: communication and privacy advantages for distributed learning](http://arxiv.org/abs/2606.20344v2)
  > **TL;DR**: Explores quantum communications for enhanced efficiency and privacy in distributed training. Develops quantum ring all-reduce using entanglement and superdense coding, reducing per-link online communication by a factor of two provably. Achieves composable privacy guarantees with low overhead.
* `disaggregation` `networking` [Hash Table Design for RDMA:Challenges and Opportunities](http://arxiv.org/abs/2606.24073v1)
  > **TL;DR**: Addresses inefficiencies in deploying hash tables on RDMA disaggregated memory systems for IoT data. Explores designs to minimize multiple remote accesses while avoiding CPU involvement. Covers reducing RDMA requests during collision handling through structure optimizations.
* `serving` `offloading` `networking` [The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing](http://arxiv.org/abs/2606.23969v1)
  > **TL;DR**: Investigates LLM serving throughput loss under GPU confidential computing on Blackwell GPUs due to serialized host/device transfers. Proposes scheduler flags and worker-thread draining to mitigate bottlenecks. Achieves up to 92% recovery of performance gap in high-concurrency runs.
* `edge` `serving` [EnerInfer: Energy-Aware On-Device LLM Inference](http://arxiv.org/abs/2606.23001v1)
  > **TL;DR**: Addresses energy and thermal inefficiency in on-device LLM inference. Proposes EnerInfer, using model-structure-aware prediction and online feedback to dynamically select efficient NPU/DDR settings under thermal constraints. Achieves up to 65% energy efficiency improvement on phones without QoE violation.

### 2026-06-23
* `serving` `multi-modal` `kernel` [Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse](http://arxiv.org/abs/2606.23581v1)
  > **TL;DR**: Addresses recomputation overhead in multimodal inference from repetitive encoding at different positions. Proposes a position-invariant KV cache with a low-rank conditioning patch, enabling reuse via RoPE re-rotation. Achieves full task accuracy with minimal footprint and bf16 reconstruction in SGLang kernel across six backbones.
* `training` `scaling` [The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model](http://arxiv.org/abs/2606.23546v1)
  > **TL;DR**: Models the energy consumption of Transformer fine-tuning using a roofline-inspired framework incorporating parallelism and hardware efficiency. Achieves accurate energy predictions across heterogeneous GPU configurations.
* `serving` `kernel` `offloading` [Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference](http://arxiv.org/abs/2606.23521v1)
  > **TL;DR**: Addresses fault tolerance in LLM inference by recovering GPU-resident state after failures without restarting. Introduces Concordia, a runtime with a persistent kernel that JIT-compiles delta-checkpoint handlers and logs deltas to CPU memory. Aims to reduce downtime and state loss.
* `serving` `offloading` `multi-modal` [LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs](http://arxiv.org/abs/2606.22983v1)
  > **TL;DR**: Addresses inefficiencies in serving interactive omni-modal LLMs due to unsuitable scheduling and KV management. LiveServe introduces interaction-aware scheduling and next-use-aware KV eviction/preloading, reducing P90 audio TTFP by 1.55× on average and up to 2.21×.
* `training` `networking` [Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo](http://arxiv.org/abs/2606.22768v1)
  > **TL;DR**: Reduces blocking communication in distributed training (DiLoCo) for low-bandwidth environments. Proposes factored gossip synchronization with non-blocking and blocking steps. Achieves substantial compute utilization improvements and robustness to failures while matching DiLoCo's training progress.
* `serving` `MoE` `disaggregation` [ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill](http://arxiv.org/abs/2606.22541v1)
  > **TL;DR**: Addresses synchronization stalls in MoE prefill serving by disaggregating attention and MoE stages and introducing an asynchronous pipeline with optimized communication and scheduling. Achieves 90% higher SLO-compliant prefill throughput over synchronous systems.
* `offline` `serving` `storage` [StickyInvoc: Rethinking Task Models for High-throughput Workflows in the LLM Era](http://arxiv.org/abs/2606.22175v1)
  > **TL;DR**: Reduces overhead in high-throughput LLM workflows by decoupling computational state management from task execution. Proposes StickyInvoc: persistent 'sticky' tasks enable model state reuse across multiple 'invocation' tasks. Achieves 3.6× speedup for a 150k-inference scientific workflow on 20 GPUs.
* `training` `MoE` `networking` [FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs](http://arxiv.org/abs/2606.19025v2)
  > **TL;DR**: Addresses communication and memory inefficiencies in distributed MoE training requiring full replicas. Proposes FoMoE, partitioning experts across workers and skipping non-resident experts via skip-token mechanism. Achieves 1.42x communication reduction over baselines and 1.4x throughput speedup in cross-geo training.
* `offline` `offloading` `scaling` [BatchGen: An Architecture for Scalable and Efficient Batch Inference](http://arxiv.org/abs/2606.21712v1)
  > **TL;DR**: Addresses inefficiencies in scalable batch inference with extreme load variation by proposing the sequence coroutine compute model for dynamic work reorganization. BatchGen reduces batch completion time by up to 2.3x on a 128-GPU cluster and outperforms offloading baselines by 9.6x on memory-constrained accelerators.
* `agentic` `serving` [SwarmX: Agentic Scheduling for Low-Latency Agentic Systems](http://arxiv.org/abs/2606.21401v1)
  > **TL;DR**: Addresses scheduling challenges for agentic applications with multiple model calls. Proposes SwarmX, using neural predictors for prompt-dependent features to enable tail-aware decisions within existing infrastructure. Reduces tail latency by up to 61.5% and sustains 2x throughput versus state-of-the-art schedulers.
* `serving` `storage` [Recency/Frequency Adaptive KV Caching for Large Language Model Serving](http://arxiv.org/abs/2606.21238v1)
  > **TL;DR**: Proposes a recency/frequency adaptive KV caching method for LLM serving to handle diverse workloads. It dynamically balances cache allocation, improving cache hit rate by up to 10.8% and reducing time to first token by up to 12.6% over vLLM.
* `serving` `recommendation` [DPIFrame: A Dual-Level Parallelism Acceleration Framework for CTR Model Inference](http://arxiv.org/abs/2606.21101v1)
  > **TL;DR**: Addresses inefficient CTR model inference on GPUs due to serial-parallel mismatch. Proposes DPIFrame with dual-level parallelism, multi-table lookup optimization, and GPU stream scheduling. Achieves 23× embedding latency reduction and 5.83× speedup over PyTorch.
* `serving` `edge` [FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation](http://arxiv.org/abs/2606.23370v1)
  > **TL;DR**: Addresses high overhead of secure LLM inference on mobile devices using ARM TrustZone. Proposes FlexServe with Recallable Resource Isolation (Flex-Mem and Flex-NPU) enabling cooperative management between secure and normal worlds. Achieves 10.05× faster TTFT over baseline.

### 2026-06-19
* `serving` `edge` `kernel` [Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving](http://arxiv.org/abs/2606.20537v1)
  > **TL;DR**: Proposes execution-state capsules for low-latency on-device LLM serving, enabling graph-bound checkpoint/restore of complete execution state. FlashRT kernel runtime uses contiguous static buffers to eliminate block-table indirection. Achieves sub-millisecond state restore and up to 27x TTFT speedup at 16k context.
* `training` `kernel` [ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters](http://arxiv.org/abs/2606.20374v1)
  > **TL;DR**: Proposes ARGUS, a low-overhead (<2%) tracing system for fine-grained observability in large-scale LLM training. It decomposes events (CPU stacks, framework, kernels) and compresses traces by ∼3,700×. Enables automated detection of anomalies like stragglers and degraded kernels in 10,000+ GPU production clusters.
* `training` `networking` `scaling` [Quantum ring all-reduce: communication and privacy advantages for distributed learning](http://arxiv.org/abs/2606.20344v1)
  > **TL;DR**: Investigates enhancing communication efficiency and privacy in large-scale distributed training. Proposes quantum ring all-reduce using pre-shared entanglement and superdense coding. Achieves up to 2× reduction in per-link online communication.
* `kernel` [The Correctness Illusion in LLM-Generated GPU Kernels](http://arxiv.org/abs/2606.20128v1)
  > **TL;DR**: Exposes the illusion of correctness in LLM-generated GPU kernel benchmarks due to limited fixed-shape testing. Proposes op-schema-aware seeded fuzzing with a high-precision CPU reference. Achieves 100% bug detection (10/10) and correct pass rate (16/16) on extended GPU corpus.
* `edge` `serving` [Edge-Inference Governors Need Memory-Clock State](http://arxiv.org/abs/2606.16106v2)
  > **TL;DR**: Addresses how to meet deadlines and reduce energy for edge ML inference in DVFS governors by modeling memory clock state. Proposes an EMC-aware scheduler deployed on Jetson Orin, reducing deadline misses from 25-28% to at most 1.3% under a 2% QoS budget and lowering energy.
* `training` `LoRA` `multi-modal` [Online Dynamic Batching with Formal Guarantees for LLM Training](http://arxiv.org/abs/2606.19989v1)
  > **TL;DR**: Addresses inefficiencies in LLM training batch formation due to late cost observability. Proposes Online Dynamic Batching (ODB), a DataLoader-side system with formal synchronization guarantees for distributed training. Achieves up to 4.43x throughput gain over standard batching in production multi-modal fine-tuning without model changes.
* `serving` `offloading` `sparse` [SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL](http://arxiv.org/abs/2606.19746v1)
  > **TL;DR**: Addresses memory bottlenecks in LLM serving for sparse attention models with long contexts. Introduces SAC, a CXL-based disaggregated KV cache system fetching only required entries on demand. Achieves 2.1x higher throughput than RDMA-based baselines.

### 2026-06-18
* `video` `serving` `scaling` [TurboServe: Serving Streaming Video Generation Efficiently and Economically](http://arxiv.org/abs/2606.19271v1)
  > **TL;DR**: Addresses inefficiencies in serving streaming video generation with session and demand heterogeneity. Proposes TurboServe, an online scheduling system using migration-aware placement, autoscaling, coalesced processing, and offloading. Achieves 37.5% lower worst-case per-chunk latency and 37.2% cost reduction.
* `training` `diffusion` [Pulse: Training Acceleration for Large Diffusion Models with Automatic Pipeline Parallelism](http://arxiv.org/abs/2606.19163v1)
  > **TL;DR**: Addresses communication bottlenecks in pipeline-parallel training of diffusion models with skip connections. Proposes PULSE, which collocates skip-connected layers and caches skip activations, with a skip-aware partitioner and schedule synthesizer. Reduces communication by 89% and improves throughput by up to 2.3x.
* `training` `MoE` `networking` [FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs](http://arxiv.org/abs/2606.19025v1)
  > **TL;DR**: Addresses full-replica inefficiency in distributed MoE training. Introduces FoMoE with expert partitioning and skip-token to reduce communication and memory. Reduces communication costs by up to 45.44x vs DDP and achieves 1.4x throughput speedup.
* `diffusion` `RL` `training` [Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training](http://arxiv.org/abs/2606.19004v1)
  > **TL;DR**: Proposes Spotlight for efficient RL post-training of Diffusion Transformers using spot GPUs. Key techniques: bandit-based exploration with stale weights, elastic sequence parallelism, preemption-aware scheduling. Achieves 4x faster training and 1.4-6.4x cost reduction.
* `serving` `scaling` `offloading` [ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving](http://arxiv.org/abs/2606.18741v1)
  > **TL;DR**: Addresses high downtime in LLM serving during model parallelism reconfiguration. Proposes ReMP with decoupled topology and dynamic KV cache migration for online adjustments. Achieves reconfiguration in 1-7 seconds for 7B-70B models, 10× to over 100× faster than restarts.
* `serving` `hardware` [Latency Prediction for LLM Inference on NPU Systems](http://arxiv.org/abs/2606.18042v2)
  > **TL;DR**: Proposes LENS, a latency predictor for LLM inference on NPUs facing challenges like undisclosed microarch and bucketing non-linearity. Uses two end-to-end measurements per bucket to compose predictions for arbitrary input-output lengths. Achieves 2.15% mean prediction error across multiple NPUs and LLMs.
* `serving` `scaling` [ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters](http://arxiv.org/abs/2606.18600v1)
  > **TL;DR**: Proposes ShuntServe to enable cost-efficient LLM serving on volatile heterogeneous spot GPU clusters. Combines a roofline-based performance estimator with dynamic programming for model placement and uses request migration with concurrent initialization for fault tolerance. Achieves 1.42x higher throughput and 31.9% cost efficiency improvement over on-demand instances.
* `serving` `scaling` [Beyond Prediction: Tail-Aware Scheduling for LLM Inference](http://arxiv.org/abs/2606.18431v1)
  > **TL;DR**: Addresses tail latency fragility in LLM inference under variable request lengths. Proposes a prediction-free scheduler using statistical boosting and cache-aware preemption. Reduces P99 token latency by 35-50% and time-to-first-token by 34-47% across workloads.

### 2026-06-17
* `serving` `hardware` [Latency Prediction for LLM Inference on NPU Systems](http://arxiv.org/abs/2606.18042v1)
  > **TL;DR**: Addresses LLM inference latency prediction on NPUs despite undisclosed microarchitecture, compiler optimizations, and bucketing. Proposes LENS, requiring only two end-to-end measurements per bucket to predict latency for any input-output length. Achieves 2.15% mean error across multiple NPUs, LLMs, and workloads.
* `serving` `scaling` [RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving](http://arxiv.org/abs/2606.17949v1)
  > **TL;DR**: RouteBalance fuses routing and load balancing for joint optimization in heterogeneous LLM serving. It uses batched predictors and state estimation for low-latency assignment. Achieves 30 req/s and 2.6-4.1× higher throughput than enhanced baselines at high load while avoiding collapse.
* `serving` `offloading` [LUMEN: Coordinated Failure Recovery for Distributed LLM Serving](http://arxiv.org/abs/2606.17787v1)
  > **TL;DR**: Proposes LUMEN for coordinated failure recovery in distributed LLM serving. System employs load-aware coordination across checkpoint placement, request redistribution, and capacity restoration during model reload. Achieves significant improvements in serving and recovery times in experiments and simulations.
* `diffusion` `video` `serving` [AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformers](http://arxiv.org/abs/2606.17566v1)
  > **TL;DR**: Addresses performance loss in TPU-based video diffusion inference from topology-unaware auto-parallelization. Proposes AoiZora, a compiler planner filtering sharding candidates via IR analysis and HLO-based physical placement optimization. Achieves up to 1.42x lower latency for Wan 2.1 denoising.
* `offloading` `disaggregation` `serving` [ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories](http://arxiv.org/abs/2606.12556v2)
  > **TL;DR**: Addresses the challenge of large KV cache footprints in LLM inference by proposing ITME, a CXL-hybrid memory expansion system for offloading, allowing proactive data movement. Results in up to 35.7% throughput improvement.
* `thinking` `kernel` `offloading` [SpecGen: Accelerating Agentic Kernel Optimization with Speculative Generation](http://arxiv.org/abs/2606.17518v1)
  > **TL;DR**: Accelerates agentic kernel optimization by forking speculative kernel generations during reasoning, enabling early termination and parallel validation. Uses dynamic GPU pool reallocation and remote KV cache storage. On H200, reduces end-to-end time over baselines and increases kernel speedup under fixed time and token budget.
* `diffusion` `edge` `serving` [RISE: Relay Inference and Online Scheduling for Efficient Edge-Device Collaborative Diffusion Model Services](http://arxiv.org/abs/2606.17378v1)
  > **TL;DR**: Addresses efficient serving of text-to-image diffusion models at the edge with diverse latency and quality needs. Proposes RISE: a training-free relay inference mechanism for workload splitting and a contextual bandit scheduler. Achieves 2.1× speedup while maintaining model quality.
* `agentic` [Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent Large Language Model Systems](http://arxiv.org/abs/2606.17182v1)
  > **TL;DR**: Addresses concurrency anomalies in multi-agent LLM state sharing. Proposes verified detectors and runtimes with prevention techniques like pessimistic locking, snapshot isolation, and sequencers. Prevents anomaly A3 in all 120 live retracted test sessions.
* `serving` `disaggregation` `hardware` [Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators](http://arxiv.org/abs/2606.17104v1)
  > **TL;DR**: Investigates how emerging AI accelerators compare to GPUs in LLM inference by evaluating distinct Prefill and Decode phases. Measures time to first token and time per output token separately on platforms like GPUs and GroqRack. Finds GroqRack achieves significantly lower TPOT during Decode.
* `serving` `disaggregation` `networking` [The Price of Anarchy in Disaggregated Inference](http://arxiv.org/abs/2606.17081v1)
  > **TL;DR**: Analyzes game-theoretic inefficiencies in disaggregated LLM inference. Proposes an adaptive controller that adjusts routing to avoid congestion during saturation. Reduces Price of Anarchy by 3.1x in the saturated phase for a 70B model with 13% throughput cost.

### 2026-06-16
* `training` `scaling` [Tangram: Hiding GPU Heterogeneity for Efficient LLM Parallelization](http://arxiv.org/abs/2606.16907v1)
  > **TL;DR**: Addresses efficient LLM training on heterogeneous GPU clusters by hiding heterogeneity. Proposes Tangram, which groups GPUs into homogeneous islands for parallelizers and composes model slices into work-balanced pipelines. Achieves up to 2.3x higher throughput than state-of-the-art heterogeneous parallelizers.
* `serving` `agentic` [CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents](http://arxiv.org/abs/2606.16824v1)
  > **TL;DR**: Addresses inefficient KVCache management for serving LLM coding agents by introducing CacheWise, which combines prefix-aware scheduling and reuse-aware eviction guided by tool call metadata predictions. Achieves up to 3.5x improvement in session completion time.
* `serving` `kernel` [SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions](http://arxiv.org/abs/2606.16332v1)
  > **TL;DR**: Investigates inefficiencies in using CPU matrix extensions (SME) for LLM inference operators. Proposes SMEPilot, an engine that dynamically selects execution mode (CPU/SME/cooperative) per operator, partitions matrix work, and reuses packed layouts. Achieves up to 3.94x end-to-end inference speedup.
* `serving` `disaggregation` [Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing](http://arxiv.org/abs/2606.16264v1)
  > **TL;DR**: Addresses the trade-off between queuing and interference in disaggregated LLM serving for SLO attainment. Proposes Tropical, an SLO-aware request multiplexing strategy that balances TTFT and TPOT. Achieves 2.09x more requests within 90% SLO attainment than existing systems.
* `serving` `offloading` `networking` [SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing](http://arxiv.org/abs/2606.16135v1)
  > **TL;DR**: Addresses high latency and memory pressure in multi-turn LLM conversations from inefficient KV cache offloading. Proposes SwiftCache for cross-model KV cache sharing via NVLink, storing only active layers in HBM. Achieves up to 69% lower P99 TTFT compared to vLLM and SGLang.
* `edge` [Beyond CPU-GPU Frequency: Memory-Clock and Tail Effects in Edge Inference Latency Estimation](http://arxiv.org/abs/2606.16106v1)
  > **TL;DR**: Identifies memory-clock shifts (median latency up to +48%) and tail miss bursts (74% conditional miss rate) in edge ML inference. Measurement-driven Pareto tail modeling reduces deadline-miss margin overshoot from 13x-29x via Gaussian to within ~2x.
* `agentic` `serving` [Green SARC: Predictive Cost and Carbon Governance for Agentic AI Systems](http://arxiv.org/abs/2606.15954v1)
  > **TL;DR**: Investigates predictive governance to bound cost and carbon in agentic AI systems. Proposes Green SARC, a framework with enforcement gates for real-time budget control. Achieves 47-55% savings in tokens, USD, and carbon emissions.
* `agentic` [CoAgent: Concurrency Control for Multi-Agent Systems](http://arxiv.org/abs/2606.15376v1)
  > **TL;DR**: Addresses concurrency in multi-agent LLM systems by proposing MTPO protocol with advisory notifications and agent plan repair. CoAgent achieves 1.4× speedup and within 5% deviation from serial correctness at near-serial token cost.
* `serving` `diffusion` `video` [Adaptive Resource Management and Quality Control for Streaming Video Generation](http://arxiv.org/abs/2606.15319v1)
  > **TL;DR**: Addresses playout continuity in real-time streaming video generation. Proposes SlackServe, a system that dynamically reallocates resources and adjusts per-chunk fidelity based on playout slack. Achieves 1.64x-3.29x higher Continuous Play Ratio and 1.61x-9.65x lower Time to First Chunk.
* `edge` `multi-modal` `serving` [Generation Quality-Latency Tradeoff-Aware Inference Offloading for Multimodal LLMs in Cloud-Edge Continuum](http://arxiv.org/abs/2606.15210v1)
  > **TL;DR**: Addresses the tradeoff between generation quality and response latency in cloud-edge MLLM inference by proposing QLMIO, an offloading decision framework. Achieves up to 58.14% latency reduction versus baselines while maintaining cloud-level task completion.
* `serving` `MoE` [Coordinated Scheduling for MoE LLM Serving](http://arxiv.org/abs/2606.15177v1)
  > **TL;DR**: Addresses serving imbalances in MoE LLMs via coordinated cross-level scheduling. Proposes Gimbal, which uses fine-grained backend pressure signals (KV cache, queue pressure) for request dispatch and a MINLP-guided expert placement policy. Achieves 42.9% lower TTFT and 33.3% lower TPOT versus vLLM.
* `serving` `multi-modal` `diffusion` [M*: A Modular, Extensible, Serving System for Multimodal Models](http://arxiv.org/abs/2606.12688v2)
  > **TL;DR**: Proposes M*, a modular serving system for composite multimodal models using Walk Graphs for flexible component composition and distributed execution. Achieves 20% lower latency for text-to-image than vLLM-Omni and 12.5x faster robotic planning than V-JEPA baseline.
* `serving` `networking` `scaling` [Solyx AI Grid: Hardware-Telemetry-Aware Routing Across Geographically Distributed GPU Clusters](http://arxiv.org/abs/2606.15050v1)
  > **TL;DR**: Presents Solyx AI Grid, a cross-site LLM inference routing system using hardware telemetry, application metrics, and WAN signals. It employs a 10-signal weighted pressure scorer for per-request placement, achieving 1.56-1.75x higher throughput at tier-2 SLO and reducing capability-mismatch leakage to 0.43% versus standard routers.
* `training` `quantization` `hardware` [NEURON-Fabric: CXL-Side Low-Bit Gradient Aggregation for Distributed Training](http://arxiv.org/abs/2606.15045v1)
  > **TL;DR**: Proposes NEURON-Fabric, a CXL-side controller for near-memory low-bit gradient aggregation in distributed training to reduce communication overhead. Integrates a switchable low-bit/FP32 datapath. Reduces gradient traffic to 3.6-5.4% of FP32 baseline while maintaining accuracy for models like DistilBERT and ResNet-18.
* `serving` `offloading` `agentic` [CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents](http://arxiv.org/abs/2606.16824v1)
  > **TL;DR**: CacheWise addresses inefficient KVCache management for serving coding agents by introducing prefix-aware scheduling and prediction-guided eviction. Implemented in vLLM, it reduces KVCache evictions by up to 2.6x and improves session completion time by up to 3.5x.

### 2026-06-15
* `serving` `edge` `scaling` [PLAIground: SLO-Driven Runtime Model Selection for Compound AI Systems in the Edge-Cloud-Space Continuum](http://arxiv.org/abs/2606.14356v1)
  > **TL;DR**: Addresses dynamic model selection to satisfy accuracy, latency, and cost SLOs in Compound AI systems across edge-cloud-space. Proposes CAIM abstraction for model-task decoupling and Pixie SLO-driven algorithm. Achieves 91.3% accuracy with SLO compliance, while fixed strategies exhibit up to 21x cost/latency violations or 4% accuracy drops.
* `serving` `scaling` `disaggregation` [Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems](http://arxiv.org/abs/2606.14350v1)
  > **TL;DR**: Proposes system-centric design of compound AI systems using distributed workflows to overcome limitations of monolithic models. Achieves within 2.5-4 pp accuracy while reducing latency by 60% and cost by 71%, though combinatorial challenges exist in configuration management.
* `serving` `networking` [STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming](http://arxiv.org/abs/2606.13968v1)
  > **TL;DR**: Proposes STREAM, a multi-tier LLM inference middleware with complex routing and dual-channel token streaming for HPC. Achieves 0.54 s median TTFT (21.1x faster than batch) for firewall-friendly delivery without rule changes.

### 2026-06-12
* `serving` `diffusion` `scaling` [GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving](http://arxiv.org/abs/2606.13501v1)
  > **TL;DR**: Addresses inefficient static parallelism in Diffusion Transformer serving. Proposes GF-DiT, a runtime with asynchronous execution and group-free collectives for dynamic GPU parallelism adaptation. Achieves up to 6.01× throughput improvement and reduces communication setup overhead from 778 ms to ~60 μs.
* `training` `networking` [ScaleAcross: Designing Multi-Data-Center Infrastructure for Geo-Distributed AI Training](http://arxiv.org/abs/2606.12963v1)
  > **TL;DR**: Designs infrastructure for geo-distributed AI training, addressing synchronization, latency, and cross-site data challenges. Proposes an EVPN-VXLAN framework with queue-pair-aware traffic distribution. Evaluation under realistic WAN conditions demonstrates improved communication behavior for AllReduce and Parameter Server workloads.
* `serving` `agentic` `offloading` [Maestro: Workload-Aware Cross-Cluster Scheduling for LLM-Based Multi-Agent Systems](http://arxiv.org/abs/2606.12950v1)
  > **TL;DR**: Addresses serving challenges in LLM-based multi-agent workflows under GPU constraints. Proposes Maestro, a hierarchical scheduler using agent role predictions for resource-aware scheduling at node, cluster, and global levels. Reduces KV-reservation HBM by 67.2%.
* `edge` `multi-modal` `networking` [SemanticXR: Low Power and Real-time Queryable Semantic Mapping with an Object-Level Device-Cloud Architecture](http://arxiv.org/abs/2606.12849v1)
  > **TL;DR**: Enables low-power, real-time semantic mapping for XR by proposing an object-level device-cloud architecture. Uses object parallelism and downsampling on server, sparse map updates on device, and depth-mapping co-design. Achieves 2.2x server latency improvement, <2.5 Mbps bandwidth, and 2% device power overhead.
* `hardware` `kernel` [Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU](http://arxiv.org/abs/2606.12765v1)
  > **TL;DR**: Characterizes Apple Metal 4.1's hidden tensor compute path via microbenchmarking, revealing fp8 matmul emulation and hardware behavior. An optimized hand-fused GEMM + bias + GELU kernel improves performance by 6.5-12.9% in the cache-resident regime.
* `serving` `multi-modal` `diffusion` [M*: A Modular, Extensible, Serving System for Multimodal Models](http://arxiv.org/abs/2606.12688v1)
  > **TL;DR**: Addresses limitations in serving composite multimodal models. Introduces M*, which represents models as dataflow graphs (Walk Graph) for modular composition and distributed optimization. Achieves 20% lower latency than vLLM-Omni for text-to-image tasks and up to 2.9x lower real-time factor for text-to-speech workloads.
* `training` `networking` [Eidola: Modeling Multi-GPU Network Communication Traffic in Distributed AI Workloads](http://arxiv.org/abs/2606.12638v1)
  > **TL;DR**: Addresses modeling irregular inter-GPU network communication traffic in distributed AI training. Introduces Eidola, a gem5 extension using annotated timing profiles for cycle-level simulation of peer-to-peer writes. Confirms reductions in polling-related memory traffic.
* `offloading` `serving` `storage` [ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories](http://arxiv.org/abs/2606.12556v1)
  > **TL;DR**: Addresses handling large KV caches beyond host memory in LLM inference by proposing ITME, a disaggregated CXL-hybrid tiered memory expansion system with proactive data movement. Achieves up to 35.7% throughput improvement.

### 2026-06-11
* `serving` `edge` `scaling` [Fair Comparison of Scheduling Algorithms on Heterogeneous Edge Clusters: A Continuous Adaptive Benchmark](http://arxiv.org/abs/2606.12343v1)
  > **TL;DR**: Proposes CMMS benchmark for scheduling algorithms on edge-cloud clusters that assign nodes and processing modes to meet AI task SLOs. Provides unified controller interface, closed-loop workload driver, and dual SLO scoring. Finds deep RL scheduler loses by 29 percentage points under intense load vs heuristic with 500× overhead.
* `MoE` `RL` `training` [Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training](http://arxiv.org/abs/2606.11867v1)
  > **TL;DR**: Addresses expert load imbalance during micro-steps in RL post-training for MoE models. Introduces ForeMoE, which exploits routing foresight and a hierarchical planner with hardware-complementary transfer for load balancing. Achieves up to 1.45× speedup on 64 GPUs.
* `serving` `quantization` `MoE` [Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation](http://arxiv.org/abs/2606.11690v1)
  > **TL;DR**: Proposes a concurrency-aware LLM cost estimation method to address GPU underutilization errors in existing calculators. Introduces vllm-cost-meter for real server measurements, revealing 2.5-24x underutilization penalties and FP8 quantization improving MoE throughput by 69-74% vs 31% for dense models.
* `quantization` `kernel` `edge` [TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs](http://arxiv.org/abs/2606.11357v1)
  > **TL;DR**: Proposes TileFuse, a kernel library for efficient quantized LLM inference on AMD NPUs. Fuses unpacking, dequantization, and GEMM/GEMV into a single kernel, co-designing layout and dataflow. Achieves up to 2.0x lower prefilling latency and 64.6% lower energy in end-to-end experiments.
* `serving` [Real-Time Language Model Jamming: A Case Study for Live Music Accompaniment Generation](http://arxiv.org/abs/2606.11886v1)
  > **TL;DR**: Addresses frame-synchronous streaming inference for real-time LM generation aligned with external signals. Introduces StreamMUSE, a client-server system that sends high-frequency requests for synchronized outputs. Achieves real-time performance by adapting configurations to varying round-trip latencies across environments.

### 2026-06-10
* `training` `MoE` [Piper: A Programmable Distributed Training System](http://arxiv.org/abs/2606.11169v1)
  > **TL;DR**: Addresses inflexibility in manual design of training strategies for large models. Proposes Piper, a system that decouples strategy from runtime using annotated scheduling directives and a compiled global DAG. Achieves performance parity with existing systems and enables improved efficiency for composed parallelism like DualPipe.
* `MoE` `serving` `disaggregation` [Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design](http://arxiv.org/abs/2606.10493v1)
  > **TL;DR**: Proposes CPU-GPU hybrid system to achieve cloud-grade SLOs for local MoE inference via stream-loading prefill, disaggregation, and optimized kernels. Achieves 4-5x lower CPU latency, 50% higher throughput, and 32K prompts within 30 seconds on consumer hardware.
* `serving` `inference` `networking` [ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling](http://arxiv.org/abs/2606.10440v1)
  > **TL;DR**: Addresses the need for high-fidelity simulation of distributed ML systems to optimize latency-sensitive collective communication. Enhances ASTRA-sim with detailed GPU modeling, InfraGraph infrastructure, and cache-line granularity. Enables scalable explorations to improve efficiency in inference optimization.
* `training` `scaling` [RATrain: A Resource-Aware Training Runtime for Large Language Models on Bandwidth-Constrained Heterogeneous Supercomputing Platforms](http://arxiv.org/abs/2606.10415v1)
  > **TL;DR**: Proposes RATrain, a resource-aware training runtime for dense LLMs on bandwidth-constrained supercomputing platforms. It schedules training states (gradient sync, parameter update) at layer-level with an MT-3000 backend and resource-aware planner. Achieves up to 1.35× speedup and 97% scaling efficiency for LLaMA-2-7B on 1024 clusters.

### 2026-06-09
* `serving` `kernel` `quantization` [AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis](http://arxiv.org/abs/2606.09682v1)
  > **TL;DR**: Introduces AutoMegaKernel, compiling LLMs into single, safe CUDA megakernels for inference. Validates schedules statically and employs agent-driven search. Achieves up to 1.33x faster batch-1 decode vs cuBLAS on L4 GPUs with int8 (W8A16) quantization.
* `serving` `multi-modal` `scaling` [FMplex: Model Virtualization for Serving Extensible Foundation Models](http://arxiv.org/abs/2606.09643v1)
  > **TL;DR**: Addresses inefficient resource usage when serving independent foundation model tasks by replicating backbones. Introduces FMplex, a virtualization system with task-isolated shared backbones and a batch-aware fair-queueing scheduler. Achieves up to 80% lower latency and hosts 6x more tasks at cluster scale.
* `training` [Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads](http://arxiv.org/abs/2606.09200v1)
  > **TL;DR**: Addresses communication overhead in multi-GPU training by enabling concurrent computation and communication. Introduces runtime controls: shared-memory-driven occupancy shaping for compute kernels and high-priority scheduling for communication kernels. Reduces total execution time by up to 25.5% without kernel modifications.
* `serving` `disaggregation` [FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location](http://arxiv.org/abs/2606.04415v2)
  > **TL;DR**: Addresses inefficient co-location of prefill and decode phases in LLM serving. Proposes FlexNPU, a transparent virtualization layer enabling dynamic phase-aware scheduling for NPUs. Reduces TTFT by over 92% on Qwen2.5-7B with comparable throughput.
* `serving` [Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving](http://arxiv.org/abs/2606.09061v1)
  > **TL;DR**: Addresses scheduling fairness and latency control in chunked-prefill LLM serving. Introduces aging-based priority calculation, LPRS for temporal constraints, and APC for prefill concurrency regulation. Achieves >10% mean latency reduction and significant P99 tail latency improvements versus FCFS.
* `quantization` `serving` `kernel` [APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing](http://arxiv.org/abs/2606.08761v1)
  > **TL;DR**: Addresses dequantization overhead in pure W4A4 LLM inference. Co-designs INT4 GEMM kernels with ρ-aware granularity adaptation to balance Tensor Core and CUDA Core workloads. Achieves up to 2.09× end-to-end speedup on A40 GPUs when deployed in vLLM.
* `serving` `quantization` `disaggregation` [SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving](http://arxiv.org/abs/2606.08635v1)
  > **TL;DR**: Proposes SpectrumKV, per-token mixed-precision KV cache transfer for prefill-decode disaggregated LLM serving. Assigns FP16, INT8, INT4 to tokens by importance and model tolerance via a probe. At 50% KV budget, perplexity change is +1.97% vs. PDTrim's +25.85% and TTFT reduced by 50-62% at b=0.5.
* `training` [FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training](http://arxiv.org/abs/2606.08476v1)
  > **TL;DR**: Addresses workload imbalance and communication overhead in context parallelism for LLM training by introducing FlashCP, which combines sharding-aware communication and a Whole-Doc strategy via a heuristic planner. Achieves up to 1.63× speedup over prior methods.
* `serving` `edge` `networking` [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](http://arxiv.org/abs/2606.04581v2)
  > **TL;DR**: Proposes Multi-SPIN, a distributed speculative inference system for edge LLM serving that jointly optimizes draft-length and bandwidth allocation in multi-access networks to maximize token goodput. Achieves up to 88% higher goodput than baselines.
* `training` `edge` `RL` [AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments](http://arxiv.org/abs/2606.08197v1)
  > **TL;DR**: Addresses inefficiency in federated fine-tuning of LLMs. Proposes AlignFed with multi-stage alignment to mitigate model drift and improve fairness in heterogeneous edge settings. Enhances resource utilization and stability in high-heterogeneity scenarios.
* `serving` `agentic` [Cost-Aware Speculative Execution for LLM-Agent Workflows: An Integrated Five-Dimension Method](http://arxiv.org/abs/2606.07846v1)
  > **TL;DR**: Reduces idle time and cost in LLM-agent workflows via cost-aware speculative execution. Introduces a five-dimension method with Bayesian probability estimation and a self-limiting rule. Validated via a synthetic suite to confirm expected decision boundaries and cancellation behavior.
* `serving` `multi-modal` `scaling` [FMplex: Model Virtualization for Serving Extensible Foundation Models](http://arxiv.org/abs/2606.09643v1)
  > **TL;DR**: Addresses inefficient deployment of multiple customized foundation model tasks with duplicated backbones. Proposes FMplex, a virtualization system enabling backbone sharing across tasks via virtual FMs and a batch-aware fair-queueing scheduler. Achieves up to 80% lower latency and hosts 6x more tasks.

### 2026-06-08
* `serving` `edge` `scaling` [Clairvoyant: Predictive SJF Scheduling to Mitigate Head-of-Line Blocking in Serial LLM Backends](http://arxiv.org/abs/2606.07248v1)
  > **TL;DR**: Mitigates head-of-line blocking in serial LLM inference backends via Clairvoyant, a sidecar proxy predicting response lengths with an XGBoost classifier for SJF scheduling. Achieves 70-76% P50 latency reduction for short requests under 100 concurrent workloads.
* `training` `networking` `scaling` [PCCL: Process Group-Aware Scalable and Generic Collective Algorithm Synthesizer](http://arxiv.org/abs/2606.07019v1)
  > **TL;DR**: Addresses inefficiency in collective communications for distributed ML training, especially with process groups. Proposes PCCL, a topology-aware algorithm synthesizer supporting arbitrary collective patterns. Synthesizes 512-NPU All-to-All algorithm in 11.68 minutes.
* `MoE` `training` `serving` [UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing](http://arxiv.org/abs/2606.04101v2)
  > **TL;DR**: Addresses load imbalance in MoE training and serving prefill causing inefficiencies. Designs UltraEP, a real-time balancer with quota-driven planning and RSN-native streaming for expert-state transfers. Attains 94.3% of force-balanced ideal throughput, 1.49x over non-balancing and reduces inter-rank imbalance to 1.01-1.04.
* `training` [StageFrontier: Synchronization-Aware Stage Accounting for Distributed ML Training](http://arxiv.org/abs/2606.06751v1)
  > **TL;DR**: Proposes StageFrontier, a synchronization-aware stage accounting system for distributed ML training. It uses coarse stage vectors and frontier timing to pinpoint delay origins with minimal overhead. Adds only 0.2% overhead on 128-GPU clusters and accurately identifies slowdown causes.
* `quantization` `kernel` [P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8](http://arxiv.org/abs/2606.06521v1)
  > **TL;DR**: Investigates precision loss in FP8 attention computation during softmax casting under Attention Sink. Proposes reverse KV iteration and S=256 scaling to prevent underflow and minimize quantization errors. Achieves 3-10x MSE improvement in kernel-faithful experiments.
* `hardware` `kernel` [DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Accelerators](http://arxiv.org/abs/2606.06515v1)
  > **TL;DR**: Proposes DxPTA for HW/SW co-design of photonic transformer accelerators via optical dataflow-guided DSE. Devises constraint-aware architecture search balancing area/power/latency. Achieves 6ms latency (within 10ms SLO) with 15.2x faster design exploration than exhaustive methods.

### 2026-06-06
* `serving` [Beyond Greedy Chunking: SLO-Aware Sliding-Window Scheduling for LLM Inference](http://arxiv.org/abs/2606.05933v1)
  > **TL;DR**: Proposes SlidingServe, a scheduling system for LLM inference that employs dynamic chunking and multi-level priority sorting to optimize throughput while meeting SLOs. Achieves up to 30% higher service capacity and reduces SLO violations by 16%-53% compared to existing systems.
* `serving` `thinking` [Latent Reasoning Guidance for Parallel Code Translation](http://arxiv.org/abs/2606.05518v1)
  > **TL;DR**: Proposes latent reasoning guidance via a Process Reward Model (PRM) to select better hidden-state trajectories during code generation, reducing test-time computation. PRM guidance increases validation rate from 32.89% to 42.1% on ParaTrans benchmark tasks.

### 2026-06-04
* `serving` [Ekka: Automated Diagnosis of Silent Errors in LLM Inference](http://arxiv.org/abs/2606.04594v1)
  > **TL;DR**: Investigates automated diagnosis of silent errors in LLM inference serving frameworks. Proposes Ekka, which aligns intermediate execution states between target and reference systems for root cause identification. Achieves 80% pass@1 diagnosis accuracy on a benchmark of real-world errors.
* `serving` `edge` [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](http://arxiv.org/abs/2606.04581v1)
  > **TL;DR**: Proposes Multi-SPIN, a distributed speculative inference framework for edge-based LLM serving, using on-device draft generation and server batch verification. Optimizes joint draft-length control and bandwidth allocation to handle device heterogeneity. Achieves up to 88% higher sum token goodput than baselines.
* `serving` `diffusion` [D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models](http://arxiv.org/abs/2606.04446v1)
  > **TL;DR**: Addresses low token acceptance in diffusion-based speculative decoding for LLM inference. Introduces D^2SD with two diffusion drafters: the first generates tokens and confidence scores to guide prefix recovery, the second proposes top-K alternative continuations in parallel; candidates are jointly verified via cascade attention. Achieves improved acceptance rates over baseline methods.
* `training` `RL` `MoE` [Schedule-Level Shared-Prefix Reuse for LLM RL Training](http://arxiv.org/abs/2606.01143v3)
  > **TL;DR**: Eliminates redundant shared-prefix computation in RL post-training for LLMs. Introduces a schedule-level reuse mechanism decoupling prefix and suffix computation for gradient accumulation and activation offloading. Achieves up to 4.395x speedup and reduces peak HBM by 59.1%.
* `serving` `disaggregation` [FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location](http://arxiv.org/abs/2606.04415v1)
  > **TL;DR**: Addresses inflexible NPU scheduling for LLM prefill-decode phases. Proposes FlexNPU, a transparent virtualization layer that enables dynamic co-location via AscendCL API interception and phase-aware scheduling. Achieves up to 26.33% throughput gain and over 92% TTFT reduction compared to static approaches.
* `training` `RL` [EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms](http://arxiv.org/abs/2606.04145v1)
  > **TL;DR**: Proposes EvalStop, a scheduler primitive that uses world feedback to detect reward overoptimization in multi-tenant RLHF platforms. It preemptively terminates jobs on eval decline, freeing GPUs and reducing wasted compute by 22% while improving job completion time by 9% with 98% precision detection.
* `MoE` `training` `serving` [UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing](http://arxiv.org/abs/2606.04101v1)
  > **TL;DR**: Addresses expert load imbalance in MoE training and serving prefill on rack-scale nodes. Proposes UltraEP, a real-time balancer using quota-driven planning and persistent tile streaming. Achieves 94.3% of ideal throughput, 1.49× improvement over non-balancing, reducing rank imbalance to near 1.00.
* `serving` `disaggregation` `networking` [NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference](http://arxiv.org/abs/2606.03910v1)
  > **TL;DR**: Addresses network-induced delays in disaggregated LLM inference by accounting for dynamic network conditions during scheduler selection. Proposes NetKV, a greedy scheduler with a network cost oracle for topological distance and congestion. Reduces mean TTFT by up to 21.2% and improves SLO attainment by 20.1 percentage points.
* `edge` `disaggregation` `serving` [E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments](http://arxiv.org/abs/2606.03770v1)
  > **TL;DR**: Addresses efficient LLM serving in resource-constrained edge/fog environments. Proposes E2LLM, a framework with prefill/decoder role-based replica groups and uses Genetic Algorithm / Dynamic Programming for optimal clustering and partitioning. Achieves >50% reduction in average waiting time vs Splitwise.
* `training` [Demystifying Pipeline Parallelism: First Theory for PipeDream](http://arxiv.org/abs/2606.03498v1)
  > **TL;DR**: Provides theoretical analysis for PipeDream's pipeline parallelism. Introduces RPD for nonconvex convergence and diagnoses delay scaling as $S^2 - S/2 + O(1)$ for $S$ stages. Experiments show PD excels on quadratics but LocalSGD improves with more stages for logistic regression.
* `agentic` `RL` `scaling` [Libra: Efficient Resource Management for Agentic RL Post-Training](http://arxiv.org/abs/2606.03077v1)
  > **TL;DR**: Addresses resource management challenges in agentic RL post-training with long-tailed, non-stationary workloads. Introduces Libra, featuring a global resource planner and causality-driven scheduler (C-MLFQ). Achieves up to 3.0× higher throughput and 2.5× faster reward convergence.
* `serving` `edge` [Brief Announcement: Generative Markov Model for Distributed Computing Systems](http://arxiv.org/abs/2606.03061v1)
  > **TL;DR**: Proposes a generative Markov model framework for optimizing collaborative AI inference in distributed systems. Distributes computation across user devices to avoid centralized bottlenecks, reducing latency and server resource consumption in inference scenarios.
* `training` [FOLD: Fuzzy Online Deduplication for Very Large Evolving Datasets via Approximate Nearest Neighbor Search](http://arxiv.org/abs/2606.03001v1)
  > **TL;DR**: Proposes FOLD, an online fuzzy deduplication system for LLM training corpora using an incrementally updated HNSW index and bitmap representations to improve Jaccard similarity. Achieves 93-97% recall and up to 2.09x higher throughput than baselines at large scales.
* `serving` [DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference](http://arxiv.org/abs/2606.02982v1)
  > **TL;DR**: Addresses token drift causing misclassification and QoS degradation in multi-tenant LLM serving. Proposes DriftSched with adaptive workload classification and drift compensation. SJF reduces median latency by ~42% and P99 by ~16% versus FIFO under GPU contention.
* `edge` `agentic` `sparse` [AURA: Action-Gated Memory for Robot Policies at Constant VRAM](http://arxiv.org/abs/2606.02775v1)
  > **TL;DR**: Addresses high memory write costs for long-horizon edge inference. AURA wraps a frozen model with a constant recurrent memory and a learned gate that writes only when inputs change action, yielding 5.2-7x fewer writes at constant 4.2KB memory without loss in task success.
* `agentic` `serving` [Agent libOS: A Library-OS-Inspired Runtime for Long-Running, Capability-Controlled LLM Agents](http://arxiv.org/abs/2606.03895v1)
  > **TL;DR**: Proposes Agent libOS, a runtime for long-running LLM agents enabling scheduling, authorization, and audit. Designs AgentProcesses with explicit capabilities and runtime primitives as the authority boundary. Includes prototype with async scheduling and 123 regression tests.

### 2026-06-02
* `serving` `agentic` `disaggregation` [Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving](http://arxiv.org/abs/2606.01839v1)
  > **TL;DR**: Eliminates prediction need in agentic LLM serving by scheduling at conversation level. ConServe routes turn-1 prefill to high-throughput hardware and pins decoding using observable inputs, reducing p95 time-to-first-effective-token by 51.08% and energy by up to 22.75%.
* `training` `networking` `scaling` [Don't Let a Few Network Failures Slow the Entire AllReduce](http://arxiv.org/abs/2606.01680v1)
  > **TL;DR**: Proposes OptCC, a fault-tolerant AllReduce algorithm for distributed training. Achieves near-optimal completion time under asymmetric bandwidth constraints by intelligent pipeline design and routing. Reduces overhead to 2-6% versus fault-free performance in experiments, versus up to 57% in state-of-the-art approaches.
* `quantization` `kernel` [TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization](http://arxiv.org/abs/2606.01556v1)
  > **TL;DR**: Addresses accuracy degradation in 4-bit LLM quantization via learnable subspace decomposition and a fused kernel for efficient execution. Preserves near-FP16 accuracy and achieves 1.8x end-to-end speedup over FP16 baseline.
* `RAG` [Self-Conditioned Positional HNSW for Overlap-Aware Retrieval in Chunked-Document RAG Systems: Method and Industrial Evidence-Quality Audit](http://arxiv.org/abs/2606.01542v1)
  > **TL;DR**: Addresses redundant chunk retrieval in RAG due to overlapping documents. Proposes SCP-HNSW with positional codes and a two-pass query for auditable context construction. A text audit reveals 574 out of 770 reviews rated 3/5, motivating evidence-quality improvements.
* `serving` `multi-modal` [Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defense](http://arxiv.org/abs/2606.01513v1)
  > **TL;DR**: Addresses efficient and compliant multimodal document generation at scale. Proposes a guardrail orchestration layer with parallel candidate generation and compliance scoring for early exit. Achieves 5 attempts within 20 seconds at 91% compliance.
* `serving` `offloading` `sparse` [Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics](http://arxiv.org/abs/2606.01502v1)
  > **TL;DR**: Reduces latency for cross-GPU LLM attention in agentic workloads. Proposes routing compressed queries instead of KV cache blocks using Multi-head Latent Attention, enabled by a cost model and predicate. Replaces ~3 ms cache splicing with tens-of-microsecond query transfers via RDMA.
* `serving` `offloading` [Fail-Closed Lowering of Resident KV Claims onto LLM Serving Runtimes](http://arxiv.org/abs/2606.01387v1)
  > **TL;DR**: Addresses the gap between KV-cache primitives and future reuse obligations in LLM serving runtimes. Proposes a fail-closed lowering framework with checker and descriptors for runtime validation. Demonstrates with vLLM patch establishing claim-scoped outcomes.
* `RL` `training` `offloading` [Schedule-Level Shared-Prefix Reuse for LLM RL Training](http://arxiv.org/abs/2606.01143v1)
  > **TL;DR**: Addresses inefficiency of recomputing shared prefixes in RL training for large language models. Proposes a decoupled schedule that reuses prefixes via K/V and gK/gV offloading. Achieves up to 4.395x speedup and 59.1% peak HBM reduction.
* `serving` `agentic` `kernel` [Leyline: KV Cache Directives for Agentic Inference](http://arxiv.org/abs/2606.01065v1)
  > **TL;DR**: Addresses inefficient KV cache management for agentic LLMs requiring policy-driven cache edits without full re-prefill. Introduces Leyline, a serving primitive with declarative directives and a RoPE-rotation kernel for in-place edits. Splice kernel increases cache-hit by 11.2 pp and reduces latency up to 241 ms.
* `serving` `scaling` [Lodestar: An Online-Learning LLM Inference Router](http://arxiv.org/abs/2606.00946v1)
  > **TL;DR**: Addresses efficient routing for LLM inference requests in distributed clusters to reduce latency and improve GPU utilization. Proposes Lodestar, an online-learning system that trains a reward predictor to select optimal GPU instances per request. Achieves 1.41x lower average TTFT and 1.47x lower P99 TTFT versus state-of-the-art baselines.
* `serving` `MoE` [ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving](http://arxiv.org/abs/2606.00735v1)
  > **TL;DR**: Addresses execution-time imbalance in distributed MoE inference caused by workload skew and hardware variability. Proposes ViBE, a frame-work combining per-GPU modeling and profiling for optimal expert placement. Reduces P90 TTFT by up to 45% and increases SLO attainment by 14%.
* `kernel` [ScanWeaver: Compiler-Driven Parallelization of Affine Recurrences via Associative Scan Lowering](http://arxiv.org/abs/2606.00601v1)
  > **TL;DR**: Addresses limited parallelization of affine recurrences like Mamba on accelerators. Proposes ScanWeaver, a compiler that transforms recurrences into associative scans and lowers them to MLIR-generated Blelloch scan execution on GPUs. Benchmarks demonstrate reduced execution time compared to sequential baselines.
* `training` `scaling` [HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity](http://arxiv.org/abs/2606.00271v1)
  > **TL;DR**: Addresses misaligned model updates in asynchronous distributed training under data and device heterogeneity. Proposes HeLoCo, a method that corrects stale pseudo-gradients using outer momentum to align with global optimization. Achieves 7.5% validation loss improvement at fixed token budget over baselines.
* `training` `MoE` `agentic` [PithTrain: A Compact and Agent-Native MoE Training System](http://arxiv.org/abs/2605.31463v1)
  > **TL;DR**: Addresses high agent-task inefficiency in evolving MoE training frameworks. Designs PithTrain with agent-native principles for automated development. Reduces Agent Turns by 62% and Active GPU Time by 64% while matching baseline throughput.
* `offloading` `kernel` [Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended](http://arxiv.org/abs/2605.30728v1)
  > **TL;DR**: Reduces GPU memory bottleneck by minimizing PCIe transfers for ML tensors via Invariant Bit Packing (IBP), a lossless compression method using warp-parallel decompression. Achieves 24% faster LLM inference and 74%/180% speedups in GNN/DLRM.
* `serving` `quantization` `kernel` [Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode](http://arxiv.org/abs/2605.30571v1)
  > **TL;DR**: Reveals that batch-1 LLM decode for physical systems fails to scale with memory bandwidth despite being memory-bound, due to kernel launch overhead. CUDA Graphs reduces latency by 1.259x on H100 at context 2048. Quantized models on L4 achieve up to 3.6x speedup but not full gains.
* `kernel` [From Roofline to Ruggedness: Decomposing and Smoothing the GEMM Performance Landscape](http://arxiv.org/abs/2605.29752v1)
  > **TL;DR**: Addresses the rugged performance variation in GEMM operations on GPUs. Introduces a two-stage approach: dynamic tile selection and a dynamic-programming-based padding-splitting optimizer. Achieves 70% roughness reduction and 30% higher mean throughput on BF16 GEMM.
* `serving` `offloading` `agentic` [Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI](http://arxiv.org/abs/2606.00866v1)
  > **TL;DR**: Addresses inefficient KV cache offloading in agentic LLM serving systems during tool-call idle windows. Proposes MORI, which ranks programs by idleness, dynamically allocates memory tiers, and enforces admission control. Achieves 20-71% higher throughput than baseline offloading approaches.

### 2026-05-29
* `training` [AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training](http://arxiv.org/abs/2605.29664v1)
  > **TL;DR**: Addresses convergence degradation in asynchronous pipeline training from parameter mismatch. Introduces AMDP, using bounded minibatches per stage, multiple concurrent pipelines, and gradient accumulation in one update. Accelerates training significantly while preserving convergence.
* `edge` `MoE` [Rotary GPU: Exploring Local Execution Paths for Large Mixture-of-Experts Models Under Limited GPU Memory](http://arxiv.org/abs/2605.29135v1)
  > **TL;DR**: Explores efficient local execution of large MoE models under tight GPU memory constraints. Proposes Rotary GPU, enabling a 35B-class MoE model to run on an 8GB laptop GPU, achieving 6.3GB VRAM usage and 21.06 tokens/s decode throughput.
* `serving` `disaggregation` `quantization` [RTP-LLM: High-Performance Alibaba LLM Inference Engine](http://arxiv.org/abs/2605.29639v1)
  > **TL;DR**: Presents RTP-LLM, a high-performance LLM inference engine using prefill-decode disaggregation, KV cache management, and adaptive quantization. Achieves 4.7x-6.3x model loading speedup and 35-37% TTFT latency reduction in production over vLLM/SGLang.

### 2026-05-28
* `serving` `networking` `edge` [SwarmHarness: Skill-Based Task Routing via Decentralized Incentive-Aligned AI Agent Networks](http://arxiv.org/abs/2605.28764v1)
  > **TL;DR**: Seeks to efficiently utilize idle compute resources for AI task serving via a decentralized protocol. Proposes SwarmHarness with DHT-based peer discovery, utility-driven task routing, and a Shapley-value credit incentive mechanism. Aims to improve resource utilization and reduce costs without centralized coordination.
* `serving` `MoE` `disaggregation` [How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving](http://arxiv.org/abs/2605.28302v1)
  > **TL;DR**: Assesses Attention-FFN disaggregation (AFD) for MoE LLM serving under TTFT/TPOT SLOs. Proposes a partitioned execution framework using kernel measurements and network simulation. Achieves 4k tokens/s on DeepSeek-V3.2 workloads where non-AFD fails.
* `serving` `offline` `networking` [SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference](http://arxiv.org/abs/2605.28095v1)
  > **TL;DR**: Addresses memory constraints in data parallelism for offline LLM inference. Proposes SiDP, a weight-sharing paradigm with on-demand remote access via WaS (weight streaming) and CaS (activation shipping). Achieves up to 1.8x KV capacity increase and 1.5x throughput gain over vLLM.
* `training` `networking` [Throughput-Optimized Networks at Scale](http://arxiv.org/abs/2605.27963v1)
  > **TL;DR**: Proposes TONS, an automated network synthesis framework for optimizing datacenter network topology and routing for AI training. Uses linear optimization and a deadlock-free routing scheme. Achieves geometric mean speedups of 2.1x and 1.6x over TPU v4/5p torus variants for uniform random and all-to-all traffic.
* `training` `multi-modal` [Addressing Variable Heterogeneity in Distributed Multimodal Training with Entrain](http://arxiv.org/abs/2605.27918v1)
  > **TL;DR**: Addresses workload heterogeneity in distributed multimodal LLM training. Introduces Entrain with static model-parallel configuration and hierarchical microbatch assignment to balance loads. Achieves up to 1.40x higher training throughput.
* `training` `multi-modal` `scaling` [Heterogeneous Parallelism for Multimodal Large Language Model Training](http://arxiv.org/abs/2605.27678v1)
  > **TL;DR**: Proposes heterogeneous parallelism for multimodal LLM training, decoupling module layouts and placements with boundary communicators for cross-layout transforms. Achieves up to 49.3% higher TFLOPS/GPU (colocated) and 13.0% higher token throughput (non-colocated) while maintaining convergence parity.
* `training` `storage` [Orbax: Distributed Checkpointing with JAX](http://arxiv.org/abs/2605.23066v2)
  > **TL;DR**: Addresses the lack of a standardized distributed checkpointing solution in JAX. Introduces Orbax, a modular library that abstracts distributed accelerator complexities for flexible checkpointing. Achieves up to 3.5× faster saving and 2× faster loading than PyTorch counterparts.
* `edge` `agentic` `hardware` [The Energy Blind Spot: NVIDIA's Flagship Edge AI Hardware Cannot Support Process-Level Energy Attribution](http://arxiv.org/abs/2605.27599v1)
  > **TL;DR**: Audit of NVIDIA's GB10 edge platform reveals no CPU energy telemetry, hindering per-process energy attribution for agentic AI. Proposes calibration via external DC metering with GPU subtraction and SCMI standards. Prior work finds CPU energy up to 44% of total in agentic workloads.

### 2026-05-27
* `scaling` `training` [Provisioning to Runtime Optimization of a 100 MW-Scale AI Cluster](http://arxiv.org/abs/2605.24461v2)
  > **TL;DR**: Addresses end-to-end power management for hyper-scale AI clusters from planning to runtime. Proposes an optimization process validated in a 150 MW facility with 83K GB200 GPUs. Shares measurements and insights from operating at this scale.
* `MoE` `offloading` `serving` [ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference](http://arxiv.org/abs/2605.27081v1)
  > **TL;DR**: Addresses high I/O overhead from expert fetches in memory-constrained MoE LLM inference. Proposes router fine-tuning (ReMoE) that biases towards recently selected experts to boost cache locality. Achieves 8.4% higher throughput and 43.6-49.8% lower TPOT, accelerating decode by up to 1.99×.
* `networking` [Extreme-Scale Interconnection Networks](http://arxiv.org/abs/2605.26960v1)
  > **TL;DR**: Investigates efficient network topologies for extreme-scale data centers supporting AI workloads. Proposes Multipass Random Leaf-Spine (MRLS) with non-minimal routing. Achieves 50% speedup in All2All collective performance versus Fat-Tree for 100k endpoints.
* `networking` `training` [Revisiting Bruck: Phase-Efficient All-to-All Communication in Reconfigurable Networks](http://arxiv.org/abs/2605.26930v1)
  > **TL;DR**: Addresses inefficient All-to-All communication in optical reconfigurable networks for distributed ML. Proposes ReTri, a bidirectional schedule leveraging balanced ternary propagation to reduce phases and amortize reconfiguration delays. Achieves up to 10× faster completion versus static systems and 2.1× over prior reconfigurable approach.
* `serving` [Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks](http://arxiv.org/abs/2605.24217v2)
  > **TL;DR**: Addresses measurement bias in LLM inference benchmarks under high concurrency. Proposes a multi-process framework with NTPOT metric to isolate serving performance. Enables accurate profiling at production scales exceeding thousands of queries per second.
* `agentic` `serving` [Agentic AI Workload Characteristics](http://arxiv.org/abs/2605.26297v1)
  > **TL;DR**: Characterizes agentic AI workloads to inform efficient serving. Using end-to-end tracing, finds token reuse via context caching makes execution decode-dominated and dependent on KV-cache persistence. Observes tool-use evolution from read/explore to execute/write, highlighting design requirements for joint model, state, and tool management.

### 2026-05-26
* `serving` `RAG` [An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG](http://arxiv.org/abs/2605.25716v1)
  > **TL;DR**: Addresses privacy and efficiency challenges in federated RAG inference. Proposes FedRAG with Scrambled Distributed Attention for secure distributed KV cache access. Achieves <0.1% utility drop and 62x lower latency versus baselines.
* `serving` `edge` `offloading` [Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment](http://arxiv.org/abs/2605.25682v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed Transformer inference on embedded edge devices. Proposes Prism, combining Segment Means compression with offline profiling to adaptively switch between local and distributed execution. Reduces latency by 65-77% and energy by 34-52% versus static distributed inference.
* `serving` `hardware` `kernel` [Bandwidth-Aware LLM Inference on Heterogeneous Many-Core Supercomputers](http://arxiv.org/abs/2605.25655v1)
  > **TL;DR**: Addresses LLM inference inefficiency on bandwidth-constrained heterogeneous supercomputers. Proposes THInfer, a framework with optimized kernels, computation fusion, and P-B-D pipeline for data locality and parallelism. Achieves 62%-84% higher throughput over GPUs for Llama models.
* `training` `serving` `LoRA` [Fine-Tuning and Serving Gemma 4 31B on Google Cloud TPU: A Technical Comparison with GPU Baselines](http://arxiv.org/abs/2605.25645v1)
  > **TL;DR**: Compares TPU vs. GPU for fine-tuning and serving Gemma 4 31B. Adapts GPU-native LoRA training to JAX/TPU stack and details vLLM-TPU inference setup. TPU achieves 1.61x faster training at 2.12x lower cost and 2x lower first-token latency.
* `diffusion` `disaggregation` `scaling` [DisagFusion: Asynchronous Pipeline Parallelism and Elastic Scheduling for Disaggregated Diffusion Serving](http://arxiv.org/abs/2605.25550v1)
  > **TL;DR**: Addresses high memory and imbalance in diffusion model deployment via disaggregated serving. Proposes DisagFusion with asynchronous pipeline parallelism and hybrid scheduling for resource rebalancing. Achieves 3.4x-20.5x throughput improvement and 18.5x latency reduction.
* `training` `scaling` `networking` [Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training](http://arxiv.org/abs/2605.25375v1)
  > **TL;DR**: Addresses pipeline scheduling for geo-distributed LLM training under heterogeneous bandwidth and electricity prices. Proposes BACE-Pipe, featuring dynamic job prioritization, bandwidth-aware pathfinding, and cost-minimizing allocation. Achieves up to 64.7% lower JCT and 30.6% lower electricity cost versus state-of-the-art baselines.
* `serving` `offloading` [Kavier: Exploring Performance, Sustainability, and Efficiency of LLM Ecosystems under Inference through Cache-Aware Discrete-Event Simulation](http://arxiv.org/abs/2605.25247v1)
  > **TL;DR**: To optimize LLM inference ecosystems' performance, sustainability, and efficiency, proposes Kavier, a cache-aware discrete-event simulator with reference architecture focusing on KV caching policies. Achieves high prediction accuracy and enables cost-efficient decision-making for serving.
* `serving` `diffusion` `scaling` [Optimus: Elastic Decoding for Efficient Diffusion LLM Serving](http://arxiv.org/abs/2605.24832v1)
  > **TL;DR**: Addresses inefficient GPU utilization in diffusion LLM serving under dynamic loads. Proposes Optimus, an elastic decoding system with chunked execution and saturation-aware scheduling that adapts granularity at runtime. Achieves up to 6.1x and 4.3x throughput improvements over AR and fixed-block decoding.
* `scaling` `training` `hardware` [Provisioning to Runtime Optimization of a +100 MW AI Cluster](http://arxiv.org/abs/2605.24461v1)
  > **TL;DR**: Addresses the bottleneck of electric power in hyper-scale AI data centers by establishing an end-to-end power management process from planning to runtime optimization. Applied to a 150 MW datacenter with 83K GB200 GPUs, sharing operational insights for state-of-the-art clusters.
* `training` `scaling` `networking` [ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training](http://arxiv.org/abs/2605.24326v1)
  > **TL;DR**: Addresses communication optimization for scaling LLM training across multiple data centers. Proposes ScaleAcross Explorer, an optimizer for parallelism placement, scheduling, and network technologies. Achieves up to 64.62% training speedup over production configurations.
* `serving` `offloading` [Resident KV Claims: A Conformance Contract for Future Reuse under Active KV Pressure](http://arxiv.org/abs/2605.24259v1)
  > **TL;DR**: Addresses unreported eviction of resident KV cache during inference. Proposes resident KV claims as a conformance contract binding reuse intent with materialization predicates and telemetry. Prototype in vLLM shows conversion of resident loss into reconstructable arbitration for 60-block claim vs 70-block prefill in 80-block pool.
* `RL` `agentic` `training` [Polar: Agentic RL on Any Harness at Scale](http://arxiv.org/abs/2605.24220v1)
  > **TL;DR**: Addresses efficient RL training for agentic LLMs across diverse harnesses. Proposes Polar, an asynchronous rollout framework that proxies LLM APIs and reconstructs token-faithful trajectories. Improves Qwen3.5-4B by up to 22.6 points on SWE-Bench.
* `serving` [Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks](http://arxiv.org/abs/2605.24217v1)
  > **TL;DR**: Identifies systemic measurement bias in LLM inference benchmarks due to client-side queuing bottlenecks. Proposes a multi-process evaluation framework and NTPOT metric to isolate serving engine performance. Enables accurate benchmarking at production scales exceeding thousands of queries per second.
* `serving` `offloading` `sparse` [Adaptive KV Cache Reuse for Fast Long-Context LLM Serving](http://arxiv.org/abs/2605.24022v1)
  > **TL;DR**: Reduces TTFT latency in long-context LLM serving by enabling efficient non-prefix KV cache reuse. Introduces CacheTune with frequency-guided critical token selection, sparse KV transfer, and hardware-aware I/O optimization. Achieves 3.72x-4.86x TTFT speedup and 3.93x-6.21x throughput improvement while maintaining generation quality.
* `training` `networking` `scaling` [A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training](http://arxiv.org/abs/2605.24006v1)
  > **TL;DR**: Addresses challenges in comparing pipeline schedules for efficient LLM training. Proposes a tabular schedule abstraction and simulation methodology blending formulas and communication considerations. Results show 1F1B achieves lower activation-memory peak than GPipe, and schedule performance varies with network bottlenecks.

### 2026-05-25
* `training` `MoE` `hardware` [HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs](http://arxiv.org/abs/2605.23764v1)
  > **TL;DR**: Addresses underutilized heterogeneous parallelism in MoE training on NPUs. Proposes HyperParallel-MoE for statically scheduled tile-level taskflow with one-sided communication and event-driven scheduling. Achieves up to 1.58x lower Dispatch-to-Combine latency.
* `serving` `offloading` [AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System](http://arxiv.org/abs/2605.23389v1)
  > **TL;DR**: Addresses iteration-level bubbles in LLM serving caused by KV-cache length variance. Proposes prefix-aware batching and GPU-Prefetch-For-GPU to align cache lengths and reduce CPU-GPU transfers. Achieves 1.98× higher throughput and 7.4× lower latency versus state-of-the-art systems.
* `serving` `networking` [XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms](http://arxiv.org/abs/2605.23348v1)
  > **TL;DR**: Addresses efficient LLM inference serving at renewable wind farms with fluctuating energy supply. Proposes XWind, a lightweight cross-site router using real-time latency, KV-cache, and queue signals to dynamically route requests. Reduces P99 latency by up to 52% compared to state-of-the-art alternatives.
* `edge` `offloading` `scaling` [Hybrid Edge-HPC Systems for Low-Latency Data-Driven Inference](http://arxiv.org/abs/2605.20532v2)
  > **TL;DR**: Resolves latency mismatch between edge inference and slow simulation-driven model updates. Proposes RBF, an edge-HPC hybrid with lightweight edge surrogates and asynchronous model integration. Enables continuous low-latency inference while improving fidelity despite irregular HPC updates.
* `training` `storage` [Orbax: Distributed Checkpointing with JAX](http://arxiv.org/abs/2605.23066v1)
  > **TL;DR**: Addresses efficient distributed checkpointing for JAX in ML training. Introduces Orbax, a modular library abstracting distributed accelerator complexities for user-friendly saving/loading. Achieves 3.5x faster saving and 2x faster loading than PyTorch.
* `offline` `storage` [Budgeted Dynamic Trace Structures for Token-Efficient Sequential Computation](http://arxiv.org/abs/2605.22879v1)
  > **TL;DR**: Reduces long traces in sequential computations under token budgets via budgeted dynamic trace structures with compaction techniques. Achieves token reduction from 3,359 to 433 with model benchmarks.

### 2026-05-22
* `agentic` `RL` `storage` [DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback](http://arxiv.org/abs/2605.22781v1)
  > **TL;DR**: Investigates reducing checkpoint and rollback latency for state exploration in LLM-powered AI agents. Proposes DeltaBox with OS-level DeltaState, including DeltaFS for layered filesystem changes and DeltaCR for incremental process rollback. Achieves millisecond-level latencies (14ms checkpoint, 5ms rollback).

### 2026-05-22
* `training` `networking` [Exploiting Multicast for Accelerating Collective Communication](http://arxiv.org/abs/2605.22428v1)
  > **TL;DR**: Proposes MultiWrite, a multicast-based semantic to eliminate redundant data transmission during collective communication (e.g., AllGather) in large model training/inference. Addresses traditional multicast limitations via NPU implementation, achieving up to 33% latency reduction on commercial hardware.
* `serving` [Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference](http://arxiv.org/abs/2605.22416v1)
  > **TL;DR**: Addresses inefficient memory handling for KV and SSM caches in hybrid Mamba-Transformer inference. Proposes Asymmetric Virtual Memory Paging (AVMP) with separate pools and on-demand migration. Improves request throughput by 1.83x to 13.3x on synthetic workloads and 2.36x on ShareGPT.
* `training` `scaling` [LiveR: Fine-Grained Elasticity via Live Reconfiguration for Model Training](http://arxiv.org/abs/2605.22014v1)
  > **TL;DR**: Aims to reduce downtime during elastic resizing in LLM training. Introduces LiveR, a runtime that asynchronously prepares new configurations and streams state without stopping training. Achieves 14x-23x faster reconfiguration and 99% training goodput with volatile resources.
* `RAG` `hardware` `storage` [NasZip: Software and Hardware Co-Design to Accelerate Approximate Nearest Neighbor Search with DIMM-Based Near-Data Processing](http://arxiv.org/abs/2605.21952v1)
  > **TL;DR**: Addresses memory-bound approximate nearest neighbor search in retrieval-augmented generation (RAG) systems. Proposes NASZIP, a hardware-software co-design with DIMM-based near-data processing, PCA-guided early exiting, and dynamic-float encoding. Achieves up to 8.4× speedup over CPU baselines at equal accuracy.
* `hardware` [CompPow: A Case for Component-level GPU Power Management](http://arxiv.org/abs/2605.21847v1)
  > **TL;DR**: Proposes CompPow, a component-level GPU power management for ML workloads via software-hardware co-design. Achieves 10% higher energy efficiency across various operations and execution patterns.
* `kernel` `training` [DynaFlow: Transparent and Flexible Intra-Device Parallelism via Programmable Operator Scheduling](http://arxiv.org/abs/2605.21603v1)
  > **TL;DR**: DynaFlow introduces efficient intra-device parallelism by decoupling logical models from physical schedules. Its programmable scheduler and memory management reduce copy overheads, achieving up to 1.29x throughput improvement across ML systems.
* `training` `networking` [High-speed Networking for Giga-Scale AI Factories](http://arxiv.org/abs/2605.21187v1)
  > **TL;DR**: Addresses predictable high-speed networking for distributed AI training at scale. Introduces Spectrum-X multiplane architecture with hardware-accelerated load balancing. Achieves 98% of theoretical line rate with low jitter-free latency under realistic workloads.

### 2026-05-21
* `serving` `MoE` `quantization` [PALS: Power-Aware LLM Serving for Mixture-of-Experts Models](http://arxiv.org/abs/2605.21427v1)
  > **TL;DR**: Proposes PALS, a power-aware LLM serving runtime that dynamically controls GPU power caps and batch sizes. Uses offline modeling and feedback control to optimize energy efficiency while meeting throughput targets. Achieves up to 26.3% energy savings and reduces QoS violations by 4x-7x.
* `serving` `disaggregation` `kernel` [Frontier: Towards Comprehensive and Accurate LLM Inference Simulation](http://arxiv.org/abs/2605.21312v1)
  > **TL;DR**: Proposes Frontier, a discrete-event simulator for accurate LLM inference serving systems. Models disaggregated execution (PDD, AFD), runtime optimizations, and stateful workloads with role-specific workers. Achieves average throughput error below 4% and scales to 1K GPUs.
* `MoE` `serving` `scaling` [NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding](http://arxiv.org/abs/2605.21100v1)
  > **TL;DR**: Addresses load imbalance and KV cache fragmentation in MoE serving systems. Proposes dynamic context parallelism (DCP) to decouple MoE communication from KV cache placement and adjust parallelism per request. Achieves up to 3.27× higher request rates and reduces P99 latency by up to 2.12×.
* `MoE` `training` `networking` [Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory](http://arxiv.org/abs/2605.20982v1)
  > **TL;DR**: Investigates routing imbalance in MoE expert parallelism during training. Introduces DODOCO to evaluate workload assumptions using real LLMs and data. Shows mock tokens overestimate routing imbalance by up to 2.35× and reveals intrinsic model-dependent bands.
* `training` `networking` `sparse` [LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging](http://arxiv.org/abs/2605.20866v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed training via LOSCAR-SGD: overlapping communication-computation with delay-corrected sparse model averaging. Achieves faster convergence and up to reduced training time with sparse updates, heterogeneous workers, and optimized merging.
* `training` `RL` `scaling` [PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR](http://arxiv.org/abs/2605.20863v1)
  > **TL;DR**: Addresses training inefficiency in RLVR for LLMs, proposing PlexRL, a cluster-level runtime orchestrating multiplexed LLM services. Achieves up to 37.58% reduction in GPU hour costs by filling job anti-correlated idle gaps through centralized scheduling and state management.
* `training` `hardware` `quantization` [Instant GPU Efficiency Visibility at Fleet Scale](http://arxiv.org/abs/2605.20799v1)
  > **TL;DR**: Introduces Overall FLOP Utilization (OFU), a hardware-level GPU efficiency metric derived from performance counters, requiring no instrumentation. Evaluated via GEMM experiments and production training jobs, OFU achieves r=0.78 correlation with MFU and identified framework-level errors. Deployed fleet-wide, it detected a 2.5x efficiency regression.
* `serving` `kernel` `offloading` [Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU](http://arxiv.org/abs/2605.20706v1)
  > **TL;DR**: Proposes LlamaWeb, a WebGPU backend for llama.cpp to optimize browser-based LLM inference. Employs static memory planning, efficient model loading, and templated GPU kernels for multi-precision support. Reduces memory by 29-33% and increases throughput by 45-69% across heterogeneous devices.
* `storage` `serving` `edge` [LatentBox: Storing AI-Generated Images at Scale via a Latent-First Design](http://arxiv.org/abs/2605.19385v2)
  > **TL;DR**: Proposes LatentBox, a latent-first storage system for AI-generated images using compressed latents and on-demand GPU reconstruction, balancing cache tiers to optimize access latency. Reduces persistent storage by 78.7% with competitive latency compared to image-based storage.
* `training` `serving` [Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference](http://arxiv.org/abs/2605.17164v2)
  > **TL;DR**: Addresses accurate performance simulation for optimizing large-scale LLM training and inference. Proposes Charon, a unified modular simulator for parallelism strategies, system optimizations, and hardware configurations. Achieves under 5.35% prediction error and improves inference throughput over baseline.
* `edge` `offloading` `RAG` [Hybrid Edge-HPC Systems for Low-Latency Data-Driven Inference](http://arxiv.org/abs/2605.20532v1)
  > **TL;DR**: Proposes RBF, a hybrid edge-HPC architecture for low-latency inference by decoupling edge surrogates from asynchronous model updates. Uses opportunistic computation to improve accuracy despite delayed HPC updates. Achieves continuous inference with improved fidelity in real-world agriculture deployment.
* `serving` `sparse` `kernel` [SSV: Sparse Speculative Verification for Efficient LLM Inference](http://arxiv.org/abs/2605.19893v2)
  > **TL;DR**: Addresses inefficiency in combining sparse attention and speculative decoding for long-context LLM inference. Proposes SSV framework with overlap-aware grouped query execution, kernel fusion, and adaptive orchestration. Achieves up to 3.49x end-to-end throughput over autogressive decoding.

### 2026-05-20
* `video` `quantization` `training` [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](http://arxiv.org/abs/2605.18739v2)
  > **TL;DR**: Proposes LongLive-2.0, an NVFP4-based parallel infrastructure for efficient long video generation training and inference. Combines sequence-parallel AR training with NVFP4 precision for reduced memory and accelerated GEMM, along with W4A4 quantization and SP inference. Achieves 2.15x training speedup and 45.7 FPS inference.
* `MoE` `training` `serving` [GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems](http://arxiv.org/abs/2605.19945v1)
  > **TL;DR**: Addresses straggler issue in MoE inference due to GPU variability and expert placement. Proposes GEM, a GPU-variability-aware expert mapping framework that balances token loads by considering expert types (consistent/temporal) and GPU speeds. Achieves up to 16.5% lower end-to-end latency over baselines.
* `MoE` `offloading` `training` [Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles](http://arxiv.org/abs/2605.19775v1)
  > **TL;DR**: Analyzes inference scaling bottlenecks in large-scale LLMs, especially for reasoning loads causing Capacity-Bound regimes. Evaluates parallelism strategies (Data, Tensor, Pipeline) across models up to 671B parameters. Shows hybrid parallelism achieves sublinear scaling gains while mitigating KV-cache fragmentation, improving compute utilization up to 50% for sparse MoE models.
* `training` `kernel` `quantization` [JanusPipe: Efficient Pipeline Parallel Training for Machine Learning Interatomic Potentials](http://arxiv.org/abs/2605.18404v2)
  > **TL;DR**: Faster pipeline parallel training for ML interatomic potentials via SymFold for memory efficiency and WaveK for bubble reduction. Achieves 1.45× higher throughput over prior systems on 32 GPUs.
* `serving` `offloading` `multi-modal` [Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption](http://arxiv.org/abs/2605.19593v1)
  > **TL;DR**: Studies multi-model LLM scheduling challenges including offloading and preemption penalties. Empirically analyzes effects of layer offloading and preemption on throughput, latency, and overheads. Identifies degradation varies by model size and GPU residency, with high cost of state reload dominating.
* `training` `networking` `scaling` [Modeling the Impact of Fiber Latency on Compute-Communication Overlap in Geo-Distributed Multi-Datacenter AI Training](http://arxiv.org/abs/2605.19169v1)
  > **TL;DR**: Studies fiber latency impact on communication-compute overlap in geo-distributed AI training. Uses discrete-event simulation to show optimal 10-100km inter-cluster distance. Achieving 25% higher overlap using hollow-core fiber versus conventional fiber.
* `edge` `thinking` `serving` [Unleashing the Power of Tree-of-Thoughts for Edge-Enabled AIGC Service Provisioning](http://arxiv.org/abs/2605.19108v1)
  > **TL;DR**: Optimizes mobile edge computing for Tree-of-Thoughts AIGC services by formulating thought assignment as a DAG and solving with a diffusion-based SAC algorithm. Achieves up to 36.09% lower delay vs baselines and 80% latency reduction over local execution under quality constraints.
* `training` `offline` [Conflict-Free Replicated Data Types for Neural Network Model Merging: A Two-Layer Architecture Enabling CRDT-Compliant Model Merging Across 26 Strategies](http://arxiv.org/abs/2605.19373v1)
  > **TL;DR**: Addresses lack of algebraic properties in neural network merging strategies for distributed systems. Proposes CRDTMergeState, a two-layer architecture enabling Conflict-Free Replicated Data Types for model merging. Ensures Strong Eventual Consistency with under 0.5 ms overhead, validated on models up to 7.24B parameters.
* `offline` `edge` `multi-modal` [INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference](http://arxiv.org/abs/2605.18853v1)
  > **TL;DR**: Explores efficient routing for edge-cloud deployment of vision-language models. Proposes INAR-VL, using input complexity signals to route requests between edge and cloud for optimal accuracy-latency tradeoff. Achieves 24% latency reduction and 26% energy savings while preserving 97% cloud-level accuracy.
* `serving` `sparse` `quantization` [SpecSA: Bridging Speculative Decoding and Sparse Attention for Efficient LLM Inference](http://arxiv.org/abs/2605.19893v1)
  > **TL;DR**: Addresses inefficiency in combining speculative decoding and sparse attention for LLM inference. Proposes SpecSA with grouped-query execution, kernel fusion, and profile-guided orchestration. Achieves up to 3.49x end-to-end throughput and 6.86x kernel speedups over baselines.
* `serving` `offline` `RAG` [Skim: Speculative Execution for Fast and Efficient Web Agents](http://arxiv.org/abs/2605.16565v2)
  > **TL;DR**: Introduces Skim, a speculative execution framework for web agents that uses offline profiling and lightweight models to bypass full inference and browser rendering. Validates outputs with a verifier to ensure correctness. Reduces per-task cost by 1.9x and latency by 33.4% without accuracy loss.

### 2026-05-19
* `training` `scaling` `networking` [A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability](http://arxiv.org/abs/2605.18750v1)
  > **TL;DR**: Proposes Runtime-Readiness-First Pipeline (RRFP) for pipeline-parallel training under runtime variability. Uses non-binding schedule hints, message-driven async communication, and ready-set arbitration to reduce idle bubbles. Achieves up to 2.77× speedup on multimodal workloads at 128 GPUs.
* `training` `video` `quantization` [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](http://arxiv.org/abs/2605.18739v1)
  > **TL;DR**: Addresses bottlenecks in long video generation systems via NVFP4-based parallel infrastructure. Introduces sequence-parallel AR training with SP-aware chunking and NVFP4 precision, and employs quantized KV cache with asynchronous VAE decoding for inference. Achieves 2.15x training speedup and 1.84x inference speedup.
* `training` `multi-modal` `scaling` [Mosaic: Towards Efficient Training of Multimodal Models with Spatial Resource Multiplexing](http://arxiv.org/abs/2605.18710v1)
  > **TL;DR**: Proposes a training system for multimodal models using temporal-spatial multiplexing to colocate modules. Develops execution engine and perf model to optimize resource allocation. Achieves up to 1.31x speedup in training.
* `serving` `offloading` [PopPy: Opportunistically Exploiting Parallelism in Python Compound AI Applications](http://arxiv.org/abs/2605.18697v1)
  > **TL;DR**: Investigates how to reduce latency in Python-based compound AI applications by automatically uncovering parallelism. Proposes PopPy, an ahead-of-time compiler with runtime support for handling language complexity, dynamic dispatch, and mutations. Achieves up to 6.4× speedups in end-to-end execution time.
* `training` `networking` `scaling` [EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet](http://arxiv.org/abs/2605.18683v1)
  > **TL;DR**: Proposes EPIC, an Ethernet-based in-network collective protocol for optimizing AI training and inference. Introduces a polymorphic abstraction with modular design, formal verification, and unified resource management. Achieves significant performance gains validated across testbeds.
* `training` `sparse` `quantization` [JanusPipe: Efficient Pipeline Parallel Training for Machine Learning Interatomic Potentials](http://arxiv.org/abs/2605.18404v1)
  > **TL;DR**: Proposes JanusPipe, a pipeline parallel training system for machine learning interatomic potentials (MLIPs) with double-backward execution. Introduces SymFold for memory efficiency and WaveK for bubble reduction. Achieves 1.51× higher throughput vs 1F1B on 32 GPUs.

### 2026-05-19
* `training` `scaling` [Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method](http://arxiv.org/abs/2605.18174v1)
  > **TL;DR**: Proposes Ringmaster LMO, an asynchronous momentum method for distributed LMO-based optimization, leveraging delay-thresholding to discard stale gradients. Addresses stragglers in heterogeneous systems, achieving optimal time complexity and outperforming baselines in experiments with increasing system heterogeneity.
* `video` `training` `kernel` [AdaptiveLoad: Towards Efficient Video Diffusion Transformer Training](http://arxiv.org/abs/2605.17923v1)
  > **TL;DR**: Proposes AdaptiveLoad for efficient training of large-scale video diffusion transformers. Introduces dual-constraint load balancing and fused CUDA kernel to address sequence length variance. Reduces computational imbalance by 20.1% and increases training throughput by 27.2%.
* `training` `scaling` `hardware` [Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training](http://arxiv.org/abs/2605.17879v1)
  > **TL;DR**: Addresses the problem of silent performance degradation in large-scale training clusters. Proposes Guard, a system combining lightweight online monitoring with offline node-sweep diagnostics. Improves FLOPs utilization by up to 1.7x and reduces training step variance from 20% to 1% in foundation model training.
* `training` `storage` [TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training](http://arxiv.org/abs/2605.17821v1)
  > **TL;DR**: Proposes TierCheck, a tiered checkpointing system using local/peer memory and remote storage to minimize training interruptions. Maintains differential checkpoints in memory for fast localized recovery, reducing checkpointing time to under 10s and enabling high-frequency checkpointing.
* `quantization` `serving` `offloading` [OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization](http://arxiv.org/abs/2605.17757v1)
  > **TL;DR**: Proposes OSCAR, an offline spectral covariance-aware rotation method for 2-bit KV cache quantization to reduce memory and improve throughput in LLM serving. Integrates with paged KV-cache systems, achieving up to 8x memory reduction, 7x throughput increase, and 3x decoding acceleration while maintaining near-BF16 accuracy.
* `storage` `disaggregation` [Send: Objects, History, and Transactions in a Single-Verb Kernel](http://arxiv.org/abs/2605.17401v1)
  > **TL;DR**: Proposes a kernel for multi-party object coordination with authenticated provenance, opaque encapsulation, and atomic commits using ephemeral/persistent objects and append-only logs. Structural guarantees include six properties against arbitrary programs under kernel trust. Achieves operator-adversarial deployability via cryptographic compiler.
* `training` `serving` `scaling` [Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference](http://arxiv.org/abs/2605.17164v1)
  > **TL;DR**: Proposes Charon, a unified fine-grained simulator for predicting performance of large-scale LLM training and inference. Models parallelism strategies, system optimizations, and hardware configurations with 5.35% prediction error. Discovered a configuration improving inference throughput over engineering-tuned baseline.
* `agentic` `serving` [S-Bus: Automatic Read-Set Reconstruction for Multi-Agent LLM State Coordination](http://arxiv.org/abs/2605.17076v1)
  > **TL;DR**: Proposes S-Bus, an HTTP middleware with a DeliveryLog for automatic read-set reconstruction, ensuring Observable-Read Isolation to prevent structural race conditions in concurrent LLM agent state coordination. Achieves zero Type-I corruptions across 427,308 HTTP-409 conflicts compared to backend systems.
* `serving` `scaling` `agentic` [GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources](http://arxiv.org/abs/2605.16867v1)
  > **TL;DR**: Proposes GoodServe, a routing system for agentic LLM inferences on heterogeneous GPUs to improve goodput. Employs predict-and-rectify with output length estimation, GPU status tracking, and runtime request migration. Increases goodput by 27.4% over existing methods.
* `agentic` `serving` `disaggregation` [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](http://arxiv.org/abs/2605.16637v1)
  > **TL;DR**: Proposes HexAGenT, a scheduler for agentic LLM workflows that models requests as online-revealed DAGs and prioritizes calls by workflow completion risk. It optimizes prefill/decode placement with KV-cache and transfer constraints. Reduces SLO scale by up to 80.5% at 99% attainment.
* `quantization` `kernel` `hardware` [Exceeding the Numerical and Performance Characteristics of IEEE-754 SGEMM with BFloat16 Tensor Cores on GPUs for Scientific Computing](http://arxiv.org/abs/2605.16617v1)
  > **TL;DR**: Investigates emulating FP32 matrix multiplications using bfloat16 Tensor Cores on GPUs. Proposes a BF16-based emulation method leveraging Blackwell GPU's scaling hardware. Achieves higher performance and efficiency than native FP32 for scientific applications.
* `offline` `storage` `networking` [TIDAL: Recovering Temporal Phase for Cloud Block Storage Placement from LLM-Derived Semantics](http://arxiv.org/abs/2605.18066v1)
  > **TL;DR**: Proposes TIDAL, a framework for complementary phase-aware placement of cloud virtual disks. Uses LLMs to derive temporal signals from disk names for load balancing, with offline-to-online design. Reduces overload frequency by 79.1% and P95 overload duration by 73.7%.
* `offline` `storage` [PipeANN-Filter: An Efficient Filtered Vector Search System on SSD](http://arxiv.org/abs/2605.17992v1)
  > **TL;DR**: Addresses SSD I/O inefficiency in filtered vector search. Proposes exploring superset of valid vectors and verifying attributes post-search using probabilistic structures (e.g., Bloom filters). Achieves improved search latency and throughput over state-of-the-art systems.
* `offline` `agentic` `scaling` [TClone: Low-Latency Forking of Live GUI Environments for Computer-Use Agents](http://arxiv.org/abs/2605.17320v1)
  > **TL;DR**: Addresses the need for low-latency forking of live GUI environments for computer-use agents. Introduces TClone with siblings containers, copy-on-write memory sharing, and asynchronous checkpointing. Reduces total task latency by 1.9x and 1.5x over KVM and CRIU.
* `serving` `offloading` `RAG` [Skim: Speculative Execution for Fast and Efficient Web Agents](http://arxiv.org/abs/2605.16565v1)
  > **TL;DR**: Proposes Skim, a speculative execution framework for web agents that uses offline profiling and lightweight verification to bypass costly LLM inference and rendering components. Achieves 1.9x cost reduction and 33.4% lower latency without accuracy loss.

### 2026-05-18
* `training` `hardware` `scaling` [Designing Datacenter Power Delivery Hierarchies for the AI Era](http://arxiv.org/abs/2605.16255v1)
  > **TL;DR**: Addresses efficient datacenter power delivery design under increasing AI accelerator densities. Proposes a framework combining hardware projections and operational data to optimize deployable capacity over time. Achieves up to 42% higher deployable capacity compared to existing designs.
* `training` `offloading` `quantization` [Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training](http://arxiv.org/abs/2605.16184v1)
  > **TL;DR**: Addresses the system bottleneck in second-order LLM training by introducing Asteria, a runtime that offloads optimizer state across storage tiers and overlaps computation. Achieves training of a 1B model on a single GPU with unified memory and reduces overhead in distributed 7B training.
* `scaling` `serving` `offline` [ADAPT: A Self-Calibrating Proactive Autoscaler for Container Orchestration](http://arxiv.org/abs/2605.15788v1)
  > **TL;DR**: Proposes ADAPT, a self-calibrating proactive autoscaler using online EWMA for container provisioning delay estimation, integrated with MPC for replica optimization. Achieves below 5% SLA violation across workloads, outperforming baselines by up to 23.7 percentage points in violation reduction.
* `training` `scaling` [A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM](http://arxiv.org/abs/2605.15617v1)
  > **TL;DR**: Introduces PrismLLM, a method for faithful emulation of large-scale LLM training on small GPU clusters using slicing and hybrid execution. Accurately replicates performance and memory behavior with 0.58% error in iteration time and <0.01% memory error, emulating up to 8192 GPUs using <1% physical resources.
* `training` `storage` [BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training](http://arxiv.org/abs/2605.09994v2)
  > **TL;DR**: Proposes BatchWeave, an object-store-native data plane for distributed foundation model training. Uses versioned manifests and conditional object writes for transactional global batches, decentralized commit, and storage-layer recovery. Achieves higher throughput than Apache Kafka and better failure isolation than colocated dataloaders in 64-GPU evaluations.

### 2026-05-15
* `agentic` `scaling` `serving` [APWA: A Distributed Architecture for Parallelizable Agentic Workflows](http://arxiv.org/abs/2605.15132v1)
  > **TL;DR**: Proposes APWA, a distributed architecture for decomposing agentic workflows into non-interfering parallel subproblems to overcome scaling bottlenecks. Achieves scalability on large tasks where prior systems fail, supporting heterogeneous data and parallel patterns.
* `edge` `networking` `training` [Multi-objective application placement in fog computing using graph neural network-based reinforcement learning](http://arxiv.org/abs/2605.14649v1)
  > **TL;DR**: Proposes a graph neural network-based reinforcement learning framework for multi-objective application placement in fog computing. Incorporates service dependencies into placement decisions using dual actor-critics. Achieves placement decisions in milliseconds with comparable Pareto front to genetic algorithms requiring hours.
* `serving` `offloading` `edge` [PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding](http://arxiv.org/abs/2605.13319v2)
  > **TL;DR**: Proposes PipeSD, a cloud-edge collaborative inference framework using speculative decoding. Implements token-batch pipeline scheduling and dual-threshold non-autoregressive verification triggering. Achieves 1.16x-2.16x speedup and reduces energy consumption by 14.3%-25.3% versus baselines.
* `kernel` `training` `offline` [Hierarchical Transformer Preconditioning for Interactive Physics Simulation](http://arxiv.org/abs/2605.13343v2)
  > **TL;DR**: Proposes Hierarchical Transformer Preconditioner for real-time physics simulation with O(N) scaling. Uses dense batched GEMMs and a cosine-Hutchinson probe objective for training. Achieves 2.7x speedup over neural SPAI and 28x over GPU IC/DILU at N=8192.
* `offline` `edge` `kernel` [SemaTune: Semantic-Aware Online OS Tuning with Large Language Models](http://arxiv.org/abs/2605.15026v1)
  > **TL;DR**: Proposes SemaTune, an LLM-based framework for online OS tuning using semantic-aware context and bounded guidance. It employs fast and slow update loops with validation to optimize host-level parameters. Achieves 72.5% performance improvement over defaults and 153.3% over baselines at $0.20 per session.

### 2026-05-14
* `training` `serving` `LoRA` [MinT: Managed Infrastructure for Training and Serving Millions of LLMs](http://arxiv.org/abs/2605.13779v1)
  > **TL;DR**: Presents MinT, a managed infrastructure for LoRA post-training and serving that scales up with large models, scales down with small adapter transfers (18.3x faster step), and scales out to support millions of addressable catalogs and improve engine loading by 8.5-8.7x.
* `serving` `disaggregation` `quantization` [KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving](http://arxiv.org/abs/2605.13734v1)
  > **TL;DR**: Proposes KVServe, an adaptive KV cache compression framework for disaggregated LLM serving. Introduces modular strategy space, Bayesian profiling for efficient offline search, and service-aware online controller. Achieves up to 9.13× job completion time speedup and 32.8× TTFT reduction.
* `serving` `scaling` [MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenters](http://arxiv.org/abs/2605.13496v1)
  > **TL;DR**: Proposes MARLIN, a multi-agent reinforcement learning framework for sustainable LLM inference serving in cloud datacenters. It co-optimizes latency, carbon emissions, water usage, and energy costs. Achieves ≥18% lower TTFT, 33% reduced carbon, 43% less water, and 11% lower energy costs.
* `training` `quantization` `scaling` [TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation](http://arxiv.org/abs/2605.13433v1)
  > **TL;DR**: Addresses training bottlenecks for large-scale generative recommendation models on NPUs. Proposes an Ascend-affinity system with jagged operator fusion, hierarchical parallelism, and quantization techniques. Achieves 54.71% MFU and near-linear (0.97) scaling on KuaiRand-27K at 0.2B parameters.
* `training` `kernel` `scaling` [Hierarchical Transformer Preconditioning for Interactive Physics Simulation](http://arxiv.org/abs/2605.13343v1)
  > **TL;DR**: Proposes a neural hierarchical transformer preconditioner for physics simulation with efficient O(N) scaling. Uses batched dense GEMMs for inference with a single CUDA Graph and a cosine-Hutchinson training objective. Achieves 2.7x speedup over neural SPAI at N=8,192.
* `serving` `offloading` `edge` [PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding](http://arxiv.org/abs/2605.13319v1)
  > **TL;DR**: Proposes PipeSD, a cloud-edge collaborative inference framework using speculative decoding. Implements token-batch pipeline scheduling and dual-threshold non-autoregressive verification triggering. Achieves 1.16x-2.16x speedup and 14.3%-25.3% energy reduction compared to baselines.

### 2026-05-13
* `training` `networking` `quantization` [NCCLZ: Compression-Enabled GPU Collectives with Decoupled Quantization and Entropy Coding](http://arxiv.org/abs/2605.12396v1)
  > **TL;DR**: Optimizes GPU collective communication for distributed deep learning by decoupling quantization and entropy coding. Integrates compression into NCCL primitives to overlap computation and communication. Achieves up to 9.65x speedup over NCCL.
* `inference` `sparse` `kernel` [AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference](http://arxiv.org/abs/2605.12110v1)
  > **TL;DR**: Addresses KV cache bottleneck in long-context LLM inference by introducing AB-Sparse, a framework with adaptive block sizes per attention head and centroid quantization. Achieves up to 5.43% accuracy improvement without throughput overhead.
* `serving` `quantization` [The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures](http://arxiv.org/abs/2605.11999v1)
  > **TL;DR**: Examines the ineffectiveness of power capping for LLM autoregressive decode phase due to memory-bound workloads. Proposes SM clock locking as an alternative energy lever to avoid firmware throttling illusions. Recovers up to 32% decode energy with minimal throughput loss across attention architectures.
* `serving` `quantization` `offloading` [Position: LLM Inference Should Be Evaluated as Energy-to-Token Production](http://arxiv.org/abs/2605.11733v1)
  > **TL;DR**: Proposes evaluating LLM inference as energy-per-token production, introducing a Token Production Function framework. Highlights KV-cache compression, quantization, and routing as energy-saving techniques. Calls for reporting joules/token and power constraints alongside latency and accuracy metrics to address scale deployment challenges.
* `storage` `RAG` `offline` [NAVIS: Concurrent Search and Update with Low Position-Seeking Overhead in On-SSD Graph-Based Vector Search](http://arxiv.org/abs/2605.11523v1)
  > **TL;DR**: Addresses high overhead in concurrent search and update for on-disk vector search by introducing selective vector reads, dynamic entrance graph updates, and an edgelist cache. Achieves up to 2.74x higher insertion throughput and 25.26% lower search latency.
* `kernel` `training` `inference` [FalconGEMM: Surpassing Hardware Peaks with Lower-Complexity Matrix Multiplication](http://arxiv.org/abs/2605.06057v2)
  > **TL;DR**: Proposes FalconGEMM, a framework automating deployment and optimization of lower-complexity matrix multiplication for DL training and inference. Features code generation, group-parallel optimizations, and analytical model for strategy selection. Achieves 7.59%-17.85% speedup over cuBLAS/CUTLASS on GPU and CPU.
* `serving` `scaling` `edge` [Kairos: A Scalable Serving System for Physical AI](http://arxiv.org/abs/2605.11381v1)
  > **TL;DR**: Proposes Kairos, a multi-robot serving system for physical AI that optimizes the generate-execute loop with asynchronous inference-action interleaving. Achieves 31.8–66.5% reduction in end-to-end task latency over existing systems, scaling with fleet size.
* `offloading` `kernel` `training` [ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference](http://arxiv.org/abs/2605.11335v1)
  > **TL;DR**: Studies reducing GPU memory footprint in distributed diffusion transformer inference. Proposes ChunkFlow, a communication-aware chunk-granular offloading runtime that co-schedules prefetches with collective communications. Achieves up to 1.28x step-time speedup over prior layerwise offloading.
* `training` `scaling` `hardware` [MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces](http://arxiv.org/abs/2605.11333v1)
  > **TL;DR**: Addresses agile performance benchmarking and co-design for distributed AI systems with Chakra, a graph-based execution trace ecosystem. Standardized traces represent compute, memory, and communication operations. Adoption by MLCommons demonstrates improvements in workload analysis and hardware-software optimization efficiency.
* `training` `scaling` [ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload](http://arxiv.org/abs/2605.11215v1)
  > **TL;DR**: Proposes ReCoVer, a resilient LLM pre-training system maintaining stochastically equivalent per-iteration gradients via fault-tolerant collectives, in-step recovery, and workload redistribution. Achieves 2.23× higher throughput vs. baselines after failures and processes 74.9% more tokens in 234 GPU-hours.
* `training` `scaling` [ShardTensor: Domain Parallelism for Scientific Machine Learning](http://arxiv.org/abs/2605.11111v1)
  > **TL;DR**: Proposes ShardTensor, a domain parallelism paradigm for SciML to scale input data beyond device limitations. Decouples spatial data from hardware, enabling high-fidelity training/inference on extreme-resolution data. Achieves strong/weak scaling, reducing latency and handling larger data sizes.
* `training` `MoE` `networking` [DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism](http://arxiv.org/abs/2605.11005v1)
  > **TL;DR**: Addresses communication bottlenecks in MoE training. Proposes DisagMoE with disaggregated attention-FFN placement and multi-stage pipeline to overlap computation-communication. Achieves up to 1.8x training speedup on 16-node clusters.

### 2026-05-12
* `training` `kernel` `hardware` [Closer in the Gap: Towards Portable Performance on RISC-V Vector Processors](http://arxiv.org/abs/2605.10860v1)
  > **TL;DR**: Investigates performance challenges with RISC-V Vector Extension (RVV) for HPC and ML workloads. Uses assembly microbenchmarks to assess compilers' auto-vectorization, revealing limitations in cost models and memory access patterns. Shows GCC 15 outperforming LLVM 21 in 4/6 applications, with LLVM excelling only in SGEMM/DGEMM via aggressive optimization.
* `serving` `MoE` `scaling` [Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference](http://arxiv.org/abs/2605.10670v1)
  > **TL;DR**: Addresses partial rank failures in MoE inference by proposing EEP, a substrate with mutable membership to handle faults without full restarts. Recovers reachability, expert coverage, and graph reintegration. Reduces recovery time from 348s to 11s and maintains 95% throughput within 52s after failure.
* `training` `sparse` `scaling` [ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism](http://arxiv.org/abs/2605.06374v2)
  > **TL;DR**: Proposes ResiHP, a resilient system for LLM training that addresses hardware failures and sequence variability. Includes a workload-aware failure detector and a dynamic scheduler adapting parallelism, partitioning, and workload. Improves throughput by 1.04-4.39× in a 256-GPU cluster under failures.
* `training` `multi-modal` `scaling` [Accelerating Compound LLM Training Workloads with Maestro](http://arxiv.org/abs/2605.10501v1)
  > **TL;DR**: Proposes Maestro, a section-centric training framework for compound LLM workloads like knowledge distillation and MLLM training. Employs component-aware parallelism configuration and wavefront scheduling to handle static and dynamic heterogeneity. Reduces GPU consumption by ~40% in production workloads.
* `agentic` `edge` [Agentic Performance at the Edge: Insights from Benchmarking](http://arxiv.org/abs/2605.10384v1)
  > **TL;DR**: Investigates how to optimize agentic AI models for edge deployment under memory, power, and latency constraints. Introduces a domain-conditioned evaluation methodology and analyzes model-tool interactions. Finds Pareto fronts in accuracy-latency space for model selection, with results showing effective tradeoffs in constrained settings.

### 2026-05-12
* `offline` `kernel` `sparse` [FusionRCG: Orchestrating Recursive Computation Graphs across GPU Memory Hierarchies](http://arxiv.org/abs/2605.10312v1)
  > **TL;DR**: Proposes FusionRCG, a framework optimizing computation graph structure and GPU memory mapping for recursive quantum chemistry integrals. Combines liveness-aware orchestration, algebraic dimensionality reduction, and adaptive multi-tier kernels. Achieves up to 3.09x speedup over GPU4PySCF while maintaining 75% parallel efficiency at 64 GPUs.
* `serving` `offloading` `edge` [GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference](http://arxiv.org/abs/2605.10124v1)
  > **TL;DR**: Proposes GELATO, a device-edge speculative decoding framework with online token offloading scheduling based on drift-plus-penalty and entropy early-exiting. Maximizes throughput under energy constraints, achieving 64.98% higher token throughput and 47.47% lower energy than SOTA.
* `training` `storage` `scaling` [Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training](http://arxiv.org/abs/2605.09994v1)
  > **TL;DR**: Proposes Lakestream, a brokerless object-store-native data plane for large foundation model training, featuring Transactional Global Batch (TGB) for consistency and DAC for scalable ingestion. Outperforms colocated dataloaders and Kafka in throughput with lower read latency on 64-GPU workloads.
* `serving` `offloading` `kernel` [KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving](http://arxiv.org/abs/2605.09735v1)
  > **TL;DR**: Addresses irregular KV-cache behavior in static-graph LLM serving. Proposes KV-RM, a runtime that decouples logical KV histories from physical storage via block paging and merge-staged transfers. Improves throughput by 1.8x and reduces tail latency by 2.4x while lowering reserved memory by 33% on two A100 GPUs.
* `kernel` `serving` `training` [Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon](http://arxiv.org/abs/2605.09708v1)
  > **TL;DR**: Presents Metal-Sci, a benchmark for evaluating LLM-driven evolutionary search of compute kernels on Apple Silicon. Proposes a harness for automatic kernel optimization using a frozen LLM in an evolutionary loop, with gated scoring for generalization. Achieves kernel speedups up to 10.7x while detecting silent regressions.
* `edge` `offloading` `serving` [Adaptive DNN Partitioning and Offloading in Heterogeneous Edge-Cloud Continuum](http://arxiv.org/abs/2605.09623v1)
  > **TL;DR**: Proposes adaptive DNN partitioning and offloading across heterogeneous edge-cloud continuum. Framework dynamically splits layers based on runtime profiling and network conditions. Reduces energy by 27.09-35.82% and latency by 6.34-22.92% compared to static partitioning.
* `training` `storage` `scaling` [From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs](http://arxiv.org/abs/2605.09370v1)
  > **TL;DR**: Analyzes operational challenges in large-scale LLM pre-training on 504 GPUs. Provides empirical evidence on failure detection, checkpointing bottlenecks, and recovery mechanisms using multi-party prometheus and log data. Achieves 100% failure detection rate and attributes NFS RPC saturation causing bandwidth underutilization.
* `training` `multi-modal` `scaling` [MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production](http://arxiv.org/abs/2605.08962v1)
  > **TL;DR**: Proposes MegaScale-Omni, a hyper-scale MLLM training system with workload resilience. Introduces decoupled parallelism, unified representations, and workload balancing for dynamic modality mixtures. Achieves 1.27×-7.57× throughput gains over SOTA under production workloads.
* `training` `scaling` [Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reduction](http://arxiv.org/abs/2605.08871v1)
  > **TL;DR**: Investigates if variance reduction improves time complexity in heterogeneous distributed training. Proposes Rennala MVR, a momentum-based variance reduction method extending Rennala SGD. Achieves better time complexity in relevant regimes and shows empirical gains on neural networks over baseline.
* `edge` `agentic` `offloading` [PAAC: Privacy-Aware Agentic Device-Cloud Collaboration](http://arxiv.org/abs/2605.08646v1)
  > **TL;DR**: Proposes PAAC for privacy-aware agentic collaboration between device and cloud by aligning planner-executor decomposition with trust boundaries. Uses typed placeholder tokens to protect sensitive data and on-device span masking. Improves accuracy by 15-36% and reduces leakage by 2-6× over baselines.
* `training` `kernel` `storage` [Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reduction](http://arxiv.org/abs/2605.08633v1)
  > **TL;DR**: Proposes a generative compression framework with historical priors for Earth observation data. Develops exascale training on supercomputer with co-optimization of model design, CUDA kernels, memory hierarchy, runtime, and parallelism. Achieves peak throughput of 2.16 EFLOP/s and 10,000x data reduction.
* `edge` `networking` `serving` [Large Language Models over Networks: Collaborative Intelligence under Resource Constraints](http://arxiv.org/abs/2605.08626v1)
  > **TL;DR**: Explores collaborative LLM inference across device, edge, and cloud to overcome resource constraints. Proposes vertical (device-cloud) and horizontal (multi-agent) collaboration frameworks with adaptive routing. Achieves improved response quality under heterogeneous compute, memory, and network limitations.
* `RL` `LoRA` `disaggregation` [MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service](http://arxiv.org/abs/2605.08527v1)
  > **TL;DR**: Addresses high computational cost of concurrent multi-tenant RL fine-tuning. Proposes MARLaaS, featuring shared base model with LoRA adapters and disaggregated asynchronous pipeline decoupling rollouts, environment, and training. Achieves 4.3× accelerator utilization improvement and 85% training time reduction.
* `training` `scaling` `kernel` [Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP](http://arxiv.org/abs/2605.08524v1)
  > **TL;DR**: Addresses the inefficiency in context parallelism for foundation model pre-training due to variable sequence lengths. Proposes FCP, a block-level sharding and bin-packing approach with flexible peer-to-peer communication. Achieves near-linear scalability on 256 GPUs with 1.13x-2.21x attention MFU improvement.
* `serving` `RL` `agentic` [FlashEvolve: Accelerating Agent Self-Evolution with Asynchronous Stage Orchestration](http://arxiv.org/abs/2605.08520v1)
  > **TL;DR**: Addresses high wall-clock cost of LLM-based agent evolution systems. Proposes FlashEvolve with asynchronous stage orchestration, artifact version tracking, and stale artifact repair policies. Achieves 3.5× to 4.9× higher proposal throughput on GEPA workloads compared to synchronous baselines.
* `inference` `privacy` `efficiency` [Private Vertical Federated Inference for Time-Series](http://arxiv.org/abs/2605.08343v1)
  > **TL;DR**: Proposes a hybrid public/private head model for vertically federated inference to balance privacy and efficiency. Splits model head into plaintext and lightweight MPC components, achieving 44.4x speedup and 91.2x communication reduction versus baselines while improving accuracy.
* `serving` `RL` [Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests](http://arxiv.org/abs/2605.08267v1)
  > **TL;DR**: Proposes execution envelopes as a shared admission contract for diverse AI backend requests to enable governance, observability, and policy hooks. Formalizes requested vs granted resources and defines lifecycle invariants. Demonstrates applicability on model deployment paths without affecting runtime execution.
* `serving` `offloading` [KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving](http://arxiv.org/abs/2605.09735v1)
  > **TL;DR**: Proposes KV-RM, a runtime for static-graph LLM serving that regularizes KV-cache movement by decoupling logical histories from physical storage and coalescing transfers. On a 2-GPU A100 node, it improves decoding throughput, reduces KV memory reservation, and eliminates tail latency spikes under production traces.

### 2026-05-11
* `serving` `kernel` `quantization` [Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation](http://arxiv.org/abs/2605.07985v1)
  > **TL;DR**: Proposes Dooly, a configuration-agnostic profiling method for LLM inference simulation. It uses taint propagation and selective profiling to build a latency database, reducing redundancy. Achieves up to 56.4% reduction in profiling GPU-hours with simulation accuracy within 5-8% MAPE.
* `hardware` `kernel` `training` [Accelerating Precise End-to-End Simulation: Latency-Sensitive Many-core System Modeling](http://arxiv.org/abs/2605.07750v1)
  > **TL;DR**: Proposes a fast end-to-end modeling approach for latency-sensitive many-core accelerators targeting large-scale systems. Models interconnect latency and SPM access to predict timing with <7% error vs RTL, achieving 115x simulation speedup. Case study optimizes FlashAttention-2 to reduce stalls.
* `training` `scaling` [A Scalable Recipe on SuperMUC-NG Phase 2: Efficient Large-Scale Training of Language Models](http://arxiv.org/abs/2605.07726v1)
  > **TL;DR**: Explores efficient distributed training strategies for LLMs on HPC systems. Combines tensor, pipeline, and sharded data parallelism with hyperparameter tuning to optimize throughput. Achieves 93% weak scaling efficiency and 82% strong scaling efficiency on 128 nodes for a 175B model.
* `training` `scaling` [HexiSeq: Accommodating Long Context Training of LLMs over Heterogeneous Hardware](http://arxiv.org/abs/2605.07569v1)
  > **TL;DR**: Proposes HexiSeq, a system for efficient long-context LLM training on heterogeneous GPU clusters. It formalizes partitioning as an optimization problem and schedules sequence shards and attention heads based on hardware capabilities. Achieves throughput improvements up to 1.72× on simulated clusters.
* `serving` `edge` `scaling` [Deadline-Driven Hierarchical Agentic Resource Sharing for AI Services and RAN Functions in AI-RAN](http://arxiv.org/abs/2605.07547v1)
  > **TL;DR**: Proposes a hierarchical agentic framework (HAF) for compute sharing in AI-RAN, combining an LLM-based agent for slow-timescale placement and a deadline-aware convex allocator for fast-timescale GPU/CPU allocation. Achieves 90.0% overall SLO fulfillment, a 20.5% improvement over baseline.
* `serving` `recommendation` `offloading` [RcLLM: Accelerating Generative Recommendation via Beyond-Prefix KV Caching](http://arxiv.org/abs/2605.07443v1)
  > **TL;DR**: Investigates high latency in generative recommendation LLMs due to personalized prompts. Proposes RcLLM with Beyond-Prefix KV caching, block decomposition, stratified storage, and selective attention. Achieves 1.31x-9.51x reduction in Time-To-First-Token versus prefix caching systems.
* `kernel` [MERBIT: A GPU-Based SpMV Method for Iterative Workloads](http://arxiv.org/abs/2605.07391v1)
  > **TL;DR**: Presents MERBIT, a GPU-based SpMV method for iterative sparse workloads like PageRank. Combines merge-path partitioning and bit-field descriptors to improve workload balance and coalesced memory access. Achieves average speedups of 1.27x (single) and 1.25x (double) over cuSPARSE COO.
* `RL` `sparse` `training` [SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication](http://arxiv.org/abs/2605.07330v1)
  > **TL;DR**: Reduces communication cost for weight synchronization in RL systems by exploiting high sparsity in parameter changes. Proposes SparseRL-Sync, a lossless sparse update method transmitting only indices and values. Achieves ~100x reduction in data transmitted at 99% sparsity.
* `edge` `kernel` `quantization` [TREA: Low-precision Time-Multiplexed, Resource-Efficient Edge Accelerator for Object Detection and Classification](http://arxiv.org/abs/2605.07321v1)
  > **TL;DR**: Proposes TREA, a low-precision edge accelerator with dual-precision SIMD MAC units and structured pruning for object detection. Features hardware-aware reductive pruning and reconfigurable CORDIC-based nonlinear functions. Achieves up to 9x latency reduction at kernel level with 50% sparsity.
* `serving` `scaling` `networking` [FATE: Future-State-Aware Scheduling for Heterogeneous LLM Workflows](http://arxiv.org/abs/2605.07238v1)
  > **TL;DR**: Proposes FATE, a future-state-aware scheduler for heterogeneous multi-stage LLM workflows that optimizes immediate cost and downstream state preservation. Uses frontier planning and conditional cost estimation to reduce end-to-end latency. Achieves 32.3% lower P95 latency over RoundRobin and 8.8% over best baseline.
* `serving` `scaling` [Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale](http://arxiv.org/abs/2605.06113v2)
  > **TL;DR**: Addresses load imbalance in data-parallel LLM serving caused by heterogeneous request growth and sticky assignments. Proposes BalanceRoute, an online routing algorithm with F-score optimization and optional lookahead. Deployed on 144-NPU cluster, improves throughput by reducing DP imbalance in production traces.
* `serving` `MoE` `networking` [Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend](http://arxiv.org/abs/2605.06055v2)
  > **TL;DR**: Proposes a relay-buffer-free communication design for MoE inference on Ascend, using direct placement and reading via pooled HBM. Removes intermediate buffers, reducing dispatch/combine latency. Improves TTFT by up to 1.5× while maintaining TPOT under latency constraints.
* `serving` `offloading` `scaling` [Regulating Branch Parallelism in LLM Serving](http://arxiv.org/abs/2605.06914v1)
  > **TL;DR**: Addresses step latency inflation from concurrent branch decoding in LLM serving. Proposes TAPER, a per-step admission controller that admits extra branches opportunistically based on predicted externality fitting within slack budget. Improves goodput by 1.77× over baselines while maintaining >95% SLO attainment on Qwen3-32B.

### 2026-05-08
* `training` `scaling` `kernel` [CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure](http://arxiv.org/abs/2605.06544v1)
  > **TL;DR**: Proposes CCL-Bench, a trace-based benchmark for evaluating LLM infrastructure by capturing reusable evidence on compute, memory, and communication efficiency. Reveals inefficiencies in parallelization and framework tuning, showing up to 3× slowdown in poorly configured setups.
* `RL` `serving` `offloading` [ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL](http://arxiv.org/abs/2605.06534v1)
  > **TL;DR**: Addresses high resource inefficiency in agentic RL rollouts during LLM training. Proposes ROSE, a cooperative elasticity system that opportunistically repurposes underutilized serving GPUs for rollouts without violating SLOs. Achieves 1.20-3.31x higher end-to-end throughput.
* `training` `sparse` `quantization` [ADELIA: Automatic Differentiation for Efficient Laplace Inference Approximations](http://arxiv.org/abs/2605.06392v1)
  > **TL;DR**: ADELIA addresses high computational cost in Laplace inference approximations by introducing an automatic differentiation-enabled multi-GPU backend that exploits model sparsity. Achieves 4.2–7.9× per-gradient speedups and reduces energy consumption by 5–8× compared to finite differences.
* `training` `scaling` [ResiHP: Taming LLM Training Failures with Dynamic Hybrid](http://arxiv.org/abs/2605.06374v1)
  > **TL;DR**: Addresses performance skew and failures in hybrid parallel LLM training. Proposes ResiHP, with workload-aware failure detection and dynamic parallelism adaptation. Improves training throughput by 1.04-4.39× under failures in a 256-GPU cluster.
* `agentic` `RL` `training` [Safactory: A Scalable Agent Factory for Trustworthy Autonomous Intelligence](http://arxiv.org/abs/2605.06230v1)
  > **TL;DR**: Addresses fragmentation in autonomous agent infrastructure. Proposes Safactory, integrating simulation, data management, and asynchronous RL evolution platforms for trustworthy autonomous agents. Achieves a unified evolutionary pipeline for continuous model improvement.
* `serving` `scaling` [Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale](http://arxiv.org/abs/2605.06113v1)
  > **TL;DR**: Addresses the data-parallel load balancing bottleneck in large-scale LLM serving. Proposes BalanceRoute, an online routing algorithm family with asymmetric F-score optimization and horizon discounting. Improves end-to-end serving throughput by balancing DP workers, reducing compute waste.
* `serving` `RL` [VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?](http://arxiv.org/abs/2605.06068v1)
  > **TL;DR**: Proposes VibeServe, an agentic loop that automatically generates bespoke LLM serving stacks for diverse scenarios. Uses outer-loop planning and inner-loop implementation/validation to create specialized systems. Matches vLLM in standard settings and outperforms in non-standard scenarios by 1.5–3× latency reduction.
* `kernel` `training` `inference` [FalconGEMM: Surpassing Hardware Peaks with Lower-Complexity Matrix Multiplication](http://arxiv.org/abs/2605.06057v1)
  > **TL;DR**: Proposes FalconGEMM, a framework for automating lower-complexity matrix multiplication to surpass hardware peaks. It features deployment, execution with group-parallel optimizations, and decision modules. Achieves 7.59% to 55.61% speedup over libraries like cuBLAS and AlphaTensor in LLM workloads.
* `MoE` `serving` `offloading` [Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend](http://arxiv.org/abs/2605.06055v1)
  > **TL;DR**: Accelerates MoE inference by eliminating relay buffers through direct expert window placement and pooled HBM. Designs prefill and decode schedules for throughput and latency. Reduces dispatch/combine latency by 1.6–3×, improves TTFT by 17–30%, and maintains TPOT.
* `MoE` `training` `networking` [MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems](http://arxiv.org/abs/2605.05888v1)
  > **TL;DR**: Addresses communication bottlenecks in Mixture-of-Experts training on multi-GPU systems. Proposes MoE-Hub, a hardware-software co-design with destination-agnostic communication for seamless overlap. Achieves 1.21x-1.98x end-to-end speedup over SOTA systems.
* `edge` `offloading` `RL` [LLM-Enhanced Deep Reinforcement Learning for Task Offloading in Collaborative Edge Computing](http://arxiv.org/abs/2605.05727v1)
  > **TL;DR**: Proposes LeDRL, a hybrid framework coupling lightweight LLM with DRL for real-time task offloading in edge computing. Uses structured prompts and self-attention for policy alignment, with reflective evaluator for semantic feedback. Achieves 17% higher task success rate and faster convergence.
* `serving` `kernel` `offloading` [Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving](http://arxiv.org/abs/2605.05696v1)
  > **TL;DR**: Investigates position-independent caching for agentic LLM serving to address cache inefficiency. Proposes Irminsul, a caching system with content-hash keying and delta-rotation for Multi-Head Latent Attention models. Achieves 63% prefill energy savings per cache hit and recovers up to 83% of prompt tokens.
* `training` `networking` `kernel` [Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi-GPU Systems](http://arxiv.org/abs/2605.05628v1)
  > **TL;DR**: Addresses inefficiency in tensor parallelism due to compute-communication phase mismatch in multi-GPU systems. Proposes CAIS, a compute-aware in-switch framework with ISA extensions, thread block coordination, and dataflow optimization. Achieves 1.38× training speedup over state-of-the-art NVLS solution.
* `MoE` `training` `networking` [Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs](http://arxiv.org/abs/2605.05607v1)
  > **TL;DR**: Addresses redundant inter-GPU communication in MoE training by proposing DySHARP, a dynamic in-switch computing solution with communication primitives and scheduling. Achieves up to 1.79× speedup over state-of-the-art.
* `serving` `edge` `quantization` [EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge](http://arxiv.org/abs/2605.05527v1)
  > **TL;DR**: Addresses deadline-aware serving of multiple DNN models on edge GPUs. Proposes EdgeServing with time-sharing and early-exit inference, using a stability score to quantify scheduling impact. Reduces SLO violations by 47% and P95 latency by 51% compared to baselines.
* `serving` `scaling` `networking` [Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism](http://arxiv.org/abs/2605.05467v1)
  > **TL;DR**: Investigates dynamic adaptation of tensor parallelism for tiered-SLO LLM serving. Proposes Nitsum, which jointly optimizes TP level and scheduling with weight reuse and KV migration. Achieves up to 5.3 times higher SLO-compliant goodput compared to state-of-the-art systems.

### 2026-05-07
* `training` `RL` `scaling` [Toward a Risk Assessment Framework for Institutional DeFi: A Nine-Dimension Approach](http://arxiv.org/abs/2605.05145v1)
  > **TL;DR**: Investigates how to optimize deep learning training systems. Introduces a new parallelization strategy combining model parallelism and efficient gradient synchronization. Achieves a 1.4x speedup compared to state-of-the-art methods on large-scale neural network training.
* `training` `MoE` `scaling` [Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism](http://arxiv.org/abs/2605.05049v1)
  > **TL;DR**: Addresses inefficiency in large-scale MoE training on HPC. Proposes Piper, a framework using resource modeling and pipelined hybrid parallelism to optimize memory, compute, and communication. Achieves 2-3.5x higher MFU than X-MoE and 1.2-9x all-to-all bandwidth improvement.
* `networking` `offloading` `hardware` [Communication Offloading on SmartNIC DPUs: A Quantitative Approach](http://arxiv.org/abs/2605.04842v1)
  > **TL;DR**: Explores offloading communication tasks to SmartNIC DPUs to save CPU resources. Designs Buddy, an engine decoupling communication from applications, running on DPUs and CPUs. Achieves up to 1.55x speedup for host-dominated workloads like Quicksilver.
* `edge` `offloading` `networking` [Delay-Aware Large-Small Model Collaboration over LEO Satellite Networks](http://arxiv.org/abs/2605.04565v1)
  > **TL;DR**: Proposes a delay-aware large-small model collaboration scheme for LEO satellite networks. Uses multi-agent RL for decentralized routing and bisection search for offloading decisions. Reduces service delay by up to 31.85% versus benchmarks.
* `training` `networking` [CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training](http://arxiv.org/abs/2605.04478v1)
  > **TL;DR**: Addresses slow/hang anomalies in collective communication during large-scale model training. Proposes CCL-D, integrating real-time probing and intelligent analysis for anomaly detection and root-cause location. Deployed on a 4,000-GPU cluster, locates faulty ranks within 6 minutes, outperforming existing solutions.
* `recommendation` `scaling` `serving` [One Pool, Two Caches: Adaptive HBM Partitioning for Accelerating Generative Recommender Serving](http://arxiv.org/abs/2605.04450v1)
  > **TL;DR**: Addresses dynamic HBM allocation conflicts between embedding caches and KV caches in generative recommender inference. Proposes HELM with adaptive memory allocation and routing to adjust allocation ratios in 32μs, reducing P99 latency by 24-38% and achieving 93.5-99.6% SLO satisfaction.
* `serving` `scaling` [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](http://arxiv.org/abs/2605.04357v1)
  > **TL;DR**: Addresses cost-efficient serving of multiple LLMs over heterogeneous GPUs. Proposes Coral, an adaptive system that jointly optimizes resource allocation and serving strategies via a two-stage decomposition. Reduces serving cost by up to 2.79x and improves goodput by 2.39x under scarce resources.
* `training` `networking` `scaling` [Resilient AI Supercomputer Networking using MRC and SRv6](http://arxiv.org/abs/2605.04333v1)
  > **TL;DR**: Addresses tail latency in large-scale AI training by proposing MRC protocol for multi-path RDMA transport and SRv6-based static routing to bypass failures. Enables resilient training on clusters exceeding 100K GPUs, withstanding network disruptions without job interruption.
* `storage` `offloading` `networking` [Revocation-Ready CP-ABE Key Management for Blockchain-Based IoT Data Sharing](http://arxiv.org/abs/2605.04280v1)
  > **TL;DR**: Proposes a blockchain-based CP-ABE key management layer for IoT data sharing, replacing online key release with ciphertext key publication. Uses epoch-bound attributes and CK-rotation for revocation without heavy re-encryption. Gateway mode reduces median decryption time by 4x under client slowdown.

### 2026-05-06
* `training` `RAG` `edge` [A Workflow-Oriented Framework for Asynchronous Human-AI Collaboration in Hybrid and Compute-Intensive HPC Environments](http://arxiv.org/abs/2605.03743v1)
  > **TL;DR**: Proposes a workflow framework for asynchronous human-AI collaboration in HPC and hybrid environments to avoid idle resources during human input checks. Integrates with SLURM, supports containerization, and enables pausing at checkpoints without halting compute jobs. Increases workflow efficiency by reducing resource idleness in MareNostrum 5 experiments.
* `inference` `serving` `speculative decoding` [SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection](http://arxiv.org/abs/2605.02888v2)
  > **TL;DR**: Proposes SpecKV, an adaptive controller for selecting speculation length γ in speculative decoding based on draft model confidence and entropy. Uses an MLP predictor to maximize tokens per step. Achieves 56% speedup over fixed γ=4 with <0.5% overhead.
* `scaling` `kernel` `offline` [Enhancing Performance Insight at Scale: A Heterogeneous Framework for Exascale Diagnostics](http://arxiv.org/abs/2605.03561v1)
  > **TL;DR**: Proposes a heterogeneous framework for exascale performance diagnostics. Combines high-performance C++ API and GPU acceleration for high-throughput analysis of execution traces. Achieves 314x speedup in trace processing and identifies 32.28% potential speedup in a workload.
* `kernel` `training` [FACT: Compositional Kernel Synthesis with a Three-Stage Agentic Workflow](http://arxiv.org/abs/2604.26666v2)
  > **TL;DR**: Proposes FACT, an agent-driven workflow synthesizing optimized kernels from PyTorch modules via pattern discovery, realization, and composition. Achieves up to 2.03x speedup on transformers over baselines by grounding in CUTLASS and auto-tuning.
* `serving` `kernel` `offloading` [VDCores: Resource Decoupled Programming and Execution for Asynchronous GPU](http://arxiv.org/abs/2605.03190v1)
  > **TL;DR**: Proposes VDCores, a decoupled programming model abstracting GPU resources as virtual cores to improve utilization of asynchronous hardware. Uses micro-operations with dependency tracking for automatic compute-memory overlap. Achieves up to 77% higher decoding throughput for LLM inference.
* `agentic` `RL` [Pact: A Choreographic Language for Agentic Ecosystems](http://arxiv.org/abs/2605.03143v1)
  > **TL;DR**: Introduces Pact, a choreographic language for self-interested agents in multi-party settings. Maps protocols to formal games with computational decision policies. Demonstrates use in multi-party coordination with bounded-rational solver.
* `training` `scaling` `storage` [From Barrier to Bridge: The Case for AI Data Center/Power Grid Co-Design](http://arxiv.org/abs/2605.03090v1)
  > **TL;DR**: Addresses AI data centers' massive power demands breaking grid stability. Proposes co-design of compute and power infrastructure for joint capacity planning, control, and protocols. Enables sustainable scaling of large-model training by coordinating megawatt-level load fluctuations.
* `RAG` `training` `offline` [AutoRAGTuner: A Declarative Framework for Automatic Optimization of RAG Pipelines](http://arxiv.org/abs/2605.02967v1)
  > **TL;DR**: Automates RAG pipeline optimization to reduce manual tuning. Proposes a declarative framework with modular architecture and Bayesian optimization for hyperparameter tuning. Achieves 95% reduction in code churn for architectural adjustments.
* `serving` `offloading` `storage` [Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving](http://arxiv.org/abs/2605.03375v1)
  > **TL;DR**: Proposes Tutti, a GPU-centric SSD-backed KV cache system for long-context LLM serving, eliminating CPU intervention via GPU object abstraction, GPU io_uring, and slack-aware scheduling. Reduces TTFT by 78.3% and serving cost by 27%, nearing DRAM-backed performance.

### 2026-05-05
* `serving` `quantization` [SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection](http://arxiv.org/abs/2605.02888v1)
  > **TL;DR**: Proposes SpecKV, an adaptive controller for speculative decoding that dynamically selects speculation length γ based on draft model signals to optimize token acceptance. Uses MLP to predict γ per step for compressed models, improving tokens per step by 56.0% over fixed γ with minimal overhead.

### 2026-05-05
* `edge` `kernel` `serving` [Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge](http://arxiv.org/abs/2605.00536v2)
  > **TL;DR**: Addresses efficient LLM inference on edge devices with resource constraints. Proposes Tempus, a temporally scalable GEMM framework using iterative execution and data tiling on AMD Versal AI Edge. Achieves 607 GOPS at 10.677 W, with 211.2× higher prominence than spatial SOTA.
* `serving` `disaggregation` `scaling` [Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference](http://arxiv.org/abs/2605.02329v1)
  > **TL;DR**: Addresses SLO violations in disaggregated LLM serving due to request length imbalance. Proposes Kairos with urgency-based prefill scheduling and slack-guided adaptive batching for decode. Achieves up to 33.8% higher end-to-end SLO attainment and 19.3% higher decode throughput.
* `offline` `offloading` `sparse` [PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers](http://arxiv.org/abs/2605.02189v1)
  > **TL;DR**: Proposes PipeMax for high-throughput offline LLM inference on GPU servers by integrating pipeline parallelism with KV cache offloading. Coordinates computation and data movement to expand effective GPU memory, enabling large-batch processing. Achieves up to 2.51x higher throughput than vLLM on an 8-GPU node.
* `agentic` `scaling` `training` [AAFLOW: Scalable Patterns for Agentic AI Workflows](http://arxiv.org/abs/2605.02162v1)
  > **TL;DR**: Proposes AAFLOW, a distributed runtime for agentic workflows to address scalability and reproducibility issues. Models workflows as operators for communication-efficient execution, utilizing Apache Arrow/Cylon for zero-copy transfer and resource-deterministic scheduling. Achieves up to 4.64× pipeline speedup.
* `training` `offloading` `multi-modal` [Cross-Layer Energy Analysis of Multimodal Training on Grace Hopper Superchips](http://arxiv.org/abs/2605.01938v1)
  > **TL;DR**: Analyzes energy-performance tradeoffs in multimodal training on NVIDIA Grace Hopper. Characterizes interactions between application, runtime, and hardware layers using high-bandwidth CPU-GPU interconnects. Achieves balanced energy efficiency and throughput via offloading, sequence parallelism, and hardware-aware scheduling.
* `sparse` `kernel` `serving` [Stochastic Sparse Attention for Memory-Bound Inference](http://arxiv.org/abs/2605.01910v1)
  > **TL;DR**: Addresses bandwidth limitation in autoregressive decoding by stochastic sparsification of KV-cache access. Proposes SANTA method for variance-reduced sampling to gather only subsets of values and keys. Achieves 1.5× decode-step attention kernel speedup over FlashInfer/FlashDecoding at 32k-token contexts.
* `serving` `offloading` `quantization` [SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving](http://arxiv.org/abs/2605.01708v1)
  > **TL;DR**: Addresses the KV cache transfer bottleneck in disaggregated LLM serving. Proposes SplitZip, a GPU-friendly lossless KV compressor exploiting exponent redundancy, with fixed-length codes and escape streams. Achieves up to 1.32× speedup in end-to-end KV transfer and 1.30× in TTFT.
* `serving` `multi-modal` `edge` [VUDA: Breaking CUDA-Vulkan Isolation for Spatial Sharing of Compute and Graphics on the Same GPU](http://arxiv.org/abs/2605.01352v1)
  > **TL;DR**: Proposes VUDA for spatial sharing of CUDA compute and Vulkan graphics on GPUs. Redirects channels and grafts page-tables to merge execution contexts, eliminating data copying. Achieves 85% higher throughput vs. temporal sharing in embodied AI workloads.
* `serving` [Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics](http://arxiv.org/abs/2605.01280v1)
  > **TL;DR**: Argues that LLM inference serving requires mathematical optimization and algorithmic foundations beyond heuristics. Proposes developing models capturing KV cache dynamics, phase asymmetry, and batching constraints to design algorithms with provable guarantees. Emerging work shows such methods can match heuristic performance with theoretical assurances.
* `edge` `networking` `scaling` [ncsim: A Lightweight Simulator for Networked Edge Computing with Wireless Interference Modeling](http://arxiv.org/abs/2605.01094v1)
  > **TL;DR**: Proposes ncsim, a simulator for DAG task scheduling in wireless edge computing with interference modeling. Jointly models compute placement and IEEE 802.11 CSMA/CA interference to avoid rank inversions. Shows interference-free models cause up to 2.7x worse makespan in 27.8%-50% scenarios.
* `offline` `quantization` [SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data](http://arxiv.org/abs/2605.01060v1)
  > **TL;DR**: Proposes SURGE, a streaming GPU encoding system for partitioned data embedding pipelines. Introduces a memory-safety bound enabling streaming two-threshold policy with O(B_min + n_max) memory. Achieves 7% throughput gain and 12.6× less memory usage vs baselines on 10M texts with bge-base encoder.
* `training` `hardware` `kernel` [VUDA: Breaking CUDA-Vulkan Isolation for Spatial Sharing of Compute and Graphics on the Same GPU](http://arxiv.org/abs/2605.01352v1)
  > **TL;DR**: Addresses GPU utilization inefficiency when interleaving CUDA compute and Vulkan graphics. Proposed VUDA system enables spatial sharing by redirecting CUDA streams to Vulkan's scheduler and grafting page tables without data copying. Achieves 85% higher throughput than temporal-sharing baselines.

### 2026-05-04
* `MoE` `serving` `networking` [Eliminating Hidden Serialization in Multi-Node Megakernel Communication](http://arxiv.org/abs/2605.00686v1)
  > **TL;DR**: Addresses hidden serialization in multi-node MoE megakernel communication for RDMA fabrics. Proposes Perseus using decoupled signaling and NIC-side ordering to reduce fences and prevent proxy stalls. Achieves up to 10.3× end-to-end speedup on multi-node MoE inference.
* `serving` `offloading` `kernel` [LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling](http://arxiv.org/abs/2605.00616v1)
  > **TL;DR**: Proposes LLM-Emu, a serving-native emulator for vLLM that replaces GPU execution with profile-sampled latency for realistic, low-cost evaluation. Achieves end-to-end latency within 5.3% error and throughput within 1.9% of real vLLM.
* `edge` `serving` `kernel` [Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge](http://arxiv.org/abs/2605.00536v1)
  > **TL;DR**: Proposes Tempus, a resource-invariant GEMM framework for edge SoCs using iterative graph execution and algorithmic tiling. Achieves 607 GOPS at 10.677 W, 211.2x higher prominence than SOTA with 22x core frugality and 6.3x I/O reduction for efficient edge LLM inference.
* `serving` `offloading` `quantization` [SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters](http://arxiv.org/abs/2605.00528v1)
  > **TL;DR**: Proposes SAGA, a distributed scheduler for AI agents that treats entire workflows as schedulable units, reusing KV cache across tool calls and employing session-affinity batching. Reduces task completion time by 1.64x over vLLM and improves GPU memory utilization by 1.22x on a 64-GPU cluster.
* `serving` `MoE` `networking` [Space Network of Experts: Architecture and Expert Placement](http://arxiv.org/abs/2605.00515v1)
  > **TL;DR**: Proposes Space-XNet for deploying MoE models in satellite networks. Uses layer partitioning along orbit direction and optimization-based expert placement based on activation frequency. Achieves at least 3× lower latency compared to baselines.
* `training` `hardware` `scaling` [Adaptation of AI-accelerated CFD Simulations to the IPU platform](http://arxiv.org/abs/2605.00462v1)
  > **TL;DR**: Assesses training ML models for CFD on IPUs, using TensorFlow and Poplar SDK. Optimizes data feeding via popdist to overcome host-side bottlenecks, achieving 34% speedup. Scales from 2 to 16 IPUs, improving throughput from 560.8 to 2805.8 samples/s.
* `serving` `networking` `scaling` [AI Inference as Relocatable Electricity Demand: A Latency-Constrained Energy-Geography Framework](http://arxiv.org/abs/2604.27855v2)
  > **TL;DR**: Proposes an energy-geography framework for geo-distributed AI inference to optimize electricity cost and carbon intensity under latency constraints. Models placement as constrained optimization over prices, carbon, capacity, and network latency. Latency relaxation reduces electricity costs by up to 40% while meeting SLOs in simulations.
* `serving` `quantization` `RL` [Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference](http://arxiv.org/abs/2605.00300v1)
  > **TL;DR**: Introduces TokenArena, a continuous benchmark for LLM endpoints measuring efficiency across speed, cost, quality, and energy. Evaluates 78 endpoints across 12 model families, showing variations in latency (order of magnitude), accuracy (up to 12.5 pts), and energy efficiency (6.2x). Highlights workload-aware pricing impact on leaderboards.
* `serving` `RL` `offloading` [SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters](http://arxiv.org/abs/2605.00528v1)
  > **TL;DR**: Introduces SAGA, a workflow-atomic scheduler for AI agent inference that treats entire workflows as schedulable units. Uses Agent Execution Graphs to predict KV cache reuse, session-affinity batching, and Agent Fair Share. Reduces task completion time by 1.64x over vLLM and improves GPU memory utilization by 1.22x.

### 2026-05-01
* `kernel` `sparse` `hardware` [Exploring Sparse Matrix Multiplication Kernels on the Cerebras CS-3](http://arxiv.org/abs/2604.27985v1)
  > **TL;DR**: Investigates sparse matrix multiplication kernel performance on the Cerebras CS-3 accelerator. Designs and optimizes SpMM/SDDMM kernels to improve I/O, memory footprint, and scalability. Achieves up to 100× speedup over CPU for SpMM with 90% sparse matrices, though degrades beyond 99% sparsity.
* `scaling` `hardware` [Affinity Tailor: Dynamic Locality-Aware Scheduling at Scale](http://arxiv.org/abs/2604.27915v1)
  > **TL;DR**: Analyzes how dynamic scheduling affects locality in multicore systems. Proposes Affinity Tailor, a userspace-guided kernel scheduler that uses compact CPU sets as affinity hints for demand-aware placement. Achieves 12% per-CPU throughput on chiplet systems and 3-7% per-GB throughput gains.
* `serving` `networking` `edge` [AI Inference as Relocatable Electricity Demand: A Latency-Constrained Energy-Geography Framework](http://arxiv.org/abs/2604.27855v1)
  > **TL;DR**: Studies digital relocation of AI inference workloads to optimize electricity cost and carbon emissions under latency constraints. Develops a geo-distributed framework modeling energy-latency tradeoffs with optimization over electricity prices, carbon intensity, and network latency. Simulation shows latency relaxation expands feasible geography but migration frictions reduce benefits.
* `scaling` `storage` [Towards the Democratization and Standardization of Dynamic Resources with MPI Spawning](http://arxiv.org/abs/2604.27430v1)
  > **TL;DR**: Proposes a unified dynamic resource management API for HPC applications, enabling seamless resource reconfiguration without process respawning. Integrates the Proteo engine into DMRlib for adaptive resource strategies. Enhances MPDATA solver performance with malleability.
* `training` `scaling` `kernel` [AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism](http://arxiv.org/abs/2604.27089v1)
  > **TL;DR**: Addresses long-context LLM training inefficiencies. Introduces AutoSP, a compiler-based system automating sequence parallelism and context-aware activation checkpointing. Achieves up to 2.7× longer context on NVIDIA hardware with negligible runtime cost.
* `training` `offloading` `LoRA` [Efficient Training on Multiple Consumer GPUs with RoundPipe](http://arxiv.org/abs/2604.27085v1)
  > **TL;DR**: Proposes RoundPipe, a pipeline parallelism schedule that dynamically dispatches computation stages across GPUs to eliminate weight binding and pipeline bubbles. Integrates transfer scheduling, synchronization, and partitioning for consumer-grade GPU clusters. Achieves 1.48–2.16x speedups in fine-tuning 1.7B to 32B models and enables Qwen3-235B LoRA fine-tuning on 8x RTX 4090s.
* `serving` `MoE` `multi-modal` [FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving](http://arxiv.org/abs/2604.26881v1)
  > **TL;DR**: Proposes FaaSMoE, a serverless framework for multi-tenant MoE serving using FaaS functions to decouple control and execution, enabling elastic expert invocation. Achieves over 3× resource reduction compared to full-model deployment.
* `hardware` `kernel` [Exploring the Efficiency of 3D-Stacked AI Chip Architecture for LLM Inference with Voxel](http://arxiv.org/abs/2604.26821v1)
  > **TL;DR**: Proposes Voxel, a compiler-aware simulation framework for exploring 3D-stacked AI chip efficiency in LLM inference. Enables software/hardware co-exploration via programmable execution plans. Validated with emulator; found cooperative factors and mapping dependencies impact efficiency (e.g., DRAM bank bandwidth optimization).
* `training` `kernel` `networking` [CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training](http://arxiv.org/abs/2604.24013v2)
  > **TL;DR**: Addresses tail latency in distributed LLM training due to communication overhead. Proposes CommFuse, replacing collective ops with decomposed P2P communication and fine-grained scheduling. Achieves higher MFU (up to 26.5% improvement) and lower latency compared to baselines.
* `training` `scaling` [COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training](http://arxiv.org/abs/2604.26687v1)
  > **TL;DR**: Introduces COPUS, a system for adaptive joint tuning of global batch size, parallelism strategy, and micro-batch size during LLM training. Uses Goodput metric combining throughput and statistical efficiency, enabling dynamic reconfiguration. Achieves 3.9-11.1% time-to-convergence speedups across models.
* `kernel` `RLHF` [FACT: Compositional Kernel Synthesis with a Three-Stage Agentic Workflow](http://arxiv.org/abs/2604.26666v1)
  > **TL;DR**: Proposes FACT, a three-stage agentic workflow for automating the generation of optimized CUTLASS kernels from PyTorch modules. Combines pattern discovery, realization, and composition to synthesize and auto-tune kernels. Achieves up to 2.79× end-to-end speedup on MiniGPT block.
* `training` `scaling` `storage` [DMRlib: Easy-coding and Efficient Resource Management for Job Malleability](http://arxiv.org/abs/2604.26624v1)
  > **TL;DR**: Proposes DMRlib, a library for enabling process malleability in MPI-like applications with minimal coding effort. Evaluates resource management for job scalability, showing up to 3x higher throughput and improved resource allocation compared to static methods.
* `serving` `scaling` [A comprehensive evaluation of spatial co-execution on GPUs using MPS and MIG technologies](http://arxiv.org/abs/2604.22430v2)
  > **TL;DR**: Evaluates spatial co-execution on GPUs using NVIDIA's MPS and MIG technologies to improve resource utilization. Compares trade-offs between flexibility (MPS) and isolation (MIG) for concurrent execution. MPS achieves up to 30% performance gain and 20% energy reduction in favorable scenarios.
* `offloading` `edge` `kernel` [DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference](http://arxiv.org/abs/2604.26557v1)
  > **TL;DR**: Addresses KV cache offloading for edge LLM inference under memory constraints. Proposes DUAL-BLADE with dual-path storage access (page-cache or NVMe-direct) and adaptive pipelining to reduce I/O bottlenecks. Reduces prefill and decode latency by up to 33.1% and 42.4%, respectively.
* `multi-modal` `offloading` `edge` [Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models](http://arxiv.org/abs/2604.26508v1)
  > **TL;DR**: Proposes a progressive semantic communication framework for edge-cloud VLM inference using adaptive token compression. Employs a Meta AutoEncoder to reduce communication costs while maintaining semantic fidelity. Achieves significant latency reduction at 1 Mbps uplink compared to full-edge/cloud solutions.
* `RL` `offloading` `sparse` [SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning](http://arxiv.org/abs/2604.26388v1)
  > **TL;DR**: Proposes SplitFT for adaptive federated split learning of LLM fine-tuning. Dynamically sets client cut layers based on resources and reduces LoRA rank to cut communication. Achieves 1.5× faster training and higher accuracy over baselines on diverse benchmarks.
* `kernel` `training` [CUDA Kernel Optimization and Counter-Free Performance Analysis for Depthwise Convolution in Cloud Environments](http://arxiv.org/abs/2604.25422v2)
  > **TL;DR**: Optimizes CUDA kernels for depthwise convolution in S4ConvD models. Introduces warp-tiled variants and counter-free profiling to analyze memory efficiency. Achieves 3.26× faster convolution and 1.29× end-to-end training speedup compared to naive CUDA.
* `edge` `offloading` `serving` [Efficient, VRAM-Constrained xLM Inference on Clients](http://arxiv.org/abs/2604.26334v1)
  > **TL;DR**: Improves VRAM-constrained xLM inference on client devices via pipelined sharding for CPU-GPU hybrid scheduling. Uses model sharding, CPU offloading, pipelined compute, and prioritized VRAM placement for TTFT/TPS gains. Achieves up to 6.7x lower TTFT, 30x higher TPS.
* `training` `quantization` `MoE` [Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference](http://arxiv.org/abs/2604.26294v1)
  > **TL;DR**: Proposes tensor and sequence parallelism (TSP) to reduce memory overhead in Transformer training/inference by folding multiple parallelism onto one device axis, via parameter shard iteration and partial output accumulation. Achieves up to 3.1x reduced activation memory versus TP+SP.
* `training` `RL` `MoE` [DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training](http://arxiv.org/abs/2604.26256v1)
  > **TL;DR**: Addresses rollout bottleneck in RL-based LLM training caused by long-tailed trajectories. Proposes DORA, an asynchronous system with multi-version streaming rollout to overlap generation and training. Achieves 2-4× training speedup on large-scale deployments without compromising convergence.
* `offloading` `serving` `kernel` [DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference](http://arxiv.org/abs/2604.26074v1)
  > **TL;DR**: Proposes DAK, a direct-access memory offloading framework for LLM inference, utilizing GPU shared memory via repurposed TMA and congestion control to optimize remote access. Achieves near-optimal bandwidth aggregation and up to 3× performance gain over baselines.
* `MoE` `serving` `kernel` [RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts](http://arxiv.org/abs/2604.26039v1)
  > **TL;DR**: Addresses runtime inefficiency in MoE inference kernels due to varying expert distributions. Proposes RaMP, a routing-aware kernel dispatch method with a hardware-derived wave cost model and polymorphic kernel design. Improves end-to-end serving speed by up to 1.41x compared to existing systems.
* `agentic` `offloading` `scaling` [Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxes](http://arxiv.org/abs/2604.28138v1)
  > **TL;DR**: Proposes Crab, a checkpoint/restore system for LLM agent sandboxes that uses eBPF-based inspection to classify OS effects per agent turn. Reduces checkpoint traffic by 87% while maintaining 100% recovery correctness and adds only 1.9% runtime overhead.
* `scaling` `training` `serving` [Affinity Tailor: Dynamic Locality-Aware Scheduling at Scale](http://arxiv.org/abs/2604.27915v1)
  > **TL;DR**: Investigates how to preserve locality in multicore scheduling for improved resource efficiency. Proposes Affinity Tailor, a userspace-guided kernel scheduler using topology-aware CPU affinity hints. Achieves up to 12% per-CPU throughput gain on chiplet systems over Linux CFS.

### 2026-04-29
* `serving` `agentic` `scaling` [Pythia: Toward Predictability-Driven Agent-Native LLM Serving](http://arxiv.org/abs/2604.25899v1)
  > **TL;DR**: Proposes Pythia, an agent-native LLM serving system exploiting structured multi-agent workflow semantics to optimize serving. Introduces a serving layer interface capturing workflow constraints, improving cache hit rates and scaling. Achieves higher throughput and job completion time over baselines.
* `edge` `serving` `networking` [SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission](http://arxiv.org/abs/2604.25777v1)
  > **TL;DR**: Proposes SpecFed for efficient federated LLM inference via speculative decoding and compressed communication. Combines speculative processing for parallel execution and top-K compression to reduce token transmission. Achieves 5.3× higher throughput while maintaining generation fidelity.
* `kernel` `training` [CUDA Kernel Optimization and Counter-Free Performance Analysis for Depthwise Convolution in Cloud Environments](http://arxiv.org/abs/2604.25422v1)
  > **TL;DR**: Optimizes CUDA kernels for depthwise convolution in S4ConvD models via warp-tiling and memory-access improvements. Proposes a counter-free performance analysis method using traffic modeling and roofline analysis. Achieves 3.26x convolution speedup and 1.29x training acceleration.
* `serving` `offloading` `kernel` [CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration](http://arxiv.org/abs/2604.25080v1)
  > **TL;DR**: Proposes CacheFlow, a KV cache restoration framework using 3D parallelism across tokens, layers, and GPUs with batch-aware scheduling to reduce TTFT. Achieves 10%-62% lower Time-To-First-Token over baselines in diverse models and workloads.
* `offline` `training` `sparse` [Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark](http://arxiv.org/abs/2604.25061v1)
  > **TL;DR**: Addresses inefficient policy learning in Spark due to rowwise Python execution and driver-side materialization. Proposes vectorized inference via mapInArrow/mapInPandas and on-executor split search. Achieves up to 7.23M rows/s inference throughput at 50M rows while preserving policy semantics.
* `quantization` `serving` [PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference](http://arxiv.org/abs/2604.24971v1)
  > **TL;DR**: Introduces PolyKV, a shared KV cache pool using quantized Keys (int8) and MSE-compressed Values (3-bit) for concurrent multi-agent LLM inference. Achieves 97.7% KV memory reduction and maintains low perplexity degradation (+0.57%) with 15 agents on Llama-3-8B.
* `serving` `RAG` [ITAS: A Multi-Agent Architecture for LLM-Based Intelligent Tutoring](http://arxiv.org/abs/2604.24808v1)
  > **TL;DR**: Addresses how to deploy LLM-based tutoring systems in real courses. Proposes ITAS, a multi-agent architecture with teaching, operational, and feedback layers, using microservices and conversational agents. Achieves stable handling of 334 chat turns without task-boundary hallucinations and captures 10,628 events.
* `edge` `quantization` `serving` [Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers](http://arxiv.org/abs/2604.24785v1)
  > **TL;DR**: Proposes a multi-dimensional benchmarking methodology to evaluate LLM inference on hardware-accelerated single-board computers. Tests NPUs, GPUs, quantized models, and assesses power efficiency, size, and throughput trade-offs. Achieves up to 32% higher token throughput per watt compared to CPU-only configurations.

### 2026-04-28
* `training` `split_learning` [A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations](http://arxiv.org/abs/2604.24468v1)
  > **TL;DR**: Surveys split learning for private LLM fine-tuning. Explores distributed model partitioning between clients and servers to enable collaborative training without data sharing. Highlights system optimizations for efficiency and privacy defenses across diverse resource constraints.
* `training` `scaling` `offloading` [Incisor: Ex Ante Cloud Instance Selection for HPC Jobs](http://arxiv.org/abs/2604.24464v1)
  > **TL;DR**: Automates cloud instance selection for HPC jobs by leveraging LLM-guided reasoning to infer hardware requirements from submission artifacts alone. Achieves 54% runtime reduction and 44% cost savings compared to expert-based selection.
* `offloading` `edge` `networking` [Exact, Efficient, and Reliable Multi-Objective and Multi-Constrained IoT Workflow Scheduling in Edge-Hub-Cloud Cyber-Physical Systems](http://arxiv.org/abs/2604.24340v1)
  > **TL;DR**: Proposes an exact multi-objective workflow scheduling method for IoT edge-hub-cloud systems using continuous-time MILP to jointly optimize latency, energy, and reliability under constraints. Achieves up to 33.96% average improvement in energy consumption while meeting practical runtime requirements.
* `training` `networking` `kernel` [Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs](http://arxiv.org/abs/2604.22228v2)
  > **TL;DR**: Improves intra-node GPU communication for MPI-based HPC by integrating CUDA Graphs into UCX to enable concurrent multi-path transfers (NVLink/PCIe). Achieves up to 2.95x bandwidth improvement in GPU-to-GPU transfers compared to single-path UCX at 512MB message size.
* `serving` `scaling` `offline` [Latency and Cost of Multi-Agent Intelligent Tutoring at Scale](http://arxiv.org/abs/2604.24110v1)
  > **TL;DR**: Evaluates latency and cost of multi-agent LLM tutoring systems under different deployment tiers and concurrency levels. Compares Performance of Standard PayGo, Priority PayGo, and Provisioned Throughput tiers with live testing up to 50 users. Priority PayGo maintains sub-4-second response times across all loads.
* `training` `quantization` `networking` [TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training](http://arxiv.org/abs/2604.24088v1)
  > **TL;DR**: Proposes TACO, an FP8-based adaptive compression framework for intermediate tensors in tensor-parallel LLM training. Integrates data-driven reshaping, Dual-Scale Quantization, and fused operator to reduce communication overhead. Achieves 1.87× throughput improvement with near-lossless accuracy.
* `training` `scaling` `networking` [FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost](http://arxiv.org/abs/2604.24073v1)
  > **TL;DR**: Addresses computational inefficiency in distributed training of sequential recommendation models. Proposes FreeScale with load-balanced samples, prioritized embedding communication overlap, and SM-Free techniques to reduce communication bottlenecks. Achieves 90.3% reduction in computational bubbles on 256 H100 GPUs.
* `scaling` `training` `serving` [KubePACS: Kubernetes Cluster Using Performant, Highly Available, and Cost Efficient Spot Instances](http://arxiv.org/abs/2604.24027v1)
  > **TL;DR**: Presents KubePACS, a Kubernetes autoscaler using spot instances to optimize cost and performance for cluster workloads. Formulates multi-objective optimization with real-time spot prices, performance data, and availability scores. Achieves up to 81% higher performance per dollar than prior solutions.
* `kernel` `serving` `offloading` [ClusterFusion++: Expanding Cluster-Level Fusion to Full Transformer-Block Decoding](http://arxiv.org/abs/2604.23553v1)
  > **TL;DR**: Addresses high latency in LLM decoding from fragmented operators and tensor materialization. Extends ClusterFusion to fuse full Transformer blocks using CUDA-level cluster collectives, adding CUDA Graph compatibility. Achieves 1.34x throughput gain on Pythia-2.8B with near-token-identical fidelity.
* `agentic` `networking` [Usable Agent Discovery for Decentralized AI Systems](http://arxiv.org/abs/2604.23080v1)
  > **TL;DR**: Studies decentralized agent discovery for AI systems under node and agent-level churn. Compares structured (Kademlia) versus gossip-based (Cyclon+Vicinity) overlays in resilience and readiness. Shows structured overlays are more robust in stable/churn regimes while gossip excels when readiness dominates.
* `edge` `inference` `serving` [Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities](http://arxiv.org/abs/2604.22906v1)
  > **TL;DR**: Surveys LLM inference challenges at the network edge due to resource constraints. Reviews system architectures, model optimization, deployment strategies, and resource management techniques. Summarizes techniques to enable efficient edge deployment for LLMs.

### 2026-04-27
* `edge` `offline` `LoRA` [Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM](http://arxiv.org/abs/2604.18655v2)
  > **TL;DR**: Proposes a hardware-aware framework for efficient on-device LLM inference with multi-LoRA support. Integrates LoRAs as runtime inputs, multi-stream decoding for stylistic variations, and DS2D for token acceleration. Achieves 4-6x improvements in memory and latency with INT4 quantization on mobile devices.
* `serving` `offloading` `scaling` [A comprehensive evaluation of spatial co-execution on GPUs using MPS and MIG technologies](http://arxiv.org/abs/2604.22430v1)
  > **TL;DR**: Evaluates spatial sharing technologies MPS and MIG on GPUs for co-execution of applications. Analyzes trade-offs between flexibility and isolation, provisioning resource allocation. MPS improves performance by 30% and reduces energy by 20% in favorable cases, but suffers 30% degradation under memory contention.
* `serving` `sparse` `kernel` [Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlation](http://arxiv.org/abs/2604.22312v1)
  > **TL;DR**: Addresses the latency bottleneck of exact Top-K in sparse-attention decoding during LLM serving. Proposes Guess-Verify-Refine (GVR), a data-aware algorithm exploiting temporal correlation across decode steps for efficient exact Top-K selection. Achieves 1.88x speedup in operator latency and up to 7.52% end-to-end throughput improvement.
* `networking` `training` [Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs](http://arxiv.org/abs/2604.22228v1)
  > **TL;DR**: Proposes multi-path GPU communication using CUDA Graphs in UCX to reduce overhead. Utilizes NVLink and PCIe paths concurrently with CUDA Graph optimization. Achieves 2.95x bandwidth gain vs. single-path UCX for GPU-GPU transfers up to 512MB.
* `networking` `offline` `multi-modal` [Chamelio: A Fast Shared Cloud Network Stack for Isolated Tenant-Defined Protocols](http://arxiv.org/abs/2604.22603v1)
  > **TL;DR**: Proposes Chamelio, a shared cloud network stack enabling tenant-defined protocols. Combines a bounded eBPF fast path with tenant slow paths and joint optimization for performance isolation. Achieves 9.2 Mreq/s comparable to TAS stack with <4% overhead over fixed stacks under adversarial load.

### 2026-04-24
* `kernel` [Leveraging SIMD for Accelerating Large-number Arithmetic](http://arxiv.org/abs/2604.21566v1)
  > **TL;DR**: Proposes DigitsOnTurbo (DoT), a SIMD-based method to accelerate large-number arithmetic by restructuring computations into data-parallel operations. Achieves up to 2.3x speedup for multiplication and 19.3% end-to-end throughput gain in scientific computing.
* `edge` `scaling` `serving` [Risk-Aware and Stable Edge Server Selection Under Network Latency SLOs](http://arxiv.org/abs/2604.21483v1)
  > **TL;DR**: Proposes a risk-aware edge server selection framework with stability control to meet network latency SLOs. Uses predictive mean/uncertainty for SLO violation risk and hysteresis to reduce switching. Reduces deadline-miss rate to 34% vs. baseline 39% and switching frequency by 88%.
* `serving` `edge` `RL` [A Task Decomposition and Planning Framework for Efficient LLM Inference in AI-Enabled WiFi-Offload Networks](http://arxiv.org/abs/2604.21399v1)
  > **TL;DR**: Addresses LLM inference offloading in WiFi networks with heterogeneous devices. Proposes an LLM-based task decomposition planner and collaborative scheduling strategy. Achieves 20% lower latency and 80% higher reward over baselines.
* `kernel` `hardware` `serving` [GraphLeap: Decoupling Graph Construction and Convolution for Vision GNN Acceleration on FPGA](http://arxiv.org/abs/2604.21290v1)
  > **TL;DR**: Introduces GraphLeap to decouple dynamic graph construction from feature update in Vision GNNs, enabling concurrent processing. Proposes an FPGA accelerator with layer-pipelined design and kNN-engine overlap. Achieves up to 95.7× CPU speedup and 8.5× GPU speedup for real-time inference.
* `training` `storage` `scaling` [Optimizing High-Throughput Distributed Data Pipelines for Reproducible Deep Learning at Scale](http://arxiv.org/abs/2604.21275v1)
  > **TL;DR**: Addresses data-loading bottlenecks in distributed GPU training for large-scale deep learning. Optimizes architecture with push-down transformations, local caching, and deterministic queues. Achieves 6x speedup, reducing training time from 22 to 3 hours, and boosts GPU utilization to over 60%.
* `RAG` `scaling` [A Cloud-Native Architecture for Human-in-Control LLM-Assisted OpenSearch in Investigative Settings](http://arxiv.org/abs/2604.21125v1)
  > **TL;DR**: Proposes a cloud-native architecture integrating LLMs into Human-in-Control workflows for translating natural-language queries to OpenSearch DSL. Designs hybrid lexical-semantic retrieval with BM25 and vector embeddings. Focuses on scalability and security under private-cloud deployments, using Enron dataset for feasibility validation.
* `hardware` `edge` `inference` [Enabling Mixed criticality applications for the Versal AI-Engines](http://arxiv.org/abs/2604.21124v1)
  > **TL;DR**: Proposes a dynamic task dispatching infrastructure for mixed criticality applications on AMD Versal AI-Engines. Enables runtime assignment of tasks with different criticality levels to AIE tiles, reducing idle time by 65.5% and maintaining low overhead (<0.002%).
* `serving` `networking` `offloading` [Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization](http://arxiv.org/abs/2604.21072v1)
  > **TL;DR**: Proposes BloomBee, a distributed LLM inference framework for internet-scale deployment, combining layer assignment, micro-batching, tensor offloading, and compression. Optimizes communication using dynamic programming, achieving 1.76x higher throughput and 43.2% lower latency in low-bandwidth networks.

### 2026-04-23
* `serving` `offloading` `sparse` [Stream-CQSA: Avoiding Out-of-Memory in Attention Computation via Flexible Workload Scheduling](http://arxiv.org/abs/2604.20819v1)
  > **TL;DR**: Develops a method to avoid OOM in exact attention computation for LLMs. Introduces Stream-CQSA, which decomposes attention into independent subsequences scheduled within memory limits. Achieves exact attention on billion-token sequences using a single GPU without approximation.
* `serving` `sparse` [FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving](http://arxiv.org/abs/2604.20503v1)
  > **TL;DR**: Addresses inefficiency in speculative decoding under dynamic LLM serving workloads. Introduces fine-grained phase management with dynamic token length adjustment, early pruning of rejected tokens, and overlapping draft/verification phases. Improves throughput by up to 53% and reduces latency by 1.92× over state-of-the-art.
* `MoE` `multi-modal` `quantization` [ReaLB: Real-Time Load Balancing for Multimodal MoE Inference](http://arxiv.org/abs/2604.19503v2)
  > **TL;DR**: Addresses load imbalance in multimodal MoE inference due to skewed expert workloads. Proposes ReaLB, a real-time load balancing method that dynamically adjusts expert computation precision per EP rank. Achieves 1.29x layer-level speedup with <1.2% accuracy loss.
* `scaling` `serving` `networking` [Predictive Autoscaling for Node.js on Kubernetes: Lower Latency, Right-Sized Capacity](http://arxiv.org/abs/2604.19705v2)
  > **TL;DR**: Proposes predictive autoscaling for Node.js on Kubernetes to maintain latency SLOs. Forecasts load using cluster-wide invariant metrics and a five-stage pipeline. Achieves median latency of 26ms vs. 154ms for KEDA and 522ms for HPA under steady ramp.
* `offloading` `disaggregation` `serving` [Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter](http://arxiv.org/abs/2604.15039v2)
  > **TL;DR**: Addresses cross-datacenter LLM serving challenges by proposing Prefill-as-a-Service (PrfaaS), which selectively offloads long-context prefill to remote clusters and uses bandwidth-aware scheduling. Achieves 54% higher throughput and 64% lower P90 TTFT than homogeneous baselines.
* `edge` `scaling` `offloading` [A Delta-Aware Orchestration Framework for Scalable Multi-Agent Edge Computing](http://arxiv.org/abs/2604.20129v1)
  > **TL;DR**: Addresses the synergistic collapse in multi-agent edge systems when scaling beyond 100 agents. Proposes DAOEF with differential neural caching, criticality-based pruning, and learned hardware affinity to reduce latency. Achieves 62% latency reduction and sub-linear scaling to 250 agents.
* `training` `RL` `networking` [UCCL-Zip: Lossless Compression Supercharged GPU Communication](http://arxiv.org/abs/2604.17172v2)
  > **TL;DR**: Addresses GPU communication bottlenecks in LLM training and inference with lossless compression integrated into communication primitives. Proposes UCCL-Zip for P2P and collective communications via split-send pipelines and fused NCCL execution. Achieves 47.5% faster RL weight synchronization and 10% lower vLLM inference latency.

### 2026-04-22
* `scaling` `edge` `serving` [Predictive Autoscaling for Node.js on Kubernetes: Lower Latency, Right-Sized Capacity](http://arxiv.org/abs/2604.19705v1)
  > **TL;DR**: Proposes a predictive autoscaling algorithm for Node.js serving on Kubernetes that forecasts load to proactively scale capacity. Uses a cluster-wide invariant metric and a five-stage pipeline to transform data for prediction. Keeps median latency at 26ms under ramp load, compared to 154ms (KEDA) and 522ms (HPA).
* `MoE` `training` `networking` [FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training](http://arxiv.org/abs/2604.19654v1)
  > **TL;DR**: Addresses load imbalance in distributed Mixture-of-Experts training by leveraging NVLink Copy Engines for nearly free intra-node token redistribution. Proposed FEPLB uses a Two-Phase Dispatch and lightweight CPU scheduler. Reduces token and GEMM stragglers by 51-70% and 50-68%, respectively, without added communication overhead.
* `MoE` `quantization` `serving` [ReaLB: Real-Time Load Balancing for Multimodal MoE Inference](http://arxiv.org/abs/2604.19503v1)
  > **TL;DR**: Proposes ReaLB, a real-time load balancing method for multimodal MoE inference by dynamically adjusting computation precision per expert using FP4, achieving 1.29x layer-level speedup with <1.2% accuracy loss.
* `edge` `offloading` `inference` [CROWDio: A Practical Mobile Crowd Computing Framework with Developer-Oriented Design, Adaptive Scheduling, and Fault Resilience](http://arxiv.org/abs/2604.19363v1)
* `training` `MoE` `networking` [UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training](http://arxiv.org/abs/2604.19241v1)
  > **TL;DR**: Addresses communication bottlenecks in expert-parallel training of MoE models. Proposes UniEP, unifying optimizations into fused MegaKernels with deterministic token ordering. Achieves 1.03-1.38× speedups over SOTA while maintaining numerical precision.
* `diffusion` `serving` `networking` [CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism](http://arxiv.org/abs/2604.14561v2)
  > **TL;DR**: Optimizes collective communication overhead in distributed diffusion transformer inference. Proposes tile-aware all-to-all decomposition, V-first scheduling for communication-computation overlap, and selective projection communication. Achieves up to 8.4x speedup on 1-8 node DiT models.
* `edge` `quantization` `LoRA` [Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM](http://arxiv.org/abs/2604.18655v1)
  > **TL;DR**: Proposes a hardware-aware framework for efficient on-device LLM inference supporting multiple LoRAs on mobile devices. Integrates application-specific LoRAs into a single frozen graph, multi-stream decoding, and Dynamic Self-Speculative Decoding. Achieves 4-6x improvements in memory and latency with INT4 quantization while maintaining accuracy.
* `kernel` `training` `MoE` [ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants](http://arxiv.org/abs/2604.18616v1)
  > **TL;DR**: Proposes Argus, an agentic GPU optimization framework using data-flow invariants for compiling high-performance LLM kernels. Integrates a DSL with tag functions and assertions for compile-time verification, enabling dense feedback. Achieves 99-104% of hand-optimized throughput for GEMM, attention, and MoE kernels.
* `serving` `offloading` `networking` [HadAgent: Harness-Aware Decentralized Agentic AI Serving with Proof-of-Inference Blockchain Consensus](http://arxiv.org/abs/2604.18614v1)
  > **TL;DR**: Introduces HadAgent, a decentralized AI serving system using Proof-of-Inference consensus to replace hash-based mining. Features a two-tier node architecture with trusted nodes for optimistic execution and harness monitoring for security. Achieves 100% tamper detection, sub-millisecond validation latency, and efficient trust convergence.

### 2026-04-21
* `serving` `offloading` `kernel` [HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing](http://arxiv.org/abs/2604.18529v1)
  > **TL;DR**: Proposes HybridGen, a CPU-GPU hybrid attention framework for efficient long-context LLM inference using tiered memory. Introduces attention logit parallelism, feedback-driven scheduler, and semantic-aware KV cache mapping. Achieves 1.41x--3.2x higher throughput compared to six KV cache management baselines.
* `edge` `serving` [AgenTEE: Confidential LLM Agent Execution on Edge Devices](http://arxiv.org/abs/2604.18231v1)
  > **TL;DR**: Aims to securely deploy LLM agents on edge devices. Introduces AgenTEE using confidential VMs on Arm CCA for runtime isolation of agent pipelines. Achieves near-native performance with <5.15% overhead compared to multi-process deployments.

### 2026-04-21
* `kernel` `training` [Matrix-Free 3D SIMP Topology Optimization with Fused Gather-GEMM-Scatter Kernels](http://arxiv.org/abs/2604.18020v1)
  > **TL;DR**: Proposes a fused gather-GEMM-scatter CUDA kernel to eliminate DRAM traffic in matrix-free 3D topology optimization. The fused implementation achieves up to 7.3x end-to-end speedup and 4.9x lower energy consumption on large-scale simulations compared to baseline.
* `kernel` `serving` `training` [GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion](http://arxiv.org/abs/2604.17861v1)
  > **TL;DR**: Addresses kernel launch overhead in small tensor operations common in inference and training. Proposes GPUOS, a GPU runtime JIT system using persistent kernels, runtime operator injection, and PyTorch integration. Achieves up to 15.3x speedup over standard PyTorch on micro-batched inference and attention workloads.
* `kernel` `sparse` `offline` [AsyncSparse: Accelerating Sparse Matrix-Matrix Multiplication on Asynchronous GPU Architectures](http://arxiv.org/abs/2604.17834v1)
  > **TL;DR**: Accelerates Sparse Matrix-Matrix Multiplication (SpMM) using asynchronous GPU features. Proposes two kernels: BCSR for structured sparsity and WCSR for irregular sparsity, leveraging Tensor Memory Accelerator and warp specialization. Achieves 1.47-6.24x speedup over prior SpMM kernels and 2.66x speedup in Qwen2.5-7B prefill.
* `serving` `sparse` [DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models](http://arxiv.org/abs/2604.17709v1)
  > **TL;DR**: Addresses poor parallel inference performance in decomposed LLMs. Introduces DeInfer, a high-performance inference system with multiple compatible optimizations. Achieves superior results in facilitating parallel inference.
* `serving` `training` `scaling` [Towards Energy Efficient Co-Scheduling in HPC](http://arxiv.org/abs/2604.17640v1)
  > **TL;DR**: Optimizes GPU allocation and co-scheduling in HPC clusters for energy efficiency. Proposes EcoSched, an online scheduler with runtime profiling and NUMA-aware placement. Achieves up to 14.8% energy savings and 40.4% EDP reduction.
* `training` `scaling` [EcoShift: Performance-Aware Power Management for Power-Constrained Heterogeneous Systems](http://arxiv.org/abs/2604.17635v1)
  > **TL;DR**: Addresses inefficient power usage in power-constrained heterogeneous HPC systems running CPU-GPU workloads. Proposes EcoShift, combining online performance prediction with dynamic programming to redistribute reclaimed power. Achieves up to 6% average performance improvement over state-of-the-art policies.
* `serving` `quantization` [SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving](http://arxiv.org/abs/2604.17627v1)
  > **TL;DR**: Proposes SLO-Guard, a crash-aware autotuner for LLM serving that uses feasible-first exploration and warm-started exploitation. It includes GPU-aware KV-cache memory guard and configuration repair. Achieves 4.4x tighter latency standard deviation and higher budget consistency compared to random search.
* `offline` `agentic` `storage` [AgileLog: A Forkable Shared Log for Agents on Data Streams](http://arxiv.org/abs/2604.14590v2)
  > **TL;DR**: Proposes AgileLog, a forkable shared log for AI agents on data streams to avoid performance interference. Designs Bolt with novel forking primitives for cheap forks and isolation. Achieves up to 3.5× throughput improvement over baseline.
* `training` `scaling` `kernel` [Flint: Compiler Enabled Cluster-Free Design Space Exploration for Distributed ML](http://arxiv.org/abs/2604.17550v1)
  > **TL;DR**: Addresses challenge in DNN systems design space exploration. Proposes Flint, leveraging compiler IR for workload representation across arbitrary cluster sizes. Enables exploration without execution, validated via case study.
* `edge` `serving` `networking` [Active Inference-Based Adaptive Routing for Heterogeneous Edge AI Services](http://arxiv.org/abs/2604.17373v1)
  > **TL;DR**: Proposes AIF-Router, an Active Inference-based routing framework for edge AI services. Uses Bayesian state inference and expected free energy minimization to autonomously balance latency, throughput, and resource utilization. Achieves stable online learning despite unreliable edge environments with dynamic workloads.
* `agentic` `serving` `offloading` [Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling](http://arxiv.org/abs/2604.17353v1)
  > **TL;DR**: Proposes Hive, a multi-agent infrastructure with Logits Cache to reuse intermediate computations and Agent-Aware Scheduling for resource allocation. Achieves 1.11×-1.76× speedup for re-sampling and reduces KV cache hotspot miss rate by 33%-51%.
* `serving` `training` `scaling` [Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda](http://arxiv.org/abs/2604.17227v1)
  > **TL;DR**: Explores cloud-native and distributed systems to enhance LLM scalability and efficiency in training and inference. Investigates techniques like autoscaling, microservices, and hybrid cloud-edge solutions for resource optimization. Highlights pathways for improved deployment with serverless inference and quantum computing support.
* `training` `networking` `scaling` [CCCL: In-GPU Compression-Coupled Collective Communication](http://arxiv.org/abs/2604.17172v1)
  > **TL;DR**: Addresses collective communication overhead in LLM training. Proposes CCCL, a compression-based library fused with NCCL to minimize memory access and eliminate data coalescing. Achieves up to 10.1% throughput improvement in vLLM disaggregation workloads.
* `serving` `scaling` `networking` [HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads](http://arxiv.org/abs/2604.17111v1)
  > **TL;DR**: Addresses resource contention in concurrent LLM agent workloads sharing a rate-limited API. Proposes HiveMind, a transparent HTTP proxy with five OS-inspired scheduling primitives like admission control and backpressure. Reduces failures from 72-100% to 0-18% and eliminates 48-100% of wasted compute.
* `storage` `quantization` `compression` [TensorHub: Rethinking AI Model Hub with Tensor-Centric Compression](http://arxiv.org/abs/2604.17104v1)
  > **TL;DR**: Proposes TensorHub to address storage challenges in AI model hubs. Uses tensor-level fingerprinting, clustering, and deduplication to reduce redundancy. Achieves substantial storage savings in experiments with minimal overhead and preserved performance.
* `sparse` `kernel` `offloading` [HieraSparse: Hierarchical Semi-Structured Sparse KV Attention](http://arxiv.org/abs/2604.16864v1)
  > **TL;DR**: Addresses computational and memory overhead of long-context LLMs via HieraSparse, a hierarchical semi-structured sparse KV attention framework with GPU acceleration. Achieves up to 1.85x prefill attention speedup and 4.57x decode speedup at equivalent sparsity.
* `serving` `kernel` `hardware` [FliX: Flipped-Indexing for Scalable GPU Queries and Updates](http://arxiv.org/abs/2604.16725v1)
  > **TL;DR**: Introduces FliX, a flipped indexing strategy for GPU-based concurrent data structures to improve query and update efficiency. Eliminates the index layer by assigning compute units to data buckets and performs batched binary searches. Achieves 6.5x reduced query latency versus GPU B-trees and 4x higher throughput per memory footprint.
* `training` `sparse` `scaling` [Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs](http://arxiv.org/abs/2604.16715v1)
  > **TL;DR**: Proposes a distributed training framework for graph transformers that optimizes parallelization strategies based on graph structure and hardware. Uses distributed sparse operations to accelerate attention by 3.8x and reduce memory by 78%. Achieves up to 6x speedup on 8 GPUs.
* `agentic` `serving` `offloading` [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](http://arxiv.org/abs/2604.16682v1)
  > **TL;DR**: Addresses power efficiency degradation in agentic AI serving due to long-lived context. Proposes KAIROS, a context-aware system that manages GPU frequency, concurrency, and request placement per agent. Achieves average 27% power reduction while meeting performance targets.
* `kernel` `hardware` [Forge-UGC: FX optimization and register-graph engine for universal graph compiler](http://arxiv.org/abs/2604.16498v1)
  > **TL;DR**: Presents Forge-UGC, a universal graph compiler to optimize transformer deployment on NPUs. Includes four phases: graph capture, optimization, IR lowering, and backend scheduling with buffer allocation. Reduces inference latency by 18.2-35.7% and energy per inference by 30.2-40.9% on models up to 8B parameters.
* `kernel` `inference` `training` [GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion](http://arxiv.org/abs/2604.17861v1)
  > **TL;DR**: Addresses kernel launch overhead in small-operation DL workloads. Introduces GPUOS, a JIT runtime using a persistent kernel with runtime operator injection and live updates. Achieves up to 15.3× speedup over PyTorch in micro-batched inference and attention.
* `kernel` `agentic` `RL` [Governed MCP: Kernel-Level Tool Governance for AI Agents via Logit-Based Safety Primitives](http://arxiv.org/abs/2604.16870v1)
  > **TL;DR**: Proposes Governed MCP, a kernel-level safety gateway for AI agent tool calls using a logit-based semantic check pipeline. Implements in Anima OS with ProbeLogits adding 65 ms overhead per call. Ablation shows ProbeLogits essential (F1 drops from 0.773 to 0.327 without it).

### 2026-04-20
* `training` `quantization` [Training Time Prediction for Mixed Precision-based Distributed Training](http://arxiv.org/abs/2604.16145v1)
  > **TL;DR**: Addresses inaccuracies in distributed training time prediction due to mixed precision. Proposes a precision-aware predictor for robust time estimation under diverse quantization settings. Attains 9.8% mean absolute percentage error, reducing errors by over 14x versus baselines.
* `training` `MoE` `scaling` [Breaking the Training Barrier of Billion-Parameter Universal Machine Learning Interatomic Potentials](http://arxiv.org/abs/2604.15821v1)
  > **TL;DR**: Addresses the challenge of training billion-parameter universal ML interatomic potentials with second derivatives. Introduces Janus, a hardware-aware distributed training framework with parallelization techniques. Achieves peak performance of 1.2 EFLOPS and over 90% parallel efficiency, reducing training time from weeks to hours.
* `edge` `RL` `networking` [CroSatFL: Energy-Efficient Federated Learning with Cross-Aggregation for Satellite Edge Computing](http://arxiv.org/abs/2604.15779v1)
  > **TL;DR**: Proposes CroSatFL, a federated learning framework for satellite edge computing to reduce energy consumption. Uses energy-aware cluster formation, straggler mitigation, and cross-aggregation to shift communication from ground stations to inter-satellite links. Reduces ground station communication count by over 100× and transmission energy by 6×.
* `kernel` `training` `scaling` [cuNNQS-SCI: A Fully GPU-Accelerated Framework for High-Performance Configuration Interaction Selection withNeural Network QQantum States](http://arxiv.org/abs/2604.15768v1)
  > **TL;DR**: Proposes cuNNQS-SCI, a fully GPU-accelerated framework for neural network quantum states to overcome CPU-GPU bottlenecks. Implements distributed de-duplication, specialized CUDA kernels, and GPU memory optimization. Achieves 2.32x speedup on 64 GPUs with 90% parallel efficiency.
* `serving` `RAG` [Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving](http://arxiv.org/abs/2604.15732v1)
  > **TL;DR**: Addresses how accuracy-driven retries under long-context LLM serving inflate latency. Proposes Lightweight Accuracy-Aware Routing (LAAR) to reduce Time-to-Correct-Answer (TTCA) via capability-based routing. Achieves up to 2.3× lower TTCA compared to standard routing.
* `training` `scaling` `storage` [DataCenterGym: A Physics-Grounded Simulator for Multi-Objective Data Center Scheduling](http://arxiv.org/abs/2604.15594v1)
  > **TL;DR**: Proposes DataCenterGym, a physics-grounded simulator for multi-objective scheduling in geo-distributed data centers. Develops Hierarchical MPC to optimize job placement considering thermal/power dynamics. Achieves improved scheduling performance over baselines under thermal-sensitive workloads.

### 2026-04-17
* `serving` `scaling` `agentic` [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](http://arxiv.org/abs/2604.15186v1)
  > **TL;DR**: Proposes Scepsy, an agentic serving system that schedules multi-LLM workflows by profiling and predicting execution shares. Constructs an Aggregate LLM Pipeline for latency/throughput prediction and uses a heuristic allocator. Achieves up to 2.4x higher throughput and 27x lower latency.
* `hardware` `networking` `offloading` [SCENIC: Stream Computation-Enhanced SmartNIC](http://arxiv.org/abs/2604.15128v1)
  > **TL;DR**: Develops SCENIC, a programmable SmartNIC with stream computation units, to offload data processing like TCP/IP and RDMA stacks. Integrates with OS for transparent application use, achieving 200G bandwidth and enabling GPU-direct hash partitioning, reducing CPU overhead.
* `disaggregation` `serving` `offloading` [Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter](http://arxiv.org/abs/2604.15039v1)
  > **TL;DR**: Proposes Prefill-as-a-Service (PrfaaS), a cross-datacenter LLM serving architecture that offloads long-context prefill to dedicated clusters and transfers KVCache over Ethernet. Combines hybrid-attention models with selective offloading, bandwidth-aware scheduling, and cache-aware placement. Achieves 54% higher throughput versus homogeneous baseline.
* `serving` `offloading` [Serving Chain-structured Jobs with Large Memory Footprints with Application to Large Foundation Model Serving](http://arxiv.org/abs/2604.14993v1)
  > **TL;DR**: Addresses GPU memory constraints in distributed LLM serving by formulating server chain composition via block placement and cache allocation. Proposes scalable algorithms for pipeline parallelism with load balancing. Achieves significant response time reduction compared to state-of-the-art.
* `training` `hardware` `networking` [SAKURAONE: An Open Ethernet-Based AI HPC System and Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment](http://arxiv.org/abs/2604.13600v2)
  > **TL;DR**: Introduces SAKURAONE, an Ethernet-based HPC cluster optimized for LLM training, featuring open networking with 800 GbE/SONiC. Demonstrates real-world workload dynamics, showing shift from large-scale training to mid-scale refinement jobs. Achieves 33.95 PFLOP/s HPL performance.
* `MoE` `serving` `kernel` [ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving](http://arxiv.org/abs/2604.14626v1)
  > **TL;DR**: Proposes ELMoE-3D, a hardware-software co-design for efficient on-premises MoE serving using elastic self-speculative decoding accelerated by hybrid-bonding. Leverages elasticity in experts and bits, with bit-sliced architecture for redundancy exploitation. Achieves 6.6× speedup and 4.4× energy efficiency gain over naive serving.
* `diffusion` `serving` `networking` [CoCoDiff: Optimizing Collective Communications for Distributed Diffusion Transformer Inference Under Ulysses Sequence Parallelism](http://arxiv.org/abs/2604.14561v1)
  > **TL;DR**: Optimizes collective communication overhead in distributed diffusion transformer inference under Ulysses parallelism. Proposes tile-aware parallel all-to-all, V-first scheduling, and selective communication to overlap communication with computation. Achieves up to 8.4x speedup on Aurora supercomputer.
* `training` `sparse` `kernel` [Incidence Constraints in Hypergraph Partitioning on GPU](http://arxiv.org/abs/2604.14411v1)
  > **TL;DR**: Accelerates hypergraph partitioning on GPUs for constrained problems by exploiting set sparsity and incidence matrix materialization. Achieves up to 940x speedup and 2-26% better connectivity versus sequential implementations.

### 2026-04-16
* `training` `storage` `networking` [An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience](http://arxiv.org/abs/2604.12973v2)
  > **TL;DR**: Addresses engineering challenges in training large-scale multilingual LLMs on HPC systems. Details transforming the Alps supercomputer into an ML platform, overcoming storage bottlenecks and network instabilities for 70B model training. Achieves deployment on one of Europe's largest open-science systems.
* `offloading` `storage` `disaggregation` [OffloadFS: Leveraging Disaggregated Storage for Computation Offloading](http://arxiv.org/abs/2604.13743v1)
  > **TL;DR**: Proposes OffloadFS, a user-level file system for offloading IO-intensive tasks to disaggregated storage nodes for near-data processing. It optimizes cache management and reduces I/O interference. Improves RocksDB and ML pre-processing performance by up to 3.36x and 1.85x over OCFS2.
* `training` `networking` `storage` [SAKURAONE: An Open Ethernet-Based AI HPC System and Its Observed Workload Dynamics in a Single-Tenant LLM Development Environment](http://arxiv.org/abs/2604.13600v1)
  > **TL;DR**: Proposes SAKURAONE, an Ethernet-based HPC cluster optimized for LLM training with open networking (800 GbE, SONiC). System achieves 33.95 PFLOP/s HPL performance, uses 800 GPUs and 2PB Lustre storage. Observes workload shift from large-scale training to mid-scale refinement jobs during project progression.
* `edge` `offloading` `serving` [Self-adaptive Multi-Access Edge Architectures: A Robotics Case](http://arxiv.org/abs/2604.13542v1)
  > **TL;DR**: Proposes a self-adaptive edge computing system for efficient neural network processing in robotics. Uses Kubernetes-orchestrated offloading to heterogeneous units with MAPE-K supervisor for scaling and offloading decisions. Achieves improved service quality in response time and power consumption over traditional setups.
* `inference` `offloading` `kernel` [A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs](http://arxiv.org/abs/2604.09752v2)
  > **TL;DR**: Addresses memory-bound challenges and kernel synchronization overhead in LLM inference on NPUs. Proposes an adaptive orchestration framework that overcomes limitations of static model deployment and speculative decoding. Achieves reduced latency by optimizing model deployment and computation graphs.
* `sparse` `kernel` `quantization` [PackSELL: A Sparse Matrix Format for Precision-Agnostic High-Performance SpMV](http://arxiv.org/abs/2604.13433v1)
  > **TL;DR**: Proposes PackSELL, a sparse matrix format for efficient SpMV on GPUs, featuring delta encoding and packed bit allocation for reduced memory and flexible precision. Achieves up to 1.63× speedup over cuSPARSE with FP16 and exceeds FP16 performance while maintaining FP32 accuracy.
* `serving` `kernel` `training` [Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel](http://arxiv.org/abs/2604.13327v1)
  > **TL;DR**: Proposes Event Tensor, a compiler abstraction for dynamic megakernels to handle dynamic shapes and data-dependent computation in LLM inference. ETC compiler applies static/dynamic scheduling to generate persistent kernels. Achieves state-of-the-art LLM serving latency with reduced warmup overhead.

### 2026-04-15
* `training` `scaling` `storage` [An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience](http://arxiv.org/abs/2604.12973v1)
  > **TL;DR**: Details engineering challenges in training the 70B parameter Apertus multilingual LLM on the Alps supercomputer. Overcame storage bottlenecks and interconnect stability issues to transform HPC infrastructure into a resilient ML platform. Achieved academia's first large-scale LLM training on European supercomputing infrastructure.
* `kernel` `hardware` `serving` [Leveraging Mathematical Reasoning of LLMs for Efficient GPU Thread Mapping](http://arxiv.org/abs/2604.10387v2)
  > **TL;DR**: Automates GPU thread mapping for non-box domains using LLM symbolic reasoning. Proposes in-context learning to derive O(1)/O(log N) mapping equations, eliminating block waste. Achieves up to 4833x speedup and 2890x energy reduction for GPU workloads.
* `scaling` `offloading` `training` [Intelligent resource prediction for SAP HANA continuous integration build workloads](http://arxiv.org/abs/2604.12673v1)
  > **TL;DR**: Proposes an ML-based approach to predict resource needs for CI build workloads to reduce memory over-allocation. Uses a LightGBM-XGBoost quantile regression ensemble for memory prediction integrated via a microservice layer. Achieves 36GB average memory savings per build with under-allocation rates below 0.3%.
* `training` `serving` `inference` [Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems](http://arxiv.org/abs/2604.12599v1)
  > **TL;DR**: Investigates architectural challenges for the full AI lifecycle (pre-training, fine-tuning, inference) on HPC systems. Proposes a hybrid cloud-native platform combining HPC and virtualized infrastructure orchestrated by Kubernetes. Enhances user productivity by bridging HPC batch processing with service-oriented workflows.
* `RL` `quantization` `edge` [Three Birds, One Stone: Solving the Communication-Memory-Privacy Trilemma in LLM Fine-tuning Over Wireless Networks with Zeroth-Order Optimization](http://arxiv.org/abs/2604.12401v1)
  > **TL;DR**: Addresses communication, memory, and privacy trilemma in federated LLM fine-tuning over wireless networks. Proposes pAirZero framework combining zeroth-order optimization with over-the-air computation, reducing peak memory cost to 25% on OPT-125M and communication loads orders of magnitude lower.
* `serving` `offline` `RAG` [Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads](http://arxiv.org/abs/2604.12301v1)
  > **TL;DR**: Proposes seven tactics to reduce cloud LLM token usage using a local model as a triage layer. Implements them in a shim and evaluates on coding workloads. Tactic combinations achieve up to 79% token savings, with workload-dependent optimal subsets.
* `serving` `offloading` `scaling` [PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving](http://arxiv.org/abs/2604.12171v1)
  > **TL;DR**: Addresses live adaptation of pipeline parallelism for LLM serving under dynamic environments. Proposes PipeLive with a redesigned KV cache layout, PageAttention extension, and incremental KV patching. Achieves under 10ms reconfiguration overhead and up to 54.7% TTFT improvement.
* `serving` `offloading` `agentic` [Aethon: A Reference-Based Replication Primitive for Constant-Time Instantiation of Stateful AI Agents](http://arxiv.org/abs/2604.12129v1)
  > **TL;DR**: Introduces Aethon, a reference-based replication primitive for efficient instantiation of stateful AI agents. Uses compositional views with layered memory and copy-on-write to avoid full materialization. Achieves near-constant-time instantiation with over 98% latency reduction compared to traditional approaches.
* `training` `scaling` [Evaluating Cross-Architecture Performance Modeling of Distributed ML Workloads Using StableHLO](http://arxiv.org/abs/2604.12090v1)
  > **TL;DR**: Investigates StableHLO as a unified workload representation for cross-architecture performance modeling of distributed ML workloads. Proposes a methodology mapping StableHLO to analytical, profiling-based, and simulator-driven predictors. Prediction errors within practical bounds for design exploration, enabling systematic comparison across GPUs and TPUs.
* `training` `networking` `sparse` [ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism](http://arxiv.org/abs/2604.11947v1)
  > **TL;DR**: Proposes Residual Bottleneck Models (ResBM) for low-bandwidth pipeline parallelism. Introduces residual encoder-decoder bottlenecks across pipeline stages to compress activations 128x. Achieves near-equivalent convergence with minimal overhead in decentralized training environments.
* `training` `offloading` `storage` [TierBPF: Page Migration Admission Control for Tiered Memory via eBPF](http://arxiv.org/abs/2604.12300v1)
  > **TL;DR**: Investigates page migration admission control for tiered memory to optimize application performance. Proposes TierBPF, an eBPF-based mechanism that considers page size and hardware topology for migration decisions. Achieves up to 17.7% geomean throughput gains across 17 workloads.
* `offloading` `storage` `kernel` [Hybrid Adaptive Tuning for Tiered Memory Systems](http://arxiv.org/abs/2604.12165v1)
  > **TL;DR**: Proposes PTMT, a hybrid offline-online framework using reinforcement learning for automated parameter tuning in memory tiering systems. Achieves 14-30% performance improvement over default configurations across four memory tiering solutions.
* `kernel` `inference` `offloading` [ProbeLogits: Kernel-Level LLM Inference Primitives for AI-Native Operating Systems](http://arxiv.org/abs/2604.11943v1)
  > **TL;DR**: Proposes ProbeLogits, a kernel-level operation for single-pass logit reading to govern LLM agent actions. Implements in Anima OS with adjustable calibration strength for policy enforcement. Achieves 65ms per classification on a 7B model with up to 97.3% accuracy on custom benchmarks.

### 2026-04-14
* `disaggregation` `serving` `scaling` [Nanvix: A Multikernel OS Design for High-Density Serverless Deployments](http://arxiv.org/abs/2604.11669v1)
  > **TL;DR**: Addresses high deployment density with strong isolation for serverless platforms. Proposes Nanvix, a multikernel OS that disaggregates ephemeral and persistent state using user VMs and shared system VMs per tenant. Achieves 20-100x fewer host servers for production traces.
* `hardware` `kernel` [CUTEv2: Unified and Configurable Matrix Extension for Diverse CPU Architectures with Minimal Design Overhead](http://arxiv.org/abs/2604.11615v1)
  > **TL;DR**: Proposes a unified CPU matrix extension architecture decoupled from the CPU pipeline for efficient AI workloads. Achieves over 90% GEMM utilization, 1.57-2.31x speedups on models like BERT and Llama3, and 30% gains from overlapped execution in 0.53mm² area.

### 2026-04-14
* `scaling` `training` `RL` [NimbusGuard: A Novel Framework for Proactive Kubernetes Autoscaling Using Deep Q-Networks](http://arxiv.org/abs/2604.11017v1)
  > **TL;DR**: Proposes NimbusGuard, a Kubernetes autoscaling framework using deep Q-networks and LSTM for workload forecasting. Achieves better performance and cost efficiency compared to traditional reactive autoscalers like HPA and KEDA.
* `serving` `multi-modal` `scaling` [RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving](http://arxiv.org/abs/2604.10907v1)
  > **TL;DR**: Proposes RouterWise for joint resource allocation and routing in multi-model LLM serving to maximize output quality under latency SLOs. Uses system profiling and dual-price formulation to optimize deployment setups and routing fractions. Achieves up to 87% higher output quality by co-optimizing resource allocation.
* `serving` `training` `offline` [Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers](http://arxiv.org/abs/2604.10769v1)
  > **TL;DR**: Explores how batch and inference workload composition affects AI data center power variability and ramping. Uses a trace-calibrated framework simulating workload arrivals, queueing, scheduling, and GPU power dynamics. Reveals U-shaped variability and hump-shaped ramping trends as inference share rises, impacting grid stability.
* `serving` `networking` `scaling` [Bipartite matching under communication constraints](http://arxiv.org/abs/2604.10744v1)
  > **TL;DR**: Investigates scheduling in data centers with local information constraints. Proposes probabilistic bipartite matching algorithms using degree-biased sampling and random thinning. Simulations show significant stability region extension in packet-level tests with production traffic.
* `storage` `networking` [CIR: Lightweight Container Image for Cross-Platform Deployment](http://arxiv.org/abs/2604.10411v1)
  > **TL;DR**: Proposes CIR, a lightweight container intermediate representation format that avoids platform-specific builds by deferring dependency assembly to deployment. Uses lazy-builders for on-demand dependency resolution, reducing image size by 95% and deployment time by 40-60%.
* `networking` `serving` [Verifying In-Network Computing Systems for Design Risks](http://arxiv.org/abs/2604.10186v1)
  > **TL;DR**: Proposes INCGuard for verifying correctness in in-network computing systems. Uses model checking with state transition representations and network exception simulations. Identifies risks in 7 systems within seconds with 67.2% code reduction for specs.
* `disaggregation` `serving` `kernel` [Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation](http://arxiv.org/abs/2604.10180v1)
  > **TL;DR**: Proposes Tessera, a kernel-granularity disaggregation system for heterogeneous GPU inference. It uses offline dependency extraction from PTX and online pipelined execution with workload-aware scheduling. Achieves up to 2.3x higher throughput and 1.6x better cost efficiency versus prior methods on diverse models and GPUs.
* `inference` `offloading` `kernel` [A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs](http://arxiv.org/abs/2604.09752v1)
  > **TL;DR**: Addresses memory-bound challenges in LLM decoding on NPUs by proposing Adaptive Inference Orchestration (A-IO). Dynamically adapts model configurations and optimizes kernel synchronization to mitigate bottlenecks. Achieves up to 2.6× higher throughput compared to static deployments.
* `serving` `sparse` `quantization` [SMART: When is it Actually Worth Expanding a Speculative Tree?](http://arxiv.org/abs/2604.09731v1)
  > **TL;DR**: Addresses the efficiency paradox in tree-based speculative decoding. Proposes SMART, a hardware-aware framework that optimizes tree expansion for end-to-end speedup using marginal benefit–cost analysis. Achieves up to 20.0% average speedup for MLLMs and 15.4% for LLMs without performance loss.
* `serving` `disaggregation` `edge` [ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving](http://arxiv.org/abs/2604.09722v1)
  > **TL;DR**: Proposes ConfigSpec, a profiling-based framework for optimal configuration selection in distributed edge-cloud speculative LLM serving. It models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost, and energy efficiency across draft models, quantization, and K lengths. Analysis shows conflicting optima: K=2 minimizes cost and energy, while device-dependent K (2-10) maximizes goodput.
* `serving` `agentic` `offline` [Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation](http://arxiv.org/abs/2604.09718v1)
  > **TL;DR**: Proposes a compile-and-execute method to reduce LLM inference costs in web automation agents. Uses one-shot LLM compilation to generate deterministic JSON blueprints, avoiding continuous inference loops. Reduces workflow cost from $150 to under $0.10 for 500 iterations, achieving O(1) amortized inference scaling.
* `agentic` `offloading` `serving` [ClawVM: Harness-Managed Virtual Memory for Stateful Tool-Using LLM Agents](http://arxiv.org/abs/2604.10352v1)
  > **TL;DR**: Proposes ClawVM, a virtual memory layer for stateful tool-using LLM agents to manage state as typed pages with multi-resolution representations and validated writeback. Eliminates policy-controllable faults when minimum-fidelity set fits token budget, adding <50 μs overhead per turn.

### 2026-04-13
* `training` `scaling` `quantization` [Sustaining Exascale Performance: Lessons from HPL and HPL-MxP on Aurora](http://arxiv.org/abs/2604.09517v1)
  > **TL;DR**: Explores sustaining exascale performance for HPL and mixed-precision HPL-MxP on Aurora. Proposes locality-aware mapping, CPU-GPU pipelining, and mixed-precision orchestration with hybrid resilience. Achieves 1.01 EF/s FP64 and 11.64 EF/s HPL-MxP (11.5× speedup).
* `training` `RL` `offloading` [TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training](http://arxiv.org/abs/2604.09107v1)
  > **TL;DR**: Proposes TensorHub, a weight transfer system for LLM RL training, using Reference-Oriented Storage that avoids physical copies and leverages GPU-resident weights. Reduces GPU stall time by up to 6.7x and accelerates weight updates by 4.8x.
* `edge` `quantization` `kernel` [EdgeFlow: Fast Cold Starts for LLMs on Mobile Devices](http://arxiv.org/abs/2604.09083v1)
  > **TL;DR**: Addresses high cold-start latency in mobile LLM inference due to wasted flash bandwidth. Proposes EdgeFlow with NPU-aware adaptive quantization, SIMD-friendly packing, and fine-grained CPU-NPU pipelining. Reduces cold-start latency by up to 4.07x compared to state-of-the-art frameworks.
* `training` `offline` `LoRA` [ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads](http://arxiv.org/abs/2604.05426v2)
  > **TL;DR**: Improves efficiency of hyperparameter tuning for multiple LoRA jobs. Proposes early termination of weak configurations, fused GEMM with rank-local parallelism, and intra-/inter-task scheduling. Achieves up to 13.8× speedup over SOTA.
* `serving` `offline` `hardware` [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](http://arxiv.org/abs/2604.09048v1)
  > **TL;DR**: Analyzes energy-efficient LLM inference on heterogeneous GPUs. Proposes Watt Counts, an open-access energy consumption dataset and benchmark for 50 LLMs across 10 GPUs. Achieves up to 70% energy reduction in server scenarios and up to 20% in batch scenarios.
* `training` `hardware` `quantization` [Fine-Grained Power and Energy Attribution on AMD GPU/APU-Based Exascale Nodes](http://arxiv.org/abs/2604.06056v2)
  > **TL;DR**: Proposes a fine-grained power and energy attribution methodology for AMD GPU/APU systems to optimize exascale workloads. Uses reconstructored power estimation from cumulative energy counters with sensor validation. Achieves 31-79% node energy reduction via mixed-precision training.
* `storage` `RAG` [Decoupling Vector Data and Index Storage for Space Efficiency](http://arxiv.org/abs/2604.09173v1)
  > **TL;DR**: Proposes DecoupleVS, a framework decoupling vector data and index metadata storage to reduce overhead in ANNS systems. Implements specialized optimizations for compression, data layouts, queries, and updates. Achieves up to 58.7% storage space reduction while maintaining competitive search performance.
* `edge` `quantization` `kernel` [EdgeFlow: Fast Cold Starts for LLMs on Mobile Devices](http://arxiv.org/abs/2604.09083v1)
  > **TL;DR**: Addresses high cold-start latency for LLM inference on mobile devices. Proposes EdgeFlow with adaptive quantization, SIMD-friendly packing, and fine-grained CPU-NPU pipelining. Reduces cold-start latency by up to 4.07x compared to llama.cpp, MNN, and llm.npu.

### 2026-04-10
* `offloading` `serving` [Taming GPU Underutilization via Static Partitioning and Fine-grained CPU Offloading](http://arxiv.org/abs/2604.08451v1)
  > **TL;DR**: Addresses GPU underutilization due to imbalanced resource demands in scientific and AI workloads. Proposes memory-offloading via Nvlink-C2C to complement static GPU partitioning (MIG), reducing interference. Achieves up to 40% higher throughput and 25% energy savings compared to MIG alone.
* `networking` `scaling` `RL` [NL-CPS: Reinforcement Learning-Based Kubernetes Control Plane Placement in Multi-Region Clusters](http://arxiv.org/abs/2604.08434v1)
  > **TL;DR**: Proposes a reinforcement learning framework for Kubernetes control-plane placement in multi-region clusters. Uses neural contextual bandits to learn optimal placement policies based on infrastructure characteristics. Achieves substantial performance improvements over baseline approaches in geographically distributed experiments.
* `serving` `hardware` `networking` [DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators](http://arxiv.org/abs/2604.04750v2)
  > **TL;DR**: Proposes DeepStack, a performance modeling tool for system-hardware co-design of distributed 3D-stacked accelerators for LLM inference. Incorporates fine-grained 3D memory modeling and parallelization strategies. Achieves 100,000x faster runtime versus simulators and 9.5x higher throughput.
* `diffusion` `serving` `scaling` [LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflows](http://arxiv.org/abs/2604.08123v1)
  > **TL;DR**: Proposes LegoDiffusion for decomposing text-to-image diffusion workflows into independently managed model nodes. Enables per-model scaling, sharing, and adaptive parallelism. Achieves up to 3x higher sustained request rates and 8x better burst tolerance compared to monolithic systems.
* `serving` `multi-modal` `RAG` [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](http://arxiv.org/abs/2604.06036v3)
  > **TL;DR**: Proposes CodecSight, a video streaming analytics system that leverages video codec metadata to optimize VLM inference. The system performs codec-guided patch pruning before ViT encoding and selective KV cache refresh during LLM prefilling, reducing GPU compute by up to 87% with minimal accuracy drop.
* `agentic` `RL` `storage` [LogAct: Enabling Agentic Reliability via Shared Logs](http://arxiv.org/abs/2604.07988v1)
  > **TL;DR**: Proposes LogAct, a shared log abstraction for reliable LLM agents, enabling action visibility, voter-based pre-execution blocking, and self-recovery via execution history analysis. Achieves efficient recovery, self-debugging, and stops all unwanted actions with only 3% drop in benign utility.
* `edge` `RL` `serving` [Administrative Decentralization in Edge-Cloud Multi-Agent for Mobile Automation](http://arxiv.org/abs/2604.07767v1)
  > **TL;DR**: Proposes AdecPilot, a decentralized edge-cloud multi-agent framework for mobile automation, enabling autonomous tactical planning at the edge. Reduces cloud token consumption by 37.5% and end-to-end latency by 88.9% compared to baselines.
* `serving` `edge` `hardware` [Reduced-Mass Orbital AI Inference via Integrated Solar, Compute, and Radiator Panels](http://arxiv.org/abs/2604.07760v1)
  > **TL;DR**: Proposes a solar-integrated satellite array architecture for efficient orbital LLM inference. Co-locates solar cells, radiators, and compute in panels to achieve high power efficiency and cooling. Supports 500k-token context at 553 tokens/sec per session, scaling to 16 MW compute.
* `serving` `offloading` `hardware` [Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC](http://arxiv.org/abs/2604.07609v1)
  > **TL;DR**: Proposes Blink, a CPU-free LLM inference architecture that offloads serving stack to GPU and SmartNIC. Uses persistent GPU kernel for batching/scheduling and SmartNIC for input delivery via RDMA. Achieves up to 8.47× lower P99 TTFT and 48.6% energy reduction.
* `serving` `diffusion` `scaling` [GENSERVE: Efficient Co-Serving of Heterogeneous Diffusion Model Workloads](http://arxiv.org/abs/2604.04335v2)
  > **TL;DR**: Addresses efficient co-serving of heterogeneous diffusion model workloads (T2I and T2V). Proposes GENSERVE with step-level resource adaptation via video preemption, elastic parallelism, and SLO-aware scheduling. Improves SLO attainment rate by up to 44% over baselines.
* `serving` `offloading` `quantization` [Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate](http://arxiv.org/abs/2604.07874v1)
  > **TL;DR**: Addresses resource underutilization in LLM inference clusters via online-offline colocation. Proposes Valve, a GPU runtime with channel-controlled compute isolation and page-fault-free memory reclamation to bound preemption latency/rate. Achieves 34.6% utilization gain with <5% TTFT/2% TPOT increase.
* `scheduling` `training` `scaling` [Qurator: Scheduling Hybrid Quantum-Classical Workflows Across Heterogeneous Cloud Providers](http://arxiv.org/abs/2604.05505v2)
  > **TL;DR**: Proposes Qurator, a scheduler for hybrid quantum-classical workflows that jointly minimizes queue time and maximizes fidelity. Uses dynamic DAGs with quantum semantics and unified fidelity estimation. Reduces queue time by 30-75% at high load while maintaining fidelity within user-specified bounds.
* `serving` `offloading` `kernel` [Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC](http://arxiv.org/abs/2604.07609v1)
  > **TL;DR**: Addresses CPU interference in LLM inference by offloading orchestration to GPU and SmartNIC. Proposes Blink, a CPU-free architecture using persistent GPU kernels and SmartNIC for direct RDMA input. Reduces P99 TTFT by 8.47× and TPOT by 3.40× while improving throughput by 2.1×.

### 2026-04-09
* `training` `inference` `storage` [Measurement of Generative AI Workload Power Profiles for Whole-Facility Data Center Infrastructure Planning](http://arxiv.org/abs/2604.07345v1)
  > **TL;DR**: Presents high-resolution power measurement methodology for generative AI workloads. Links workload power profiles to whole-facility energy demand using bottom-up event-driven modeling. Enables reproducible power profiling at 0.1s resolution across training, fine-tuning, and inference jobs.
* `inference` `distributed` `kernel` [Making Room for AI: Multi-GPU Molecular Dynamics with Deep Potentials in GROMACS](http://arxiv.org/abs/2604.07276v1)
  > **TL;DR**: Investigates efficient multi-GPU molecular dynamics simulation with AI-driven potentials. Extends GROMACS with domain-decomposed DeepMD-kit integration for concurrent inference across nodes using MPI collectives. Achieves strong-scaling efficiency of 66% on 16 GPUs and 40% on 32 GPUs.

### 2026-04-09
* `serving` `offloading` `LoRA` [InfiniLoRA: Disaggregated Multi-LoRA Serving for Large Language Models](http://arxiv.org/abs/2604.07173v1)
  > **TL;DR**: Proposes InfiniLoRA, a disaggregated system to decouple LoRA execution from base-model inference, reducing memory cost and latency. Uses shared LoRA Server with parallelism, SLO-driven provisioning, and optimized kernels. Achieves 3.05x higher serviceable request rate under SLOs.
* `serving` `scaling` [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](http://arxiv.org/abs/2604.07144v1)
  > **TL;DR**: Addresses runtime dynamics in LLM serving via online self-evolving policies. Introduces Autopoiesis, an LLM-driven workflow for continuous policy synthesis that observes real-time conditions and rewrites code. Achieves up to 53% higher throughput over state-of-the-art systems.
* `serving` `scaling` [Scheduling the Unschedulable: Taming Black-Box LLM Inference at Scale](http://arxiv.org/abs/2604.06970v1)
  > **TL;DR**: Addresses client-side scheduling for black-box LLM inference with unpredictable output token counts. Proposes a three-layer architecture with allocation, ordering, and overload control, utilizing coarse token priors. Achieves 100% deadline satisfaction and goodput of 4.2±1.6 requests/s with short P95 latency within tens of ms.
* `training` `scaling` `sparse` [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](http://arxiv.org/abs/2604.06956v1)
  > **TL;DR**: Addresses communication bottlenecks in training large recommendation models on 1,500+ accelerators. Proposes NestPipe with dual-buffer and frozen-window pipelining to overlap data movement with dense computation without staleness. Achieves 3.06x speedup and 94.07% scaling efficiency on 1,536 workers.
* `offline` `edge` `training` [Beyond End-to-End: Dynamic Chain Optimization for Private LLM Adaptation on the Edge](http://arxiv.org/abs/2604.06819v1)
  > **TL;DR**: Addresses memory constraints for federated fine-tuning of LLMs on edge devices. Proposes ChainFed, a sequential layer-by-layer optimization with dynamic co-tuning and global perceptive updates. Achieves up to 46.46% higher accuracy compared to existing methods.
* `video` `serving` `multi-modal` [CodecFlow: Codec-Guided End-to-End Optimization for Streaming Video Analytics](http://arxiv.org/abs/2604.06036v2)
  > **TL;DR**: Optimizes end-to-end streaming video analytics with VLMs using codec metadata to guide patch pruning in vision encoding and selective KV cache refresh in LLM prefilling. Achieves up to 3x throughput improvement and 87% GPU compute reduction with only 0-8% accuracy drop.
* `offloading` `storage` `serving` [Nexus: Transparent I/O Offloading for High-Density Serverless Computing](http://arxiv.org/abs/2604.06682v1)
  > **TL;DR**: Proposes Nexus, a hypervisor that transparently offloads I/O processing from guest VMs to a shared backend via zero-copy memory. This reduces CPU and memory overhead, enabling asynchronous I/O optimizations like pre-fetching. Achieves 44% lower CPU, 31% less memory consumption, and 37% higher density.
* `hardware` `storage` `scaling` [SwarmIO: Towards 100 Million IOPS SSD Emulation for Next-generation GPU-centric Storage Systems](http://arxiv.org/abs/2604.06668v1)
  > **TL;DR**: Proposes SwarmIO, an SSD emulator for GPU-centric storage systems, optimizing for ultra-high IOPS (up to 40 MIOPS). Solves scalability challenges with efficient frontend ingestion and reduced software overhead. Achieves 303.9x speedup over baselines and end-to-end speedups up to 9.7x in case studies.
* `serving` `kernel` `offline` [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](http://arxiv.org/abs/2604.06664v1)
  > **TL;DR**: Addresses slow cold-start in LLM serving autoscaling due to CUDA graph capture latency. Proposes Foundry, a template-based system that persists graph topology and execution context offline. Reduces cold-start for Qwen3-235B-A22B from 10 minutes to 3.9 seconds (99%).
* `kernel` `quantization` `sparse` [Sparsity-Aware Roofline Models for Sparse Matrix-Matrix Multiplication](http://arxiv.org/abs/2604.06637v1)
  > **TL;DR**: Evaluates roofline modeling for sparse matrix-matrix multiplication (SpMM) to improve performance prediction. Proposes sparsity-aware models accounting for structure, locality, and traffic. Achieves more accurate performance analysis for SpMM kernels across diverse matrices.
* `serving` `LoRA` `offloading` [ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache](http://arxiv.org/abs/2604.06370v1)
  > **TL;DR**: Addresses inefficient KV cache usage in multi-LoRA agent serving when sharing contexts. Proposes ForkKV with CoW-disaggregated KV cache management via DualRadixTree and ResidualAttention kernel. Achieves 3.0x throughput over SOTA systems with negligible quality loss.
* `serving` `scaling` [Scheduling the Unschedulable: Taming Black-Box LLM Inference at Scale](http://arxiv.org/abs/2604.06970v1)
  > **TL;DR**: Investigates client-side scheduling for black-box LLM inference with output token prediction. Proposes a three-layer decomposition: allocation, ordering, and overload control. Achieves 100% deadline satisfaction under congestion with short P95 latency within tens of milliseconds of isolated tiers.
* `offloading` `storage` `networking` [Nexus: Transparent I/O Offloading for High-Density Serverless Computing](http://arxiv.org/abs/2604.06682v1)
  > **TL;DR**: Addresses I/O overhead in serverless computing by transparently offloading communication fabric to a shared host backend. Proposes Nexus, a KVM-based hypervisor using shared memory and prefetching. Reduces CPU by 44%, memory by 31%, and improves start latency by 39-10%.

### 2026-04-08
* `training` `hardware` `scaling` [Fine-Grained Power and Energy Attribution on AMD GPU/APU-Based Exascale Nodes](http://arxiv.org/abs/2604.06056v1)
  > **TL;DR**: Addresses power and energy attribution challenges in large-scale GPU/APU systems for HPC workloads. Proposes a methodology using controlled workloads to reconstruct power from energy counters and validate sensors. Achieves significant energy savings (up to 79% reduction) via mixed precision optimizations on exascale nodes.
* `video` `multi-modal` `serving` [CoStream: Codec-Guided Resource-Efficient System for Video Streaming Analytics](http://arxiv.org/abs/2604.06036v1)
  > **TL;DR**: Reduces cost of video streaming analytics with multimodal inference. Uses video codec metadata to guide patch pruning for ViT encoding and key-value cache refresh for LLM prefilling, operating on compressed streams. Achieves up to 3x throughput improvement and 87% GPU compute reduction.
* `training` `kernel` `scaling` [GTaP: A GPU-Resident Fork-Join Task-Parallel Runtime with a Pragma-Based Interface](http://arxiv.org/abs/2604.05982v1)
  > **TL;DR**: Proposes GTaP, a GPU-resident runtime for fork-join task parallelism using persistent kernels and state machines. Introduces Execution-Path-Aware Queueing to reduce warp divergence. Achieves up to 1.8× speedup on Fibonacci vs. naive GPU alternatives and outperforms a 72-core CPU in many cases.
* `training` `LoRA` `scaling` [ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads](http://arxiv.org/abs/2604.05426v1)
  > **TL;DR**: Addresses inefficiency in concurrent LoRA hyperparameter tuning for LLM fine-tuning. Proposes ALTO, a system for early termination, co-location via fused grouped GEMM and rank-local adapter parallelism, and inter-task scheduling. Achieves up to 13.8× speedup over prior systems.
* `training` `scaling` `hardware` [Minos: Systematically Classifying Performance and Power Characteristics of GPU Workloads on HPC Clusters](http://arxiv.org/abs/2604.03591v2)
  > **TL;DR**: Proposes Minos, a systematic approach to classify GPU workloads for power and performance optimization in HPC clusters. Uses low-cost profiling to group workloads and predict frequency capping behavior. Reduces profiling time by 89% and achieves 4% mean power prediction error.

### 2026-04-08
* `training` `offloading` [MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU](http://arxiv.org/abs/2604.05091v1)
  > **TL;DR**: Presents MegaTrain, a system using CPU memory for storing parameters and optimizer states, with GPUs as transient compute engines. Uses pipelined double-buffered execution and stateless layer templates to overcome CPU-GPU bandwidth limits. Achieves 1.84× training throughput over DeepSpeed ZeRO-3 for 14B models.

### 2026-04-07
* `serving` `hardware` `scaling` [DeepStack: Scalable and Accurate Design Space Exploration for Distributed 3D-Stacked AI Accelerators](http://arxiv.org/abs/2604.04750v1)
  > **TL;DR**: Proposes DeepStack, a hardware-system co-design tool for distributed 3D-stacked LLM inference accelerators. Features fine-grained 3D memory modeling, parallelization strategies, and hierarchical search for cross-stack optimization. Achieves 9.5x higher throughput through co-optimized parallelism and architecture search.
* `training` `energy` `scaling` [The Energy Cost of Execution-Idle in GPU Clusters](http://arxiv.org/abs/2604.04745v1)
  > **TL;DR**: Characterizes the execution-idle state in GPU clusters as a high-power low-activity phase. Proposes automatic downscaling and load imbalance strategies to mitigate energy waste. Reduces execution-idle energy by 19.7% of in-execution time and 10.7% total energy.
* `training` `scaling` [Sampling Parallelism for Fast and Efficient Bayesian Learning](http://arxiv.org/abs/2604.04736v1)
  > **TL;DR**: Proposes sampling parallelism for Bayesian neural networks to reduce training time and memory consumption. Distributes sample evaluations across multiple GPUs, complementary to data parallelism. Achieves near-perfect scaling when sample count scales proportionally to GPUs, reducing convergence epochs via augmentation diversity.
* `serving` `offloading` `networking` [Communication-Efficient Collaborative LLM Inference over LEO Satellite Networks](http://arxiv.org/abs/2604.04654v1)
  > **TL;DR**: Proposes a communication-efficient collaborative LLM inference scheme for LEO satellite networks by splitting models across satellites and overlapping inference with transmission. Achieves up to 42% lower inference delay and 71% reduced communication overhead while maintaining accuracy loss under 1%.
* `kernel` `inference` `quantization` [LP-GEMM: Integrating Layout Propagation into GEMM Operations](http://arxiv.org/abs/2604.04599v1)
  > **TL;DR**: Proposes LP-GEMM, a method to eliminate redundant data repacking in sequential GEMM operations by propagating layouts across kernels. Integrates layout propagation into GEMM decomposition. Shows 2.25x speedup over OpenBLAS on x86 for sequential GEMMs in ML workloads.
* `video` `serving` `scaling` [GENSERVE: Efficient Co-Serving of Heterogeneous Diffusion Model Workloads](http://arxiv.org/abs/2604.04335v1)
  > **TL;DR**: Addresses co-serving latency SLO violations for heterogeneous diffusion model workloads (text-to-image and text-to-video). Proposes step-level resource adaptation with preemption, elastic parallelism, and SLO-aware scheduling. Achieves up to 44% higher SLO attainment than baselines.
* `scaling` `training` `kernel` [Minos: Systematically Classifying Performance and Power Characteristics of GPU Workloads on HPC Clusters](http://arxiv.org/abs/2604.03591v1)
  > **TL;DR**: Proposes Minos, a low-cost profiling method to classify GPU workloads on HPC clusters by power and performance characteristics. Groups workloads into classes to reduce profiling overhead and enable optimization. Achieves 89% reduction in profiling time and less than 4% error in power/performance predictions.
* `serving` `inference` `offloading` [AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems](http://arxiv.org/abs/2604.03425v1)
  > **TL;DR**: Addresses the challenge of scaling long-sequence encrypted Transformer inference on multi-GPU systems. Proposes AEGIS, which co-designs device placement for token-coherent and modulus-coherent data to minimize communication and overlaps collectives with computation. Achieves 57.9-81.3% comm reduction and 3.86x speedup on 4 GPUs.

### 2026-04-06
* `serving` `offloading` `storage` [TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing](http://arxiv.org/abs/2604.03143v1)
  > **TL;DR**: Addresses KV cache redundancy in multi-agent LLM serving with synchronized rounds. Proposes TokenDance, which enables collective KV cache sharing via diff-aware storage and a KV Collector. Achieves 17.5x KV cache storage reduction and 2.7x more concurrent agents than vLLM with SLO compliance.
* `multi-modal` `offloading` `edge` [MSAO: Adaptive Modality Sparsity-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference](http://arxiv.org/abs/2604.02945v1)
  > **TL;DR**: Proposes MSAO, an edge-cloud collaborative framework for MLLM inference that uses modality sparsity-aware offloading and speculative execution. Achieves 30% latency reduction, 30-65% resource savings, and 1.5x-2.3x throughput gains.
* `networking` `serving` `quantization` [Analyzing Reverse Address Translation Overheads in Multi-GPU Scale-Up Pods](http://arxiv.org/abs/2604.02473v1)
  > **TL;DR**: Investigates performance impact of Reverse Address Translation on multi-GPU collective communication. Models Link MMUs/TLBs in simulators to show 1.4x degradation for small collectives. Proposes fused pre-translation kernels and TLB prefetching to hide latency for inference throughput.
* `storage` `offloading` `hardware` [WIO: Upload-Enabled Computational Storage on CXL SSDs](http://arxiv.org/abs/2604.02442v1)
  > **TL;DR**: Proposes reversible storage-side compute with migratable actors on CXL SSDs to address I/O bottlenecks. Implements agility-aware scheduler for dynamic computation offloading across thermal/power constraints. Achieves up to 2× throughput improvement and 3.75× write latency reduction.

### 2026-04-03
* `scaling` `offloading` [Intelligent Cloud Orchestration: A Hybrid Predictive and Heuristic Framework for Cost Optimization](http://arxiv.org/abs/2604.02131v1)
  > **TL;DR**: Addresses cloud cost optimization under dynamic workloads with a hybrid framework. Combines LSTM-based predictive scaling and heuristic task allocation for efficient resource provisioning. Achieves reduced infrastructure costs near ML-based models while maintaining fast response times.
* `MoE` `training` [FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models](http://arxiv.org/abs/2604.01762v1)
  > **TL;DR**: Proposes FourierMoE, a spectral-domain MoE adaptation method with frequency-adaptive routing and complex coefficients to combat task interference and representational deficiency in multi-task LLM fine-tuning. Achieves superior performance over baselines on 28 benchmarks with fewer parameters.
* `MoE` `offloading` `serving` [DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72](http://arxiv.org/abs/2604.01621v1)
  > **TL;DR**: Reduces synchronization overhead in MoE model inference. Proposes Distributed Weight Data Parallelism (DWDP) with expert weight offloading and prefetching on peer GPUs. Improves end-to-end output TPS/GPU by 8.8% for DeepSeek-R1 on GB200 NVL72 at 8K input, 1K output.
* `training` [ModTrans: Translating Real-world Models for Distributed Training Simulator](http://arxiv.org/abs/2604.01607v1)
  > **TL;DR**: Addresses the challenge of importing real-world models into distributed training simulators. Proposes ModTrans, a translator that converts any real-world model format to ASTRA-sim input. Achieves negligible translation cost.
* `kernel` `training` `agentic` [CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe](http://arxiv.org/abs/2604.01489v1)
  > **TL;DR**: Presents CuTeGen, an agentic framework for auto-generation of optimized GPU kernels using CuTe abstraction. Employs structured generate-test-refine workflow with execution validation and staged optimization. Achieves competitive performance with optimized libraries on matrix multiplication and activation workloads.
* `storage` `hardware` `networking` [DAXFS: A Lock-Free Shared Filesystem for CXL Disaggregated Memory](http://arxiv.org/abs/2604.01620v1)
  > **TL;DR**: Presents DAXFS, a lock-free shared filesystem for CXL memory enabling multi-host coordination via cmpxchg atomic operations without centralized control. Features include a CAS-based hash overlay for concurrent writes and a decentralized MH-clock eviction algorithm. Achieves up to 2.68x higher random write throughput compared to tmpfs under single-host DRAM-backed DAX.

### 2026-04-02
* `training` `MoE` `scaling` [Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer](http://arxiv.org/abs/2604.00785v1)
  > **TL;DR**: Proposes Optimus, a training library for large MoE models on Aurora supercomputer. Implements custom GPU kernels, EP-Aware sharded optimizer for up to 1.71x speedup, and scales to 12288 GPUs with 90% efficiency training 220B model.
* `edge` `serving` [CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing](http://arxiv.org/abs/2603.27420v2)
  > **TL;DR**: Addresses carbon emissions in edge-based inference. Proposes CarbonEdge with carbon-aware scheduling that integrates emissions estimation and green scheduling via a tunable trade-off metric. Reduces emissions by 22.9% and improves carbon efficiency to 245.8 inferences per gram CO2 versus baseline.
* `training` `scaling` `offline` [Reclaiming Idle CPU Cycles on Kubernetes: Sparse-Domain Multiplexing for Concurrent MPI-CFD Simulations](http://arxiv.org/abs/2604.00377v1)
  > **TL;DR**: Presents a multiplexing framework to reclaim idle CPU during MPI synchronization barriers in Kubernetes clusters. Uses PMPI profiling and proportional CPU allocation for concurrent simulations, achieving up to 3.74x throughput with 4 concurrent jobs.
* `serving` `networking` `disaggregation` [TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving](http://arxiv.org/abs/2604.00368v1)
  > **TL;DR**: Proposes TENT, a declarative data-movement engine for disaggregated LLM serving. Dynamically decomposes and sprays data slices across heterogeneous interconnects using telemetry. Achieves 1.36x higher throughput and 26% lower P90 TTFT in LLM inference compared to baselines.
* `training` `networking` `MoE` [From Skew to Symmetry: Node-Interconnect Multi-Path Balancing with Execution-time Planning for Modern GPU Clusters](http://arxiv.org/abs/2604.00317v1)
  > **TL;DR**: Proposes NIMBLE, a runtime communication system that dynamically balances traffic across intra-node and inter-node paths in GPU clusters for skewed communication patterns. It uses capacity-normalized min-congestion optimization and CUDA-aware RDMA pipelining. Achieves up to 5.2x higher throughput on All-to-Allv workloads and 1.35x improvement on LLM MoE training vs baselines.
* `serving` `kernel` `offloading` [MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation](http://arxiv.org/abs/2604.00235v1)
  > **TL;DR**: Addresses the IO-bound challenge of long-context decoding in LLMs by proposing MAC-Attention, a computation reuse scheme with match-amend-complete stages. Reduces KV cache accesses by up to 99% and token latency by over 60% at 128K context, achieving up to 14.3x attention speedup.

### 2026-04-01
* `edge` `offline` `storage` [Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras](http://arxiv.org/abs/2603.29631v1)
  > **TL;DR**: Proposes an on-device epsilon-net filter to retain semantically novel frames in edge camera streams, improving cross-modal retrieval accuracy while reducing storage. Achieves 45.6% Hit@5 using an 8M encoder at 2.7 mW, outperforming offline methods like k-means and uniform sampling.
* `serving` `quantization` `kernel` [ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing](http://arxiv.org/abs/2603.27914v2)
  > **TL;DR**: Proposes ITQ3_S, a 3-bit weight quantization format with FWHT-based rotation-domain smoothing for efficient LLM inference. The method integrates weight pre-rotation, ternary quantization, and fused inverse FWHT in CUDA kernels, achieving FP16-competitive perplexity with 1.5x throughput over 4-bit baselines on Blackwell GPUs.
* `serving` `networking` `storage` [1.5 Million Messages Per Second on 3 Machines: Benchmarking and Latency Optimization of Apache Pulsar at Enterprise Scale](http://arxiv.org/abs/2603.29113v1)
  > **TL;DR**: Addresses high-latency issues in Apache Pulsar message queuing at enterprise scale. Optimizes hardware (NVMe, network) and JVM garbage collection (ZGC Generational) to resolve root causes like fdatasync spikes. Achieves 1.5M msg/s at 3.88 ms latency on 3 nodes, 4.7x improvement.
* `serving` `offloading` `hardware` [Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference](http://arxiv.org/abs/2603.29002v1)
  > **TL;DR**: Proposes a heterogeneous system to accelerate the memory processing pipeline in disaggregated LLM inference by offloading sparse/irregular operations to FPGAs and retaining compute-intensive tasks on GPUs. Achieves 1.04–2.2× speedup and 1.11–4.7× energy reduction compared to GPU baselines.
* `serving` `quantization` `kernel` [GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference](http://arxiv.org/abs/2603.28708v1)
  > **TL;DR**: Proposes a GPU-accelerated inference pipeline using TensorRT with hybrid precision (FP16/FP32) for transformer models. Achieves up to 64.4x speedup over CPUs and sub-10ms latency while reducing memory usage by 63% and preserving numerical fidelity (>0.9998 cosine similarity).
* `networking` `edge` [Trust-Aware Routing for Distributed Generative AI Inference at the Edge](http://arxiv.org/abs/2603.28622v1)
  > **TL;DR**: Addresses robust distributed inference for generative AI at the edge with untrusted devices. Proposes G-TRAC, a trust-aware routing framework combining risk-bounded path selection with hybrid trust architecture. Achieves high inference completion rates and sub-millisecond median routing latency in heterogeneous environments.
* `training` `federated` `memory` [FeDMRA: Federated Incremental Learning with Dynamic Memory Replay Allocation](http://arxiv.org/abs/2603.28455v1)
  > **TL;DR**: Proposes federated class-incremental learning (FCIL) with dynamic memory replay allocation to mitigate catastrophic forgetting in non-IID distributed healthcare data. Introduces strategy balancing exemplar storage across clients based on data heterogeneity. Improves model accuracy by 12.7% over baselines on medical image datasets.
* `edge` `serving` `scaling` [CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computing](http://arxiv.org/abs/2603.27420v1)
  > **TL;DR**: Proposes CarbonEdge, a carbon-aware DL inference framework for edge computing, introducing a scheduling algorithm with carbon efficiency metrics and model partitioning. Achieves 22.9% lower emissions and 1.3x higher carbon efficiency (245.8 vs 189.5 inferences/g CO2) with minimal overhead.
* `storage` `RL` [Efficiently Reproducing Distributed Workflows in Notebook-based Systems](http://arxiv.org/abs/2603.26965v1)
  > **TL;DR**: Addresses reproducibility inefficiency in distributed workflow notebooks. Proposes NBRewind, a dual-kernel system with cell-level checkpointing and dataflow analysis. Achieves minimal overhead in checkpointing with portable cross-site reproducibility.
* `hardware` `training` [Hardware-Agnostic and Insightful Efficiency Metrics for Accelerated Systems: Definition and Implementation within TALP](http://arxiv.org/abs/2603.26576v1)
  > **TL;DR**: Proposes hardware-agnostic efficiency metrics for CPU-accelerator systems, extending POP metrics with separate host/device hierarchies. Implements in TALP monitoring library to quantify offloading, load balance, and orchestration inefficiencies. Validated on HPC applications providing actionable optimization insights.
* `serving` `multi-modal` `scheduling` [Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference](http://arxiv.org/abs/2603.26498v1)
  > **TL;DR**: Addresses high latency and resource monopolization in multimodal LLM inference due to heterogeneous workloads. Proposes RPS-Serve, a modality-aware scheduler that prioritizes requests based on resource demand (video/image/text) and applies aging. Reduces time-to-first-token by 54% overall and 78.5% for latency-critical requests.
* `serving` `edge` `networking` [UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Models](http://arxiv.org/abs/2603.26469v1)
  > **TL;DR**: Proposes UNIFERENCE, a discrete-event simulation framework for developing and benchmarking distributed AI inference algorithms with heterogeneous devices. Integrates with PyTorch Distributed for seamless transition from simulation to deployment. Achieves 98.6% accuracy in runtime profiling across diverse hardware setups.
* `hardware` `training` `networking` [A Lightweight High-Throughput Collective-Capable NoC for Large-Scale ML Accelerators](http://arxiv.org/abs/2603.26438v1)
  > **TL;DR**: Proposes a collective-capable Network-on-Chip with Direct Compute Access for ML accelerators, enabling in-network reductions and multicast. Achieves 2.9x speedup in multicast, 2.5x in reduction operations, and up to 3.8x performance gain over baseline unicast NoC.

### 2026-03-27
* `offline` `networking` `scaling` [SHADOW: Seamless Handoff And Zero-Downtime Orchestrated Workload Migration for Stateful Microservices](http://arxiv.org/abs/2603.25484v1)
  > **TL;DR**: Proposes SHADOW for zero-downtime migration of stateful microservices in Kubernetes. Uses ShadowPod with concurrent message replay and ExchangeFence for identity swap. Reduces service downtime from median 38.5s to zero and migration time by up to 77% in 280 runs.
* `scaling` `training` [PRISM: Dynamic Primitive-Based Forecasting for Large-Scale GPU Cluster Workloads](http://arxiv.org/abs/2603.25378v1)
  > **TL;DR**: Proposes PRISM, a primitive-based forecasting framework for volatile GPU cluster workloads, combining temporal decomposition and spectral refinement. Reduces burst-phase errors by up to 45% in production traces, enabling efficient scheduling and resource management for AI infrastructure.
* `training` `multi-modal` `scaling` [DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization](http://arxiv.org/abs/2603.25120v1)
  > **TL;DR**: Proposes DFLOP, a data-driven framework for multimodal LLM training pipelines that profiles runtime behavior to address computation skew caused by heterogeneous inputs. Employs predictive scheduling to balance workloads, achieving up to 3.6x faster training over existing frameworks.
* `serving` `offloading` `storage` [GateANN: I/O-Efficient Filtered Vector Search on SSDs](http://arxiv.org/abs/2603.21466v2)
  > **TL;DR**: Proposes GateANN, a filtered vector search system that reduces I/O for SSD-based graph ANNS by decoupling graph traversal from vector retrieval and checking filter predicates in memory. Avoids index rebuilds and achieves up to 10x fewer SSD reads and 7.6x higher throughput.

### 2026-03-26
* `edge` `quantization` `hardware` [LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load](http://arxiv.org/abs/2603.23640v1)
  > **TL;DR**: Benchmarks sustained LLM inference performance on edge devices under thermal and power constraints. Evaluates Qwen 2.5 1.5B (4-bit quantized) across mobile and NPU/GPU platforms. Quantifies trade-offs: RTX 4050 sustains 131.7 tok/s at 34.1W; Hailo-10H achieves 6.9 tok/s at <2W with 19x energy proportionality.

### 2026-03-25
* `training` `scaling` `networking` [Communication-Aware Diffusion Load Balancing for Persistently Interacting Objects](http://arxiv.org/abs/2603.23329v1)
  > **TL;DR**: Proposes communication-aware diffusion load balancing for applications with persistently communicating objects. Leverages the communication graph to minimize across-node communication while distributing load. Achieves improved load balance for Particle-in-Cell on 8 nodes, reducing communication costs versus other strategies.

### 2026-03-25
* `serving` `RAG` `offloading` [PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving](http://arxiv.org/abs/2603.23049v1)
  > **TL;DR**: Addresses high prefill latency in RAG serving due to long inputs. Proposes PCR, with prefix-tree caching, layer-wise overlapping, and queue-based prefetching to enhance KV-cache reuse and reduce data transfer overhead. Achieves 2.47× speedup in average TTFT.
* `serving` `quantization` [Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference](http://arxiv.org/abs/2603.22774v1)
  > **TL;DR**: Identifies CPU bottlenecks in multi-GPU LLM inference causing GPU underutilization. Shows adequate CPU core provisioning reduces kernel launch delays, stalled communication, and tokenization latency. Restores responsiveness and cuts time-to-first-token latency by 1.36-5.40× without extra GPUs.
* `serving` `networking` `scaling` [WWW.Serve: Interconnecting Global LLM Services through Decentralization](http://arxiv.org/abs/2603.20661v2)
  > **TL;DR**: Proposes WWW.Serve, a decentralized framework for global LLM serving that allows flexible participant policies and self-organizing request dispatch. Achieves up to 1.5x better SLO attainment and 27.6% lower latency compared to centralized approaches while preserving decentralization benefits.
* `training` `offloading` `networking` [Communication-Efficient Approximate Gradient Coding](http://arxiv.org/abs/2603.22514v1)
  > **TL;DR**: Proposes communication-efficient approximate gradient coding schemes to mitigate stragglers in distributed learning. Uses structured graph-based designs and randomization to reduce communication vectors. Achieves tight error bounds and shows convergence in experiments.

### 2026-03-24
* `serving` `scaling` `RAG` [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](http://arxiv.org/abs/2603.21354v1)
  > **TL;DR**: Proposes the Workload-Router-Pool (WRP) architecture for optimizing LLM inference. Framework integrates workload characterization, routing dispatch strategies, and resource pool management. Enables dynamic adaptation to workload shifts and achieves optimized inference efficiency based on prior vLLM system improvements.
* `training` `agentic` `sparse` [ARYA: A Physics-Constrained Composable & Deterministic World Model Architecture](http://arxiv.org/abs/2603.21340v1)
  > **TL;DR**: Presents ARYA, a composable and deterministic world model architecture using nano-modules and a cognitive daemon. Achieves sub-20-second training cycles via sparse activation, resolving capability-efficiency tension. Matches state-of-the-art on 6/9 benchmarks with zero neural parameters.
* `serving` `offloading` `networking` [CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands](http://arxiv.org/abs/2603.21257v1)
  > **TL;DR**: Addresses network bottlenecks in distributed KVCache retrieval for LLM serving. Proposes CALVO, which decouples KVCache loading from GPU computation and models network delay in scheduling. Achieves 61.67% higher SLO attainment in long-context workloads.
* `storage` `offline` `RAG` [GateANN: I/O-Efficient Filtered Vector Search on SSDs](http://arxiv.org/abs/2603.21466v1)
  > **TL;DR**: Proposes GateANN, an SSD-based graph system for filtered vector search. Decouples graph traversal from vector retrieval via graph tunneling, allowing filter checks in memory to avoid SSD reads for non-matching nodes. Reduces SSD reads by 10× and increases throughput by 7.6×.

### 2026-03-24
* `training` `scaling` `multi-cloud` [Adviser: An Intuitive Multi-Cloud Platform for Scientific and ML Workflows](http://arxiv.org/abs/2603.20941v1)
  > **TL;DR**: Addresses complexity in leveraging multi-cloud resources for scientific and ML workflows. Introduces Adviser, a platform automating environment setup, resource provisioning, and data movement using reusable workflow artifacts. Demonstrates rapid exploration of cost-performance tradeoffs with up to 30% cost savings in glaciology simulations.
* `serving` `scaling` `RAG` [Compass: Optimizing Compound AI Workflows for Dynamic Adaptation](http://arxiv.org/abs/2603.20821v1)
  > **TL;DR**: Proposes Compass, a framework for dynamic optimization of compound AI workflows to balance latency, accuracy, and cost under varying loads. Combines offline configuration discovery with runtime adaptation using queuing-based policies. Achieves 71.6% higher SLO compliance and improves latency and accuracy over static baselines.
* `edge` `multi-modal` `serving` [RoboECC: Multi-Factor-Aware Edge-Cloud Collaborative Deployment for VLA Models](http://arxiv.org/abs/2603.20711v1)
  > **TL;DR**: Proposes RoboECC, an edge-cloud collaborative deployment framework for Vision-Language-Action models using model-hardware co-aware segmentation and network-aware adjustment. Achieves up to 3.28x speedup with 2.55x~2.62x overhead.
* `serving` `scaling` `networking` [WWW.Serve: Interconnecting Global LLM Services through Decentralization](http://arxiv.org/abs/2603.20661v1)
  > **TL;DR**: Proposes WWW.Serve, a decentralized framework for global LLM serving that enables flexible participation policies and self-organizing request dispatch without centralized coordination. Improves SLO attainment by 1.5x and reduces latency by 27.6% compared to centralized approaches.
* `scaling` `hardware` [Modernizing Amdahl's Law: How AI Scaling Laws Shape Computer Architecture](http://arxiv.org/abs/2603.20654v1)
  > **TL;DR**: Reformulates Amdahl's Law for modern AI with heterogeneous hardware. Introduces a model considering resource allocation across specialized accelerators vs programmable compute given empirical scaling laws. Shows optimal specialized investment collapses to zero beyond a critical scalable fraction, explaining GPU dominance despite efficiency advantages of domain-specific AI accelerators.
* `edge` `hardware` `training` [DGNNFlow: A Streaming Dataflow Architecture for Real-Time Edge-based Dynamic GNN Inference in HL-LHC Trigger Systems](http://arxiv.org/abs/2603.20364v1)
  > **TL;DR**: Proposes DGNNFlow, a streaming dataflow architecture for real-time edge-based dynamic GNN inference in HL-LHC trigger systems. Introduces hardware support for dynamic edge embeddings and resolves data dependencies on FPGA. Achieves up to 6.3x speedup and 0.22x power consumption vs GPU.

### 2026-03-23
* `edge` `offloading` `serving` [Speculative Policy Orchestration: A Latency-Resilient Framework for Cloud-Robotic Manipulation](http://arxiv.org/abs/2603.19418v1)
  > **TL;DR**: Addresses network latency in cloud-robotic manipulation by proposing SPO, which pre-computes and buffers waypoints at the edge with adaptive horizon scaling and safety verification. Reduces network-induced idle time by over 60% compared to blocking remote inference.

### 2026-03-20
* `agentic` `serving` `offloading` [Act While Thinking: Accelerating LLM Agents via Pattern-Aware Speculative Tool Execution](http://arxiv.org/abs/2603.18897v1)
  > **TL;DR**: Addresses high latency in LLM agents due to serial tool execution. Proposes PASTE, a pattern-aware speculative tool execution method exploiting stable control flows and data dependencies. Achieves 48.5% reduction in task completion time and 1.8x throughput improvement.

### 2026-03-20
* `serving` `inference` `scaling` [From Servers to Sites: Compositional Power Trace Generation of LLM Inference for Infrastructure Planning](http://arxiv.org/abs/2603.18383v1)
  > **TL;DR**: Proposes a compositional framework for generating power traces of LLM inference workloads, modeling state transitions and power distributions to synthesize server-to-site power profiles. Achieves median absolute energy error below 5% across configurations while preserving temporal autocorrelation.
* `training` `quantization` `storage` [Adaptive Domain Models: Bayesian Evolution, Warm Rotation, and Principled Training for Geometric and Neuromorphic AI](http://arxiv.org/abs/2603.18104v1)
  > **TL;DR**: Proposes an adaptive domain model (ADM) training architecture that reduces memory to twice inference footprint. Leverages geometric algebra, b-posit arithmetic, and program hypergraphs for grade-preserving updates and exact gradients. Achieves depth-independent memory and enables continuous adaptation via warm rotation.

### 2026-03-19
* `serving` `networking` `disaggregation` [Multi-stage Flow Scheduling for LLM Serving](http://arxiv.org/abs/2603.17456v1)
  > **TL;DR**: Addresses network contention in multi-stage LLM serving workflows for TTFT SLO attainment. Proposes MFS scheduler with Defer-and-Promote principle and Reverse Multi-Level Queue to prioritize flows. Improves TTFT SLO attainment by 1.2x--2.4x over baselines.
* `serving` `quantization` `kernel` [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](http://arxiv.org/abs/2603.17435v1)
  > **TL;DR**: Introduces ZipServ, a lossless compression framework for LLM inference using Tensor-Core-Aware Triple Bitmap Encoding and fused decompression-GEMM kernel. Achieves up to 30% model size reduction and 1.22x end-to-end inference speedup over vLLM.
* `serving` `offloading` `scaling` [The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency](http://arxiv.org/abs/2603.17280v1)
  > **TL;DR**: Analyzes how context window size affects LLM inference energy efficiency, deriving the 1/W law: tokens per watt halve when context doubles. Proposes FleetOpt, a two-pool context-length routing topology to optimize GPU utilization. Achieves 2.5x better tokens per watt over homogeneous fleets.
* `storage` `kernel` `recommendation` [HierarchicalKV: A GPU Hash Table with Cache Semantics for Continuous Online Embedding Storage](http://arxiv.org/abs/2603.17168v1)
  > **TL;DR**: Challenges GPU hash table dictionary assumption for embedding storage by introducing HierarchicalKV with cache semantics. Proposes cache-aligned buckets, in-line score-driven upsert, dual-bucket selection, and triple-group concurrency. Achieves up to 3.9B KV/s find throughput and 1.4x higher than baselines.

### 2026-03-18
* `training` `sparse` `kernel` [Unifying Optimization and Dynamics to Parallelize Sequential Computation: A Guide to Parallel Newton Methods for Breaking Sequential Bottlenecks](http://arxiv.org/abs/2603.16850v1)
  > **TL;DR**: Proposes scalable parallel Newton methods to overcome sequential bottlenecks in dynamical systems like RNNs. Combines quasi-Newton and trust-region approaches for efficiency and stability. Achieves faster convergence with memory-efficient quasi-Newton variants and proves linear convergence rates based on Lyapunov exponents.
* `serving` `offloading` `scaling` [FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism](http://arxiv.org/abs/2603.16514v1)
  > **TL;DR**: Proposes FleetOpt, an analytical framework for cost-optimal LLM inference fleet provisioning using compress-and-route to mitigate KV-cache waste. Designs a two-pool architecture with optimal context boundary and extractive compression gateway. Reduces GPU costs by 6-82% versus homogeneous fleets.

### 2026-03-18
* `training` `offloading` `quantization` [An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU](http://arxiv.org/abs/2603.16428v1)
  > **TL;DR**: Addresses efficient fine-tuning of large language models on single GPUs. Proposes SlideFormer with asynchronous sliding window engine, heterogeneous memory management, and optimized kernels. Achieves 1.40x-6.27x throughput improvement and halves memory usage for 123B+ models on single RTX 4090.
* `training` `networking` `quantization` [Biased Compression in Gradient Coding for Distributed Learning](http://arxiv.org/abs/2603.16353v1)
  > **TL;DR**: Introduces COCO-EF, combining biased compression with error-compensated gradient coding for distributed learning to reduce communication costs and mitigate stragglers. Achieves improved convergence over baselines by incorporating prior errors in compression and coding.
* `scaling` `serving` `training` [Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing](http://arxiv.org/abs/2603.15042v2)
  > **TL;DR**: Addresses unpredictable tail latency and low utilization in GPU sharing for data centers. Proposes DetShare, using GPU coroutines for fine-grained resource allocation via lightweight context migration. Reduces P99 tail latency by 15.1% and increases training throughput by 79.2% compared to temporal sharing.
* `serving` `offloading` `networking` [Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking](http://arxiv.org/abs/2603.12831v2)
  > **TL;DR**: Addresses interference when colocating latency-sensitive (SLO-bound) and best-effort LLM services. Proposes OmniServe using CPU-GPU Attention Piggybacking to offload BE attention to CPUs with async communication. Achieves up to 9.85× higher BE throughput and 1.48× SLO attainment for LS services.
* `serving` `scaling` `disaggregation` [inference-fleet-sim: A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference](http://arxiv.org/abs/2603.16054v1)
  > **TL;DR**: Proposes inference-fleet-sim, a queueing-theory-based GPU fleet capacity planner for LLM inference. Combines M/G/c queueing models with discrete-event simulation to optimize fleet configuration for cost and P99 TTFT SLO. Evaluated on real traces, shows improvements over simple analysis in fleet sizing and topology choices.
* `serving` `training` `quantization` [Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing](http://arxiv.org/abs/2603.15042v2)
  > **TL;DR**: Proposes DetShare, a GPU sharing system using GPU coroutines for fine-grained resource allocation without kernel modifications. Achieves semantic and performance determinism, reducing P99 tail latency by 15.1% and inference latency by 69.1% while improving training throughput by 79.2%.

### 2026-03-17
* `disaggregation` `scaling` `hardware` [DUET: Disaggregated Hybrid Mamba-Transformer LLMs with Prefill and Decode-Specific Packages](http://arxiv.org/abs/2603.15530v1)
  > **TL;DR**: Proposes DUET, a disaggregated accelerator with specialized prefill and decode packages for hybrid Mamba-Transformer LLMs. Achieves 4x faster time to first token and 1.4x higher throughput over B200 GPU.
* `serving` `scaling` [LMetric: Simple is Better - Multiplication May Be All You Need for LLM Request Scheduling](http://arxiv.org/abs/2603.15202v1)
  > **TL;DR**: Introduces LMetric, a scheduling algorithm for LLM inference using multiplication of KV cache utilization and load indicators. Achieves 92% TTFT reduction and 21% TPOT improvement over vLLM and production schedulers without hyperparameter tuning.
* `serving` `training` `offloading` [Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing](http://arxiv.org/abs/2603.15042v1)
  > **TL;DR**: Addresses unpredictable tail latency and performance interference in GPU sharing for ML tasks. Proposes DetShare with GPU coroutines for fine-grained resource allocation via context migration. Reduces P99 tail latency by 15.1% and inference latency by 69.1% over baselines.
* `serving` `offloading` [Fold-CP: A Context Parallelism Framework for Biomolecular Modeling](http://arxiv.org/abs/2603.14806v1)
  > **TL;DR**: Proposes Fold-CP, a context parallelism framework to scale biomolecular structure prediction by distributing inference/training across GPUs. Implements custom primitives for dense triangular updates and local attention. Achieves O(N²/P) memory scaling, enabling 30,000-residue predictions on 64 GPUs.
* `edge` `serving` `quantization` [Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference](http://arxiv.org/abs/2603.14577v1)
  > **TL;DR**: Proposes CORAL, an online optimizer using covariance for efficient edge inference hardware configuration. It dynamically adapts DVFS and concurrency to meet power and throughput targets, achieving 96-100% of optimal performance on NVIDIA Jetson.
* `serving` `offloading` `scaling` [LMetric: Simple is Better - Multiplication May Be All You Need for LLM Request Scheduling](http://arxiv.org/abs/2603.15202v1)
  > **TL;DR**: Proposes LMetric, a KV cache-aware and load-balanced LLM request scheduler using simple multiplicative scoring of prefill tokens and batch size. Eliminates hyperparameter tuning and reduces TTFT by up to 92% and TPOT by 21% compared to vLLM-v1 on real-world workloads.
* `serving` `offloading` `training` [Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing](http://arxiv.org/abs/2603.15042v1)
  > **TL;DR**: Proposes DetShare, a GPU sharing system with GPU coroutines for fine-grained resource allocation via context migration. Ensures semantic and performance determinism. Reduces P99 inference latency by 15.1% and training throughput improvement by 79.2%.

### 2026-03-17
* `networking` `hardware` `offloading` [The Forward-In-Time-Only Assumption in SmartNIC Resource Management: A Critique of Wave and the Case for Bilateral Interaction](http://arxiv.org/abs/2603.13750v1)
  > **TL;DR**: Critiques SmartNIC resource management's forward-only model. Proposes bilateral swap primitive via Open Atomic Ethernet to resolve latency and atomicity issues without optimization stacks. Achieves reduced PCIe latency by eliminating 350% degradation compared to Wave.
* `edge` `serving` `offloading` [Audo-Sight: AI-driven Ambient Perception Across Edge-Cloud for Blind and Low Vision Users](http://arxiv.org/abs/2603.13668v1)
  > **TL;DR**: Proposes Audo-Sight, an edge-cloud AI system for ambient perception, using intent-aware dynamic routing and a response fusion engine for low-latency speech output. Achieves 80% faster urgent task responses and 50% faster complete responses compared to cloud-only baselines.
* `MoE` `training` `serving` [NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL](http://arxiv.org/abs/2603.13606v1)
  > **TL;DR**: Proposes NCCL EP, a unified expert parallelism communication library based on NCCL's Device API for MoE models. Offers Low-Latency and High-Throughput modes for inference and training, leveraging RDMA+NVLink and hierarchical communication. Achieves competitive kernel performance in multi-node H100 clusters.

### 2026-03-16
* `training` `RL` `scaling` [ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning](http://arxiv.org/abs/2603.13019v1)
  > **TL;DR**: Addresses resource inefficiency in agentic RL training by proposing ARL-Tangram, with action-level orchestration and elastic scheduling for shared external resources. Improves average action completion time by 4.3× and reduces external resource usage by 71.2%.
* `serving` `offloading` [Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking](http://arxiv.org/abs/2603.12831v1)
  > **TL;DR**: Proposes OmniServe, a CPU-GPU attention piggybacking system for hybrid LLM loads. Offloads BE attention to CPUs via asynchronous streams and uses dynamic batching for dense modules. Improves LS SLO attainment by 1.48× and BE throughput by 9.85×.
* `serving` `offloading` `multi-modal` [Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity](http://arxiv.org/abs/2603.12707v1)
  > **TL;DR**: Addresses the inefficiency of uniform GPU usage for MLLM inference. Proposes HeteroServe, a system that exploits modality-level partitioning and cross-tier GPU heterogeneity. Achieves 37% higher Tokens/$ and up to 54% throughput gain over vLLM.
* `serving` `scaling` [SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity](http://arxiv.org/abs/2603.07917v2)
  > **TL;DR**: Addresses efficient LLM inference scheduling under demand uncertainty and hybrid resource usage. Proposes SageSched with output-length prediction, compute-memory cost modeling, and uncertainty-aware policy. Achieves over 28.7% efficiency improvement in diverse setups.
* `serving` `kernel` `MoE` [TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition](http://arxiv.org/abs/2603.12465v1)
  > **TL;DR**: Introduces TaxBreak, a trace-based method to decompose host-side overheads in LLM inference into framework, CUDA, and kernel launch times. Proposes Host-Device Balance Index (HDBI) to guide optimization targets. CPU thread optimization reduces orchestration overhead by 29% and latency by 14% for MoE models.
* `kernel` `training` [KernelFoundry: Hardware-aware evolutionary GPU kernel optimization](http://arxiv.org/abs/2603.12440v1)
  > **TL;DR**: Proposes KernelFoundry, an evolutionary framework for hardware-aware GPU kernel optimization using MAP-Elites search and meta-prompt evolution. Generates efficient SYCL/CUDA kernels with 2.3x average speedup on KernelBench.
* `agentic` `serving` `scaling` [AgentRM: An OS-Inspired Resource Manager for LLM Agent Systems](http://arxiv.org/abs/2603.13110v1)
  > **TL;DR**: Proposes AgentRM, an OS-inspired resource manager for LLM agent systems. Uses MLFQ scheduling with zombie reaping and three-tier context management. Reduces P95 latency by 86%, decreases lane waste by 96%, and increases throughput by 168%.

### 2026-03-13
* `serving` `multi-modal` `disaggregation` [Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models](http://arxiv.org/abs/2603.12118v1)
  > **TL;DR**: Addresses inefficiencies in serving any-to-any multimodal models by proposing a distributed system with task abstraction for disaggregation, record-and-replay execution for dependency tracking, and direct tensor forwarding. Achieves up to 3.81× higher throughput and 5.79× lower tail latency.
* `edge` `serving` `multi-modal` [RAPID: Redundancy-Aware and Compatibility-Optimal Edge-Cloud Partitioned Inference for Diverse VLA Models](http://arxiv.org/abs/2603.07949v2)
  > **TL;DR**: Addresses inefficiencies in edge-cloud collaborative inference for Vision-Language-Action models. Proposes RAPID framework with redundancy-aware partitioning and compatibility optimization. Achieves 1.73x speedup with 5–7% overhead.
* `training` `networking` `scaling` [NCCLbpf: Verified, Composable Policy Execution for GPU Collective Communication](http://arxiv.org/abs/2603.11438v1)
  > **TL;DR**: Proposes NCCLbpf, eBPF-based framework for verified, composable policy execution in NCCL GPU collective communication. Embedds userspace eBPF runtime for static verification, cross-plugin maps, and hot-reloads, reducing decision overhead to 80-130 ns and improving AllReduce throughput by up to 27%.
* `training` `scaling` `quantization` [Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure](http://arxiv.org/abs/2603.11101v1)
  > **TL;DR**: Proposes a cloud-based thousand-GPU distributed training infrastructure for embodied intelligence to address bottlenecks. Optimizes data pipeline, combines FlashAttention, data packing, FP8 quantization, and high-performance networking. Reduces training time from 15 hours to 22 minutes (40x speedup).
* `training` `networking` `kernel` [NCCLbpf: Verified, Composable Policy Execution for GPU Collective Communication](http://arxiv.org/abs/2603.11438v1)
  > **TL;DR**: Addresses safety and performance issues in NCCL GPU communication plugins by introducing NCCLbpf, a verified eBPF extension framework. Implements load-time verification, composable policies, and hot-reloading. Reduces AllReduce latency by up to 27% for 4-128 MiB messages with 80-130 ns decision overhead.

### 2026-03-12
* `serving` `security` [CacheSolidarity: Preventing Prefix Caching Side Channels in Multi-tenant LLM Serving Systems](http://arxiv.org/abs/2603.10726v1)
  > **TL;DR**: Addresses security vulnerabilities from prefix caching in multi-tenant LLM serving. Proposes CacheSolidarity, which monitors cache reuse and selectively isolates prefixes to prevent side-channel attacks. Achieves 70% higher cache reuse and 30% lower latency compared to isolation-based defenses.
* `quantization` `kernel` [Double-Precision Matrix Multiplication Emulation via Ozaki-II Scheme with FP8 Quantization](http://arxiv.org/abs/2603.10634v1)
  > **TL;DR**: Proposes a novel method to emulate FP64 matrix multiplication using FP8 units via Ozaki-II scheme, overcoming prior limitations. Reduces the number of required FP8 matrix multiplications compared to Ozaki-I, enabling efficient FP64 emulation on modern GPUs with limited FP64 support.
* `training` `networking` `scaling` [CD-Raft: Reducing the Latency of Distributed Consensus in Cross-Domain Sites](http://arxiv.org/abs/2603.10555v1)
  > **TL;DR**: Proposes CD-Raft, an optimized Raft protocol reducing consensus latency for cross-domain data synchronization in distributed training. Optimizes leader positioning and RTT for reads/writes. Reduces avg latency by 32.9% and tail latency by 49.24% in YCSB benchmarks.
* `offline` `edge` `inference` [COHORT: Hybrid RL for Collaborative Large DNN Inference on Multi-Robot Systems Under Real-Time Constraints](http://arxiv.org/abs/2603.10436v1)
  > **TL;DR**: Proposes COHORT, a hybrid RL framework for collaborative DNN inference on multi-robot edge systems. Uses offline RL with AWR and online MAPPO adaptation to schedule tasks across robots. Reduces battery usage by 15.4% and increases GPU utilization by 51.67% while meeting deadlines.
* `serving` `sparse` `kernel` [S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance](http://arxiv.org/abs/2603.10353v1)
  > **TL;DR**: Addresses high-latency issues in LLM attention computation through Sparsity-aware Head-Parallel Load Balance (S-HPLB), which optimizes head-adaptive sparsity budgets and deployment across GPUs to reduce resource bubbles. Achieves 2.88× faster average attention computation latency without quality degradation.
* `serving` `RL` `offloading` [AgentServe: Algorithm-System Co-Design for Efficient Agentic AI Serving on a Consumer-Grade GPU](http://arxiv.org/abs/2603.10342v1)
  > **TL;DR**: Explores efficient serving for agentic LLMs on consumer-grade GPUs amidst heterogeneous prefill-decode contention. Introduces AgentServe with cold/resume prefill isolation, dynamic budgeting, and CUDA Green Context slots. Achieves up to 2.8× TTFT and 2.7× TPOT improvements over baselines.
* `serving` `offloading` `storage` [Pooling Engram Conditional Memory in Large Language Models using CXL](http://arxiv.org/abs/2603.10087v1)
  > **TL;DR**: Proposes using CXL memory pools to store Engram embeddings for LLM serving. Integrates offloaded Engram storage with SGLang runtime via fine-grained CXL access, achieving near-DRAM performance. Reduces cost while maintaining inference latency comparable to DRAM setups.

### 2026-03-11
* `edge` `sparse` `scaling` [Multi-DNN Inference of Sparse Models on Edge SoCs](http://arxiv.org/abs/2603.09642v1)
  > **TL;DR**: Addresses high SLO violations in edge-based multi-DNN inference with sparse models. Introduces model stitching to recombine sparse subgraphs without retraining. SparseLoom reduces SLO violations by up to 74%, improves throughput 2.31x, and lowers memory overhead by 28%.
* `inference` `offloading` `kernel` [Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference](http://arxiv.org/abs/2603.09555v1)
  > **TL;DR**: Proposes a compiler-first approach for state-space models that achieves portable O(1) autoregressive caching without custom kernels. Implements full inference path in XLA for CPU/GPU/TPU, showing up to 15% MFU on TPU prefills and matching PyTorch/CUDA accuracy.
* `serving` `offloading` `storage` [Provuse: Platform-Side Function Fusion for Performance and Efficiency in FaaS Environments](http://arxiv.org/abs/2603.06170v2)
  > **TL;DR**: Reduces latency and resource overhead in FaaS environments via automatic runtime function fusion. Proposes Provuse, a platform-side optimization that consolidates function execution transparently without code changes. Achieves average 26.33% latency reduction and 53.57% RAM usage reduction.
* `training` `sparse` `offloading` [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](http://arxiv.org/abs/2603.08163v2)
  > **TL;DR**: Investigates feasibility of open-permission globally distributed LLM pre-training. Proposes SparseLoCo optimizer for dynamic peer participation with blockchain-coordinated resource tracking. Achieves competitive performance at 72B scale with 1.1T tokens despite untrusted nodes and churn.
* `MoE` `training` [Scalable Training of Mixture-of-Experts Models with Megatron Core](http://arxiv.org/abs/2603.07685v2)
  > **TL;DR**: Addresses systems challenges in scaling MoE training by co-designing optimizations in memory (recomputation, offloading), communication (dispatchers, overlapping), and computation (Grouped GEMM, CUDA Graphs). Achieves up to 1,233 TFLOPS/GPU for DeepSeek-V3-685B on thousands of GPUs.
* `kernel` `training` [Flash-KMeans: Fast and Memory-Efficient Exact K-Means](http://arxiv.org/abs/2603.09229v1)
  > **TL;DR**: Addresses IO bottlenecks in GPU-based k-means clustering. Introduces fused distance-argmin computation and contention-free updates via explicit inverse mapping. Achieves up to 17.9× speedup over baselines on H200 GPUs.
* `edge` `offloading` `kernel` [PIM-SHERPA: Software Method for On-device LLM Inference by Resolving PIM Memory Attribute and Layout Inconsistencies](http://arxiv.org/abs/2603.09216v1)
  > **TL;DR**: Addresses memory attribute and layout inconsistencies in on-device LLM inference for PIM systems. Proposes PIM-SHERPA with DRAM double buffering and online weight rearrangement to resolve cacheable region conflicts. Achieves 47.8-49.7% memory capacity savings on Llama 3.2.
* `offline` `edge` `networking` [Two Teachers Better Than One: Hardware-Physics Co-Guided Distributed Scientific Machine Learning](http://arxiv.org/abs/2603.09032v1)
  > **TL;DR**: Proposes EPIC, a distributed SciML framework with hardware-aware local encoding and physics-guided central decoding. Transmits latent features instead of raw data, using cross-attention to reduce communication costs. Achieves 8.9× lower latency and 33.8× lower energy consumption across 10 datasets.
* `MoE` `serving` `inference` [The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference](http://arxiv.org/abs/2603.08960v1)
  > **TL;DR**: Identifies the 'qs inequality' criterion to predict when MoE models incur a throughput penalty versus quality-matched dense models during decoding due to reuse fragmentation. Analysis shows MoE disadvantages in long-context inference, e.g., DeepSeek-V3 has 4.5x lower throughput than dense baseline at 128k context.
* `serving` `edge` `scaling` [Serving Compound Inference Systems on Datacenter GPUs](http://arxiv.org/abs/2603.08797v1)
  > **TL;DR**: Proposes JigsawServe, a framework for serving compound inference systems by adaptively choosing model variants and spatially partitioning GPUs per task to optimize latency, accuracy, and cost. Analytically increases serviceable demand by 11.3x and empirically uses only 43.3% GPU resources while meeting SLOs.
* `edge` `serving` `offloading` [FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation](http://arxiv.org/abs/2603.09046v1)
  > **TL;DR**: Proposes FlexServe, a fast and secure LLM serving system for mobile devices using flexible resource isolation in TrustZone. Introduces Flex-Mem and Flex-NPU for efficient switching between unprotected/protected modes, with LLM-aware memory management and secure inference pipeline. Achieves up to 24.3× end-to-end speedup in multi-model workflows.
* `offline` `offloading` `serving` [The Missing Memory Hierarchy: Demand Paging for LLM Context Windows](http://arxiv.org/abs/2603.09023v1)
  > **TL;DR**: Addresses context window waste in LLMs by introducing a demand paging system. Proposes Pichay, a transparent proxy for evicting stale content and fault-driven pinning. Reduces context consumption by up to 93% in production and achieves a 0.0254% fault rate in simulation.

### 2026-03-10
* `training` `networking` [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](http://arxiv.org/abs/2603.08163v1)
  > **TL;DR**: Explores decentralized LLM pre-training with untrusted peers over the internet. Employs SparseLoCo, a communication-efficient optimizer supporting dynamic participation via blockchain protocol. Achieved competitive performance with centralized models at 72B scale using 1.1T tokens.
* `edge` `multi-modal` `serving` [RAPID: Redundancy-Aware and Compatibility-Optimal Edge-Cloud Partitioned Inference for Diverse VLA models](http://arxiv.org/abs/2603.07949v1)
  > **TL;DR**: Addresses high inference costs for Vision-Language-Action models in edge-cloud settings. Proposes RAPID, a redundancy-aware and compatibility-optimal partitioned inference framework to reduce visual noise interference and preserve motion continuity. Achieves 1.73x speedup with 5-7% overhead.
* `serving` `quantization` `offloading` [SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity](http://arxiv.org/abs/2603.07917v1)
  > **TL;DR**: Addresses efficient LLM serving amidst demand uncertainty and compute-memory hybridity. Proposes SageSched, combining output length prediction with compute/memory cost modeling for uncertainty-aware scheduling. Achieves 28.7% higher efficiency versus baseline schedulers.
* `serving` `networking` `kernel` [ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs](http://arxiv.org/abs/2603.07770v1)
  > **TL;DR**: Addresses inefficient cross-NUMA memory access in CPU-based LLM inference. Proposes ArcLight with fine-grained tensor parallelism and optimized memory management & thread scheduling. Achieves 46% higher throughput versus mainstream frameworks on many-core CPUs.
* `training` `MoE` `scaling` [Scalable Training of Mixture-of-Experts Models with Megatron Core](http://arxiv.org/abs/2603.07685v1)
  > **TL;DR**: Addresses systems challenges in scaling MoE model training. Proposes integrated optimizations for memory, communication, and computation, including Parallel Folding and low-precision support. Achieves up to 1,233 TFLOPS/GPU for DeepSeek-V3-685B on NVIDIA GB300/GB200 clusters.
* `edge` `thinking` `RL` [Agentic AI-Driven UAV Network Deployment: A LLM-Enhanced Exact Potential Game Approach](http://arxiv.org/abs/2603.07456v1)
  > **TL;DR**: Proposes LLM-enhanced game theory for UAV network deployment. Combines spatial-scale optimization with log-linear learning and gradient methods, using LLM for utility weight generation. Reduces energy by 20%, latency by 30%, and improves throughput.
* `hardware` `kernel` `offloading` [Mitigating the Memory Bottleneck with Machine Learning-Driven and Data-Aware Microarchitectural Techniques](http://arxiv.org/abs/2603.07683v1)
  > **TL;DR**: Proposes data-aware microarchitectural techniques using ML to mitigate memory bottlenecks. Introduces four ML-driven mechanisms including RL-based prefetching and off-chip prediction. Achieves significant performance and energy efficiency improvements over prior approaches.

### 2026-03-10
* `training` `scaling` [AIReSim: A Discrete Event Simulator for Large-scale AI Cluster Reliability Modeling](http://arxiv.org/abs/2603.07041v1)
  > **TL;DR**: Proposes AIReSim, a discrete event simulator to model and optimize reliability mechanisms (failure recovery, scheduling, repair) in large-scale AI clusters. Enables systematic evaluation of parameter tuning for reliability and capacity planning. Achieves improved cluster utilization by optimizing failure mitigation strategies.
* `RAG` `offline` `serving` [Configurable Runtime Orchestration for Dynamic Data Retrieval in Distributed Systems](http://arxiv.org/abs/2603.06980v1)
  > **TL;DR**: Proposes a runtime orchestration framework for dynamic data retrieval in distributed systems. Generates execution graphs from configuration at request time with dependency-aware scheduling and parallel execution to reduce latency. Achieves low-latency orchestration without redeploying code when integrations change.
* `training` `networking` `scaling` [NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning](http://arxiv.org/abs/2603.06798v1)
  > **TL;DR**: Proposes NEST, a network- and memory-aware device placement framework that optimizes distributed deep learning. Uses structured dynamic programming with operator graphs to jointly model parallelism, topology, and memory. Achieves up to 2.43× higher throughput over baselines.

### 2026-03-09
* `MoE` `serving` `scaling` [MoEless: Efficient MoE LLM Serving via Serverless Computing](http://arxiv.org/abs/2603.06350v1)
  > **TL;DR**: Addresses expert load imbalance in MoE LLM serving by introducing a serverless framework with proactive straggler identification and optimized expert scaling/placement. MoEless reduces inference latency by 43% and cost by 84% compared to state-of-the-art solutions.
* `offloading` `kernel` `scaling` [Provuse: Platform-Side Function Fusion for Performance and Efficiency in FaaS Environments](http://arxiv.org/abs/2603.06170v1)
  > **TL;DR**: Proposes Provuse, a platform-side function fusion for FaaS to reduce latency and redundant resource usage by transparently consolidating functions at runtime. Achieves 26.33% lower latency and 53.57% RAM reduction in evaluations.
* `edge` `RAG` `offloading` [Knowledge-driven Reasoning for Mobile Agentic AI: Concepts, Approaches, and Directions](http://arxiv.org/abs/2603.05831v1)
  > **TL;DR**: Proposes a knowledge-driven framework for mobile agentic AI under SWAP-C constraints, using reusable decision structures synchronized over bandwidth-limited links. Achieves perfect mission reliability at lower cost than knowledge-free or cloud-centric approaches with a 3B-parameter model.
* `serving` `multi-modal` `scaling` [StreamWise: Serving Multi-Modal Generation in Real-Time at Scale](http://arxiv.org/abs/2603.05800v1)
  > **TL;DR**: Addresses real-time multi-modal generation serving under strict latency constraints. Proposes StreamWise with adaptive quality control, model parallelism, and resource-aware scheduling on heterogeneous hardware. Achieves sub-second startup delay for high-quality streaming under $45.
* `serving` `training` `scaling` [Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks](http://arxiv.org/abs/2603.05692v1)
  > **TL;DR**: Investigates parallelization strategies for dense LLM deployment to optimize latency-throughput tradeoffs. Analyzes Tensor Parallelism (TP) and Pipeline Parallelism (PP) hybrids for different workload objectives. TP achieves up to 50% lower latency, while PP improves inference throughput by 2×, depending on batching and model size.

### 2026-03-06
* `edge` `training` `scaling` [Scaling Real-Time Traffic Analytics on Edge-Cloud Fabrics for City-Scale Camera Networks](http://arxiv.org/abs/2603.05217v1)
  > **TL;DR**: Proposes a scalable edge-cloud system for real-time traffic analytics on city-scale camera networks. Integrates capacity-aware scheduling and continuous federated learning to balance DNN inference and ST-GNN forecasting across edge devices and cloud. Handles 1000 streams, achieving stable 2000 FPS on Jetson Orin and low latency.
* `serving` `RL` `scaling` [PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning](http://arxiv.org/abs/2603.05087v1)
  > **TL;DR**: Optimizes resource efficiency and SLO compliance for prompt-tuning-as-a-service. Proposes PromptTuner with a Prompt Bank for initial prompts and elastic Workload Scheduler for resource allocation. Reduces SLO violations by 4.0x-7.9x and costs by 1.6x-4.5x.
* `training` `networking` `offloading` [The Semantic Arrow of Time, Part III: RDMA and the Completion Fallacy](http://arxiv.org/abs/2603.04774v1)
  > **TL;DR**: Addresses the completion fallacy in RDMA where data delivery does not guarantee semantic commitment. Analyzes industrial deployments and proposes a protocol with a mandatory reflecting phase to bridge the gap. Shows the gap can cause arbitrarily large delays in data centers.
* `serving` `disaggregation` [SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference](http://arxiv.org/abs/2603.04716v1)
  > **TL;DR**: Proposes a method to determine optimal prefill-decode disaggregated resource allocation for LLM inference under SLO constraints. Combines theoretical M/M/1 queuing modeling for prefill throughput and empirical decode batch sizing. Accurately predicts resource allocation in experiments meeting throughput and SLOs.
* `training` `kernel` `hardware` [DuaLip-GPU Technical Report](http://arxiv.org/abs/2603.04621v1)
  > **TL;DR**: Addresses inefficiency in large-scale LP solvers for ranking/allocation problems. Redesigns DuaLip-GPU with GPU-centric architecture, operator-driven formulation, and optimized kernels for sparse constraints. Achieves 10x speedup over prior CPU solver on matching workloads.

### 2026-03-05
* `networking` `training` `scaling` [The Ghost in the Datacenter: Link Flapping, Topology Knowledge Failures, and the FITO Category Mistake](http://arxiv.org/abs/2603.03736v1)
  > **TL;DR**: Addresses link flapping failures in datacenter networks during large-scale training that corrupt network topology. Proposes Open Atomic Ethernet with transaction-based failure detection and recovery. Prevents ghost failures in clusters scaling to ~3M GPUs, reducing disruptions reported in LLaMA 3 training (419 events in 54 days).
* `training` `scaling` `hardware` [HyperParallel: A Supernode-Affinity AI Framework](http://arxiv.org/abs/2603.03731v1)
  > **TL;DR**: Proposes HyperParallel, a supernode-affinity framework with hierarchical memory management (HyperOffload), fine-grained MPMD parallelism (HyperMPMD), and declarative sharding (HyperShard) to optimize training/inference on accelerator-rich architectures. Achieves significant efficiency gains and reduced tuning overhead.
* `serving` `offloading` `hardware` [Accelerating OpenPangu Inference on NPU via Speculative Decoding](http://arxiv.org/abs/2603.03383v1)
  > **TL;DR**: Proposes an end-to-end speculative inference acceleration scheme for OpenPangu-7B on NPU to overcome memory bottlenecks. Tailors speculative decoding for domestic NPU hardware lacking native support. Achieves significant inference speedup (quantitative result implied but not specified).

### 2026-03-04
* `training` `LoRA` `storage` [MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing](http://arxiv.org/abs/2603.02885v1)
  > **TL;DR**: Addresses GPU underutilization and stalls in multi-task PEFT fine-tuning. Proposes MuxTune with spatial-temporal backbone multiplexing and hierarchical co-scheduling. Achieves 2.33× higher throughput and 5.29× memory reduction vs. baselines.
* `training` `storage` `scaling` [Why Atomicity Matters to AI/ML Infrastructure: Snapshots, Firmware Updates, and the Cost of the Forward-In-Time-Only Category Mistake](http://arxiv.org/abs/2603.02603v1)
  > **TL;DR**: Addresses atomicity flaws in AI/ML training checkpointing and updates. Proves the infeasibility of temporal snapshots for global state recovery under asynchrony, formalizes mixed-epoch errors, and introduces a bilateral protocol replacing temporal assumptions. Quantifies inconsistency growth as exponential with persistence domains.
* `kernel` `serving` `inference` [GPUTOK: GPU Accelerated Byte Level BPE Tokenization](http://arxiv.org/abs/2603.02597v1)
  > **TL;DR**: Addresses CPU bottleneck in tokenization during large-context LLM inference. Proposes GPU-accelerated byte-level BPE tokenizer using cuCollections map and CUB reductions. Achieves up to 7.6x speedup vs HuggingFace GPT-2 tokenizer for 131k-token inputs.
* `serving` `MoE` `scheduling` [Multi-Layer Scheduling for MoE-Based LLM Reasoning](http://arxiv.org/abs/2602.21626v2)
  > **TL;DR**: Proposes a multi-layer scheduling framework for MoE-based LLM serving to improve resource utilization and reduce latency. It features request-level (SJF, priority-aware aging), engine-level (load-aware dispatching), and expert-level (hotspot alleviation) scheduling. Achieves up to 17.8% lower TTFT and 13.3% lower TPOT vs. vLLM.
* `kernel` `training` `networking` [CUCo: An Agentic Framework for Compute and Communication Co-design](http://arxiv.org/abs/2603.02376v1)
  > **TL;DR**: Addresses the need for efficient joint compute-communication kernels in LLM training/inference. Proposes CUCo, an agent-driven framework for automated high-performance kernel generation. Achieves up to 1.57× latency reduction compared to SotA baselines.

### 2026-03-03
* `serving` `scaling` `networking` [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](http://arxiv.org/abs/2602.22593v2)
  > **TL;DR**: Addresses dynamic parallelism switching for LLM serving under varying workloads. Proposes Flying Serving, integrating zero-copy weights management, KV cache adaptation, and a deadlock-free scheduler. Achieves up to 4.79× performance improvement under high load.
* `training` `multi-modal` `scaling` [Trident: Adaptive Scheduling for Heterogeneous Multimodal Data Pipelines](http://arxiv.org/abs/2603.02075v1)
  > **TL;DR**: Addresses non-stationary bottlenecks in multimodal data pipelines for foundation model training. Proposes Trident, an adaptive scheduler with anomaly filtering, memory-safe Bayesian optimization, and joint operator scheduling. Achieves up to 2.01x throughput improvement over static designs.
* `serving` `offline` [Beyond Microservices: Testing Web-Scale RCA Methods on GPU-Driven LLM Workloads](http://arxiv.org/abs/2603.02057v1)
  > **TL;DR**: Evaluates root cause analysis (RCA) methods for failures in LLM inference deployments. Tests 24 RCA methods under controlled failure injections, showing multi-source approaches achieve highest accuracy. Reveals metric-based methods vary by fault type, while trace-based largely fail; motivates tailored observability.
* `agentic` `edge` `RAG` [HeRo: Adaptive Orchestration of Agentic RAG on Heterogeneous Mobile SoC](http://arxiv.org/abs/2603.01661v1)
  > **TL;DR**: Introduces HeRo, a heterogeneous-aware framework for optimizing latency in agentic RAG workflows on mobile SoCs. Combines profiling-based modeling with shape-aware partitioning, criticality-based mapping, and bandwidth-aware concurrency control. Achieves up to 10.94× lower end-to-end latency.
* `hardware` `kernel` `scaling` [TeraPool: A Physical Design Aware, 1024 RISC-V Cores Shared-L1-Memory Scaled-up Cluster Design with High Bandwidth Main Memory Link](http://arxiv.org/abs/2603.01629v1)
  > **TL;DR**: Proposes TeraPool, a physically implementable cluster design with 1024 RISC-V cores sharing L1 memory via hierarchical interconnect. Achieves near-gigahertz frequencies (910MHz) and energy efficiency of 9-13.5pJ per access, delivering 1.89 TFLOPS peak performance.
* `serving` `quantization` `offloading` [Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification](http://arxiv.org/abs/2603.01399v1)
  > **TL;DR**: Proposes Quasar, a training-free framework using low-bit quantization for verification in speculative decoding to reduce memory bandwidth. Halves memory traffic while maintaining acceptance length, achieving 1.28× end-to-end throughput improvement on OpenPangu and Qwen3.
* `MoE` `offloading` `training` [TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading](http://arxiv.org/abs/2603.01058v1)
  > **TL;DR**: Investigates how to optimize Mixture-of-Experts inference throughput via heterogeneous offloading. Proposes TriMoE, a GPU-CPU-NDP architecture with AMX-enabled CPU for warm experts, bottleneck-aware scheduling, and dynamic rebalancing. Achieves up to 2.83x speedup over state-of-the-art solutions.
* `training` `scaling` `networking` [SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs](http://arxiv.org/abs/2603.00357v1)
  > **TL;DR**: Addresses frequent failures in large-scale LLM pretraining on 100k+ GPUs. Proposes SPARe: fault-tolerance via stacked parallelism and adaptive reordering to mask node failures during gradient sync. Reduces time-to-train by 40~50% at 600k GPU scale.
* `serving` `scaling` `networking` [Token Management in Multi-Tenant AI Inference Platforms](http://arxiv.org/abs/2603.00356v1)
  > **TL;DR**: Proposes token pools, a control-plane abstraction for multi-tenant LLM serving platforms to manage capacity in token/KV cache units. Allows burst handling, priority tiers, and fair allocation without runtime changes. Maintains bounded P99 latency during overload by throttling spot traffic.
* `serving` `scaling` [Token Management in Multi-Tenant AI Inference Platforms](http://arxiv.org/abs/2603.00356v1)
  > **TL;DR**: Proposes token pools for multi-tenant LLM inference to balance resource utilization and SLAs. Models capacity in token/KV cache/concurrency units combined with priority-aware throttling. During overload, maintains bounded P99 latency for guaranteed workloads while baseline degrades unboundedly.

### 2026-03-02
* `serving` `quantization` `storage` [Data Driven Optimization of GPU efficiency for Distributed LLM Adapter Serving](http://arxiv.org/abs/2602.24044v1)
  > **TL;DR**: Optimizes GPU efficiency for distributed LLM adapter serving by developing a data-driven pipeline. Combines a Digital Twin for emulation, ML models for throughput prediction, and greedy placement to minimize GPUs needed. Reduces GPU count by up to 40% while avoiding starvation and memory errors.
* `serving` `RL` `scaling` [Green or Fast? Learning to Balance Cold Starts and Idle Carbon in Serverless Computing](http://arxiv.org/abs/2602.23935v1)
  > **TL;DR**: Addresses the trade-off between cold start latency and idle resource carbon emissions in serverless LLM serving. Proposes LACE-RL, a reinforcement learning framework that dynamically adjusts pod keep-alive durations based on workload and carbon intensity. Achieves 51.69% fewer cold starts and 77.08% lower idle emissions compared to static policies.
* `training` `kernel` `hardware` [A task-based data-flow methodology for programming heterogeneous systems with multiple accelerator APIs](http://arxiv.org/abs/2602.21897v2)
  > **TL;DR**: Addresses efficient programming for heterogeneous systems with multiple accelerator APIs. Proposes task-based data-flow methodology with Task-Aware APIs and unified thread management via nOS-V. Reduces programming complexity and enables transparent integration, improving performance on multi-CPU/GPU systems with up to 7x speedup demonstrated in experiments.
* `training` `quantization` `scaling` [veScale-FSDP: Flexible and High-Performance FSDP at Scale](http://arxiv.org/abs/2602.22437v2)
  > **TL;DR**: Addresses limitations of FSDP for structure-aware training and non-element-wise optimizers. Proposes veScale-FSDP with RaggedShard format and structure-aware planning. Achieves 5-66% higher throughput and 16-30% lower memory usage, scaling to tens of thousands of GPUs.
* `training` `networking` `agentic` [Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents](http://arxiv.org/abs/2602.23556v1)
  > **TL;DR**: Addresses inefficient communication in distributed GNN training via adaptive prefetching of remote nodes. Proposes Rudder, an LLM agent using in-context learning for dynamic prefetch decisions without pretraining. Achieves up to 91% training speedup and 50% communication reduction over baseline.

### 2026-02-27
* `storage` `RAG` `recommendation` [STELLAR: Storage Tuning Engine Leveraging LLM Autonomous Reasoning for High Performance Parallel File Systems](http://arxiv.org/abs/2602.23220v1)
  > **TL;DR**: Develops STELLAR, an autonomous storage tuner using LLMs for parallel file systems. Combines retrieval-augmented generation, tool execution, and multi-agent design to navigate tuning cycles. Achieves near-optimal parameter configurations within five attempts, reducing exploration cost.
* `serving` `offloading` `disaggregation` [LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure](http://arxiv.org/abs/2602.23036v1)
  > **TL;DR**: Introduces LLMServingSim 2.0, a system-level simulator to model hardware-software interactions in heterogeneous and disaggregated LLM serving. It jointly simulates scheduling, data movement, and interconnects with profile-based hardware modeling. Achieves 0.97% average error in performance metrics compared to real deployments.
* `scaling` `networking` `storage` [Workload Buoyancy: Keeping Apps Afloat by Identifying Shared Resource Bottlenecks](http://arxiv.org/abs/2602.22852v1)
  > **TL;DR**: Introduces buoyancy, a workload performance metric combining application metrics and system-level resource contention insights. Designed for resource-aware orchestration in multi-tenant systems to diagnose bottlenecks. Achieves 19.3% better bottleneck indication versus traditional heuristics.
* `edge` `serving` `caching` [Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching](http://arxiv.org/abs/2602.22812v1)
  > **TL;DR**: Accelerates LLM inference on edge devices via distributed prompt caching with partial matching and Bloom-filter-based catalog to reduce state-sharing overhead. Reduces TTFT by 93.12% and TTLT by 50.07% on Raspberry Pi with Gemma-3 270M model.
* `training` `networking` [Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study](http://arxiv.org/abs/2602.22760v1)
  > **TL;DR**: Investigates energy-efficient LLM pretraining during renewable energy curtailment windows. Proposes a geo-distributed federated training system that switches between local and multi-site synchronization to match clean energy availability. Reduces operational emissions to 5-12% of baselines while maintaining model quality.
* `training` `networking` `scaling` [Dynamic Hierarchical Birkhoff-von Neumann Decomposition for All-to-All GPU Communication](http://arxiv.org/abs/2602.22756v1)
  > **TL;DR**: Proposes a dynamic hierarchical Birkhoff-von Neumann decomposition to optimize all-to-all communication in GPU clusters. Balances intra-server traffic and applies hierarchical decomposition to reduce scheduling complexity. Reduces mean frame length by up to 2.7× under hotspot traffic.
* `training` `RL` `hardware` [RLHFless: Serverless Computing for Efficient RLHF](http://arxiv.org/abs/2602.22718v1)
  > **TL;DR**: Proposes RLHFless, a serverless computing framework for efficient synchronous RLHF training. Adapts to dynamic resource demands with prefix pre-computation, cost-aware actor scaling, and workload balancing. Achieves up to 1.35x speedup and 44.8% cost reduction versus state-of-the-art.
* `RL` `training` `storage` [Tackling Privacy Heterogeneity in Differentially Private Federated Learning](http://arxiv.org/abs/2602.22633v1)
  > **TL;DR**: Proposes a privacy-aware client selection strategy for differentially private federated learning in heterogeneous privacy settings. Formulates selection as a convex optimization problem minimizing training error. Achieves up to 10% higher accuracy on CIFAR-10 compared to baselines.
* `serving` `scaling` [FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving](http://arxiv.org/abs/2602.22593v1)
  > **TL;DR**: Addresses how to dynamically switch parallelism configurations in LLM serving to adapt to changing workloads. Proposes Flying Serving with virtualized model weight views, KV cache preservation, pre-initialized communicators, and a deadlock-free scheduler. Achieves up to 4.79× throughput improvement under high load.
* `serving` `offloading` `disaggregation` [DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference](http://arxiv.org/abs/2602.21548v2)
  > **TL;DR**: Addresses KV-Cache storage I/O bottleneck in agentic LLM inference. Proposes DualPath system with storage-to-decode KV-Cache loading and RDMA transfer to prefill engines, plus global scheduler. Achieves up to 1.87× offline throughput and 1.96× online serving throughput without SLO violations.
* `training` `networking` `scaling` [CCCL: Node-Spanning GPU Collectives with CXL Memory Pooling](http://arxiv.org/abs/2602.22457v1)
  > **TL;DR**: Proposes CCCL, a collective communication library that uses CXL shared memory pools for GPU operations across nodes. It addresses synchronization, data interleaving, and parallelization, achieving up to 1.94× speed for collectives and 1.11× training speedup with 2.75× cost savings versus InfiniBand.
* `training` `scaling` `quantization` [veScale-FSDP: Flexible and High-Performance FSDP at Scale](http://arxiv.org/abs/2602.22437v1)
  > **TL;DR**: Addresses inefficiencies in FSDP systems for block-wise quantized training and non-element-wise optimizers at scale. Proposes veScale-FSDP with RaggedShard and structure-aware planning for flexible sharding and improved efficiency. Achieves 5-66% higher throughput and 16-30% lower memory usage versus existing FSDP systems.
* `storage` `training` `networking` [GetBatch: Distributed Multi-Object Retrieval for ML Data Loading](http://arxiv.org/abs/2602.22434v1)
  > **TL;DR**: Addresses high per-request overhead in distributed ML data loading. Proposes GetBatch API for batch retrieval as a single deterministic, fault-tolerant operation. Achieves 15x throughput improvement and 3.7x reduction in per-object tail latency.
* `agentic` `offloading` `storage` [Contextual Memory Virtualisation: DAG-Based State Management and Structurally Lossless Trimming for LLM Agents](http://arxiv.org/abs/2602.22402v1)
  > **TL;DR**: Proposes Contextual Memory Virtualisation (CMV), a DAG-based state management system with lossless trimming for LLM agents. Uses version-controlled snapshots to enable context reuse and a three-pass trimming algorithm for session compaction. Achieves 20% to 86% token reduction in real-world coding sessions.

### 2026-02-26
* `training` `storage` [LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models](http://arxiv.org/abs/2602.22158v1)
  > **TL;DR**: Proposes LLMTailor for efficient checkpointing in LLM training by selectively saving only layers with significant updates, reducing storage and time costs. Achieves 4.3x smaller checkpoint size for Llama3.1-8B and 2.8x faster checkpoint time for Qwen2.5-7B.
* `kernel` `training` [PASTA: A Modular Program Analysis Tool Framework for Accelerators](http://arxiv.org/abs/2602.22103v1)
  > **TL;DR**: Introduces PASTA, a low-overhead modular framework for program analysis on GPU accelerators. Abstracts profiling APIs and DL frameworks for runtime event capture. Achieves up to 1.3e4x faster analysis than conventional tools through GPU acceleration.
* `RAG` `offline` `serving` [IOAgent: Democratizing Trustworthy HPC I/O Performance Diagnosis Capability via LLMs](http://arxiv.org/abs/2602.22017v1)
  > **TL;DR**: Develops IOAgent, an AI tool using RAG and tree-based methods for automated HPC I/O performance diagnosis from Darshan traces. Introduces TraceBench for evaluation. Matches or exceeds state-of-the-art tools in accuracy, functioning effectively with diverse LLMs.
* `serving` `kernel` `hardware` [A task-based data-flow methodology for programming heterogeneous systems with multiple accelerator APIs](http://arxiv.org/abs/2602.21897v1)
  > **TL;DR**: Proposes a task-based data-flow methodology with Task-Aware APIs to unify multiple accelerator programming models. Integrates CUDA, SYCL, Triton via OpenMP/OmpSs-2 runtime and nOS-V threading library, reducing thread contention and enabling efficient kernel execution. Demonstrates improved multi-accelerator utilization with reduced performance variability.
* `training` `scaling` `multi-modal` [DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism](http://arxiv.org/abs/2602.21788v1)
  > **TL;DR**: Proposes Dynamic Hybrid Parallelism (DHP) to adaptively reconfigure communication groups/parallelism degrees for efficient MLLM training under heterogeneous data. Achieves 1.36× training throughput speedup over Megatron-LM/DeepSpeed with near-linear scaling on NPU clusters.
* `serving` `MoE` `scaling` [Multi-Layer Scheduling for MoE-Based LLM Reasoning](http://arxiv.org/abs/2602.21626v1)
  > **TL;DR**: Proposes a multi-layer scheduling framework for MoE-based LLM serving, addressing request-level, engine-level, and expert-level optimization. Designs algorithms for request prioritization, load-aware dispatching, and expert routing to reduce latency. Achieves 17.8% TTFT reduction and 13.3% TPOT reduction over vLLM.
* `offloading` `disaggregation` `networking` [DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference](http://arxiv.org/abs/2602.21548v1)
  > **TL;DR**: Addresses the KV-Cache storage I/O bottleneck in agentic LLM inference. Proposes DualPath with dual-path KV-Cache loading via RDMA over compute network and dynamic scheduling. Achieves 1.87× higher offline throughput and 1.96× online serving throughput without SLO violation.
* `serving` `disaggregation` `offloading` [BiScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS](http://arxiv.org/abs/2602.18755v2)
  > **TL;DR**: Proposes BiScale, a two-tier energy optimization framework for disaggregated LLM serving that jointly optimizes placement and DVFS across prefill/decode phases. Uses predictive models, phase-aware placement, MPC for prefill, and slack-aware adaptation for decode. Reduces energy by up to 39% in prefill and 48% in decode.

### 2026-02-25
* `training` `sparse` `offloading` [Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking](http://arxiv.org/abs/2602.21196v1)
  > **TL;DR**: Proposes UPipe, a context parallelism technique using headwise chunking to reduce activation memory in self-attention layers. Achieves up to 87.5% reduction in activation memory for 32B Transformers, supporting 5M token context length on an 8xH100 node.
* `inference` `kernel` `quantization` [Scaling State-Space Models on Multiple GPUs with Tensor Parallelism](http://arxiv.org/abs/2602.21144v1)
  > **TL;DR**: Addresses scaling inference for state-space models (SSMs) across GPUs. Proposes communication-efficient tensor parallelism with SSM state cache, optimized partitioning, and quantized AllReduce. Achieves 1.6-4x throughput gains on 2-4 GPUs and 10-18% improvement via quantization.
* `serving` `MoE` `disaggregation` [ReviveMoE: Fast Recovery for Hardware Failures in Large-Scale MoE LLM Inference Deployments](http://arxiv.org/abs/2602.21140v1)
  > **TL;DR**: Proposes ReviveMoE for rapid failure recovery in large-scale MoE LLM inference without restarting instances. Supports collocated and disaggregated (MoE/attention separation) architectures via xDeepServe and XCCL. Reduces recovery latency by avoiding weight reloading and recompilation, enabling sub-second restoration.
* `training` `storage` `scaling` [GPU Memory and Utilization Estimation for Training-Aware Resource Management: Opportunities and Limitations](http://arxiv.org/abs/2602.17817v2)
  > **TL;DR**: Investigates GPU memory and utilization estimation for robust collocation of deep learning training tasks. Evaluates three paradigms—analytical models, CPU-side libraries, and ML-based estimators—in accuracy, generalizability, and overhead. Demonstrates memory prediction errors below 5% for unseen models.
* `training` `networking` [Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training](http://arxiv.org/abs/2602.20656v1)
  > **TL;DR**: Addresses performance bottleneck in distributed LLM training due to computation-communication imbalance. Proposes Lagom, which co-tunes communication parameters via a unified cost model and priority-based search. Achieves 1.03-1.33x speedup over NVIDIA NCCL.

### 2026-02-24
* `training` `RL` `scaling` [A Formal Framework for Predicting Distributed System Performance under Faults](http://arxiv.org/abs/2602.19088v1)
  > **TL;DR**: Introduces a formal framework for predicting performance of distributed systems under faults. Combines system models with fault injectors using statistical analysis to estimate latency and throughput. Accurately predicts performance in varied fault scenarios, matching real deployments.
* `serving` `networking` `offloading` [WANSpec: Leveraging Global Compute Capacity for LLM Inference](http://arxiv.org/abs/2602.18931v1)
  > **TL;DR**: Addresses latency in global LLM inference due to uneven GPU utilization. Proposes WANSpec, which offloads draft model execution to under-utilized data centers via speculative decoding. Reduces draft model forward passes in high-demand data centers by over 50% without increasing latency.
* `edge` `offloading` `networking` [Carbon-aware decentralized dynamic task offloading in MIMO-MEC networks via multi-agent reinforcement learning](http://arxiv.org/abs/2602.18797v1)
  > **TL;DR**: Proposes CADDTO-PPO, a carbon-aware multi-agent reinforcement learning framework for dynamic task offloading in MIMO-MEC networks. Uses decentralized PPO with parameter sharing for local power/offloading decisions and adaptive prioritization of green energy slots. Achieves lowest carbon intensity and near-zero packet overflow with O(1) inference overhead.
* `disaggregation` `scaling` `serving` [BiScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS](http://arxiv.org/abs/2602.18755v1)
  > **TL;DR**: Proposes BiScale, a two-tier framework for energy-efficient disaggregated LLM serving. Combines phase-aware placement with hierarchical DVFS control: MPC for prefill, slack-aware adaptation for decode. Reduces prefill energy by 39% and decode by 48% vs DistServe while meeting SLOs.

### 2026-02-23
* `scaling` `edge` `networking` [Green by Design: Constraint-Based Adaptive Deployment in the Cloud Continuum](http://arxiv.org/abs/2602.18287v1)
  > **TL;DR**: Proposes constraint-based adaptive deployment for cloud-edge continuum to reduce energy and emissions. Uses dynamically learned green-aware constraints for scheduling. Achieves significant reduction in energy consumption and greenhouse gas emissions in realistic cloud-native applications.
* `training` `networking` [Joint Training on AMD and NVIDIA GPUs](http://arxiv.org/abs/2602.18007v1)
  > **TL;DR**: Addresses training efficiency in mixed AMD-NVIDIA GPU clusters. Proposes CPU-Forwarding Communication and Device-Direct Communication with CPU-offloading P2P to enable direct cross-vendor transfers. Achieves 98% throughput of homogeneous NVIDIA systems on LLaMA-8B and Qwen2-7B.
* `training` `scaling` `storage` [GPU Memory and Utilization Estimation for Training-Aware Resource Management: Opportunities and Limitations](http://arxiv.org/abs/2602.17817v1)
  > **TL;DR**: Studies GPU memory and utilization estimation for collocating DL training tasks to improve resource efficiency. Analyzes analytical, library-based, and ML estimators for accuracy and overhead. Evaluates tradeoffs on synthetic datasets with MLPs, CNNs, and Transformers, revealing limitations in generalization and hardware dependence.
* `edge` `offloading` [Collaborative Processing for Multi-Tenant Inference on Memory-Constrained Edge TPUs](http://arxiv.org/abs/2602.17808v1)
  > **TL;DR**: Addresses high latency from model swapping for memory-constrained Edge TPUs in multi-tenant inference. Proposes SwapLess, an adaptive system using queueing models to optimize CPU-TPU partitioning and core allocation. Reduces mean latency by up to 77.4% for multi-tenant workloads.
* `serving` `offline` `storage` [Closing Africa's Early Warning Gap: AI Weather Forecasting for Disaster Prevention](http://arxiv.org/abs/2602.17726v1)
  > **TL;DR**: Proposes a GPU-based AI weather forecasting system with PostgreSQL caching for low-latency queries. Implements event loop isolation, direct database writes, and coordinate management to eliminate transfer bottlenecks and enable regional deployment at 2,000-4,545x lower cost than radar infrastructure.
* `serving` `scaling` `RL` [TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs](http://arxiv.org/abs/2602.18109v1)
  > **TL;DR**: Develops TempoNet, a reinforcement learning scheduler with sparse attention transformers for real-time dispatch. Uses slack quantization and latency-aware attention for near-linear scaling and sub-millisecond inference. Achieves consistent deadline fulfillment gains over baselines in multi-processor settings.

### 2026-02-20
* `edge` `scaling` [Visual Insights into Agentic Optimization of Pervasive Stream Processing Services](http://arxiv.org/abs/2602.17282v1)
  > **TL;DR**: Addresses dynamic scaling for stream processing services on edge devices. Proposes an agent-driven platform that learns service-specific scaling policies to adapt to fluctuating demands. Achieves low latency by optimizing resource usage without service cannibalization.
* `training` `scaling` `networking` [Trivance: Latency-Optimal AllReduce by Shortcutting Multiport Networks](http://arxiv.org/abs/2602.17254v1)
  > **TL;DR**: Proposes Trivance, a latency-optimal AllReduce algorithm exploiting bidirectional rings to reduce congestion and steps compared to Bruck's. Applies to torus networks. Improves performance by 5-30% for messages up to 128MiB.
* `training` `LoRA` `RL` [Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation](http://arxiv.org/abs/2602.16936v1)
  > **TL;DR**: Addresses heterogeneous federated fine-tuning challenges. Proposes Fed-PLoRA with Parallel One-Rank Adaptation and Select-N-Fold strategy to reduce initialization and aggregation noise. Improves accuracy by up to 10.3% on GLUE benchmarks over baselines.
* `serving` `offloading` `networking` [Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks](http://arxiv.org/abs/2602.16760v1)
  > **TL;DR**: Proposes privacy-aware split LLM inference with intermediate activation offloading to untrusted cloud over WAN. Introduces asymmetric layer split and lookahead decoding to amortize network latency, achieving 8.7-9.3 tok/s for Mistral 7B on ~80ms link with formal zero quality degradation.
* `offline` `inference systems specific techniques` `storage` [The Compute ICE-AGE: Invariant Compute Envelope under Addressable Graph Evolution](http://arxiv.org/abs/2602.16736v1)
  > **TL;DR**: Proposes a deterministic graph-based substrate for AI inference with invariant compute cost under scaling. Uses bounded local state evolution instead of probabilistic recomposition. Achieves invariant traversal latency ~0.25-0.32ms and stable CPU utilization across 1M-25M node scales.

### 2026-02-19
* `serving` `offloading` `scaling` [FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving](http://arxiv.org/abs/2602.16603v1)
  > **TL;DR**: Addresses head-of-line blocking in LLM serving during prefill by decoupling preemption granularity from scheduling frequency. Introduces operator-level preemption and event-driven scheduling to enable fine-grained interruption with minimal overhead. Achieves up to 5.6× higher maximum goodput.
* `edge` `serving` `inference` [How Reliable is Your Service at the Extreme Edge? Analytical Modeling of Computational Reliability](http://arxiv.org/abs/2602.16362v1)
  > **TL;DR**: Proposes an analytical framework for computational reliability in edge-based distributed inference systems. Models device volatility and derives reliability expressions for different workload configurations. Validations show close agreement (within 5%) between predictions and empirical measurements in emulated environments.
* `serving` `scaling` `offloading` [LLM-Driven Intent-Based Privacy-Aware Orchestration Across the Cloud-Edge Continuum](http://arxiv.org/abs/2602.16100v1)
  > **TL;DR**: Proposes dynamic pipeline reconfiguration for LLM serving in heterogeneous serverless environments. Uses online adjustment of pipeline configurations to adapt to changing workloads with minimal downtime. Achieves <50 ms downtime and <10% overhead on TTFT and TPOT during migration.
* `kernel` `storage` `training` [Distributed Order Recording Techniques for Efficient Record-and-Replay of Multi-threaded Programs](http://arxiv.org/abs/2602.15995v1)
  > **TL;DR**: Proposes Distributed Clock (DC) and Distributed Epoch (DE) schemes for efficient record-and-replay of OpenMP programs. Integrates techniques into ReOMP to reduce synchronization on shared-memory accesses. Achieves 2-5x efficiency improvement over traditional methods and integrates with MPI-level replay tools.

### 2026-02-18
* `edge` `kernel` [FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations](http://arxiv.org/abs/2602.15379v1)
  > **TL;DR**: Proposes FlashMem, a memory streaming framework for mobile GPUs that statically schedules and dynamically streams model weights using 2.5D texture memory. Reduces memory consumption by 2.0–8.4× and achieves speedup of 1.7–75.0× for on-device DNN inference.
* `training` `networking` `scaling` [Co-Design and Evaluation of a CPU-Free MPI GPU Communication Abstraction and Implementation](http://arxiv.org/abs/2602.15356v1)
  > **TL;DR**: Proposes an MPI-based GPU communication API for CPU-free, high-performance messaging in ML/HPC. Utilizes HPE Slingshot network capabilities for direct GPU communication. Achieves 50% lower medium message latency and 28% speedup in halo-exchange scaling to 8,192 GPUs.

### 2026-02-17
* `agentic` `RL` [Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows](http://arxiv.org/abs/2602.14849v1)
  > **TL;DR**: Addresses unreliable side effects from agent tool calls in LLM workflows. Introduces Atomix, a runtime with epoch-based transactional semantics and progress-aware commit. Improves task success by 10-30% under failures via transactional retry and compensation.
* `serving` `disaggregation` `scaling` [Efficient Multi-round LLM Inference over Disaggregated Serving](http://arxiv.org/abs/2602.14516v1)
  > **TL;DR**: Addresses performance issues in multi-round LLM inference under prefill-decode disaggregation. Proposes AMPD, a framework that adaptively schedules prefill workloads and optimizes resource allocation. Achieves significantly higher SLO attainment than state-of-the-art baselines.
* `edge` `RL` `serving` [Floe: Federated Specialization for Real-Time LLM-SLM Inference](http://arxiv.org/abs/2602.14302v1)
  > **TL;DR**: Addresses high latency and privacy in LLM edge inference. Proposes Floe, a federated framework combining cloud LLM with edge SLMs via heterogeneity-aware LoRA and logit-level fusion. Reduces inference latency by 35% on edge devices while maintaining privacy.
* `sparse` `kernel` `hardware` [Parallel Sparse and Data-Sparse Factorization-based Linear Solvers](http://arxiv.org/abs/2602.14289v1)
  > **TL;DR**: Addresses efficient large-scale linear solvers for ill-conditioned systems. Reviews advances in parallel sparse direct solvers by reducing communication costs and computational complexity via low-rank compression. Delivers high speed and reliability on heterogeneous hardware for multiphysics and ML applications.
* `LoRA` `multi-modal` `edge` [ML-ECS: A Collaborative Multimodal Learning Framework for Edge-Cloud Synergies](http://arxiv.org/abs/2602.14107v1)
  > **TL;DR**: Proposes ML-ECS, a collaborative multimodal learning framework for edge-cloud environments, leveraging low-rank LoRA parameters and fused representations for efficient communication. Achieves 5.44-12.08% Rouge-LSum improvement with 0.65% parameter communication volume.
* `training` `scaling` `networking` [TEG: Exascale Cluster Governance via Non-Equilibrium Thermodynamics and Langevin Dynamics](http://arxiv.org/abs/2602.13789v1)
  > **TL;DR**: Proposes TEG, a decentralized cluster governance using non-equilibrium thermodynamics and Langevin dynamics to manage exascale AI workloads. Replaces centralized scheduler with Langevin agents for O(1) decision complexity and Landau phase transitions for deadlock resolution. Achieves asymptotic Nash convergence.
* `RL` `LoRA` `training` [Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity](http://arxiv.org/abs/2602.13486v1)
  > **TL;DR**: Investigates rank collapse in federated LoRA due to client heterogeneity. Proposes raFLoRA, a rank-partitioned aggregation method that weights client contributions per rank partition. Achieves up to 18% accuracy gain compared to FedLoRA baselines.
* `agentic` `serving` `storage` [ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System](http://arxiv.org/abs/2602.13692v1)
  > **TL;DR**: Proposes ThunderAgent, a program-aware agentic inference system for multi-turn LLM workflows. Introduces unified resource abstraction and program-aware scheduler to optimize KV cache hits, memory imbalances, and tool preparation. Achieves 1.5-3.6x throughput improvement in serving and up to 4.2x disk memory savings.

### 2026-02-16
* `storage` `networking` `scaling` [Bloom Filter Look-Up Tables for Private and Secure Distributed Databases in Web3 (Revised Version)](http://arxiv.org/abs/2602.13167v1)
  > **TL;DR**: Proposes a decentralized database scheme using Bloom Filters (BFLUT) with OrbitDB/IPFS/IPNS for secure cryptographic key management in Web3. Encodes/distributes keys to prevent exposure even if nodes are compromised. Reduces unauthorized access risks and achieves high performance at scale.
* `kernel` `RL` `training` [OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization](http://arxiv.org/abs/2602.12305v1)
  > **TL;DR**: Proposes OptiML, an end-to-end framework for optimizing CUDA kernels for LLMs. Uses Mixture-of-Thoughts generation and Monte Carlo Tree Search with hardware-aware rewards to navigate transformation space. Achieves verified performance improvements over LLM baselines with interpretable trajectories.

### 2026-02-14
* `serving` `scaling` [OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration](http://arxiv.org/abs/2602.12151v1)
  > **TL;DR**: Addresses spatial-temporal workload heterogeneity in LLM serving. Proposes OServe with workload-aware scheduling and adaptive model deployment switching. Achieves up to 2× performance improvement over state-of-the-art systems.
* `serving` `disaggregation` `RAG` [PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving](http://arxiv.org/abs/2602.12029v1)
  > **TL;DR**: Addresses redundant KV cache generation in multi-LLM agent systems. Proposes PrefillShare, a technique to share frozen prefill modules and KV caches across models via factorization and fine-tuning, with routing in disaggregated serving. Achieves 4.5× lower p95 latency and 3.9× higher throughput.
* `offloading` `edge` `networking` [An Auction-Based Mechanism for Optimal Task Allocation and Resource Aware Containerization](http://arxiv.org/abs/2602.11998v1)
  > **TL;DR**: Proposes an auction-based mechanism for optimal task offloading and containerization in IoT. Uses Docker Swarm with Manager and Worker nodes for distributed computation. Achieves improved offloading efficiency and resource optimization for computation-intensive tasks.
* `serving` `networking` `scaling` [GORGO: Maximizing KV-Cache Reuse While Minimizing Network Latency in Cross-Region LLM Load Balancing](http://arxiv.org/abs/2602.11688v1)
  > **TL;DR**: Optimizes LLM serving across regions to minimize Time-to-First-Token by balancing KV-cache reuse and network latency. Proposes GORGO, a centralized proxy with network-aware routing that tracks request prefixes and cluster states. Achieves 2.5x faster median TTFT and reduces P99 TTFT compared to baselines.
* `MoE` `training` `scaling` [LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training](http://arxiv.org/abs/2602.11686v1)
  > **TL;DR**: Addresses load imbalance in Mixture-of-Experts (MoE) training due to dynamic routing. Proposes LAER-MoE with Fully Sharded Expert Parallel (FSEP) for parameter partitioning and expert re-layout, plus fine-grained communication scheduling and load balancing planning. Achieves up to 1.69× training acceleration vs. state-of-the-art systems.
* `LoRA` `edge` `training` [LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detection](http://arxiv.org/abs/2602.11655v1)
  > **TL;DR**: Proposes a continuous learning system for edge-based malware detection using LoRA adapters for parameter-efficient fine-tuning and knowledge sharing. Aggregates only LoRA modules across devices for cross-device generalization, achieving 20-25% accuracy gains on unseen attacks with minimal model size increase (~0.6-1.8MB).
* `serving` `offloading` `networking` [Efficient Remote Prefix Fetching with GPU-native Media ASICs](http://arxiv.org/abs/2602.09725v2)
  > **TL;DR**: Addresses performance degradation of remote KV cache reuse in bandwidth-limited networks. Proposes KVFetcher, a system using GPU-native video codecs for efficient transmission and pipelined decoding. Achieves up to 3.51× reduction in time-to-first-token while preserving accuracy.
* `serving` `kernel` `offloading` [PAM: Processing Across Memory Hierarchy for Efficient KV-centric LLM Serving System](http://arxiv.org/abs/2602.11521v1)
  > **TL;DR**: Addresses KV cache memory bottlenecks in LLM serving by proposing PAM, a hierarchical memory system with PIM devices. Introduces context-locality-based KV token distribution, cross-device PAMattention, and dynamic workload balancing. Achieves 1.6-4.8× higher throughput than state-of-the-art systems.
* `training` `RL` `networking` [RL over Commodity Networks: Overcoming the Bandwidth Barrier with Lossless Sparse Deltas](http://arxiv.org/abs/2602.11456v1)
  > **TL;DR**: Addresses slow synchronization in distributed RL fine-tuning over commodity networks by leveraging sparsity of parameter updates. Proposes SparrowRL with sparse delta checkpoints, pipelined streaming, and overlapping transfer to reduce payload by 79× for Qwen3-8B, improving throughput by 2.4-9.5× and narrowing RDMA gap to 8.91%.

### 2026-02-12
* `RL` `kernel` `training` [Fine-Tuning GPT-5 for GPU Kernel Generation](http://arxiv.org/abs/2602.11000v1)
  > **TL;DR**: Addresses challenges in generating optimized GPU kernels via LLMs. Develops a reinforcement learning framework (Makora) for fine-tuning GPT-5 on Triton code generation. Achieves 77% correctness (+33.3 pp) and 2.12x geometric mean speedup over TorchInductor on kernel problems.
* `serving` `scaling` [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](http://arxiv.org/abs/2602.10729v1)
  > **TL;DR**: Investigates cost-efficient LLM serving via algorithm-system co-design. BOute uses multi-objective Bayesian optimization to jointly optimize query routing across heterogeneous LLMs and model deployment on heterogeneous GPUs. Achieves up to 157% higher cost-efficiency and reduces serving costs by 38% on average.
* `kernel` [Flash-SD-KDE: Accelerating SD-KDE with Tensor Cores](http://arxiv.org/abs/2602.10378v1)
  > **TL;DR**: Accelerates score-debiased kernel density estimation (SD-KDE) by restructuring computation for Tensor Core optimization. Exposes matrix-multiplication structure to enable GPU acceleration via Tensor Cores. Achieves up to 47× speedup over a GPU baseline and completes 1M-sample task in 2.3 seconds on a single GPU.
* `training` `hardware` `sparse` [Execution-Centric Characterization of FP8 Matrix Cores, Asynchronous Execution, and Structured Sparsity on AMD MI300A](http://arxiv.org/abs/2602.10262v1)
  > **TL;DR**: Characterizes system-level performance of AMD MI300A accelerator features (FP8, asynchronous engines, structured sparsity) for HPC-AI workloads. Uses microbenchmarks to quantify occupancy thresholds, concurrency trade-offs, and sparsity benefits. Provides guidance achieving up to 2× speedup in transformer kernels via optimized scheduling.

### 2026-02-11
* `serving` `offloading` `networking` [Efficient Remote Prefix Fetching with GPU-native Media ASICs](http://arxiv.org/abs/2602.09725v1)
  > **TL;DR**: Proposes KVFetcher for remote KV cache reuse in LLM inference using GPU-native video codecs. Introduces codec-friendly tensor layout and efficient pipelining for transmission/decompression. Reduces TTFT by up to 3.51× in bandwidth-limited networks compared to SOTA.
* `training` `networking` `scaling` [Harvest: Adaptive Photonic Switching Schedules for Collective Communication in Scale-up Domains](http://arxiv.org/abs/2602.09188v1)
  > **TL;DR**: Proposes Harvest, a method for synthesizing adaptive photonic switching schedules to optimize collective communication in LLM training. It balances reconfiguration overhead against congestion and propagation delays, reducing AllReduce completion time by up to 40% compared to static interconnects.
* `training` `serving` `scaling` [Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide](http://arxiv.org/abs/2602.09109v1)
  > **TL;DR**: Explores hybrid parallelization strategies for efficient LLM training and inference. Systematically analyzes collective operations, communication-computation overlap, and automated strategy search using cost models. Achieves optimal resource utilization by guiding parallelism selection for dynamic model deployment.
* `offline` `scaling` `agentic` [AgentCgroup: Understanding and Controlling OS Resources of AI Agents](http://arxiv.org/abs/2602.09345v1)
  > **TL;DR**: Analyzes OS-level resource dynamics in sandboxed AI agents and identifies mismatches in existing controls. Proposes AgentCgroup, an eBPF-based hierarchical cgroup controller with in-kernel enforcement and adaptive policies for memory and CPU. Reduces resource waste and improves isolation.

### 2026-02-10
* `training` `quantization` `networking` [DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce](http://arxiv.org/abs/2602.08923v1)
  > **TL;DR**: Addresses the inefficiency of gradient quantization in multi-hop all-reduce for large model training. Proposes DynamiQ, a quantization framework with novel partial sum representation and a fused kernel for decompress-accumulate-recompress. Achieves up to 34.2% faster training while maintaining 99.9% baseline accuracy.
* `serving` `offloading` `hardware` [Equilibria: Fair Multi-Tenant CXL Memory Tiering At Scale](http://arxiv.org/abs/2602.08800v1)
  > **TL;DR**: Proposes Equilibria, an OS framework for fair multi-tenant CXL memory tiering to address resource allocation and interference in datacenter workloads. Achieves up to 52% performance improvement for production workloads over state-of-the-art solutions.
* `serving` `training` `offline` [Evaluating Kubernetes Performance for GenAI Inference: From Automatic Speech Recognition to LLM Summarization](http://arxiv.org/abs/2602.04900v2)
  > **TL;DR**: Evaluates Kubernetes for GenAI inference workflows, combining Kueue for batch inference job scheduling and Dynamic Accelerator Slicer for parallelism. Proposes GAIE-based optimized routing for online LLM summarization. Achieves 15% lower makespan, 36% faster job completion, 82% improved time to first token.
* `serving` `scaling` [PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping](http://arxiv.org/abs/2602.08747v1)
  > **TL;DR**: Addresses the inefficiency in goodput for DNN inference pipelines under latency constraints. Proposes PARD system with proactive request dropping using runtime pipeline information and adaptive prioritization. Achieves 16%-176% higher goodput and reduces drop rates 1.6-17x versus state-of-the-art.
* `training` `scaling` [Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research](http://arxiv.org/abs/2602.08387v1)
  > **TL;DR**: Proposes Modalities, a PyTorch-native framework for efficient LLM training and ablation studies. Integrates parallelization strategies for large-scale pretraining and configurable experiments with declarative modular design. Enables trillion-token scale training and systematic ablations with reduced overhead.
* `offline` `agentic` `disaggregation` [Fork, Explore, Commit: OS Primitives for Agentic Exploration](http://arxiv.org/abs/2602.08199v1)
  > **TL;DR**: Addresses the need for isolated environments in AI agent exploration with atomic commit/rollback. Proposes branch contexts with BranchFS (filesystem isolation) and branch() syscall (process isolation). Achieves sub-350 μs branch creation time independent of file system size.
* `offloading` `edge` `serving` [Accuracy-Delay Trade-Off in LLM Offloading via Token-Level Uncertainty](http://arxiv.org/abs/2602.07958v1)
  > **TL;DR**: Proposes token-level uncertainty-aware offloading for LLM inference in mobile edge computing. Designs greedy offloading algorithm (GOA) that dynamically decides local vs. edge execution based on uncertainty metrics. GOA reduces latency by 20-40% while maintaining accuracy in multi-user scenarios.
* `serving` `hardware` `scaling` [Equilibria: Fair Multi-Tenant CXL Memory Tiering At Scale](http://arxiv.org/abs/2602.08800v1)
  > **TL;DR**: Addresses memory tiering fairness and interference in multi-tenant CXL environments for hyperscaler workloads. Introduces Equilibria with per-container controls and regulated promotion/demotion policies. Achieves up to 52% higher performance over baseline in production workloads.
* `agentic` `RL` `offline` [Fork, Explore, Commit: OS Primitives for Agentic Exploration](http://arxiv.org/abs/2602.08199v1)
  > **TL;DR**: Proposes branch contexts, a new OS abstraction for agentic exploration, enabling isolated copy-on-write environments for filesystem and process state with atomic commit/rollback. Implements BranchFS for workspace isolation and a syscall for process management. Achieves sub-350 μs branch creation and under 1 ms commit for small changes.

### 2026-02-10
* `serving` `kernel` [Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization](http://arxiv.org/abs/2602.07306v1)
  > **TL;DR**: Proposes Parallel Track Transformers to reduce synchronization overhead in multi-GPU LLM inference. Restructures computation to minimize cross-device dependencies, achieving a 16x sync reduction. Integrated into serving stacks shows 15-30% lower TTFT and up to 31.9% higher throughput.
* `edge` `multi-modal` `serving` [Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks](http://arxiv.org/abs/2602.07215v1)
  > **TL;DR**: Proposes a multi-agentic AI framework for latency- and fairness-aware multi-modal LM inference in mobile edge networks. Uses coordinated agents for prompt routing and deployment, leveraging natural language reasoning. Achieves 80% reduction in average latency and 0.90 fairness index on a city-wide testbed.

### 2026-02-09
* `serving` `RL` `scaling` [Reinforcement Learning-Based Dynamic Management of Structured Parallel Farm Skeletons on Serverless Platforms](http://arxiv.org/abs/2602.06555v1)
  > **TL;DR**: Proposes AI-driven dynamic scaling for serverless farm pattern execution to improve QoS and resource usage. Uses reinforcement learning policies coupled with monitoring for autoscaling worker pools on OpenFaaS. Achieves better QoS and stable scaling compared to model-based approaches.
* `serving` `scaling` [DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving](http://arxiv.org/abs/2602.06502v1)
  > **TL;DR**: Addresses the conflict between KV cache affinity and load balancing in distributed LLM serving. Proposes DualMap, a dual-mapping scheduler with SLO-aware routing, hotspot rebalancing, and dual-hash-ring scaling. Achieves 2.25× higher request capacity under TTFT SLO constraints.
* `training` `offloading` `scaling` [FCDP: Fully Cached Data Parallel for Communication-Avoiding Large-Scale Training](http://arxiv.org/abs/2602.06499v1)
  > **TL;DR**: Addresses communication bottlenecks in large-scale model training with limited bandwidth. Proposes FCDP, a data parallel approach caching forward-pass parameters in host memory to reduce inter-node communication. Achieves up to 100x higher throughput than ZeRO-3 in commodity clusters.
* `training` `sparse` [TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism](http://arxiv.org/abs/2602.05754v2)
  > **TL;DR**: Addresses pipeline bubble overhead in training by adaptive parameter freezing. Introduces TimelyFreeze, which models pipeline as DAG and uses linear programming to optimize freeze ratios under accuracy constraints. Achieves 40% throughput gain on LLaMA-8B without accuracy loss.
* `training` `sparse` `optimization` [Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers](http://arxiv.org/abs/2602.06079v1)
  > **TL;DR**: Proposes Canzona, a unified asynchronous framework for distributed matrix-based optimizers to eliminate redundancy in LLM training. Features alpha-Balanced Static Partitioning and Asynchronous Compute with Micro-Group Scheduling. Achieves 1.57x end-to-end speedup and 5.8x optimizer latency reduction on 256 GPUs.
* `serving` `kernel` `offloading` [PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference](http://arxiv.org/abs/2602.06072v1)
  > **TL;DR**: Addresses attention inefficiency in batched LLM inference with heterogeneous sequence lengths. Proposes PackInfer, a kernel-level framework that packs requests into balanced groups and reorganizes KV cache layouts. Improves throughput by 20% over FlashAttention and reduces latency by 13.0-20.1%.
* `kernel` `sparse` [FlashSketch: Sketch-Kernel Co-Design for Fast Sparse Sketching on GPUs](http://arxiv.org/abs/2602.06071v1)
  > **TL;DR**: Addresses inefficiency of sparse sketching on GPUs due to irregular memory access. Proposes BlockPerm-SJLT sketch design and FlashSketch kernel for GPU optimization, with tunable sparsity for efficiency-robustness trade-off. Achieves 1.7x geomean speedup over prior GPU sketches.
* `edge` `quantization` `sparse` [HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference](http://arxiv.org/abs/2602.06069v1)
  > **TL;DR**: Develops HQP, a hybrid quantization-pruning framework for ultra-low-latency edge AI. Uses sensitivity-aware pruning with FIM approximation and 8-bit quantization under strict accuracy constraints. Achieves 3.12× speedup and 55% model size reduction on Jetson platforms with <1.5% accuracy drop.

### 2026-02-06
* `training` `pipeline` [TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism](http://arxiv.org/abs/2602.05754v1)
  > **TL;DR**: Addresses pipeline bubble overhead in distributed training throughput via adaptive parameter freezing. Introduces TimelyFreeze, modeling the pipeline as a DAG and using linear programming to optimize freeze ratios under accuracy constraints. Achieves 40% training throughput gain on LLaMA-8B without accuracy loss.
* `thinking` `RL` `scaling` [ORACL: Optimized Reasoning for Autoscaling via Chain of Thought with LLMs for Microservices](http://arxiv.org/abs/2602.05292v1)
  > **TL;DR**: Investigates using LLMs as adaptive resource allocators for autoscaling microservices. Proposes ORACL framework with chain-of-thought reasoning on telemetry data to diagnose issues and recommend allocations. Achieves 15% higher root-cause accuracy and 24x faster training.
* `training` `offloading` [Horizon-LM: A RAM-Centric Architecture for LLM Training](http://arxiv.org/abs/2602.04816v2)
  > **TL;DR**: Proposes Horizon-LM, a RAM-centric LLM training architecture that uses CPU as primary parameter store and GPUs as transient compute engines. Eliminates persistent GPU modules and autograd graphs, employing manual recomputation. Achieves 12.2× higher throughput than DeepSpeed ZeRO-3 on single A100 with predictable memory scaling.
* `offline` `kernel` `scaling` [From Sequential to Parallel: Reformulating Dynamic Programming as GPU Kernels for Large-Scale Stochastic Combinatorial Optimization](http://arxiv.org/abs/2602.05179v1)
  > **TL;DR**: Addresses computational bottlenecks in stochastic programming by batching scenarios. Develops parallel GPU kernels for dynamic programming, exposing parallelism across scenarios and layers. Achieves 4-5 orders of magnitude speedup, enabling large-scale exact integer second-stage models.
* `serving` `offline` `scaling` [Evaluating Kubernetes Performance for GenAI Inference: From Automatic Speech Recognition to LLM Summarization](http://arxiv.org/abs/2602.04900v1)
  > **TL;DR**: Evaluates Kubernetes for GenAI inference workflows involving ASR and LLM summarization. Uses Kueue for batch job scheduling, DAS for parallel execution, and Gateway API Inference Extension for online serving optimization. Achieves 36% faster job completion, 15% makespan reduction, and 82% improved time to first token.
* `RAG` `serving` `offloading` [ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation](http://arxiv.org/abs/2602.02579v3)
  > **TL;DR**: Addresses KV cache recomputation inefficiency in RAG prefill by introducing query-driven token selection and dual-stage pipeline. ProphetKV prioritizes semantically relevant tokens, achieving 96%-101% accuracy at 20% recomputation ratio and up to 50.9% higher accuracy over prior methods.
* `training` `offloading` [Horizon-LM: A RAM-Centric Architecture for LLM Training](http://arxiv.org/abs/2602.04816v2)
  > **TL;DR**: Addresses memory constraints in large model training by repurposing GPUs as transient compute engines and using CPU-hosted parameters. Introduces Horizon-LM with CPU-master/GPU-worker model, explicit recomputation, and pipelining. Achieves up to 12.2× higher throughput versus DeepSpeed ZeRO-3 on a single A100 GPU.

### 2026-02-05
* `training` `offloading` [Horizon-LM: A RAM-Centric Architecture for LLM Training](http://arxiv.org/abs/2602.04816v1)
  > **TL;DR**: Proposes Horizon-LM, a RAM-centric LLM training system that uses CPUs as persistent parameter store and GPUs as transient compute engines. Employs explicit recomputation, manual gradient propagation, and pipelined double-buffering to decouple model scale from GPU memory. Achieves 12.2× higher throughput than DeepSpeed ZeRO-3 on a single A100.
* `RL` `agentic` [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](http://arxiv.org/abs/2601.21972v2)
  > **TL;DR**: Investigates decentralized LLM collaboration using multi-agent reinforcement learning. Proposes two multi-agent actor-critic methods (CoLLM-CC and CoLLM-DC) to optimize fine-tuning with reduced sample variance. Achieves comparable performance in short-horizon tasks but requires 2× more samples for Monte Carlo approaches in long-horizon settings.
* `training` `offloading` `scaling` [Horizon-LM: A RAM-Centric Architecture for LLM Training](http://arxiv.org/abs/2602.04816v1)
  > **TL;DR**: Proposes Horizon-LM, a memory-centric training architecture using CPUs as persistent parameter stores and GPUs as transient compute engines. Eliminates GPU-resident modules and autograd graphs via explicit recomputation and pipelined execution. Achieves up to 12.2× higher throughput than DeepSpeed ZeRO-3 while enabling 120B parameter training on a single GPU with host RAM.
* `serving` `RAG` `offloading` [ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation](http://arxiv.org/abs/2602.02579v2)
  > **TL;DR**: Addresses inefficient KV cache reuse in RAG prefill stage. Introduces ProphetKV, a query-driven selective recomputation method with dual-stage prioritization, dedicating budget to context-query fusion. Retains 96%-101% accuracy with only 20% recomputation, outperforming state-of-the-art by 8.8%-50.9%.

### 2026-02-04
* `training` `scaling` [Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods](http://arxiv.org/abs/2602.03802v1)
  > **TL;DR**: Investigates if synchronous SGD is optimal for distributed training under heterogeneous compute or adversarial worker participation. Proves that m-Synchronous SGD achieves near-optimal time complexity, challenging the need for asynchronous methods.
* `training` `scaling` `networking` [Improved Analysis of the Accelerated Noisy Power Method with Applications to Decentralized PCA](http://arxiv.org/abs/2602.03682v1)
  > **TL;DR**: Improves convergence analysis of Accelerated Noisy Power Method for decentralized PCA. Provides tighter noise bounds, preserving accelerated rates with milder perturbation conditions. Achieves accelerated convergence with communication costs comparable to non-accelerated methods.
* `training` `scaling` [Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation](http://arxiv.org/abs/2602.03515v1)
  > **TL;DR**: Addresses gradient staleness in asynchronous pipeline parallelism for distributed LLM training. Proposes basis rotation to align Hessian eigenbasis and rectify delayed gradients. Achieves 76.8% fewer iterations for 1B-parameter LLM training compared to baseline.
* `MoE` `offloading` [DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs](http://arxiv.org/abs/2602.03495v1)
  > **TL;DR**: Proposes DALI, a workload-aware offloading framework for efficient MoE inference on local PCs. It features dynamic expert assignment, residual-based prefetching, and workload-aware caching to address load imbalance and inefficiencies. Achieves significant speedups in prefill and decoding phases over existing frameworks.
* `RL` `training` `scaling` [ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning](http://arxiv.org/abs/2602.02192v2)
  > **TL;DR**: Proposes ECHO-2, a distributed RL framework for cost-efficient rollout generation in post-training LLMs. Uses overlapped execution, peer-assisted broadcast, and cost-aware worker activation to alleviate dissemination bottlenecks. Reduces costs by up to 2.1× while maintaining RL rewards comparable to baselines.
* `MoE` `serving` `networking` [PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching](http://arxiv.org/abs/2602.00509v2)
  > **TL;DR**: Addresses MoE inference inefficiencies due to computational stragglers and network congestion. PROBE introduces predictive prefetching and joint optimization of expert replication/token assignment. Reduces prefill latency by 1.32x and improves decoding throughput by 1.26x under dynamic workloads.
* `offloading` `kernel` `training` [HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures](http://arxiv.org/abs/2602.00748v2)
  > **TL;DR**: Proposes HyperOffload, a graph-driven hierarchical memory management framework for LLMs on supernodes. Uses compiler-assisted IR integration and static scheduling to hide remote memory latency. Reduces peak device memory by 26% with maintained performance.
* `serving` `scaling` `disaggregation` [Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control](http://arxiv.org/abs/2602.02987v1)
  > **TL;DR**: Addresses state-dependent contention in LLM inference with prefill and decode phases under heterogeneous workloads. Develops a queueing-based control framework with gate-and-route policies for cross-GPU cluster scheduling. Achieves asymptotically optimal resource allocation, outperforming standard heuristics by empirical evaluation.
* `RAG` `serving` `offloading` [ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation](http://arxiv.org/abs/2602.02579v1)
  > **TL;DR**: ProphetKV optimizes KV cache reuse in RAG prefill by prioritizing user-query-relevant tokens via semantic-driven selection and dual-stage recomputation, retaining 96%–101% accuracy with only 20% recomputation budget, outperforming baselines by 8.8%–50.9% on accuracy metrics.

### 2026-02-03
* `scaling` `networking` `multi-modal` [sVIRGO: A Scalable Virtual Tree Hierarchical Framework for Distributed Systems](http://arxiv.org/abs/2602.02438v1)
  > **TL;DR**: Proposes sVIRGO, a virtual tree hierarchical framework for scalable coordination in distributed systems. Uses configurable layers and dynamic role mapping to support thousands of regions with multiple active coordinators. Achieves near-zero recovery latency and bounded communication overhead under adversarial conditions.
* `inference` `kernel` `video` [Enabling AI Deep Potentials for Ab Initio-quality Molecular Dynamics Simulations in GROMACS](http://arxiv.org/abs/2602.02234v1)
  > **TL;DR**: Integrates AI deep potentials into GROMACS for ab initio-quality MD simulations. Utilizes DeePMD-kit with C++/CUDA backend for multi-model inference. Achieves up to 4.23x higher throughput on NVIDIA A100 GPUs with DPA2 over DPA3 architectures.
* `serving` `disaggregation` `multi-modal` [vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models](http://arxiv.org/abs/2602.02204v1)
  > **TL;DR**: Proposes a disaggregated serving system for any-to-any multimodal models, decomposing architectures into interconnected stages with independent engines. Features unified connectors and per-stage optimization. Reduces job completion time by up to 91.4% over baselines.
* `training` `RL` `offloading` [ECHO-2: A Large Scale Distributed Rollout Framework for Cost-efficient Reinforcement Learning](http://arxiv.org/abs/2602.02192v1)
  > **TL;DR**: Proposes ECHO-2, a distributed framework for RL post-training of LLMs using remote workers, pipeline broadcast, and cost-aware resource management to mitigate latency. Reduces training costs by up to 20% while maintaining reward quality in wide-area environments.
* `training` `networking` `scaling` [Grappa: Gradient-Only Communication for Scalable Graph Neural Network Training](http://arxiv.org/abs/2602.01872v1)
  > **TL;DR**: Grappa introduces gradient-only communication for distributed GNN training to reduce network overhead. It uses periodic repartitioning and a lightweight coverage-corrected gradient aggregation. Achieves 4x average speedup in training time on real and synthetic graphs.
* `training` `scaling` [BOA Constrictor: Squeezing Performance out of GPUs in the Cloud via Budget-Optimal Allocation](http://arxiv.org/abs/2602.01404v1)
  > **TL;DR**: Proposes BOA Constrictor, a budget-optimal scheduler for cloud-based GPU training jobs. It optimally allocates rented GPUs to minimize job completion time under a fixed budget, reducing average JCT by 1.6-2× compared to heuristics.
* `offloading` `kernel` `quantization` [HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures](http://arxiv.org/abs/2602.00748v1)
  > **TL;DR**: Proposes HyperOffload, a compiler-driven memory management framework for LLMs on supernode architectures. Uses graph-based IR cache operators to model data movement, enabling global analysis and static scheduling of transfers to hide latency. Achieves up to 26% reduction in peak memory usage during inference.
* `MoE` `serving` `networking` [PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching](http://arxiv.org/abs/2602.00509v1)
  > **TL;DR**: Addresses stragglers and network congestion in Mixture-of-Experts inference. Proposes PROBE with predictive prefetching, balance planning, and co-scheduling to co-balance computation and communication. Achieves up to 1.32x lower prefill latency and 1.26x higher decoding throughput.
* `edge` `RL` `thinking` [FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards](http://arxiv.org/abs/2602.00453v1)
  > **TL;DR**: Proposes FedMOA, a federated GRPO framework for multi-objective alignment of personalized reasoning LLMs under heterogeneous rewards. Uses an online adaptive weighting mechanism and task-aware aggregation to stabilize local training and prioritize high-quality updates. Achieves up to 2.2% higher accuracy on reasoning benchmarks.
* `kernel` `sparse` `quantization` [Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation](http://arxiv.org/abs/2602.00294v1)
  > **TL;DR**: Proposes a symmetric tensor decomposition method to compute self-attention with constant cost per token, reducing memory and computation. Achieves orders-of-magnitude reductions in resource usage, enabling unbounded token generation at fixed cost.
* `training` `scaling` [Training LLMs with Fault Tolerant HSDP on 100,000 GPUs](http://arxiv.org/abs/2602.00277v1)
  > **TL;DR**: Proposes Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP) to address frequent failures in large-scale synchronous training. Uses data parallel replicas for fault isolation, with a Fault Tolerant All Reduce protocol and non-blocking catch-up. Achieves 80% effective training time (vs 44% baseline) on 100,000 GPUs.
* `serving` `edge` `networking` [VoxServe: Streaming-Centric Serving System for Speech Language Models](http://arxiv.org/abs/2602.00269v1)
  > **TL;DR**: Addresses high-throughput, low-latency streaming serving for SpeechLMs. Proposes a model-execution abstraction with streaming-aware scheduling and asynchronous pipeline. Achieves 10-20× higher throughput at comparable latency over existing systems while maintaining streamability.

### 2026-02-02
* `kernel` `hardware` [AscendCraft: Automatic Ascend NPU Kernel Generation via DSL-Guided Transcompilation](http://arxiv.org/abs/2601.22760v1)
  > **TL;DR**: Addresses NPU kernel development inefficiencies for Ascend chips. Proposes AscendCraft, a DSL-guided transcompilation approach using LLMs to generate optimized AscendC kernels. Achieves 98.1% compilation success rate and 46.2% of kernels matching/exceeding PyTorch eager performance.
* `serving` `offline` [SQUAD: Scalable Quorum Adaptive Decisions via ensemble of early exit neural networks](http://arxiv.org/abs/2601.22711v1)
  > **TL;DR**: Proposes SQUAD, a quorum-based early-exit inference scheme combining ensemble learning and hierarchical diversity to improve uncertainty estimation and reduce latency. Achieves up to 70.60% lower inference latency than static ensembles while maintaining accuracy.
* `serving` `offloading` [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](http://arxiv.org/abs/2601.22705v1)
  > **TL;DR**: Addresses middle-phase thrashing in KV cache during agentic LLM batch inference. Proposes CONCUR, a congestion control-inspired mechanism for agent-level admission. Achieves up to 4.09x throughput improvement on Qwen3-32B.
* `training` `networking` `scaling` [HetCCL: Accelerating LLM Training with Heterogeneous GPUs](http://arxiv.org/abs/2601.22585v1)
  > **TL;DR**: Addresses inefficiency in multi-vendor GPU clusters during LLM training due to lack of cross-vendor collective communication. Introduces HetCCL, a library with unified RDMA-based communication leveraging NCCL and RCCL backends. Achieves performance matching homogeneous setups while scaling in heterogeneous environments, reducing training costs.
* `edge` `scaling` [Coordinating Power Grid Frequency Regulation Service with Data Center Load Flexibility](http://arxiv.org/abs/2601.22487v1)
  > **TL;DR**: Proposes EcoCenter, a framework for GPU data centers to provide frequency regulation to reduce grid carbon emissions. Introduces Exogenous Carbon metric and maximizes regulation participation. Achieves carbon savings exceeding operational emissions, reducing fossil-fuel reserves.
* `training` `scaling` `networking` [AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism](http://arxiv.org/abs/2601.22442v1)
  > **TL;DR**: Proposes AsyncMesh, an asynchronous optimization method for data and pipeline parallelism to overcome communication bottlenecks in distributed training. Introduces weight look-ahead for pipelines and sparse averaging with EMA correction for data parallelism. Reduces communication overhead by 2-5 times while matching synchronous baseline convergence.
* `serving` `offloading` `networking` [Towards Resiliency in Large Language Model Serving with KevlarFlow](http://arxiv.org/abs/2601.22438v1)
  > **TL;DR**: Addresses slow recovery from hardware faults in LLM serving. Introduces KevlarFlow with decoupled model parallelism, traffic rerouting, and background KV cache replication. Reduces MTTR by 20x and improves p99 latency by 2.8x with negligible overhead.
* `scaling` `serving` `RL` [SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning](http://arxiv.org/abs/2601.22397v1)
  > **TL;DR**: Proposes SAIR, an autoscaling framework for multi-stage ML inference pipelines using an LLM as an in-context reinforcement learning controller. Combines Pareto-dominance reward shaping, surprisal-guided experience retrieval, and fine-grained GPU rate control. Achieves up to 50% lower P99 latency and 97% lower effective cost.

### 2026-01-30
* `serving` `inference` `energy` [Where Do the Joules Go? Diagnosing Inference Energy Consumption](http://arxiv.org/abs/2601.22076v1)
  > **TL;DR**: Investigates the factors causing variations in energy consumption during generative AI inference. Proposes a diagnostic framework linking algorithm, software, and hardware layers to latent metrics like memory/utilization. Measures up to 100× energy differences for video vs image generation on H100/B200 GPUs.
* `serving` `disaggregation` `scaling` [Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference](http://arxiv.org/abs/2601.22001v1)
  > **TL;DR**: Identifies memory and interconnect bottlenecks in AI agent inference. Proposes disaggregated serving with specialized accelerators, scale-up networking, and optical I/O for compute-memory decoupling. Addresses shifting operational intensity/capacity footprint to sustain efficiency in agentic workflows.
* `RL` `training` `disaggregation` [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](http://arxiv.org/abs/2601.21972v1)
  > **TL;DR**: Addresses decentralized LLM collaboration limitations using Multi-Agent Actor-Critic. Proposes CoLLM-CC/Centralized Critic and CoLLM-DC/Decentralized Critics for efficient parallel inference. Achieves competitive results with reduced sample requirements and improved convergence in sparse-reward tasks.
* `training` `kernel` [DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training](http://arxiv.org/abs/2601.21824v1)
  > **TL;DR**: Addresses the performance loss in deterministic attention backward pass during LLM training. Proposes DASH, a DAG-based scheduling approach with optimized compute and gradient-reduction phases. Improves throughput by up to 1.28× compared to FlashAttention-3 baseline on NVIDIA H800 GPUs.
* `serving` `scaling` [EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference](http://arxiv.org/abs/2601.21758v1)
  > **TL;DR**: Addresses scheduling challenge for mixed LLM inference workloads with long batch and short interactive requests. Proposes EWSJF scheduler with unsupervised partitioning, dynamic routing, and adaptive optimization. Achieves over 30% higher throughput and up to 4x lower Time-To-First-Token for short requests versus FCFS.
* `serving` `offloading` `multi-modal` [ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management](http://arxiv.org/abs/2601.21473v1)
  > **TL;DR**: Addresses GPU memory exhaustion in large-scale multi-agent simulations by proposing ScaleSim, which uses invocation distance-based prefetching and eviction for agent states. Enables private agent states with modular interfaces. Achieves 1.74x speedup over SGLang.
* `training` `storage` `scaling` [Nimbus: A Unified Embodied Synthetic Data Generation Framework](http://arxiv.org/abs/2601.21449v1)
  > **TL;DR**: Introduces Nimbus, a unified, modular framework for high-throughput embodied synthetic data generation to support foundation model training. It uses a decoupled architecture with asynchronous stages, dynamic scheduling, and distributed fault tolerance. Boosts end-to-end throughput 2-3X over baselines.
* `serving` `multi-modal` `kernel` [Native LLM and MLLM Inference at Scale on Apple Silicon](http://arxiv.org/abs/2601.19139v2)
  > **TL;DR**: Proposes vllm-mlx: efficient LLM/MLLM inference on Apple Silicon using native MLX integration. Features content-based prefix caching to skip redundant vision encoding for identical images. Achieves up to 87% higher throughput than llama-cpp on text models and 28x speedup for repeated image queries.
* `MoE` `serving` `edge` [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](http://arxiv.org/abs/2601.21198v1)
  > **TL;DR**: Addresses high memory footprint of MoE models for on-device inference. Proposes ZipMoE, a system combining lossless compression and cache-aware scheduling to shift MoE inference to compute-centric workflow. Achieves up to 72.77% latency reduction and 6.76× higher throughput.
* `training` `kernel` `quantization` [Axe: A Simple Unified Layout Abstraction for Machine Learning Compilers](http://arxiv.org/abs/2601.19092v2)
  > **TL;DR**: Introduces Axe Layout, a unified hardware-aware abstraction for tensor layout mapping across devices and memory hierarchies. Proposes a multi-granularity DSL and compiler that integrates thread-local control with collective operators. Achieves near hand-tuned kernel performance on GPUs and multi-device setups.

### 2026-01-29
* `edge` `serving` `networking` [Agentic Fog: A Policy-driven Framework for Distributed Intelligence in Fog Computing](http://arxiv.org/abs/2601.20764v1)
  > **TL;DR**: Proposes Agentic Fog, a policy-driven framework for distributed intelligence in fog computing with LLM-based reasoning. Uses decentralized potential game coordination for adaptive control. Achieves lower average latency and better adaptation to demand than greedy heuristics under dynamic conditions.
* `diffusion` `serving` `hardware` [Beyond GEMM-Centric NPUs: Enabling Efficient Diffusion LLM Sampling](http://arxiv.org/abs/2601.20706v1)
  > **TL;DR**: Addresses NPU inefficiency in diffusion LLM sampling due to irregular memory access. Proposes optimized non-GEMM vector primitives, in-place memory reuse, and decoupled mixed-precision memory hierarchy. Achieves up to 2.53x speedup over NVIDIA GPU.
* `video` `networking` `scaling` [OnePiece: A Large-Scale Distributed Inference System with RDMA for Complex AI-Generated Content (AIGC) Workflows](http://arxiv.org/abs/2601.20655v1)
  > **TL;DR**: Introduces OnePiece, a distributed inference system with RDMA for multi-stage AIGC workflows, featuring fine-grained microservices and double-ring buffers for deadlock-free RDMA. Reduces GPU consumption by 16x in image-to-video generation.
* `kernel` `training` `networking` [AutoOverlap: Enabling Fine-Grained Overlap of Computation and Communication with Chunk-Based Scheduling](http://arxiv.org/abs/2601.20595v1)
  > **TL;DR**: Reduces communication overhead in GPU workloads with fine-grained computation-communication overlap using a chunk-based scheduler. AutoOverlap compiler transforms Triton kernels for chunk-level alignment, achieving 1.3x average speedup and up to 4.7x on multi-GPU tasks.
* `training` `serving` `scaling` [Rethinking Thread Scheduling under Oversubscription: A User-Space Framework for Coordinating Multi-runtime and Multi-process Workloads](http://arxiv.org/abs/2601.20435v1)
  > **TL;DR**: Proposes USF, a user-space scheduling framework for coordinating multi-runtime workloads under CPU oversubscription. Introduces SCHED_COOP to reduce interference by cooperative thread switching. Shows up to 2.4x performance gains in PyTorch inference for LLaMA-3 and other workloads.
* `training` `quantization` `scaling` [Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT](http://arxiv.org/abs/2601.20408v1)
  > **TL;DR**: Proposes OptiKIT, an automated framework for enterprise LLM optimization to overcome compute constraints and expertise scarcity. Features distributed resource allocation and staged execution for model compression/tuning. Achieves over 2x GPU throughput improvement in production.
* `serving` `offloading` `hardware` [SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips](http://arxiv.org/abs/2601.20309v1)
  > **TL;DR**: Addresses LLM inference KV cache memory exhaustion and SLO violations under high load. Proposes SLO-aware rotary scheduler (RotaSched) and full-duplex KV transfer engine (DuplexKV) for Superchips. Achieves up to 74.7% higher TTFT SLO attainment than prior systems.
* `diffusion` `serving` `networking` [StreamFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs](http://arxiv.org/abs/2601.20273v1)
  > **TL;DR**: Proposes StreamFusion, a distributed inference engine for Diffusion Transformers. Introduces topology-aware sequence parallelism, Torus Attention for overlapping communication with computation, and one-sided communication to reduce synchronization. Achieves up to 1.77× speedup over prior methods.
* `training` `sparse` `scaling` [Communication-Avoiding Linear Algebraic Kernel K-Means on GPUs](http://arxiv.org/abs/2601.17136v2)
  > **TL;DR**: Presents distributed-memory algorithms for large-scale Kernel K-means using communication-efficient linear algebra primitives on multi-GPU systems. Enables clustering of million-scale datasets with up to 3.6× speedup on 256 GPUs and reduces time from hours to seconds.
* `training` `networking` `hardware` [NET4EXA: Pioneering the Future of Interconnects for Supercomputing and AI](http://arxiv.org/abs/2601.19413v1)
  > **TL;DR**: Develops BXIv3, a next-gen interconnect for HPC/AI systems to meet demands of large-scale LLM training. Combines commercial switches with custom FPGA-based NICs via co-design. Targets deployment in exascale systems for 2025, building on proven BXI technology used in TOP15 supercomputers.
* `serving` `networking` `offloading` [Rethinking Thread Scheduling under Oversubscription: A User-Space Framework for Coordinating Multi-runtime and Multi-process Workloads](http://arxiv.org/abs/2601.20435v1)
  > **TL;DR**: Proposes a user-space scheduling framework (USF) with SCHED_COOP policy to reduce interference in oversubscribed multi-process workloads targeting AI inference. Enables cooperative thread switching to mitigate preemption issues. Shows up to 2.4x speedup in PyTorch inference with LLaMA-3.

### 2026-01-28
* `edge` `serving` `networking` [Modular Foundation Model Inference at the Edge: Network-Aware Microservice Optimization](http://arxiv.org/abs/2601.19563v1)
  > **TL;DR**: Proposes a microservice-based framework for edge deployment of foundation model inference with static core services and dynamic light service orchestration. Uses network-aware placement and online control for QoS guarantees under resource constraints. Achieves over 84% on-time task completion with moderate costs.
* `agentic` `serving` `RL` [Optimizing FaaS Platforms for MCP-enabled Agentic Workflows](http://arxiv.org/abs/2601.14735v2)
  > **TL;DR**: Proposes FAME, a FaaS-based system for orchestrating MCP-enabled agentic workflows. Uses composable LangGraph agents (Planner/Actor/Evaluator) with DynamoDB for state persistence, Lambda wrappers for MCP serving, and S3 caching. Achieves 13x latency reduction, 88% token savings, and 66% cost reduction.
* `training` `RL` `networking` [Revisiting Parameter Server in LLM Post-Training](http://arxiv.org/abs/2601.19362v1)
  > **TL;DR**: Revisits parameter servers for LLM post-training under imbalanced workloads. Proposes On-Demand Communication (ODC), replacing collective operations in FSDP with point-to-point communication to reduce synchronization barriers. Achieves 36% throughput speedup over FSDP by improving device utilization.
* `serving` `multi-modal` `kernel` [Native LLM and MLLM Inference at Scale on Apple Silicon](http://arxiv.org/abs/2601.19139v1)
  > **TL;DR**: Presents vllm-mlx for efficient LLM and multimodal inference on Apple Silicon using native MLX. Features content-based prefix caching for vision encoding and continuous batching. Achieves 28x speedup on repeated image queries and 525 tokens/second text throughput.
* `kernel` `training` `scaling` [Axe: A Simple Unified Layout Abstraction for Machine Learning Compilers](http://arxiv.org/abs/2601.19092v1)
  > **TL;DR**: Proposes Axe Layout, a unified layout abstraction for ML compilers to map logical tensors to physical space. Designs a multi-granularity DSL and compiler for collective operators. Achieves near hand-tuned kernel performance on GPUs and multi-device environments.

### 2026-01-27
* `serving` `scaling` `networking` [A Universal Load Balancing Principle and Its Application to Large Language Model Serving](http://arxiv.org/abs/2601.17855v1)
  > **TL;DR**: Addresses barrier-synchronized load balancing to reduce idle time in LLM serving. Introduces an integer-optimization scheduler with worst-case guarantees for heterogeneous workloads. Improves throughput and reduces idle time by 40% in production LLM traces.
* `serving` `kernel` [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](http://arxiv.org/abs/2601.17768v1)
  > **TL;DR**: Addresses non-determinism in LLM inference arising from dynamic batching and GPU kernels. Proposes LLM-42, a scheduler using verified speculation with a fast path and lightweight verify-rollback loop to enforce determinism. Achieves near-zero overhead for deterministic requests while maintaining high throughput.
* `training` `kernel` `hardware` [An MLIR Lowering Pipeline for Stencils at Wafer-Scale](http://arxiv.org/abs/2601.17754v1)
  > **TL;DR**: Develops an MLIR compiler pipeline for automatic code generation targeting the Cerebras wafer-scale engine for stencil computations. Leverages domain-specific optimizations to bridge the gap between mathematical models and distributed execution. Achieves up to 20× speedup over GPU/CPU clusters without application code changes.
* `training` `kernel` `scaling` [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](http://arxiv.org/abs/2601.17654v1)
  > **TL;DR**: Addresses joint optimization of dynamic and static energy consumption in large model training. Proposes Kareus, a system using fine-grained kernel scheduling and frequency scaling with multi-pass multi-objective optimization. Reduces energy by 28.3% at same time or time by 27.5% at same energy.
* `storage` `offloading` `hardware` [Conduit: Programmer-Transparent Near-Data Processing Using Multiple Compute-Capable Resources in Solid State Drives](http://arxiv.org/abs/2601.17633v1)
  > **TL;DR**: Proposes Conduit, a near-data processing framework for SSDs that uses compiler-based vectorization and runtime offloading across multiple compute resources. It reduces energy by 46% and improves performance by 1.8x over prior techniques in data-intensive workloads.
* `training` `networking` `scaling` [Scaling All-to-all Operations Across Emerging Many-Core Supercomputers](http://arxiv.org/abs/2601.17606v1)
  > **TL;DR**: Proposes novel all-to-all algorithms for MPI on emerging many-core supercomputers to enhance collective operations performance. Achieves up to 3x speedup over system MPI at 32 nodes, critical for large-scale machine learning and FFT applications.
* `RL` `networking` `offline` [Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federation](http://arxiv.org/abs/2601.17133v1)
  > **TL;DR**: Proposes KNEXA-FL, a federated learning framework for P2P LLM collaboration with a central matchmaker using contextual bandits. Enables direct knowledge exchange via secure distillation without model access. Improves Pass@1 by ~50% over random P2P baselines and maintains stable convergence.
* `kernel` `training` [Sawtooth Wavefront Reordering: Enhanced CuTile FlashAttention on NVIDIA GB10](http://arxiv.org/abs/2601.16032v2)
  > **TL;DR**: Analyzes and optimizes cache performance for FlashAttention on NVIDIA Grace Blackwell. Proposes Sawtooth Wavefront Reordering to reduce L2 cache misses in CUDA and CuTile implementations. Achieves 50% reduced L2 misses and 60% higher throughput.

### 2026-01-26
* `training` `storage` [DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers](http://arxiv.org/abs/2601.16956v1)
  > **TL;DR**: Addresses checkpointing overheads in distributed LLM training. Proposes DataStates-LLM with composable State Providers enabling non-blocking asynchronous snapshots and optimized serialization. Achieves 4× higher checkpoint throughput and 2.2× faster end-to-end training time.
* `quantization` `kernel` `hardware` [W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs](http://arxiv.org/abs/2601.16536v1)
  > **TL;DR**: Enables efficient W4A16 mixed-precision matrix multiplication for LLMs on Ascend NPUs. Designs a tailored kernel using vector cores for dequantization and cube cores for GEMM, with Split-K parallelization. Achieves up to 1.74x speedup over data-parallel approaches in LLM decoding scenarios.
* `training` `kernel` `scaling` [Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple](http://arxiv.org/abs/2601.16294v1)
  > **TL;DR**: Proposes space-filling curves for communication-avoiding GEMM optimization. Uses generalized Hilbert curves to partition GEMM computations for data locality and communication avoidance. Achieves up to 2x geometric mean speedup over vendor libraries on multiple CPUs.

### 2026-01-23
* `RL` `multi-modal` `edge` [DSFedMed: Dual-Scale Federated Medical Image Segmentation via Mutual Distillation Between Foundation and Lightweight Models](http://arxiv.org/abs/2601.16073v1)
  > **TL;DR**: Proposes DSFedMed, a dual-scale federated framework using mutual distillation between foundation and lightweight models for medical image segmentation. Uses synthetic medical images and learnability-guided selection to reduce communication by 90% and inference time, achieving 2% higher Dice scores than baselines.
* `kernel` `serving` `training` [Sawtooth Wavefront Reordering: Enhanced CuTile FlashAttention on NVIDIA GB10](http://arxiv.org/abs/2601.16032v1)
  > **TL;DR**: Analyzes and improves cache performance for FlashAttention on NVIDIA GB10. Proposes Sawtooth Wavefront Reordering to reduce L2 misses. Achieves 50% L2 miss reduction and up to 60% throughput increase.

### 2026-01-22
* [DeepFedNAS: A Unified Framework for Principled, Hardware-Aware, and Predictor-Free Federated Neural Architecture Search](http://arxiv.org/abs/2601.15127v1)
* `serving` `offloading` [RadixMLP -- Intra-batch Deduplication for Causal Transformers](http://arxiv.org/abs/2601.15013v1)
  > **TL;DR**: Addresses redundancy in causal transformer batch inference caused by shared prefixes. Proposes RadixMLP, which compresses shared segments into a trie structure for position-wise computation, reducing redundant MLP activations. Achieves 1.44-1.59× speedups in reranking workloads and up to 5× on synthetic benchmarks.
* `serving` `RAG` `agentic` [Optimizing FaaS Platforms for MCP-enabled Agentic Workflows](http://arxiv.org/abs/2601.14735v1)
  > **TL;DR**: Addresses challenges in scalable cloud deployment of agentic workflows using FaaS platforms. Proposes FAME, a FaaS-based architecture with composable agents, automated memory persistence, and optimizations like tool caching and function fusion. Achieves up to 13× latency reduction and 66% cost savings.
* `kernel` `scaling` `training` [JAXMg: A multi-GPU linear solver in JAX](http://arxiv.org/abs/2601.14466v1)
  > **TL;DR**: Introduces JAXMg, a multi-GPU linear solver for JAX that scales dense linear algebra beyond single-GPU memory limits. Interfaces JAX with NVIDIA's cuSOLVERMg via XLA for JIT-compatible distributed solvers. Enables Cholesky solves on matrices 4-12× larger than single-GPU capacity across 4-16 GPUs.

### 2026-01-21
* `serving` [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](http://arxiv.org/abs/2601.09258v2)
  > **TL;DR**: Proposes LatencyPrism, a zero-intrusion latency monitoring system for LLM inference that breaks down pipeline latency, alerts anomalies, guarantees SLO adherence. Deployed on thousands of XPUs, enables millisecond alerts with 0.98 F1-score for anomaly detection without service restarts.
* `offloading` `serving` `kernel` [ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management](http://arxiv.org/abs/2601.13631v1)
  > **TL;DR**: Accelerates LLM prefill through optimized KV cache offloading. ContiguousKV aligns pruning granularity with I/O management via ContiguousChunk and uses asynchronous prefetching to eliminate idle bubbles. Achieves 3.85x speedup in Re-Prefill phase over IMPRESS while preserving quality.
* `training` `scaling` [A Kubernetes custom scheduler based on reinforcement learning for compute-intensive pods](http://arxiv.org/abs/2601.13579v1)
  > **TL;DR**: Addresses suboptimal placement of compute-intensive ML training pods in Kubernetes. Proposes two RL-based schedulers (SDQN and SDQN-n), built on DQN, that consolidate pods onto fewer nodes. Reduces average CPU utilization by 10% (SDQN) and over 20% (SDQN-n) compared to default scheduler.
* `storage` `offline` [The Energy-Throughput Trade-off in Lossless-Compressed Source Code Storage](http://arxiv.org/abs/2601.13220v1)
  > **TL;DR**: Studies energy-throughput trade-offs in compressed key-value storage for large-scale source code datasets. Designs tunable compression configurations for space, time, and energy efficiency, with parallelism. Achieves high compression ratios, order-of-magnitude gains in retrieval throughput and energy efficiency, enabling sustainable archival systems.
* `agentic` `serving` [Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference](http://arxiv.org/abs/2601.12967v1)
  > **TL;DR**: Addresses latency bottlenecks in tool-based agentic LLM inference caused by tool calls and poor KV cache utilization. Proposes Sutradhara, an orchestrator-engine co-design enabling tool-aware prompt splitting, streaming tool dispatch, and cache management. Reduces median FTR latency by 15% and E2E latency by 10%.
* `training` `edge` `offloading` [CooperLLM: Cloud-Edge-End Cooperative Federated Fine-tuning for LLMs via ZOO-based Gradient Correction](http://arxiv.org/abs/2601.12917v1)
  > **TL;DR**: Proposes CooperLLM, a federated fine-tuning framework using cloud-guided ZOO gradient correction for edge devices. Integrates pipeline scheduling and adaptive compression to reduce memory by 86.4% and accelerate convergence by 8.8× while maintaining privacy.
* `training` `RL` `scaling` [Unleashing Efficient Asynchronous RL Post-Training via Staleness-Constrained Rollout Coordination](http://arxiv.org/abs/2601.12784v1)
  > **TL;DR**: Proposes StaleFlow, a system for efficient asynchronous RL post-training. It jointly addresses data staleness via a global consistency protocol and data skewness via data servers with staleness-aware strategies. Achieves 1.42-2.68× higher throughput without compromising convergence.
* `scaling` [Opportunistic Scheduling for Optimal Spot Instance Savings in the Cloud](http://arxiv.org/abs/2601.12266v1)
  > **TL;DR**: Studies cost-minimization for delay-sensitive jobs using spot and on-demand cloud instances. Proposes a scheduling policy based on queuing theory and optimization that exploits the knapsack structure at high delays. Achieves near-optimal cost while meeting average delay constraints.
* `disaggregation` `serving` `scaling` [Power Aware Dynamic Reallocation For Inference](http://arxiv.org/abs/2601.12241v1)
  > **TL;DR**: Addresses power as the limiting factor in LLM inference with disaggregated prefill/decode phases. Proposes RAPID, a framework for power-aware GPU role and power budget management via static and dynamic reallocation. Achieves 2x SLO improvement at peak load under power caps.
* `kernel` `storage` `quantization` [Canonicalization of Batched Einstein Summations for Tuning Retrieval](http://arxiv.org/abs/2601.12220v1)
  > **TL;DR**: Develops a canonicalization algorithm for batched Einstein summations to enable reuse of optimizations. Uses graph coloring to normalize einsum expressions and proposes functional array operands for transferability. Achieves 4.7× geomean speedup on einsums in TCCG and FEM benchmarks.
* `edge` `serving` `kernel` [Mitigating GIL Bottlenecks in Edge AI Systems](http://arxiv.org/abs/2601.10582v2)
  > **TL;DR**: Addresses Python GIL bottlenecks for AI inference on edge devices. Proposes a lightweight profiling tool and adaptive runtime using a Blocking Ratio metric to optimize thread counts. Achieves 96.5% of optimal performance and avoids multiprocessing's ~8x memory overhead.
* `serving` `disaggregation` [RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation](http://arxiv.org/abs/2601.11822v1)
  > **TL;DR**: Proposes RAPID-Serve, a technique for concurrent prefill/decode execution on GPUs to balance latency SLOs and throughput. Uses intra-GPU disaggregation with adaptive resource management. Achieves up to 4.1x unconstrained throughput improvement and 32x higher throughput under SLO constraints.
* `RAG` `storage` `networking` [GPU-Resident Inverted File Index for Streaming Vector Databases](http://arxiv.org/abs/2601.11808v1)
  > **TL;DR**: Proposes SIVF, a GPU-resident inverted file index enabling lock-free in-place mutation for streaming vector databases in dynamic RAG systems. Uses slab-based allocation and address translation tables. Reduces deletion latency up to 13,300× and improves ingestion throughput up to 105×.
* `serving` `offloading` `edge` [Nixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs](http://arxiv.org/abs/2601.11743v1)
  > **TL;DR**: Addresses efficient temporal multiplexing of ML workloads on memory-constrained consumer GPUs. Proposes Nixie, a system service coordinating GPU memory allocation and kernel launches with MLFQ-inspired scheduling. Reduces latency of interactive tasks by up to 3.8× and saves 66.8% CPU pinned memory.
* `edge` `serving` `networking` [HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network](http://arxiv.org/abs/2601.11676v1)
  > **TL;DR**: Improves distributed LLM inference on lossy edge networks via semantic-aware neuron grouping, parallel execution, and load balancing. HALO achieves 3.41x end-to-end speedup on Raspberry Pi clusters while maintaining accuracy comparable to optimal conditions.
* `serving` `offloading` `edge` [WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching](http://arxiv.org/abs/2601.11652v1)
  > **TL;DR**: Addresses wasted drafting time and verification interference in edge-cloud distributed speculative LLM serving. Proposes WISP with dynamic drafting and SLO-aware batching. Achieves up to 4.1x higher system capacity and 3.7x better goodput than baselines.
* `serving` `scaling` [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](http://arxiv.org/abs/2601.09258v2)
  > **TL;DR**: Presents LatencyPrism, a non-intrusive system for online latency breakdown and SLO guarantee in distributed LLM inference. It provides real-time monitoring and anomaly detection without code changes or restarts. Achieves anomaly detection F1-score of 0.98 and millisecond-level alerts.
* `offloading` `serving` `kernel` [ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management](http://arxiv.org/abs/2601.13631v1)
  > **TL;DR**: Proposes ContiguousKV, a KV cache offloading system that aligns pruning granularity with I/O operations via ContiguousChunk and introduces asynchronous prefetching to accelerate Re-Prefill phase. Achieves 3.85x speedup over IMPRESS while maintaining output quality.
* `edge` `kernel` `offloading` [Mitigating GIL Bottlenecks in Edge AI Systems](http://arxiv.org/abs/2601.10582v2)
  > **TL;DR**: Addresses Python GIL bottlenecks in edge AI systems causing performance degradation at high thread counts. Proposes an adaptive runtime with Blocking Ratio metric to distinguish I/O wait vs. contention, achieving 93.9% efficiency vs. 20% degradation in naive approaches. Evaluated on edge devices with memory constraints (512MB-2GB RAM).
* `serving` `offloading` `edge` [Nixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs](http://arxiv.org/abs/2601.11743v1)
  > **TL;DR**: Addresses inefficient GPU resource utilization for concurrent ML workloads on consumer GPUs. Proposes Nixie, a system service with memory allocation coordination and a MLFQ-inspired scheduler for job prioritization. Reduces latency by up to 3.8x and saves 66.8% CPU pinned memory.

### 2026-01-16
* `edge` `serving` `quantization` [Mitigating GIL Bottlenecks in Edge AI Systems](http://arxiv.org/abs/2601.10582v1)
  > **TL;DR**: Addresses Python's GIL bottleneck in edge AI systems. Proposes a lightweight profiling tool and adaptive runtime using a Blocking Ratio metric to optimize thread-pool scaling. Achieves 96.5% of optimal performance and 93.9% average efficiency without overprovisioning.
* `training` `networking` `scaling` [Fundamental Limits of Coded Polynomial Aggregation](http://arxiv.org/abs/2601.10028v1)
  > **TL;DR**: Extends coded polynomial aggregation for straggler mitigation in distributed computing. Proposes a framework with pre-specified non-straggler patterns and exact recovery conditions. Achieves fewer worker responses than prior methods, with feasibility threshold validated by sharp transition in simulations.
* `edge` `kernel` `serving` [Mitigating GIL Bottlenecks in Edge AI Systems](http://arxiv.org/abs/2601.10582v1)
  > **TL;DR**: Addresses GIL-induced throughput degradation in Python-based edge AI systems. Proposes an adaptive runtime with Blocking Ratio metric to distinguish I/O wait from contention. Achieves 96.5% of optimal performance, outperforming multiprocessing and asyncio in memory-constrained edge devices.

### 2026-01-15
* `agentic` `serving` `RL` [AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems](http://arxiv.org/abs/2601.09393v1)
  > **TL;DR**: Introduces AI-NativeBench, a white-box benchmark suite for AI-native agentic systems using Model Context Protocol. It analyzes system-level execution, revealing engineering factors like a parameter paradox and inference dominance. Shows lightweight models can exceed flagship adherence, challenging traditional cost assumptions.
* `serving` `kernel` `scaling` [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](http://arxiv.org/abs/2601.09258v1)
  > **TL;DR**: Proposes LatencyPrism, a zero-intrusion latency sculpting system for LLM inference that breaks down latency, alerts anomalies in milliseconds, and guarantees SLOs without code changes. Achieves 0.98 F1-score in anomaly detection with low overhead.
* `offloading` `training` `edge` [Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning](http://arxiv.org/abs/2601.09076v1)
  > **TL;DR**: Enables efficient split federated learning on edge devices via zeroth-order optimization, reducing memory and compute costs. HERON-SFL uses ZO for clients and FO on server, avoiding backpropagation. Reduces client peak memory by 64% and compute by 33% per step while matching accuracy.
* `training` `offloading` `kernel` [Bridging the Gap: Empowering Small Models in Reliable OpenACC-based Parallelization via GEPA-Optimized Prompting](http://arxiv.org/abs/2601.08884v1)
  > **TL;DR**: Optimizes prompts for small LLMs to generate high-quality OpenACC pragmas for GPU offloading via genetic algorithm (GEPA). Achieves up to 100% compilation success rate and 21% more programs with GPU speedups, making small models as effective as larger ones for parallel code generation.
* `serving` `scaling` `monitoring` [LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference](http://arxiv.org/abs/2601.09258v1)
  > **TL;DR**: Proposes LatencyPrism, a non-intrusive runtime system for real-time latency decomposition and anomaly detection in LLM serving. Uses batch-level monitoring to distinguish normal variations from SLO-threatening anomalies without service modifications. Achieves 0.98 F1-score in anomaly detection.

### 2026-01-14
* `serving` `MoE` `networking` [MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm](http://arxiv.org/abs/2601.08800v1)
  > **TL;DR**: Proposes MixServe, an automatic distributed serving system for MoE models using hybrid parallelism and fused communication to reduce latencies. It combines tensor and expert parallelism with overlapped AR-A2A communication, improving TTFT by 1.08~3.80x and throughput by 5.2%~50.3% over baselines.
* `agentic` `training` `scaling` [MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era](http://arxiv.org/abs/2601.07526v2)
  > **TL;DR**: Addresses infrastructure gaps for large-scale agent training and evaluation. Proposes MegaFlow, a distributed orchestration system with three decoupled services for independent scaling and resource allocation. Orchestrates tens of thousands of concurrent agent tasks with high stability and resource utilization.
* `edge` `offline` `networking` [Hierarchical Online-Scheduling for Energy-Efficient Split Inference with Progressive Transmission](http://arxiv.org/abs/2601.08135v1)
  > **TL;DR**: Addresses energy-latency-accuracy trade-offs in device-edge collaborative DNN inference. Proposes ENACHI, a two-tier Lyapunov framework with task-level partitioning decisions and packet-level power control via progressive transmission. Achieves 43.12% higher accuracy with 62.13% lower energy under tight deadlines.
* `serving` `training` `quantization` [Coordinated Cooling and Compute Management for AI Datacenters](http://arxiv.org/abs/2601.08113v1)
  > **TL;DR**: Proposes a hierarchical control framework to optimize GPU parallelism, frequency, and cooling for AI datacenters. Balances latency and thermal constraints by jointly modeling computation and cooling. Reduces energy consumption by 40% while meeting latency SLOs in real Azure traces.
* `edge` `serving` `networking` [Where to Split? A Pareto-Front Analysis of DNN Partitioning for Edge Inference](http://arxiv.org/abs/2601.08025v1)
  > **TL;DR**: Examines DNN partitioning for edge inference as multi-objective optimization. Proposes ParetoPipe, a framework using Pareto front analysis to optimize latency-throughput trade-offs under network variability. Achieves systematic identification of partitioning strategies balancing objectives on heterogeneous devices.

### 2026-01-13
* `training` `scaling` `kernel` [Beyond Single-GPU: Scaling PDLP to Distributed Multi-GPU Systems](http://arxiv.org/abs/2601.07628v1)
  > **TL;DR**: Scales Primal-Dual Hybrid Gradient algorithm for massive linear programming to distributed multi-GPU systems. Uses 2D grid partitioning, NCCL communication, and fused CUDA kernels to distribute memory/computation. Achieves strong scalability with FP64 accuracy on real-world datasets.
* `edge` `serving` `kernel` [Peformance Isolation for Inference Processes in Edge GPU Systems](http://arxiv.org/abs/2601.07600v1)
  > **TL;DR**: Analyzes GPU isolation mechanisms (MPS, MIG, Green Contexts) for predictable LLM inference latency on edge devices. Evaluates partitioning impact and isolation on A100 and Jetson Orin. Finds MIG offers high isolation; Green Contexts enable fine-grained SM allocation with low overhead for edge.
* `agentic` `scaling` `training` [MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era](http://arxiv.org/abs/2601.07526v1)
  > **TL;DR**: Addresses lack of infrastructure for large-scale agent training and evaluation. Proposes MegaFlow, a distributed orchestration system with three abstracted services for scheduling and resource allocation. Handles tens of thousands of concurrent agent tasks with high stability and resource efficiency.
* `training` `RL` `LoRA` [OpenTinker: Separating Concerns in Agentic Reinforcement Learning](http://arxiv.org/abs/2601.07376v1)
  > **TL;DR**: Proposes OpenTinker, an infrastructure for RL agent training that separates concerns into composable components. Uses a centralized scheduler for managing LoRA-based/full-parameter RL and fine-tuning workloads on shared resources. Demonstrates effectiveness in multi-agent training scenarios.
* `edge` `offloading` [SC-MII: Infrastructure LiDAR-based 3D Object Detection on Edge Devices for Split Computing with Multiple Intermediate Outputs Integration](http://arxiv.org/abs/2601.07119v1)
  > **TL;DR**: Proposes SC-MII, a split-computing system for LiDAR-based 3D object detection on edge devices, offloading partial DNN processing to an edge server via multiple intermediate outputs. Achieves 2.19x speed-up and 71.6% reduction in edge processing time with minimal accuracy drop.
* `training` `scaling` [SkyNomad: On Using Multi-Region Spot Instances to Minimize AI Batch Job Cost](http://arxiv.org/abs/2601.06520v1)
  > **TL;DR**: Addresses minimizing cost of AI batch jobs (training, inference) under deadlines using spot instances. Proposes SkyNomad, a multi-region scheduler that probes availability, predicts lifetimes, and optimizes spot utilization via cost modeling. Achieves 1.25-3.96x cost savings with deadline guarantees.
* `networking` `offloading` [Rethinking Inter-Process Communication with Memory Operation Offloading](http://arxiv.org/abs/2601.06331v1)
  > **TL;DR**: Addresses inefficiencies in IPC runtimes due to memory copies in data-intensive services. Proposes a unified suite integrating hardware- and software-based offloading with asynchronous pipelining and selective cache injection. Achieves up to 2.1x throughput improvement and 72% latency reduction.
* `serving` `scaling` `MoE` [AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving](http://arxiv.org/abs/2601.06288v1)
  > **TL;DR**: Proposes AIConfigurator, a unified performance-modeling system for optimizing LLM inference configurations across frameworks without GPU profiling. Uses decomposed primitives, kernel database, and abstraction layer for rapid search. Achieves up to 50% higher performance for MoE models and completes searches in 30 seconds.
* `networking` `storage` `offloading` [Employ SmartNICs' Data Path Accelerators for Ordered Key-Value Stores](http://arxiv.org/abs/2601.06231v1)
  > **TL;DR**: Proposes a high-performance key-value store using SmartNIC accelerators for ordered range queries and point lookups. Leverages data path accelerators for lock-free indexing and defers value retrieval to minimize PCIe transfers. Achieves 33 MOPS for point lookups and 13 MOPS for range queries.
* `edge` `serving` `offloading` [Peformance Isolation for Inference Processes in Edge GPU Systems](http://arxiv.org/abs/2601.07600v1)
  > **TL;DR**: Evaluates GPU isolation mechanisms (MIG, Green Contexts) for predictable inference latency in edge systems. Tests partitioning impact and temporal isolation on A100 and Jetson Orin. Green Contexts enables fine-grained SM allocation with low overhead but lacks memory isolation.
* `networking` `serving` [Rethinking Inter-Process Communication with Memory Operation Offloading](http://arxiv.org/abs/2601.06331v1)
  > **TL;DR**: Proposes a unified IPC runtime suite for memory operation offloading to reduce CPU overhead in data-intensive services. Integrates hardware/software memory offloading with shared-memory communication via asynchronous pipelining and cache injection. Achieves 22% instruction reduction and 2.1x throughput gains in real workloads.

### 2026-01-12
* `training` `scaling` `networking` [Self-Evolving Distributed Memory Architecture for Scalable AI Systems](http://arxiv.org/abs/2601.05569v1)
  > **TL;DR**: Proposes a unified memory management framework across computation, communication, and deployment layers for distributed AI systems. Features dynamic matrix partitioning, memory-aware peer selection, and runtime adaptation. Achieves 87.3% memory utilization efficiency and 30.2% lower communication latency vs baseline.
* `MoE` `training` `kernel` [MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs](http://arxiv.org/abs/2601.05296v1)
  > **TL;DR**: Addresses memory bottlenecks in Mixture-of-Experts (MoE) training. Proposes MoEBlaze with co-designed kernels and token dispatch to eliminate intermediate buffers, reducing activation memory. Achieves 4x speedups and 50% memory savings.

### 2026-01-09
* `agentic` `serving` [Nalar: An agent serving framework](http://arxiv.org/abs/2601.05109v1)
  > **TL;DR**: Proposes Nalar, an agent-serving framework handling dynamic multi-step agent workflows with managed state and adaptive routing. Achieves up to 2.9× speedups and 34-74% tail latency reduction across agentic workloads.
* `training` `storage` `hardware` [Cognitive Infrastructure: A Unified DCIM Framework for AI Data Centers](http://arxiv.org/abs/2601.04750v1)
  > **TL;DR**: Proposes DCIM 3.0, a unified framework for AI data centers using knowledge graphs and predictive analytics for efficient resource management. Integrates thermal modeling and UDCP to optimize infrastructure automation and sustainability. Achieves 30% reduction in cooling energy consumption.

### 2026-01-08
* `serving` `scaling` [Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale](http://arxiv.org/abs/2601.04071v1)
  > **TL;DR**: Addresses GPU sharing inefficiencies for SLO adherence in inference serving. Proposes Hummingbird with microsecond-scale preemption on closed-source GPUs to harvest idle slices. Achieves 9.7x better SLO attainment for high-priority tasks over spatial sharing.
* `MoE` `serving` `edge` [A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems](http://arxiv.org/abs/2601.03992v1)
  > **TL;DR**: Proposes a scheduling framework for efficient MoE inference on edge GPU-NDP systems. Key optimizations include tensor parallelism for expert partitioning, load-balancing scheduling, and dataset-free pre-fetching. Achieves 2.41x average and up to 2.56x speedup in end-to-end latency versus state-of-the-art.

### 2026-01-07
* `serving` `agentic` [Software-Defined Agentic Serving](http://arxiv.org/abs/2601.03197v1)
  > **TL;DR**: Addresses inflexibility in multi-agent LLM serving pipelines. Proposes a software-defined networking inspired framework for dynamically controlling communication based on runtime state. Achieves efficient and responsive agent systems.
* `MoE` `serving` `offloading` [Making MoE-based LLM Inference Resilient with Tarragon](http://arxiv.org/abs/2601.01310v2)
  > **TL;DR**: Addresses poor failure resilience in MoE-based LLM inference. Proposes Tarragon with reconfigurable datapath, asynchronous KV cache checkpointing, and shadow experts to confine failures. Reduces failure-induced stalls from ~64s to 0.3-0.4s (160-213× improvement) compared to MegaScale-Infer.
* `training` `LoRA` `kernel` [Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth](http://arxiv.org/abs/2601.02609v1)
  > **TL;DR**: Addresses memory and compute inefficiencies in LLM fine-tuning with Chronicals, a framework featuring fused Triton kernels, Cut Cross-Entropy, LoRA+, and sequence packing. Achieves 3.51x speedup over Unsloth on full fine-tuning of Qwen2.5-0.5B at 41,184 tokens/sec.
* `training` [First Provably Optimal Asynchronous SGD for Homogeneous and Heterogeneous Data](http://arxiv.org/abs/2601.02523v1)
  > **TL;DR**: Develops provably optimal asynchronous SGD methods for distributed learning with heterogeneous worker speeds. Introduces Ringmaster ASGD (handling homogeneous data) and Ringleader ASGD (handling heterogeneous data) based on a framework to manage staleness, plus ATA for adaptive task allocation. Achieves optimal time complexity matching synchronous methods while reducing waiting time.

### 2026-01-06
* `training` `scaling` `memory` [Placement Semantics for Distributed Deep Learning: A Systematic Framework for Analyzing Parallelism Strategies](http://arxiv.org/abs/2601.02311v1)
  > **TL;DR**: Introduces placement semantics to systematically analyze parallelism strategies by predicting memory and communication for distributed training. Derives conditions for correctness and composition rules. Matches exact results: e.g., ZeRO-3 uses 8× less memory with 1.5× communication cost vs data parallelism.
* `serving` `offloading` `storage` [RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference](http://arxiv.org/abs/2601.01712v1)
  > **TL;DR**: Addresses serving latency for long-sequence generative recommendation models under strict SLOs. Proposes RelayGR, which pre-infers reusable prefixes and maintains KV caches in HBM across pipeline stages via selective triggering, affinity routing, and memory expansion. Achieves 1.5× longer sequences and 3.6× higher throughput under fixed P99 SLO.
* `training` `hardware` `scaling` [DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster](http://arxiv.org/abs/2601.01500v1)
  > **TL;DR**: Proposes DiT-HC, a system for efficient training of DiT visual generation model on HPC CPU clusters. Introduces communication-free tensor parallelism, optimized kernels for vector/matrix units, and customized MPI backend. Achieves 90.6% weak scaling efficiency on 256 nodes and up to 87.7x speedups.
* `MoE` `serving` [Making MoE based LLM inference resilient with Tarragon](http://arxiv.org/abs/2601.01310v1)
  > **TL;DR**: Addresses resilience in MoE-based LLM serving by confining worker failures. Proposes Tarragon with reconfigurable datapath, asynchronous KV cache checkpointing, and shadow experts for low-overhead recovery. Reduces failure-induced stalls by 160-213x compared to MegaScale-Infer.
* `serving` `offloading` `sparse` [Warp-Cortex: An Asynchronous, Memory-Efficient Architecture for Million-Agent Cognitive Scaling on Consumer Hardware](http://arxiv.org/abs/2601.01298v1)
  > **TL;DR**: Proposes Warp-Cortex for memory-efficient large-scale multi-agent language model inference. Uses weight sharing and topological synapse-based context sparsification to reduce KV-cache memory from O(N*L) to O(N*k). Achieves 100 concurrent agents at 2.2 GB VRAM on a single GPU.
* `RL` `training` `networking` [OrchestrRL: Dynamic Compute and Network Orchestration for Disaggregated RL](http://arxiv.org/abs/2601.01209v1)
  > **TL;DR**: Addresses compute imbalance and network bottlenecks in disaggregated RL for LLMs. Proposes OrchestrRL with adaptive compute scheduler and RFabric reconfigurable network fabric. Achieves 1.40x higher throughput on 48 H800 GPUs.

### 2026-01-05
* `edge` `serving` `networking` [FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding](http://arxiv.org/abs/2601.00644v1)
  > **TL;DR**: Addresses high communication overhead in edge-cloud LLM speculative decoding for evolving models. Proposes FlexSpec with a shared-backbone architecture and channel-aware adaptive speculation. Reduces latency by 25-60% compared to conventional SD under varying conditions.
* `serving` `offloading` [Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving](http://arxiv.org/abs/2601.00397v1)
  > **TL;DR**: Proposes Revati, a time-warp emulator for transparent GPU-free performance modeling of LLM serving systems. Executes real serving code by virtualizing CUDA calls and fast-forwarding time via predicted kernel durations. Achieves <5% error in predictions and runs 5-17x faster than GPU execution.
* `RL` `agentic` `RAG` [Bio-inspired Agentic Self-healing Framework for Resilient Distributed Computing Continuum Systems](http://arxiv.org/abs/2601.00339v1)
  > **TL;DR**: Proposes ReCiSt, a bio-inspired agentic self-healing framework for distributed systems using LM-powered agents to autonomously diagnose faults and reconfigure resources. Achieves self-healing within tens of seconds with agent CPU usage as low as 10%.
* `offloading` `scaling` `serving` [Towards Fully-fledged GPU Multitasking via Proactive Memory Scheduling](http://arxiv.org/abs/2512.24637v2)
  > **TL;DR**: Targets GPU memory bottleneck for hosting large-scale tasks via demand paging; proposes MSched, an OS-level scheduler enabling proactive working set migration via kernel argument prediction. Achieves up to 57.88× speedup over conventional demand paging for LLMs under memory oversubscription.

### 2026-01-01
* `serving` `kernel` `RAG` [Vulcan: Instance-Optimal Systems Heuristics Through LLM-Driven Search](http://arxiv.org/abs/2512.25065v1)
  > **TL;DR**: Proposes Vulcan, which synthesizes instance-optimal heuristics for resource-management tasks using LLM-driven evolutionary search. Vulcan provides task-agnostic interfaces to enable LLMs to generate executable policies for specific workloads/hardware. Demonstrates up to 69% performance improvement over state-of-the-art algorithms for cache eviction.
* `training` `serving` `networking` [Reliable and Resilient Collective Communication Library for LLM Training and Serving](http://arxiv.org/abs/2512.25059v1)
  > **TL;DR**: Addresses network failures during large-scale LLM training and serving. Designs R^2CCL, a fault-tolerant communication library with rapid connection migration and resilient collective algorithms. Achieves >12x speedup over baselines with 1% training and 3% inference overheads.
* `scaling` `networking` [AI-Driven Cloud Resource Optimization for Multi-Cluster Environments](http://arxiv.org/abs/2512.24914v1)
  > **TL;DR**: Addresses inefficient resource management in multi-cluster cloud environments. Introduces an AI-driven framework with predictive learning and cross-cluster coordination for dynamic resource allocation. Achieves up to 25% higher resource efficiency and faster stabilization during workload fluctuations.
* `serving` `networking` `storage` [Adaptive Resource Orchestration for Distributed Quantum Computing Systems](http://arxiv.org/abs/2512.24902v1)
  > **TL;DR**: Proposes ModEn-Hub architecture for orchestrating distributed quantum computing resources. Integrates a photonic interconnect with a quantum network orchestrator that schedules non-local gates and manages shared quantum memory and ebit caching. Achieves 90% teleportation success versus 30% in baseline while maintaining higher entanglement attempts per node.
* `training` `edge` `offloading` [Distributed Bilevel Optimization with Dual Pruning for Resource-limited Clients](http://arxiv.org/abs/2512.24667v1)
  > **TL;DR**: Proposes a resource-adaptive distributed bilevel optimization framework for low-resource clients, using a second-order free hypergradient estimator. Achieves asymptotically optimal convergence rate of O(1/√(Cₓ*Q)), validated on two tasks.
* `training` `storage` [Understanding LLM Checkpoint/Restore I/O Strategies and Patterns](http://arxiv.org/abs/2512.24511v1)
  > **TL;DR**: Investigates efficient I/O strategies for LLM checkpoint/restore operations during training. Proposes microbenchmarks and techniques including file system-aware aggregation and coalescing with liburing. Achieves up to 7.6× higher write throughput compared to existing systems.
* `offloading` `sparse` `quantization` [PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression](http://arxiv.org/abs/2512.24449v1)
  > **TL;DR**: Proposes PackKV, a lossy compression framework for KV cache in long-context LLM inference, reducing memory footprint. Combines tailored compression algorithms with system co-design to minimize decompression overhead. Achieves 153.2% higher K cache and 179.6% V cache memory reduction vs. SOTA quantization at same accuracy drop.
* `offline` `video` `quantization` [RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analytics](http://arxiv.org/abs/2512.24386v1)
  > **TL;DR**: Proposes RedunCut for dynamic model size selection in live video analytics. Uses measurement-driven sampling and data-driven accuracy prediction. Reduces compute cost by 14-62% at fixed accuracy for road-vehicle, drone, and surveillance videos.
* `edge` `serving` `kernel` [Squeezing Edge Performance: A Sensitivity-Aware Container Management for Heterogeneous Tasks](http://arxiv.org/abs/2512.23952v1)
  > **TL;DR**: Optimizes container-based resource management for heterogeneous tasks on edge servers. Proposes a sensitivity-aware framework using profiling-based latency modeling and MINLP optimization decomposed into convex subproblems. Achieves over 14% latency reduction and higher energy efficiency versus baselines.
* `edge` `training` `RL` [Vulcan: Instance-Optimal Systems Heuristics Through LLM-Driven Search](http://arxiv.org/abs/2512.25065v1)
  > **TL;DR**: Proposes Vulcan, a system using LLM-driven evolutionary search to synthesize instance-optimal heuristics for resource management. Separates policy and mechanism via LLM-friendly interfaces for synthesizing OS/distributed system policies. Achieves up to 69% higher performance over state-of-the-art in cache eviction and memory tiering.
* `serving` `offloading` [MSched: GPU Multitasking via Proactive Memory Scheduling](http://arxiv.org/abs/2512.24637v1)
  > **TL;DR**: Addresses high overheads in GPU multitasking due to HBM capacity limits and demand paging. Proposes MSched, an OS-level scheduler with proactive memory migration using template-based working set prediction and global page placement. Improves performance by up to 11.05x for DL workloads and 57.88x for LLMs.

### 2025-12-30
* `networking` `scaling` `storage` [Local Rendezvous Hashing: Bounded Loads and Minimal Churn via Cache-Local Candidates](http://arxiv.org/abs/2512.23434v1)
  > **TL;DR**: Proposes Local Rendezvous Hashing to reduce load imbalance and churn in distributed systems. Restricts hash selection to cache-local physical nodes to improve locality. Achieves a max/avg load of 1.0947 vs. 1.2785 and 6.8× faster throughput than multi-probe hashing for K=50M keys.
* `scaling` `serving` [An SLO Driven and Cost-Aware Autoscaling Framework for Kubernetes](http://arxiv.org/abs/2512.23415v1)
  > **TL;DR**: Addresses SLO violations and cost inefficiencies in Kubernetes autoscaling for cloud-native applications. Proposes an SLO-driven, cost-aware framework with AIOps principles and lightweight forecasting. Reduces SLO violation duration by 31%, lowers cost by 18%, and improves response time by 24%.
* `serving` `edge` `scaling` [Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL](http://arxiv.org/abs/2512.23310v1)
  > **TL;DR**: Proposes Splitwise, a Lyapunov-assisted DRL framework for adapting LLM inference partitions between edge devices and cloud to reduce latency and energy. Divides layers into sub-blocks for fine-grained decisions. Achieves 1.4x-2.8x lower latency and up to 41% energy savings versus baselines.
* `serving` `quantization` `offloading` [Viability and Performance of a Private LLM Server for SMBs: A Benchmark Analysis of Qwen3-30B on Consumer-Grade Hardware](http://arxiv.org/abs/2512.23029v1)
  > **TL;DR**: Investigates deploying private LLM servers for small businesses using quantized MoE model on consumer hardware. Benchmarks quantized Qwen3-30B performance under load, focusing on latency, tokens/second and scalability. Achieves comparable performance to cloud services while reducing cost and preserving privacy.
* `serving` `offloading` `edge` [Argus: Token Aware Distributed LLM Inference Optimization](http://arxiv.org/abs/2512.22925v1)
  > **TL;DR**: Addresses efficiency challenges in distributed LLM inference across edge-cloud systems with variable token lengths. Proposes Argus with token-length prediction and Lyapunov-based offloading optimization. Achieves robust performance in dynamic environments with 30-50% latency reduction.
* `training` `networking` `scaling` [OptiNIC: A Resilient and Tail-Optimal RDMA NIC for Distributed ML Workloads](http://arxiv.org/abs/2512.22743v1)
  > **TL;DR**: Addresses tail latency in collective communication for distributed ML workloads. Proposes OptiNIC, an RDMA transport eliminating retransmissions and in-order delivery, using adaptive timeouts and shifting loss recovery to the ML pipeline. Achieves 3.5x lower 99th-percentile latency and 2x higher time-to-accuracy.
* `multi-modal` `serving` [Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference](http://arxiv.org/abs/2512.22695v1)
  > **TL;DR**: Analyzes energy inefficiency in multimodal LLM inference due to modality inflation. Breaks down stages (vision encoding, prefill, decoding) and identifies GPU underutilization. Proposes stage-wise DVFS optimization, achieving up to 94% overhead reduction with minimal performance impact.
* `RL` `MoE` `disaggregation` [RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure](http://arxiv.org/abs/2512.22560v1)
  > **TL;DR**: Addresses inefficiencies in agentic RL training workloads on disaggregated infrastructure. Proposes RollArc with hardware-affinity workload mapping, fine-grained asynchrony, and statefulness-aware computation. Achieves 1.35-2.05× end-to-end training time reduction on large-scale MoE models.
* `training` `RL` [Role-Based Fault Tolerance System for LLM RL Post-Training](http://arxiv.org/abs/2512.22492v1)
  > **TL;DR**: Addresses fault tolerance for RL post-training of LLMs by proposing RobustRL with role-based isolation for trainer, rollout, and management roles. Introduces role-aware failure detection, non-disruptive recovery, warm standbys, isolated replacement, and dynamic UCX-based reconnection. Achieves 80% ETTR (vs 60%) and 8.4%-17.4% faster training time under failure.
* `serving` `quantization` [Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving](http://arxiv.org/abs/2512.22420v1)
  > **TL;DR**: Addresses the trade-off of fixed-length speculative decoding in LLM inference under varying loads. Proposes Nightjar, a learning-based algorithm that adaptively adjusts speculative length or disables SD. Achieves 14.8% higher throughput and 20.2% lower latency compared to standard SD.
* `serving` `scaling` `storage` [Efficient Multi-Model Orchestration for Self-Hosted Large Language Models](http://arxiv.org/abs/2512.22402v1)
  > **TL;DR**: Proposes Pick and Spin, a Kubernetes-based orchestration framework for self-hosted LLMs with Helm deployment, scale-to-zero automation, and hybrid routing. Achieves 30% lower latency and 33% lower GPU cost per query versus static deployments.
* `storage` `kernel` [Valori: A Deterministic Memory Substrate for AI Systems](http://arxiv.org/abs/2512.22280v1)
  > **TL;DR**: Addresses floating-point non-determinism in AI vector embedding storage. Proposes Valori, a deterministic memory substrate using fixed-point arithmetic (Q16.16) and replayable state machine modeling. Guarantees bit-identical memory states and search results across hardware platforms.
* `edge` `serving` `scaling` [Scalable Cloud-Native Architectures for Intelligent PMU Data Processing](http://arxiv.org/abs/2512.22231v1)
  > **TL;DR**: Proposes a cloud-native framework for low-latency PMU stream processing using distributed edge-cloud orchestration and containerized services. Achieves sub-second response times at scale for real-time power grid analytics.

### 2025-12-29
* `RL` `RAG` `serving` [Agentic Structured Graph Traversal for Root Cause Analysis of Code-related Incidents in Cloud Applications](http://arxiv.org/abs/2512.22113v1)
  > **TL;DR**: Proposes PRAXIS, an LLM-driven orchestrator for root cause analysis using structured graph traversal over service and code dependency graphs. Agents traverse graphs as policies to localize failures. Improves accuracy by 3.1x and reduces tokens by 3.8x versus ReAct baselines.
* `MoE` `training` `networking` [FUSCO: High-Performance Distributed Data Shuffling via Transformation-Communication Fusion](http://arxiv.org/abs/2512.22036v1)
  > **TL;DR**: Addresses inefficiency in distributed data shuffling for MoE model parallelism. Develops FUSCO, a communication library that fuses transformation and communication with lightweight load balancing. Achieves up to 3.84× speedup over NCCL and reduces training latency by 1.17-1.39×.
* `serving` `offloading` `networking` [Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models](http://arxiv.org/abs/2512.21884v1)
  > **TL;DR**: Addresses optimal resource allocation for geographically-distributed LLM inference. Proposes an offline MILP formulation and online algorithm for block placement and request routing. Reduces inference time by up to 50% compared to PETALS in diverse geographical settings.
* `edge` `offloading` `serving` [LIME:Accelerating Collaborative Lossless LLM Inference on Memory-Constrained Edge Devices](http://arxiv.org/abs/2512.21835v1)
  > **TL;DR**: Proposes LIME for collaborative lossless LLM inference on memory-constrained edge devices. Uses interleaved pipeline parallelism, model offloading, and adaptive scheduling. Achieves 1.7–3.7x speedup over state-of-the-art baselines on Jetson devices for LLaMA3.3-70B inference.
* `video` `multi-modal` `offloading` [Hyperion: Low-Latency Ultra-HD Video Analytics via Collaborative Vision Transformer Inference](http://arxiv.org/abs/2512.21730v1)
  > **TL;DR**: Proposes Hyperion, a cloud-device collaborative framework for low-latency Ultra-HD video analytics using vision transformers. Integrates importance scoring, dynamic scheduling, and weighted ensembling to optimize transmission and computation. Achieves 1.61x higher frame rate and 20.2% accuracy gain.
* `offline` `training` `networking` [Embedding Samples Dispatching for Recommendation Model Training in Edge Environments](http://arxiv.org/abs/2512.21615v1)
  > **TL;DR**: Reduces embedding transmission cost in distributed edge training for deep learning recommendation models (DLRMs). Proposes ESD with HybridDis, a hybrid dispatch algorithm optimizing sample-to-worker assignment. Achieves up to 36.76% lower transmission cost and 1.74× training speedup.
* `kernel` `storage` `hardware` [nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures](http://arxiv.org/abs/2512.21571v1)
  > **TL;DR**: Introduces nncase, an end-to-end compiler that employs e-graph-based term rewriting to optimize LLM deployment across heterogeneous targets. It integrates Auto Vectorize, Auto Distribution, and Auto Schedule modules for joint computation-data movement optimization. Outperforms MLC LLM and Intel IPEX, matching hand-optimized llama.cpp on CPUs.
* `MoE` `serving` `offloading` [Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism](http://arxiv.org/abs/2512.21487v1)
  > **TL;DR**: Addresses memory-intensive MoE inference by proposing FinDEP, a fine-grained task scheduling algorithm for disaggregated expert parallelism. Optimizes computation/communication task overlap, variable granularity, and scheduling, achieving up to 1.61x throughput improvement over baselines.
* `training` `kernel` `hardware` [Demystifying ARM SME to Optimize General Matrix Multiplications](http://arxiv.org/abs/2512.21473v1)
  > **TL;DR**: Optimizes GEMM performance for deep learning on ARM SME architectures. Proposes MpGEMM with cache-aware partitioning, data packing, and SME-specific micro-kernels. Achieves 1.23x average speedup over vendor library on Apple M4 Pro with DeepSeek/LLaMA workloads.

### [🔥Daily Arxiv: LLM Systems 👉 paper.tju.chat 👈](https://paper.tju.chat)

</div>

**👍Conference Papers on LMSys**: [conference.md](conference.md)

**⚠️NOTE**: Update papers up to last day every morning (8:00 UTC+8) automatically.

**🙋WANT**: Keyword subscription (email); Functional web page.

---
### 2025-12-27
* `training` `offline` `storage` [Optimizing Frequent Checkpointing via Low-Cost Differential for Distributed Training Systems](http://arxiv.org/abs/2509.04084v3)
  > **TL;DR**: Proposes \sysname and \sysnameplus to reduce checkpointing overhead in distributed training. Uses compressed gradients as differential checkpoints and batched writes for efficient persistence, with dynamic tuning. Reduces training time by 89.2% with per-iteration checkpointing.
* `training` `MoE` `quantization` [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](http://arxiv.org/abs/2505.09343v2)
  > **TL;DR**: Addresses hardware scalability challenges in LLM training via co-design with DeepSeek-V3. Proposes MLA attention, MoE optimization, FP8 precision, and network topology innovations. Achieves cost-efficient training on 2048 H800 GPUs with 20% higher throughput versus FP16.
* `kernel` `sparse` `hardware` [Parallel GPU-Enabled Algorithms for SpGEMM on Arbitrary Semirings with Hybrid Communication](http://arxiv.org/abs/2504.06408v2)
  > **TL;DR**: Proposes GPU-accelerated distributed SpGEMM with hybrid communication for efficiency. Implements on CombBLAS with dynamic switching between host/device communication paths. Achieves 2-3x speedup over CPU-only CombBLAS and PETSc for large sparse matrices.
* `kernel` `sparse` `hardware` [Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication](http://arxiv.org/abs/2506.22714v2)
  > **TL;DR**: Proposes Libra, a framework to accelerate sparse matrix multiplication on heterogeneous GPUs by optimal task distribution balancing Tensor Cores and CUDA cores. Achieves up to 2.9x speedup in GNN applications.
* `kernel` `training` [LEGO: A Layout Expression Language for Code Generation of Hierarchical Mapping](http://arxiv.org/abs/2505.08091v2)
  > **TL;DR**: Presents LEGO, a layout expression language for optimizing data movement in GPU computations. Generates complex indexing expressions for hierarchical mappings, integrated with Triton, MLIR, and CUDA. Achieves performance competitive with Triton for data mapping optimizations and thread layouts.
* `offloading` `kernel` `storage` [The Impact of Partial Computations on the Red-Blue Pebble Game](http://arxiv.org/abs/2506.10854v2)
  > **TL;DR**: Studies how partial computation steps reduce I/O cost in red-blue pebble game models. Extends game to allow step-by-step input aggregation, enabling smaller I/O costs. Shows I/O cost can decrease by up to a linear factor in certain settings, but remains NP-hard to optimize.
* `training` `hardware` `scaling` [Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs](http://arxiv.org/abs/2503.11901v4)
  > **TL;DR**: Characterizes resilience of A100 and H100 GPUs in large-scale AI training systems using operational data. Analyzes error rates, recovery mechanisms, and failure impacts across 1,056 GPUs. Projects 5% overprovisioning needed to handle failures at scale, with H100 showing higher memory error rates but critical hardware improvements.
* `training` `storage` `offline` [Redox: Improving I/O Efficiency of Model Training Through File Redirection](http://arxiv.org/abs/2505.16280v2)
  > **TL;DR**: Addresses slow I/O in model training by redirecting file reads. Proposes Redox with batch reading and opportunistic prefetch to minimize wasted disk reads. Achieves up to 4.57x faster training than PyTorch.
* `training` `multi-modal` `scaling` [Efficient Distributed MLLM Training with Cornstarch](http://arxiv.org/abs/2503.11367v3)
  > **TL;DR**: Addresses inefficiency in training multimodal LLMs due to heterogeneity. Proposes Cornstarch framework with frozen-aware pipeline parallelism and token workload-balanced context parallelism. Achieves 2.26x higher training throughput versus state-of-the-art.
* `serving` `scheduling` `kernel` [Optimal Scheduling Algorithms for LLM Inference: Theory and Practice](http://arxiv.org/abs/2508.01002v2)
  > **TL;DR**: Proposes SLO-Aware LLM Inference (SLAI) scheduler optimizing prefill-decode phases with dynamic resource allocation and deadline prioritization. Reduces median TTFT by 53% and increases serving capacity by 26% under TBT constraints.
* `kernel` `training` `RL` [CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning](http://arxiv.org/abs/2507.14111v9)
  > **TL;DR**: Proposes CUDA-L1, a contrastive RL framework for automated CUDA kernel optimization. Uses reinforcement learning with speedup-based rewards to train an LLM-based optimizer, discovering and combining optimization techniques. Achieves average 3.12x speedup on KernelBench kernels over baselines.
* `serving` `RAG` `offloading` [Understanding and Optimizing Multi-Stage AI Inference Pipelines](http://arxiv.org/abs/2504.09775v4)
  > **TL;DR**: Proposes HERMES, a multi-stage LLM inference simulator for complex pipelines (RAG, KV retrieval, reasoning). Models heterogeneous hardware and advanced batching to analyze latency, memory contention, and communication. Achieves insights on optimal batching and architecture for 53% lower latency in hybrid deployments.
* `training` `recommendation` `scaling` [MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan](http://arxiv.org/abs/2505.12663v2)
  > **TL;DR**: Addresses inefficient training of generative recommendation models in industry. Proposes MTGenRec with dynamic hash tables, sequence balancing, and feature deduplication for embedding management. Achieves 1.6×–2.4× higher training throughput and scales to over 100 GPUs.
* `serving` `edge` `scaling` [FailLite: Failure-Resilient Model Serving for Resource-Constrained Edge Environments](http://arxiv.org/abs/2504.15856v2)
  > **TL;DR**: Proposes FailLite, a failure-resilient model serving system for edge environments using heterogeneous replica variants, warm/cold replicas, and progressive failover. Achieves 175.5ms mean time to recovery with 0.6% accuracy drop for 27 models.
* `training` `offline` `LoRA` [From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow](http://arxiv.org/abs/2509.12443v3)
  > **TL;DR**: Proposes an autonomous agentic AI workflow for translating and optimizing Fortran to Kokkos C++. Specialized LLM agents collaborate to generate performance-portable code across hardware. Achieves optimized codes surpassing Fortran baselines at a cost of a few U.S. dollars with paid models.
* `training` [Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation](http://arxiv.org/abs/2503.20191v2)
  > **TL;DR**: Presents Maya, a GPU runtime emulation system for optimizing deep learning training workloads. It intercepts device API calls from unmodified training code to model performance without semantic gaps. Achieves <5% prediction error and reduces training costs by up to 56%.
* `offloading` `edge` `serving` [To Offload or Not To Offload: Model-driven Comparison of Edge-native and On-device Processing In the Era of Accelerators](http://arxiv.org/abs/2504.15162v3)
  > **TL;DR**: Addresses when to offload ML inference tasks to edge servers versus local device processing. Develops queuing theory-based analytical models for latency prediction and adaptive offloading. Achieves 2.2% mean absolute error in latency modeling across varying workloads and network conditions.
* `training` `scaling` `kernel` [Mapple: A Domain-Specific Language for Mapping Distributed Programs](http://arxiv.org/abs/2507.17087v2)
  > **TL;DR**: Proposes Mapple, a declarative programming interface for mapping distributed programs onto high-performance systems. It resolves dimensionality mismatches via decompose primitives to minimize communication, reducing mapper code by 14× while improving performance up to 1.34× over expert implementations.
* `offloading` `edge` `serving` [Inference Offloading for Cost-Sensitive Binary Classification at the Edge](http://arxiv.org/abs/2509.15674v2)
  > **TL;DR**: Optimizes edge-based binary classification by dynamically offloading uncertain samples to a remote model. Proposes H2T2, an online learning policy adapting confidence thresholds for offloading decisions. Achieves up to 1.7% accuracy gain with 45% cost reduction compared to baselines under distribution shifts.
* `RL` `training` `LoRA` [EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models](http://arxiv.org/abs/2506.02001v2)
  > **TL;DR**: Proposes EcoLoRA, a communication-efficient FL framework for LLM fine-tuning using round-robin LoRA segment sharing, adaptive sparsification, and encoding. Reduces communication time by up to 79% and total training time by 65% without performance loss.
* `scaling` `storage` `networking` [GPUnion: Autonomous GPU Sharing on Campus](http://arxiv.org/abs/2507.18928v2)
  > **TL;DR**: Addresses campus-wide GPU underutilization by designing GPUnion, an autonomous sharing platform with container-based dispatching, provider-first architecture, and resilient execution via checkpointing/migration. Achieves 30% higher GPU utilization and 94% successful workload migration.
* `training` `serving` `quantization` [AIMeter: Measuring, Analyzing, and Visualizing Energy and Carbon Footprint of AI Workloads](http://arxiv.org/abs/2506.20535v2)
  > **TL;DR**: Presents AIMeter, a toolkit for measuring energy, power, hardware metrics, and carbon emissions in AI workloads. Integrates with AI frameworks to provide fine-grained time-series data and correlation analysis for bottleneck identification. Achieves lightweight overhead and enables sustainability benchmarking.
* `training` [MinatoLoader: Accelerating Machine Learning Training Through Efficient Data Preprocessing](http://arxiv.org/abs/2509.10712v2)
  > **TL;DR**: Addresses inefficient data preprocessing causing GPU idleness in ML training. Proposes MinatoLoader, which prioritizes fast-to-preprocess samples and parallelizes slow samples. Achieves up to 7.5× training acceleration and 90.45% GPU utilization vs. PyTorch's 46.4%.
* `training` `storage` `scaling` [FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline](http://arxiv.org/abs/2507.10367v3)
  > **TL;DR**: Investigates inefficiency of client-side metadata caching in distributed file systems for DL pipelines. Proposes FalconFS with stateless-client architecture using server-side path resolution, hybrid indexing, and concurrent request merging. Achieves up to 12.81× throughput in model training and supports 10,000 NPUs in production.
* `sparse` `inference` `training` [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](http://arxiv.org/abs/2507.03117v2)
  > **TL;DR**: Proposes BLaST, a method for block sparsity in Transformers to reduce data movement costs. Achieves 95% weight sparsity with <2.25% accuracy loss, demonstrating 2.2x inference speedup for Llama and 4.45x memory reduction.
* `serving` `scaling` `offline` [Minos: Exploiting Cloud Performance Variation with Function-as-a-Service Instance Selection](http://arxiv.org/abs/2505.12928v3)
  > **TL;DR**: Presents Minos, a system that improves FaaS performance by terminating slow instances after a benchmark test and reusing fast instances. It uses instance selection to reduce execution time and cost, achieving up to 13% speedup in workflow-intensive parts and 4% overall cost reduction.
* `training` `sparse` `scaling` [Intelligent Sampling of Extreme-Scale Turbulence Datasets for Accurate and Efficient Spatiotemporal Model Training](http://arxiv.org/abs/2508.03872v3)
  > **TL;DR**: Proposes SICKLE framework for intelligent sparse subsampling to reduce data volume in extreme-scale turbulence model training. Features maximum entropy sampling and scalable training. Achieves up to 38x lower energy consumption while improving accuracy.
* `serving` `multi-modal` `scaling` [ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving](http://arxiv.org/abs/2502.00937v3)
  > **TL;DR**: Addresses high-cost, low-throughput serving of large multimodal models (LMMs). Proposes ModServe, a modular system with stage disaggreation, modality-aware scheduling, and autoscaling. Achieves 3.3-5.5× higher throughput (25-41.3% cost saving) while meeting tail latency SLOs.
* `offline` `training` [The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution](http://arxiv.org/abs/2501.12407v5)
  > **TL;DR**: Addresses inefficiency in heterogeneous GPU/CPU resource utilization for data processing. Proposes the streaming batch model with dynamic partitioning for pipelining in Ray Data. Improves batch inference throughput by 2.5-12x and multimodal training by 31%.
* `kernel` `training` `hardware` [Linear Layouts: Robust Code Generation of Efficient Tensor Computation Using $\mathbb{F}_2$](http://arxiv.org/abs/2505.23819v3)
  > **TL;DR**: Proposes Linear Layouts, a tensor layout modeling approach using linear algebra over $\mathbb{F}_2$ for flexible and performant tensor computation. Integrates with Triton compiler to optimize operators/kernels, reducing engineering effort and fixing bugs in existing layout systems.
* `serving` `scaling` [Hierarchical Prediction-based Management for LMaaS Systems](http://arxiv.org/abs/2504.03702v2)
  > **TL;DR**: Proposes PreServe, a hierarchical prediction-based management framework for LMaaS. It uses workload and load predictors to anticipate demand, enabling proactive resource scaling and request routing. Reduces tail latency by 41.3% and resource consumption by 49.38% with minimal overhead.
* `training` `RL` `scaling` [Scaling Multi Agent Reinforcement Learning for Underwater Acoustic Tracking via Autonomous Vehicles](http://arxiv.org/abs/2505.08222v2)
  > **TL;DR**: Proposes GPU-accelerated iterative distillation for scaling multi-agent RL training by transferring high-fidelity simulations to simplified environments. Achieves a 30,000x speedup over Gazebo, enabling large-scale curriculum training with Transformer-based policies invariant to agent count.
* `training` `MoE` `scaling` [SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling](http://arxiv.org/abs/2504.19925v2)
  > **TL;DR**: Addresses inefficiency in MoE training due to token imbalance and expert load variability. Decouples expert parameter placement from optimizer state, statically partitioning optimizer across nodes and dynamically adjusting parameters without migration overhead. Achieves 30.5% and 25.9% faster convergence versus DeepSpeed and FlexMoE.
* `RAG` `storage` `networking` [Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant](http://arxiv.org/abs/2509.12384v2)
  > **TL;DR**: Investigates vector database performance for retrieval-augmented generation (RAG) in HPC systems. Measures Qdrant's insertion, indexing, and query latency scaling on 32 workers with biological text embeddings. Reports query latency reduction and throughput scaling on Polaris supercomputer.
* `serving` `disaggregation` `networking` [MoLink: Distributed and Efficient Serving Framework for Large Models](http://arxiv.org/abs/2507.05043v2)
  > **TL;DR**: Addresses efficient distributed serving of large models on heterogeneous consumer-grade GPUs with limited network conditions. Introduces MoLink, incorporating techniques like model chaining and pipelined execution over weak links. Achieves up to 458% throughput improvement and 151% cost-profit margin gain versus state-of-the-art systems.
* `kernel` `video` [TC-GS: A Faster Gaussian Splatting Module Utilizing Tensor Cores](http://arxiv.org/abs/2505.24796v2)
  > **TL;DR**: Proposes TC-GS for accelerating Gaussian Splatting rendering via mapping alpha computation to Tensor Core matrix multiplication. Introduces coordinate transformation to reduce precision errors. Achieves 2.18x speedup over existing methods, totaling 5.6x faster than baselines.
* `MoE` `training` `networking` [FAST: An Efficient Scheduler for All-to-All GPU Communication](http://arxiv.org/abs/2505.09764v2)
  > **TL;DR**: Addresses slow scheduling for skewed All-to-All communications in MoE training. Proposes FAST, an efficient scheduler using intra-server rebalancing and one-to-one scale-out transfers to avoid incast. Reduces synthesis time by orders of magnitude versus prior work in NVIDIA H200/AMD MI300X clusters.
* `training` `MoE` `offloading` [Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices](http://arxiv.org/abs/2508.19078v2)
  > **TL;DR**: Proposes FLUX for federated fine-tuning of MoE LLMs on resource-constrained devices. Features quantization-based profiling, adaptive expert merging, and dynamic expert assignment. Achieves up to 4.75x speedup in time-to-accuracy.
* `training` `quantization` `scaling` [Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimation](http://arxiv.org/abs/2505.01616v3)
  > **TL;DR**: Proposes Phantora, a hybrid GPU cluster simulator for ML training performance estimation. Executes unmodified ML frameworks in containerized emulated environments, intercepting GPU and comms ops. Reduces estimation overhead by eliminating trace capture, supporting modern frameworks on a single GPU.
* `networking` `training` `scaling` [Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering](http://arxiv.org/abs/2504.19519v2)
  > **TL;DR**: Addresses communication bottleneck in multi-GPU training of generative models. Proposes FlashOverlap, using signaling and reordering for tile-wise overlapping of computation and communication without interference. Achieves up to 1.65x speedup over existing methods.
* `training` `quantization` `kernel` [Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs](http://arxiv.org/abs/2501.05408v3)
  > **TL;DR**: Addresses the challenge of optimizing dynamic dependencies in DL programs with temporal relationships. Introduces Tempo, a system with symbolic dependence graphs enabling compiler optimizations like tiling and fusion. Achieves 7x speedup and 16x lower peak memory usage for models like Llama-3.2-3B.
* `serving` `diffusion` `kernel` [PATCHEDSERVE: A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving](http://arxiv.org/abs/2501.09253v2)
  > **TL;DR**: Addresses inefficient diffusion model serving under mixed-resolution workloads. Proposes patch-based workflow with specialized caching and SLO scheduling using lightweight latency prediction. Achieves 30.1% higher SLO satisfaction than state-of-the-art.
* `RL` `LoRA` `scaling` [Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning](http://arxiv.org/abs/2502.15436v2)
  > **TL;DR**: Proposes Fed-SB, a federated fine-tuning method using LoRA-SB adapter updates with a shared small matrix. Reduces communication costs by 230x while maintaining performance via exact adapter averaging, and enhances privacy by lowering noise requirements.
* `training` `offloading` `scaling` [NotebookOS: A Replicated Notebook Platform for Interactive Training with On-Demand GPUs](http://arxiv.org/abs/2503.20591v2)
  > **TL;DR**: Proposes NotebookOS, a replicated notebook platform for interactive deep learning training by oversubscribing server resources and allocating GPUs only during active cell execution. Achieves 1,187 GPU hours saved in 17.5 hours, improving utilization and interactivity.
* `serving` `kernel` [Lobster: A GPU-Accelerated Framework for Neurosymbolic Programming](http://arxiv.org/abs/2503.21937v2)
  > **TL;DR**: Proposes Lobster, a GPU-accelerated framework for neurosymbolic programming via compilation to an intermediate language with optimization passes. Achieves 3.9x speedup over Scallop in 9 applications.
* `training` `scaling` `networking` [Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters](http://arxiv.org/abs/2504.17096v2)
  > **TL;DR**: Addresses efficient distributed training over heterogeneous, geo-distributed clusters. Proposes Sailor with exploration algorithm, simulator, and framework supporting heterogeneity. Optimizes throughput and cost via dynamic resource utilization.
* `kernel` [Segmented Operations using Matrix Multiplications](http://arxiv.org/abs/2506.23906v2)
  > **TL;DR**: Proposes MMV-RAM, a computational model integrating matrix multiplication units for efficient parallel operations. Designs algorithms like segmented scan/sum using MMUs for speculative blocked computations. Achieves speed-ups on Ascend 910B, demonstrated with LLM-relevant kernels like sparse matrix products.
* `serving` `scaling` `training` [Scheduler-Driven Job Atomization](http://arxiv.org/abs/2509.19086v1)
  > **TL;DR**: Addresses GPU cluster inefficiency by atomizing jobs into subjobs for dynamic scheduling. Proposes scheduler-driven job atomization with bidirectional gap advertising and job signaling to fit time-capacity windows. Reduces job rejection rates and improves utilization by minimizing fragmentation.
* `serving` `quantization` [Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism](http://arxiv.org/abs/2506.01979v2)
  > **TL;DR**: Proposes SpecBranch, a speculative decoding framework with branch parallelism and hybrid drafting to reduce mutual waiting bubbles. Introduces adaptive draft lengths and rollback-aware branches. Achieves 1.8×–4.5× speedups over auto-regressive decoding and 50% fewer rollback tokens.
* `networking` `training` `inference` [Whack-a-Mole: Deterministic Packet Spraying Across Multiple Network Paths](http://arxiv.org/abs/2509.18519v1)
  > **TL;DR**: Proposes Whack-a-Mole, a deterministic packet spraying algorithm for multipath transport to reduce tail latency and imbalance in distributed AI training/inference. Uses bit-reversal counter and dynamic allocation to achieve O(log m) discrepancy bound, improving GPU utilization and collective completion time.
* `kernel` [XaaS Containers: Performance-Portable Representation With Source and IR Containers](http://arxiv.org/abs/2509.17914v1)
  > **TL;DR**: Proposes Source and IR Containers for performance-portable HPC deployment, delaying compilation until runtime to incorporate hardware-specific optimizations. Uses LLM-assisted discovery of specializations and compile-time optimization to achieve peak performance. Prototype shows performance comparable to system-specialized builds.
* `scaling` `storage` `training` [Enhancing Cluster Scheduling in HPC: A Continuous Transfer Learning for Real-Time Optimization](http://arxiv.org/abs/2509.22701v1)
  > **TL;DR**: Proposes continuous transfer learning for real-time cluster task scheduling optimization. Dynamically adapts scheduling policy during operations to reduce retraining. Achieves over 99% accuracy with reduced overhead and latency on HPC workload traces.
* `MoE` `training` `serving` [LongCat-Flash Technical Report](http://arxiv.org/abs/2509.01322v2)
  > **TL;DR**: Presents LongCat-Flash, a 560B MoE model with novel architectures (Zero-computation Experts and Shortcut-connected MoE) for efficient training and inference. Achieves training on 20T tokens in 30 days and inference at 100+ TPS costing $0.70 per million output tokens.
* `edge` `scheduling` `networking` [KubeDSM: A Kubernetes-based Dynamic Scheduling and Migration Framework for Cloud-Assisted Edge Clusters](http://arxiv.org/abs/2501.07130v2)
  > **TL;DR**: Aims to improve resource management in cloud-assisted edge clusters for latency-sensitive apps. Proposes KubeDSM, a Kubernetes-based framework with batch scheduling and live migration to minimize fragmentation. Achieves higher average edge ratio and lower standard deviation in edge ratios.
* `diffusion` `edge` `inference` [Conditional Prior-based Non-stationary Channel Estimation Using Accelerated Diffusion Models](http://arxiv.org/abs/2509.15182v1)
  > **TL;DR**: Proposes a diffusion-based method for non-stationary wireless channel estimation, using conditional priors and accelerated sampling to improve accuracy. Achieves lower NMSE than baselines across all SNRs on 3GPP benchmarks.
* `training` `scaling` [Cost-Performance Analysis: A Comparative Study of CPU-Based Serverless and GPU-Based Training Architectures](http://arxiv.org/abs/2509.14920v1)
  > **TL;DR**: Compares serverless (SPIRT) and GPU-based training architectures for distributed ML. SPIRT leverages parallel batch processing and in-database operations via RedisAI to reduce costs and training times. Achieves significant reductions in training time and communication overhead versus traditional systems.
* `scaling` [Precomputed Dominant Resource Fairness](http://arxiv.org/abs/2507.08846v3)
  > **TL;DR**: Addresses efficient computation of fair multi-resource allocation in distributed systems. Proposes Precomputed Dominant Resource Fairness, an approximation algorithm reducing steps to compute Dominant Resource Fairness allocations. Achieves comparable fairness with fewer computational iterations.
* `MoE` `training` `inference` [When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Models](http://arxiv.org/abs/2509.12141v2)
  > **TL;DR**: Proposes a blockchain-based trustworthy distributed MoE framework for LMs to enhance security against data manipulation attacks during training and inference. Integrates edge computing with decentralized verification, improving robustness while maintaining learning performance.
* `scaling` `training` [Cloud abstractions for AI workloads](http://arxiv.org/abs/2501.09562v2)
  > **TL;DR**: Proposes HarmonAIze, a framework for cloud abstraction to coordinate AI workload optimization between tenants and providers. Focuses on partitioning, scheduling, and fault tolerance for improved resource efficiency and resiliency. Achieves enhanced performance in multi-tenant environments.
* `training` `scaling` `scheduling` [GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management](http://arxiv.org/abs/2509.11134v1)
  > **TL;DR**: Addresses inefficiencies in GPU scheduling for LLM workloads with preemptible spot instances. Proposes GFS framework with demand forecasting, dynamic spot quota allocation, and preemption-aware scheduling. Reduces evictions by 33.0% and queuing delays by 44.1% for LP tasks.
* `training` `offloading` `sparse` [Chameleon: Taming Dynamic Operator Sequences for Memory-Intensive LLM Training](http://arxiv.org/abs/2509.11076v1)
  > **TL;DR**: Addresses memory-intensive LLM training under dynamic operator sequences in Eager Mode. Proposes Chameleon with online profiler, swap policy generation, and optimized execution. Reduces profiling overhead by 84.25%, enables 4x larger models, improves performance by up to 38.94% over recomputation.
* `scaling` `offloading` `networking` [Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training](http://arxiv.org/abs/2505.12815v2)
  > **TL;DR**: Addresses slow autoscaling and recovery in multi-party distributed training under churn. Proposes Chaos with multi-neighbor state replication, model sharding, and greedy shard assignment that reduces MINLP to MILP. Achieves scale-out delay reduction vs. Pollux/Elan and handles events within 20ms with lowest idle time.
* `serving` `edge` `scaling` [SynergAI: Edge-to-Cloud Synergy for Architecture-Driven High-Performance Orchestration for AI Inference](http://arxiv.org/abs/2509.12252v1)
  > **TL;DR**: Proposes SynergAI, a framework for architecture-aware scheduling of inference workloads across heterogeneous edge-to-cloud infrastructures. Integrates offline and online policies to dynamically allocate tasks, minimizing QoS violations. Reduces violations by 2.4x compared to state-of-the-art solutions.
* `training` `networking` `sparse` [Towards Communication-Efficient Decentralized Federated Graph Learning over Non-IID Data](http://arxiv.org/abs/2509.08409v1)
  > **TL;DR**: Addresses high communication costs in decentralized federated graph learning. Proposes Duplex, a framework that jointly optimizes network topology sparsification and graph neighbor sampling. Reduces training time by 20.1-48.8% and communication costs by 16.7-37.6% while improving accuracy by 3.3-7.9%.
* `video` `multi-modal` `kernel` [Déjà Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse](http://arxiv.org/abs/2506.14107v2)
  > **TL;DR**: Proposes Déjà Vu to accelerate ViT-based Video-Language Models via inter-frame computation reuse. Introduces ReuseViT to detect reuse opportunities and employs memory-compute joint compaction to convert FLOP savings. Achieves up to 2.64× embedding speedup within 2% accuracy error.
* `kernel` `hardware` [Dynamic Memory Management on GPUs with SYCL](http://arxiv.org/abs/2504.18211v2)
  > **TL;DR**: Addresses efficient dynamic memory allocation in GPU kernels for open standards. Ports Ouroboros from CUDA to SYCL for cross-platform support. Achieves performance comparable to CUDA implementation on multiple backends.
* `MoE` `training` `hardware` [Accelerating Frontier MoE Training with 3D Integrated Optics](http://arxiv.org/abs/2510.15893v1)
  > **TL;DR**: Explores using 3D-stacked photonics to overcome interconnect limitations for large-scale MoE training. Proposes GPUs with 3D co-packaged optics for high-bandwidth multi-rack scaling. Achieves 8× scale-up capability and 2.7× reduction in training time for trillion-parameter models.
* `hardware` `kernel` `training` [ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques](http://arxiv.org/abs/2507.11506v2)
  > **TL;DR**: Proposes ELK, a DL compiler framework to optimize compute, communication, and I/O for inter-core connected AI chips. Uses inductive operator scheduling and cost-aware on-chip memory allocation to generate globally optimized execution plans. Achieves 94% of ideal roofline performance.
* `training` `scaling` `MoE` [Scaling Intelligence: Designing Data Centers for Next-Gen Language Models](http://arxiv.org/abs/2506.15006v3)
  > **TL;DR**: Proposes a co-design framework for data centers to scale training of trillion-parameter LLMs. Evaluates network topologies, parallelism strategies, and hardware specs to optimize Model FLOPS Utilization. Achieves accurate performance predictions (within 10% error) and improves throughput with FullFlat networks for both dense and sparse (MoE) models.
* `training` `networking` `storage` [Hiding Latencies in Network-Based Image Loading for Deep Learning](http://arxiv.org/abs/2503.22643v2)
  > **TL;DR**: Addresses data loading bottlenecks in deep learning caused by network/storage latencies. Proposes storing data in NoSQL databases and using out-of-order incremental prefetching for high-latency networks. Achieves high-throughput image loading even over intercontinental networks.
* `edge` `networking` `storage` [Ratio1 -- AI meta-OS](http://arxiv.org/abs/2509.12223v1)
  > **TL;DR**: Proposes Ratio1, a decentralized meta-OS for distributed AI execution across edge devices. Integrates blockchain, federated learning, and novel storage/homomorphic encryption to pool idle resources into a global supercomputer. Lowers deployment barriers and improves cost-efficiency compared to centralized systems.
* `kernel` `hardware` `training` [Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis](http://arxiv.org/abs/2501.12084v2)
  > **TL;DR**: Analyzes performance of NVIDIA Hopper GPU for compute-intensive workloads. Microbenchmarks memory subsystem, tensor cores (FP8, async wgmma), TMA for async data movement, and DPX instructions. TMA achieves 1.5x matmul speedup; FP8 doubles FP16 performance; DPX accelerates bio-algorithm 4.75x.
* `training` `RL` `edge` [Semi-decentralized Federated Time Series Prediction with Client Availability Budgets](http://arxiv.org/abs/2509.03660v1)
  > **TL;DR**: Proposes FedDeCAB, a semi-decentralized client selection method for federated time-series prediction with client availability budgets. Uses probabilistic client ranking and neighbor model parameter sharing to handle disconnections, reducing communication overhead. Achieves effective convergence under data heterogeneity, with 30% lower communication costs in experiments on real-world taxi and vessel trajectory datasets.
* `training` `scaling` `kernel` [ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling](http://arxiv.org/abs/2505.04802v2)
  > **TL;DR**: Scales vision foundation models for climate downscaling via Reslim architecture and TILES algorithm, reducing self-attention complexity to near-linear. Achieves 4.1 exaFLOPS on 65,536 GPUs with 74-98% strong scaling efficiency for 0.9 km resolution.
* `hardware` `storage` `kernel` [Disaggregated Design for GPU-Based Volumetric Data Structures](http://arxiv.org/abs/2503.07898v2)
  > **TL;DR**: Proposes a disaggregated design for GPU-based volumetric data structures to balance locality with occupancy, communication, and fusion. The design reduces communication overhead on multi-GPU systems, mitigates register pressure, and enables kernel fusion. Achieves up to 3× speedup over state-of-the-art LBM solvers.
* `training` `kernel` `hardware` [COMET: A Framework for Modeling Compound Operation Dataflows with Explicit Collectives](http://arxiv.org/abs/2509.00599v1)
  > **TL;DR**: Proposes COMET, a framework for modeling and optimizing dataflows of compound operations (e.g., GEMM-Softmax) on ML accelerators, with explicit collective communication modeling. Achieves up to 3.46× speedup for GEMM-LayerNorm compared to unfused baselines.
* `networking` `training` [FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern](http://arxiv.org/abs/2510.15882v1)
  > **TL;DR**: Addresses heterogeneous inter-GPU communication bottleneck in LLM training. Proposes FlexLink, a collective communication framework that aggregates NVLink, PCIe, and RDMA NICs with two-stage adaptive load balancing. Boosts AllGather bandwidth by 27% via offloading 2-22% traffic to idle links.
* `serving` `MoE` `networking` [Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference](http://arxiv.org/abs/2509.00217v1)
  > **TL;DR**: Co-optimizes parallelism degrees and sharding dimensions for distributed LLM inference via RL with attention-based policy. Achieves up to 3.5x throughput gains over baselines on MoE models with 1.6T parameters on H100 clusters.
* `serving` `kernel` `sparse` [TinyServe: Query-Aware Cache Selection for Efficient LLM Serving](http://arxiv.org/abs/2509.12211v1)
  > **TL;DR**: Reduces LLM serving KV cache overhead via query-aware page selection. Introduces metadata-driven KV sparsity and integrated CUDA kernel for selective loading. Achieves 3.4x speedup and 2x memory savings with minimal accuracy loss.
* `scheduling` `kernel` `hardware` [IsoSched: Preemptive Tile Cascaded Scheduling of Multi-DNN via Subgraph Isomorphism](http://arxiv.org/abs/2509.12208v1)
  > **TL;DR**: Develops IsoSched for preemptive multi-DNN scheduling on tile spatial architectures. Combines ILP-based formulation, layer concatenate/split, and accelerated subgraph isomorphism with CSR encoding. Achieves higher latency-bound throughput and critical task satisfaction over baselines, with speedups in energy efficiency.
* `training` `storage` `recommendation` [AGILE: Lightweight and Efficient Asynchronous GPU-SSD Integration](http://arxiv.org/abs/2504.19365v3)
  > **TL;DR**: Proposes AGILE, an asynchronous GPU-centric I/O library with HBM cache to overlap computation and I/O for memory-intensive workflows like recommendation systems. Achieves up to 1.88× speedup by eliminating thread stalls and integrating software caching.
* `networking` `kernel` `training` [LCI: a Lightweight Communication Interface for Efficient Asynchronous Multithreaded Communication](http://arxiv.org/abs/2505.01864v2)
  > **TL;DR**: Proposes LCI, a lightweight communication library for efficient asynchronous multithreaded communication in distributed systems. Features atomic data structures, fine-grained locks, and network optimizations. Outperforms existing libraries on Infiniband and Slingshot-11, exceeding multi-process performance in multithreaded scenarios.
* `scheduling` `scaling` `edge` [A User-centric Kubernetes-based Architecture for Green Cloud Computing](http://arxiv.org/abs/2509.13325v1)
  > **TL;DR**: Proposes a Kubernetes-based system to reduce cloud computing emissions by scheduling workloads based on green energy availability. Employs a carbon intensity forecaster to exploit regional and temporal energy variations. Achieved 13% reduction in emissions compared to round-robin scheduler.
* `training` `scaling` `networking` [BandPilot: Towards Performance-Aware GPU Dispatching in AI Clusters](http://arxiv.org/abs/2506.15595v3)
  > **TL;DR**: Proposes BandPilot, a performance-aware GPU dispatching system for AI clusters. Uses a Transformer to predict NCCL bandwidth and a heuristic search for allocations. Achieves 12-31% higher bandwidth efficiency over topology-aware dispatchers.
* `training` `kernel` `networking` [MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications](http://arxiv.org/abs/2504.09014v3)
  > **TL;DR**: Proposes MSCCL++, a GPU communication library separation to improve portability and performance. Introduces primitive and higher-level interfaces enabling custom optimizations. Achieves up to 5.4× speedup in collective communication and 15% improvement in AI inference workloads.
* `agentic` `RAG` [ASIC-Agent: An Autonomous Multi-Agent System for ASIC Design with Benchmark Evaluation](http://arxiv.org/abs/2508.15940v1)
  > **TL;DR**: Proposes ASIC-Agent, a multi-agent system for ASIC design that enhances LLMs with specialized sub-agents and a vector database. Integrates RTL generation, verification, and chip hardening tools within a sandbox. When powered by Claude 4 Sonnet, it successfully automates ASIC design tasks, accelerating workflows.
* `edge` `networking` `serving` [Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey](http://arxiv.org/abs/2505.01821v4)
  > **TL;DR**: Surveys edge-cloud collaborative computing for AI/LLMs, focusing on model optimization (compression, adaptation) and AI-driven resource management to reduce latency and improve energy efficiency in applications like autonomous driving and healthcare.
* `kernel` `quantization` [SGEMM-cube: Emulating FP32 GEMM on Ascend NPUs Using FP16 Cube Units with Precision Recovery](http://arxiv.org/abs/2507.23387v3)
  > **TL;DR**: Proposes SGEMM-cube, an algorithm using FP16 units to emulate FP32 GEMM via operand decomposition and tunable scaling. Includes cache-aware blocking and double-buffering to achieve 77% of theoretical FP32 peak performance on Ascend NPU lacking native support.
* `edge` `offloading` `RL` [Oranits: Mission Assignment and Task Offloading in Open RAN-based ITS using Metaheuristic and Deep Reinforcement Learning](http://arxiv.org/abs/2507.19712v2)
  > **TL;DR**: Addresses mission assignment and task offloading for edge-based autonomous vehicles. Proposes Oranits with metaheuristic (CGG-ARO) and DRL (MA-DDQN) approaches for cooperative offloading. MA-DDQN improves mission completions by 11.0% and overall benefit by 12.5%.
* `RAG` `serving` [From Data Center IoT Telemetry to Data Analytics Chatbots -- Virtual Knowledge Graph is All You Need](http://arxiv.org/abs/2506.22267v2)
  > **TL;DR**: Designs a chatbot for IoT data analytics using LLMs with dynamically generated Virtual Knowledge Graphs (VKGs) to translate natural language to SPARQL queries. Achieves 92.5% accuracy and reduces latency by 85% to 3.03s compared to direct LLM-to-NoSQL.
* `serving` `training` `scaling` [An Empirical Study of Production Incidents in Generative AI Cloud Services](http://arxiv.org/abs/2504.08865v2)
  > **TL;DR**: Analyzes production incidents in generative AI cloud services to understand reliability challenges. Examines symptoms, root causes, and resolutions from incident data. Identifies research gaps in incident detection and mitigation for scalable GenAI systems.
* `RL` `offloading` `training` [Flexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models](http://arxiv.org/abs/2508.10349v1)
  > **TL;DR**: Proposes FlexP-SFL for personalized federated fine-tuning of foundation models via split learning. Allows clients to offload model portions to server based on resource constraints, with alignment strategy for personalized models. Achieves higher accuracy and efficiency than baselines.
* `training` `networking` [Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training](http://arxiv.org/abs/2501.11771v3)
  > **TL;DR**: Characterizes performance overheads of GPU TEEs in distributed data parallel ML training. Focuses on encryption/decryption and MAC costs during ring-all-reduce operations. Shows a runtime increase of up to 41.6x per training iteration with 4 GPUs versus non-TEE training.
* `networking` `scaling` `storage` [Faster Multi-Source Reachability and Approximate Distances via Shortcuts, Hopsets and Matrix Multiplication](http://arxiv.org/abs/2507.13470v2)
  > **TL;DR**: Introduces centralized and parallel algorithms for multi-source reachability and approximate distances. Leverages shortcut constructions, matrices, and small-separator graphs for efficient computation. Achieves time complexity $̂ O(n^{1 + √{2}{3} ω(σ)})$ improving over prior bounds in key $σ$ ranges.
* `training` `offloading` `kernel` [Performant Automatic BLAS Offloading on Unified Memory Architecture with OpenMP First-Touch Style Data Movement](http://arxiv.org/abs/2501.00279v4)
  > **TL;DR**: Develops SCILIB-Accel, an automatic tool to offload BLAS operations to GPUs on unified memory architectures using a first-touch policy and dynamic binary instrumentation. Minimizes data transfers and requires no code modifications. Achieves up to 3x speedup on quantum physics benchmarks.
* `training` `sparse` `kernel` [AMPED: Accelerating MTTKRP for Billion-Scale Sparse Tensor Decomposition on Multiple GPUs](http://arxiv.org/abs/2507.15121v2)
  > **TL;DR**: Accelerates MTTKRP for billion-scale sparse tensor decomposition on multi-GPU systems. Introduces a partitioning strategy with dynamic load balancing to distribute computation and minimize GPU idle time. Achieves 5.1x geometric mean speedup on real-world tensors over state-of-the-art baselines using 4 GPUs.
* `serving` `MoE` `disaggregation` [xDeepServe: Model-as-a-Service on Huawei CloudMatrix384](http://arxiv.org/abs/2508.02520v5)
  > **TL;DR**: Proposes xDeepServe, a Transformerless disaggregated architecture for serving MoE LLMs on SuperPod-scale hardware. Decomposes models into attention, feedforward, and MoE units executed independently with XCCL for efficient communication. Achieves scalable inference across hundreds of NPUs without performance sacrifice.
* `storage` `quantization` [Floating-Point Data Transformation for Lossless Compression](http://arxiv.org/abs/2506.18062v2)
  > **TL;DR**: Proposes Typed Data Transformation (TDT) for lossless compression of floating-point data by grouping correlated bytes. TDT improves geometric mean compression ratio by 1.16× and throughput by up to 3.79× over zstd, with applications to language model weights storage.
* `kernel` `hardware` `quantization` [High-Performance and Power-Efficient Emulation of Matrix Multiplication using INT8 Matrix Engines](http://arxiv.org/abs/2508.03984v2)
  > **TL;DR**: Proposes high-performance emulation methods for SGEMM/DGEMM using INT8 matrix engines. Leverages low-precision engines to emulate higher precision matrix multiplications. Achieves up to 3.0x speedup and 154% power efficiency improvement over native implementations.
* `training` `MoE` `multi-modal` [VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo](http://arxiv.org/abs/2508.02317v3)
  > **TL;DR**: Addresses inefficiency in training omni-modal LLMs with heterogeneous architectures. Proposes VeOmni, a modular framework using model-centric distributed recipes decoupling communication from computation for 3D parallelism. Achieves 2,800 tokens/sec/GPU throughput on a 30B MoE model at 160K context length on 128 GPUs.
* `training` `scaling` [Tesserae: Scalable Placement Policies for Deep Learning Workloads](http://arxiv.org/abs/2508.04953v1)
  > **TL;DR**: Proposes novel placement policies for DL training jobs using graph matching to minimize migration overhead and improve packing. Integrates into Tesserae scheduler, achieving up to 1.62× lower average job completion time and 1.15× lower makespan.
* `recommendation` `training` `sparse` [Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training](http://arxiv.org/abs/2508.03854v1)
  > **TL;DR**: Proposes a 2D sparse parallelism for large-scale deep learning recommendation model training to address embedding table scalability. Combines model and data parallelism with a momentum-scaled optimizer to reduce communication and memory use. Achieves near-linear scaling up to 4000 GPUs.
* `offloading` `training` `quantization` [ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates](http://arxiv.org/abs/2505.12242v3)
  > **TL;DR**: Addresses GPU stalls during offloaded LLM fine-tuning by prioritizing parameter updates. Proposes ZenFlow with asynchronous, decoupled gradient processing exploiting spatial-temporal locality for low-overhead selection. Achieves 5x speedup, 85% stall reduction, and 2x lower PCIe traffic.
* `training` `scaling` `quantization` [DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM](http://arxiv.org/abs/2505.00982v2)
  > **TL;DR**: Proposes DHO$_2$, a distributed hybrid order optimizer for DNN training acceleration. Utilizes model parallelism and ADMM for efficient curvature information calculation and updates, reducing memory burden per device. Achieves near-linear memory reduction and $1.4\times\sim2.1\times$ training time speedup compared to conventional optimizers.
* `hardware` `kernel` [TeraNoC: A Multi-Channel 32-bit Fine-Grained, Hybrid Mesh-Crossbar NoC for Efficient Scale-up of 1000+ Core Shared-L1-Memory Clusters](http://arxiv.org/abs/2508.02446v1)
  > **TL;DR**: Proposes TeraNoC, a hybrid mesh-crossbar NoC for scaling shared-L1-memory clusters to 1000+ cores. Balances traffic via router remapper and multi-channel design to reduce area and power. Achieves 37.8% area reduction and up to 98.7% higher area efficiency versus crossbar-only.
* `quantization` `networking` `training` [FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication](http://arxiv.org/abs/2508.03760v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed training of LLMs. Proposes FlashCommunication V2 with bit splitting for arbitrary bit-width transmission and spike reserving for 2-bit quantization. Achieves up to 3.2× speedup in AllReduce and 2× in All2All communication.
* `scaling` `networking` `storage` [Self-assessment approach for resource management protocols in heterogeneous computational systems](http://arxiv.org/abs/2508.02202v1)
  > **TL;DR**: Proposes a heuristics-based self-assessment approach for resource management in heterogeneous systems. Dynamically weights requirements, computes node capacity for requests, and supports extensible resource types. Achieves straightforward resource estimation with scalability and extensibility.
* `serving` `scaling` [Learning Unified System Representations for Microservice Tail Latency Prediction](http://arxiv.org/abs/2508.01635v1)
  > **TL;DR**: Proposes USRFNet, a deep learning model using GNNs and gMLPs to unify traffic and resource features for microservice P95 latency prediction. Achieves higher accuracy in predicting tail latency during large-scale stress tests than SOTA baselines.
* `serving` `diffusion` `sparse` [MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models](http://arxiv.org/abs/2503.11972v2)
  > **TL;DR**: Addresses the trade-off between latency and quality in diffusion model serving. Proposes MoDM, a caching system that uses smaller models for cache-hit requests and larger models for cache-misses. Achieves 2.5x reduction in average serving time while preserving image quality.
* `training` `scaling` [CarbonScaling: Extending Neural Scaling Laws for Carbon Footprint in Large Language Models](http://arxiv.org/abs/2508.06524v1)
  > **TL;DR**: Extends neural scaling laws to include carbon footprint in LLM training. Integrates scaling, hardware evolution, parallelism optimization, and carbon models. Finds hardware scaling has diminishing returns for large LLMs and that critical batch size optimization reduces carbon by up to 40% for billion-parameter models.
* `training` `quantization` `offloading` [Compression-Induced Communication-Efficient Large Model Training and Inferencing](http://arxiv.org/abs/2508.00960v1)
  > **TL;DR**: Proposes phantom parallelism to reduce energy consumption in large model training. Introduces new forward/backward propagation operators as custom autograd operations, minimizing bandwidth and FLOP counts. Achieves ~50% energy reduction vs. conventional tensor parallelism on FFNs across 256 GPUs.
* `serving` `scaling` `offloading` [Tetris: Efficient Intra-Datacenter Calls Packing for Large Conferencing Services](http://arxiv.org/abs/2508.00426v1)
  > **TL;DR**: Addresses inefficient call packing in datacenter conferencing services leading to hot Media Processors. Proposes Tetris, which optimizes initial call assignments using historical data and migrates calls via linear optimization to balance CPU. Reduces participants on hot MPs by 2.5X using real trace of 10M+ calls.
* `networking` `edge` [Service Discovery-Based Hybrid Network Middleware for Efficient Communication in Distributed Robotic Systems](http://arxiv.org/abs/2508.00947v1)
  > **TL;DR**: Addresses communication inefficiency in distributed robotic systems. Proposes RIMAOS2C, a service discovery-based middleware with Message Bridge for optimized data flow and shared memory. Improves large-data transmission efficiency by 36-40% and reduces latency variation by 42-906%.
* `RL` `edge` `networking` [Satellite Federated Fine-Tuning for Foundation Models in Space Computing Power Networks](http://arxiv.org/abs/2504.10403v3)
  > **TL;DR**: Proposes a satellite-ground collaborative federated fine-tuning framework for large remote sensing foundation models. Decomposes and allocates model components to satellites and ground stations with tailored communication strategies. Achieves 33% reduction in training time.
* `training` `RL` `edge` [Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages](http://arxiv.org/abs/2508.00041v1)
  > **TL;DR**: Proposes Developmental Federated Tuning (DevFT) for resource-efficient LLM fine-tuning on edge devices. Stages submodel growth with knowledge transfer and introduces layer grouping/fusion for efficiency. Achieves 4.59× faster convergence and 10.67× lower communication overhead.
* `training` `storage` `scaling` [Data Readiness for Scientific AI at Scale](http://arxiv.org/abs/2507.23018v1)
  > **TL;DR**: Proposes a scientific data readiness framework for AI training at scale on HPC. Introduces Data Readiness Levels and Processing Stages to standardize workflows supporting transformer-based foundation models. Enables scalable and reproducible preprocessing for scientific domains.
* `offloading` `serving` `inference` [A Survey on Large Language Model Acceleration based on KV Cache Management](http://arxiv.org/abs/2412.19442v3)
  > **TL;DR**: Survey on Key-Value cache management techniques for accelerating LLM inference. Covers token-level, model-level, and system-level optimizations (e.g., KV cache quantization, offloading, scheduling). Highlights reduced computational and memory demands for real-world LLM deployments with long-context and real-time constraints.
* `training` `scaling` `networking` [The Performance of Low-Synchronization Variants of Reorthogonalized Block Classical Gram--Schmidt](http://arxiv.org/abs/2507.21791v1)
  > **TL;DR**: Addresses the communication bottleneck in distributed QR factorization for Krylov solvers. Proposes low-synchronization variants BCGSI+P-1S and BCGSI+P-2S of Gram-Schmidt orthogonalization. Achieves 4× speedup for BCGSI+P-1S and 2× for BCGSI+P-2S compared to classical BCGSI+.
* `MoE` `serving` `disaggregation` [MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism](http://arxiv.org/abs/2504.02263v4)
  > **TL;DR**: Proposes MegaScale-Infer for efficient MoE model serving by disaggregating attention/FFN modules and using ping-pong pipeline parallelism. Achieves up to 1.90x higher per-GPU throughput than state-of-art systems.
* `kernel` [MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation](http://arxiv.org/abs/2507.17773v2)
  > **TL;DR**: Presents MultiKernelBench, a multi-platform benchmark for LLM-based deep learning kernel generation. Introduces a modular backend abstraction for hardware extensibility and category-aware prompting for improved generation. Evaluates seven LLMs across 285 tasks on GPU, NPU, TPU platforms.
* `RAG` `networking` `storage` [CleANN: Efficient Full Dynamism in Graph-based Approximate Nearest Neighbor Search](http://arxiv.org/abs/2507.19802v1)
  > **TL;DR**: Proposes CleANN, a dynamic graph-based ANNS index with semi-lazy memory cleaning and adaptive consolidation for efficient concurrent updates and searches. Achieves 7-1200x throughput improvement while maintaining query quality on million-scale in-memory datasets.
* `serving` `agentic` `multi-modal` [Efficient and Scalable Agentic AI with Heterogeneous Systems](http://arxiv.org/abs/2507.19635v1)
  > **TL;DR**: Addresses efficient deployment of AI agent workloads on heterogeneous hardware. Proposes a framework for optimizing execution graphs with cost models, an MLIR-based compilation system, and a dynamic orchestration system. Achieves significant TCO benefits by leveraging older GPUs with newer accelerators.
* `edge` `offloading` `serving` [Deadline-Aware Joint Task Scheduling and Offloading in Mobile Edge Computing Systems](http://arxiv.org/abs/2507.18864v1)
  > **TL;DR**: Proposes deadline-aware joint scheduling and offloading for mobile edge computing to meet task deadlines efficiently. Designs an optimal scheduling algorithm with O(n log n) complexity and an online variant for dynamic arrivals. Achieves improvements in service ratio and scheduling cost.
* `serving` `offline` `hardware` [PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via Pool-Based Pipeline Parallelism](http://arxiv.org/abs/2507.18748v1)
  > **TL;DR**: Investigates how to efficiently serve video analytics inference on heterogeneous GPU clusters using pipeline parallelism. Proposes PPipe, a system with MILP-based control plane and adaptive batching for pipeline parallelism across GPU tiers. Achieves 32.2%-75.1% higher throughput than baselines.
* `kernel` `training` `serving` [CUTHERMO: Understanding GPU Memory Inefficiencies with Heat Map Profiling](http://arxiv.org/abs/2507.18729v1)
  > **TL;DR**: Introduces cuThermo, a profiling tool for GPU memory inefficiencies. Uses heat maps based on warp counts to identify access patterns. Achieves up to 721.79% performance improvement on optimized applications.
* `serving` `edge` `networking` [AI Flow: Perspectives, Scenarios, and Approaches](http://arxiv.org/abs/2506.12479v3)
  > **TL;DR**: Proposes AI Flow, a device-edge-cloud framework for efficient large model inference. Introduces familial models with aligned features for collaborative execution and emergent intelligence via enhanced networking. Reduces latency and improves scalability for ubiquitous AI services.
* `edge` `serving` `RL` [FCPO: Federated Continual Policy Optimization for Real-Time High-Throughput Edge Video Analytics](http://arxiv.org/abs/2507.18047v1)
  > **TL;DR**: Proposes a federated continual RL approach (FCPO) for edge video analytics to optimize batch sizes, resolutions, and processing threads in dynamic environments. Achieves 5× higher throughput and 60% lower latency with 10× less memory than SOTA RL schedulers.
* `scaling` `inference` `networking` [C-Koordinator: Interference-aware Management for Large-scale and Co-located Microservice Clusters](http://arxiv.org/abs/2507.18005v1)
  > **TL;DR**: Addresses resource competition and interference in large-scale co-located microservice clusters. Proposes C-Koordinator with CPI-based interference prediction and mitigation strategies. Reduces application latency by up to 36.1% across all response time percentiles.
* `training` `scaling` `networking` [PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training](http://arxiv.org/abs/2507.17904v1)
  > **TL;DR**: Examines optimizing geo-distributed LLM training under power constraints and network latency. Proposes PowerTrip, a dynamic site selection using power-to-cost heuristic and marginal gain. Reduces time-to-accuracy by up to 50% compared to baselines using real power traces.
* `serving` `offloading` `scaling` [Multiprocessor Scheduling with Memory Constraints: Fundamental Properties and Finding Optimal Solutions](http://arxiv.org/abs/2507.17411v1)
  > **TL;DR**: Addresses scheduling computational DAGs on multiprocessors with memory constraints. Proposes an ILP-based holistic scheduling algorithm optimizing workload balancing and data movement. Achieves better solutions than classical scheduling baselines, reducing memory-related overhead.
* `edge` `hardware` `offline` [CHAMP: A Configurable, Hot-Swappable Edge Architecture for Adaptive Biometric Tasks](http://arxiv.org/abs/2507.17793v1)
  > **TL;DR**: Proposes CHAMP, a modular edge system with hot-swappable FPGA accelerators and custom OS for adaptive AI tasks. Achieves near-linear throughput scaling from 1 to 5 accelerators on USB3 bus, optimizing resource use for field-deployable multi-stage biometric pipelines.
* `serving` `offloading` [KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider](http://arxiv.org/abs/2506.02634v4)
  > **TL;DR**: Characterizes KV Cache workload patterns in LLM serving from a cloud provider and develops a workload-aware cache eviction policy. Optimizes serving performance under real-world traces, achieving up to 20% higher throughput with limited cache capacity.
* `kernel` `sparse` `offline` [Efficient Column-Wise N:M Pruning on RISC-V CPU](http://arxiv.org/abs/2507.17301v1)
  > **TL;DR**: Proposes column-wise N:M pruning at tile level for CNNs on RISC-V, fusing im2col with data packing to reduce memory overhead. Uses AITemplate profiling to select optimal convolution implementations. Achieves up to 4.0x ResNet inference throughput increase with <2.1% accuracy drop.
* `training` `kernel` `RAG` [PathWeaver: A High-Throughput Multi-GPU System for Graph-Based Approximate Nearest Neighbor Search](http://arxiv.org/abs/2507.17094v1)
  > **TL;DR**: Develops PathWeaver, a multi-GPU system for graph-based ANNS with pipelining and data selection to minimize redundant computations. Introduces pipelined path extension and direction-guided selection for early filtering. Achieves 3.24× geomean speedup at 95% recall vs. state-of-the-art.
* `kernel` `hardware` `networking` [AcceleratedKernels.jl: Cross-Architecture Parallel Algorithms from a Unified, Transpiled Codebase](http://arxiv.org/abs/2507.16710v1)
  > **TL;DR**: Presents AcceleratedKernels.jl, a unified transpiled codebase for cross-architecture parallel algorithms, enabling efficient CPU-GPU cooperation and specialized MPI. Achieved GPU sorting throughputs of 538-855 GB/s using 200 A100 GPUs with 4.93x speedup via GPU interconnects.
* `edge` `offloading` `quantization` [An Experimental Study of Split-Learning TinyML on Ultra-Low-Power Edge/IoT Nodes](http://arxiv.org/abs/2507.16594v1)
  > **TL;DR**: Explores split-learning for TinyML on ultra-low-power edge nodes. Proposes a testbed with ESP32-S3 boards using wireless offloading (ESP-NOW, BLE, UDP/IP) of quantized MobileNetV2 partitions. Achieves 3.7 s round-trip latency with ESP-NOW compared to 10s+ for BLE.
* `hardware` `quantization` `kernel` [Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks](http://arxiv.org/abs/2507.10789v2)
  > **TL;DR**: Conducts microarchitectural analysis of NVIDIA Blackwell GPU using microbenchmarks to understand subsystems including tensor cores supporting FP4/FP6. Compares with Hopper architecture for latency, throughput, and power efficiency. Highlights FP4/FP6 support enabling efficient quantized inference for LLMs.
* `training` `offline` `edge` [ACME: Adaptive Customization of Large Models via Distributed Systems](http://arxiv.org/abs/2507.14802v1)
  > **TL;DR**: Proposes ACME, a distributed system for cost-efficient customization of Transformer models on edge devices. Uses bidirectional single-loop collaboration and Pareto-optimal backbone/header generation to handle user/data heterogeneity. Reduces data transmission by 94% and improves accuracy by 10% over centralized systems.
* `edge` `scaling` [Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning](http://arxiv.org/abs/2507.14597v1)
  > **TL;DR**: Addresses proactive autoscaling for edge data stream processing under workload fluctuations. Combines GRU load forecasting with transfer learning to adapt offline models to online environments and adjust operator parallelism. Achieves 1.3% SMAPE prediction error, outperforming baselines with lower training time.
* `edge` `sparse` `offline` [Edge Intelligence with Spiking Neural Networks](http://arxiv.org/abs/2507.14069v1)
  > **TL;DR**: Surveys Spiking Neural Networks for edge intelligence, focusing on lightweight inference, resource-aware training under non-stationary data, and secure deployment. Discusses neuron models, learning algorithms, and benchmarking for hardware-aware optimization. Highlights low-power event-driven computation.
* `training` `storage` [Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication](http://arxiv.org/abs/2507.13522v1)
  > **TL;DR**: Introduces Checkmate, a system for zero-overhead model checkpointing in DNN training by replicating gradients to a shadow cluster. Utilizes multicast to deliver gradients to a CPU-based cluster that maintains checkpoints. Achieves per-iteration checkpointing with throughput comparable to no-checkpoint baseline and reduces repeated work by 80-97.1%.
* `serving` `scaling` `RL` [Autonomous Resource Management in Microservice Systems via Reinforcement Learning](http://arxiv.org/abs/2507.12879v1)
  > **TL;DR**: Proposes reinforcement learning for autonomous microservice resource management to optimize allocation, reduce latency, and improve throughput. Achieves significant improvements in response speed and throughput under varying loads while optimizing resource usage and energy consumption.
* `serving` `scaling` [PolyServe: Efficient Multi-SLO Serving at Scale](http://arxiv.org/abs/2507.17769v1)
  > **TL;DR**: Addresses efficient LLM serving with multiple latency SLO requirements. Introduces PolyServe, a scheduler grouping requests into bins per SLO, using load-based routing and bin sharing. Achieves 1.23x goodput gain versus baselines and 92.5% of optimal goodput.
* `edge` `networking` `scaling` [CRAFT: Latency and Cost-Aware Genetic-Based Framework for Node Placement in Edge-Fog Environments](http://arxiv.org/abs/2507.12445v1)
  > **TL;DR**: Proposes CRAFT, a genetic algorithm-based node placement framework for edge-fog environments to minimize latency and cost in IoT systems. Achieves 2.77% latency and 31.15% cost reduction in simulations.
* `training` `scaling` `networking` [Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release](http://arxiv.org/abs/2507.17766v1)
  > **TL;DR**: Proposes IOTA, a decentralized LLM training architecture using orchestrator-driven layer distribution across miners. Implements activation compression (128x reduction) and butterfly all-reduce for scalability. Achieves linear scaling by partitioning parameters across contributors while ensuring fair incentives.
* `edge` `scaling` `storage` [ARRC: Explainable, Workflow-Integrated Recommender for Sustainable Resource Optimization Across the Edge-Cloud Continuum](http://arxiv.org/abs/2507.12032v1)
  > **TL;DR**: Proposes ARRC, a recommender system for resource optimization in edge-cloud continuum, using explainable agents integrated into operator workflows. Reduces operator workload by over 50% and improves compute utilization by up to 7.7x while maintaining error rates below 5%.
* `edge` `offloading` `networking` [MOFCO: Mobility- and Migration-Aware Task Offloading in Three-Layer Fog Computing Environments](http://arxiv.org/abs/2507.12028v1)
  > **TL;DR**: Proposes MOFCO, a mobility-aware task offloading algorithm for fog computing environments. Uses heuristic-aided evolutionary game theory to solve MINLP formulation for task placement and resource allocation. Reduces system cost (latency + energy) by average 19% and up to 43%.
* `kernel` [NineToothed: A Triton-Based High-Level Domain-Specific Language for Machine Learning](http://arxiv.org/abs/2507.11978v1)
  > **TL;DR**: Introduces NineToothed, a high-level DSL with serial semantics for ML kernels. It automatically transforms serial tensor-oriented code into parallel code using arrange-and-apply paradigm and code generation. Maintains Triton-like performance while significantly simplifying kernel development.
* `serving` `kernel` [BlockBPE: Parallel BPE Tokenization](http://arxiv.org/abs/2507.11941v1)
  > **TL;DR**: Addresses slow CPU-bound BPE tokenization in LLM inference. Proposes BlockBPE, a GPU-optimized parallel byte-pair encoding algorithm that skips regex pre-tokenization, enabling thread-block merges. Achieves up to 2.5x higher throughput versus HuggingFace Tokenizers on batch inference.
* `serving` `offline` `scaling` [Making Serverless Computing Extensible: A Case Study of Serverless Data Analytics](http://arxiv.org/abs/2507.11929v1)
  > **TL;DR**: Proposes Proteus, an extensible serverless platform enabling domain-specific optimizations via decision workflows. Focuses on improving resource sharing and performance for serverless data analytics workloads without compromising generality. Achieves better query execution efficiency in preliminary results.
* `training` [DeInfoReg: A Decoupled Learning Framework for Better Training Throughput](http://arxiv.org/abs/2506.18193v2)
  > **TL;DR**: Proposes DeInfoReg, a decoupled learning framework with information regularization to transform long gradient flows into shorter ones, enabling pipeline parallelism across GPUs for improved throughput. Achieves higher training throughput and better resource utilization compared to standard backpropagation.
* `networking` `scaling` `serving` [Arcturus: A Cloud Overlay Network for Global Accelerator with Enhanced Performance and Stability](http://arxiv.org/abs/2507.10928v1)
  > **TL;DR**: Proposes Arcturus, a cloud-native global accelerator overlay network using heterogeneous cloud resources. Features two-plane design: forwarding plane for adaptive proxy network and scheduling plane for load coordination. Achieves 1.7× speedup over commercial services, 71% cost reduction, and 80% resource efficiency at millions of RPS.
* `edge` `offloading` `serving` [A Model Aware AIGC Task Offloading Algorithm in IIoT Edge Computing](http://arxiv.org/abs/2507.11560v1)
  > **TL;DR**: Proposes MADDPG-MATO, a multi-agent deep reinforcement learning algorithm for AI-generated content task offloading in IIoT edge computing. Minimizes latency and energy by model-aware scheduling, considering model switching overheads. Achieves 6.98% lower latency and 7.12% lower energy consumption versus baselines.
* `disaggregation` `networking` `hardware` [Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure](http://arxiv.org/abs/2507.07223v2)
  > **TL;DR**: Addresses scaling bottlenecks in AI infrastructure due to inter-GPU communication. Proposes a modular data center architecture with disaggregated memory/compute using CXL and hybrid interconnects (XLink). Demonstrates improved scalability and throughput with efficiency gains in communication overheads.
* `serving` `offloading` `quantization` [MQFQ-Sticky: Fair Queueing For Serverless GPU Functions](http://arxiv.org/abs/2507.08954v1)
  > **TL;DR**: Addresses high latency in GPU-accelerated FaaS due to limited concurrency and cold starts. Proposes MQFQ-Sticky, a fair queueing and memory management system adapting I/O scheduling principles for GPUs. Reduces function latency by 2x-20x compared to existing policies.
* `training` `scaling` `RL` [Efficient Long Context Fine-tuning with Chunk Flow](http://arxiv.org/abs/2503.02356v3)
  > **TL;DR**: Addresses inefficiencies in fine-tuning LLMs on datasets with variable sequence lengths. Proposes ChunkFlow, a method that reorganizes sequences into uniform chunks and uses state-aware scheduling. Achieves up to 4.53x speedup over Megatron-LM.
* `edge` `kernel` `video` [Accelerating Transposed Convolutions on FPGA-based Edge Devices](http://arxiv.org/abs/2507.07683v1)
  > **TL;DR**: Proposes MM2IM, a hardware-software accelerator combining matrix multiplication with col2IM to optimize transposed convolutions in generative AI models on edge devices. Achieves up to 4.2x speedup on TCONV layers from generative models and 2.4x energy reduction on DCGAN and pix2pix.
* `serving` `offline` `networking` [Towards Efficient and Scalable Distributed Vector Search with RDMA](http://arxiv.org/abs/2507.06653v1)
  > **TL;DR**: Addresses scalability bottlenecks in distributed vector search systems. Proposes CoTra with data partitioning, asynchronous execution, and task pushing leveraging RDMA and system optimizations. Achieves 9.8–13.4x throughput scaling on 16 machines with 2.12–3.58x baseline improvement at 0.95 recall.
* `edge` `efficiency` `MoE` [The AI Shadow War: SaaS vs. Edge Computing Architectures](http://arxiv.org/abs/2507.11545v1)
  > **TL;DR**: Investigates the performance and efficiency trade-offs between centralized cloud and decentralized edge architectures for AI inference. Highlights innovations like mixture-of-experts and test-time training on edge devices, achieving a 10,000x efficiency gain on ARM processors compared to cloud processing.
* `training` `diffusion` `sparse` [FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Models](http://arxiv.org/abs/2507.06449v1)
  > **TL;DR**: FedPhD addresses efficient distributed training of diffusion models with federated learning. It uses hierarchical learning with structured pruning and heterogeneity-aware aggregation to reduce costs. Achieves 88% lower comm. costs with 34% better FID using 56% resources.
* `training` `multi-modal` `edge` [Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach](http://arxiv.org/abs/2502.06355v3)
  > **TL;DR**: Proposes MPSL, a parallel split learning approach for fine-tuning multimodal transformers on edge devices. Uses lightweight client tokenizers and a unified modality-agnostic encoder to reduce computation. Cuts client-side computation by 250× and improves communication scalability versus federated learning.
* `training` `scaling` [CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training](http://arxiv.org/abs/2504.00598v2)
  > **TL;DR**: Optimizes intra-operator parallelism plans for large model training by identifying parallel-preserving and repetitive subgraphs to reduce search space. Uses dynamic programming to find optimal strategies, achieving speedups for GPT and LLAMA.
* `training` `sparse` `scaling` [Distributed Equivariant Graph Neural Networks for Large-Scale Electronic Structure Prediction](http://arxiv.org/abs/2507.03840v1)
  > **TL;DR**: Addresses memory and scalability limitations in training equivariant GNNs for large-scale electronic structure prediction. Proposes a distributed implementation with GPU communication and graph partitioning to reduce embedding exchanges. Achieves strong scaling up to 128 GPUs and weak scaling up to 512 GPUs with 87% efficiency.
* `training` `networking` `scaling` [Collective Communication Profiling of Modern-day Machine Learning Workloads](http://arxiv.org/abs/2507.07117v1)
  > **TL;DR**: Analyzes network bottlenecks in distributed ML workloads' collective communications (e.g., AllReduce). Instruments NCCL to profile operations across models (DeepSeek, GPT) under varied configurations. Observes bursty traffic causing congestion, suggesting framework and topology redesigns.
* `training` `networking` `scaling` [Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications](http://arxiv.org/abs/2507.03114v1)
  > **TL;DR**: Characterizes compute-communication overlap in GPU-accelerated distributed deep learning training. Evaluates impact of overlapping strategies, hardware features like numeric precision and power capping on performance and power. Finds overlapping causes average 18.9% compute slowdown but is still 10.2% faster than sequential execution.
* `edge` `quantization` `hardware` [Red grape detection with accelerated artificial neural networks in the FPGA's programmable logic](http://arxiv.org/abs/2507.02443v1)
  > **TL;DR**: Addresses slow object detection in robotics by deploying quantized ANNs on FPGAs using FINN framework. Achieves 6611 FPS inference with 98% success rate, accelerating attention mechanisms for realtime robotic tasks.
* `training` `storage` `offline` [The Artificial Scientist -- in-transit Machine Learning of Plasma Simulations](http://arxiv.org/abs/2501.03383v3)
  > **TL;DR**: Proposes an in-transit ML workflow for large-scale plasma simulations to bypass file system bottlenecks. Data is streamed directly from simulation to ML framework for asynchronous transformation and training. Achieves efficient streaming on Frontier exascale system at petabyte scale.
* `training` `hardware` `networking` [SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan](http://arxiv.org/abs/2507.02124v1)
  > **TL;DR**: Introduces SAKURAONE HPC cluster for large-scale LLM training. Features 800 GbE with SONiC open networking and RoCEv2 for high-speed interconnect. Achieves 339.86 PFLOP/s on HPL-MxP (FP8), demonstrating suitability for low-precision AI workloads.
* `training` `scaling` `hardware` [Evolving HPC services to enable ML workloads on HPE Cray EX](http://arxiv.org/abs/2507.01880v1)
  > **TL;DR**: Addresses extending HPC services to support ML workloads on Alps infrastructure. Proposes user environments, performance screening tools, observability, storage enhancements, and service plane for multi-workload deployment. Achieves scalable support for 10,752 GPUs to meet dynamic AI/ML community needs.
* `training` `networking` `scaling` [Distributed Training under Packet Loss](http://arxiv.org/abs/2507.07114v1)
  > **TL;DR**: Proposes a distributed training framework that maintains model accuracy and convergence under packet loss. It features unbiased gradient aggregation and bounded-drift parameter broadcasts. Achieves at most 0.8% perplexity change for LLAMA2 7B with 64 GPUs under 10% packet loss.
* `serving` `training` `scaling` [Not All Water Consumption Is Equal: A Water Stress Weighted Metric for Sustainable Computing](http://arxiv.org/abs/2506.22773v2)
  > **TL;DR**: Proposes SCARF, a framework for assessing water impact of computing with water-stress weighted metrics. Introduces Adjusted Water Impact (AWI) considering spatiotemporal water stress variations. Case studies show significant reductions in water impact for LLM serving via optimized scheduling and siting.
* `quantization` `training` [Enabling mixed-precision in spectral element codes](http://arxiv.org/abs/2503.02134v2)
  > **TL;DR**: Proposes a methodology for mixed-precision in spectral element CFD codes using Verificarlo and computer arithmetic. Applied to Nekbone and Neko, achieving up to 1.62× faster time-to-solution and 2.43× lower energy-to-solution while matching double-precision accuracy.
* `storage` [yProv4ML: Effortless Provenance Tracking for Machine Learning Systems](http://arxiv.org/abs/2507.01078v1)
  > **TL;DR**: Addresses the lack of transparency and lineage tracking in ML systems like LLM training. Proposes yProv4ML, a framework that captures provenance data in standard PROV-JSON format with low code changes. Enables effortless hyperparameter and epoch tracking for model selection.
* `training` `storage` `scaling` [Provenance Tracking in Large-Scale Machine Learning Systems](http://arxiv.org/abs/2507.01075v1)
  > **TL;DR**: Addresses efficient scaling of large AI models by introducing yProv4ML, a provenance tracking library for monitoring resource usage and inefficiencies during training. Enables JSON-based data collection compliant with standards. Achieves better resource utilization for energy-efficient distributed training.
* `training` `storage` `hardware` [eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems](http://arxiv.org/abs/2506.02007v2)
  > **TL;DR**: Proposes eACGM, an eBPF-based monitoring framework for AI/ML systems, capturing GPU/network/CUDA metrics without instrumentation. Uses GMM for anomaly detection of latency/failures/inefficiencies. Achieves low-overhead anomaly detection in distributed training, supporting fault diagnosis.
* `serving` `edge` `networking` [Real-Time In-Network Machine Learning on P4-Programmable FPGA SmartNICs with Fixed-Point Arithmetic and Taylor](http://arxiv.org/abs/2507.00428v1)
  > **TL;DR**: Investigates low-latency ML inference on smart network devices at the network edge. Proposes P4-programmable FPGA SmartNICs using fixed-point arithmetic and dynamic table lookups for model weights. Enables real-time deployment with dynamic reconfiguration and efficient edge processing for network tasks.
* `training` `networking` `scaling` [CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training](http://arxiv.org/abs/2507.00217v1)
  > **TL;DR**: Proposes CrossPipe, a framework for optimizing LLM training across datacenters by co-optimizing pipeline parallelism and data parallelism overlapping to mitigate network latency. Uses solver-based and greedy scheduling algorithms, reducing training time by up to 33.6% under memory constraints.
* `training` `offloading` `scaling` [PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization](http://arxiv.org/abs/2503.01328v2)
  > **TL;DR**: Reduces activation memory constraints in pipeline parallelism for large model training by selectively offloading activations, enabling better scalability. Combines memory offload with throughput optimization, achieving up to 19% speedup with lower memory consumption.
* `kernel` `storage` `inference` [Verifying Properties of Index Arrays in a Purely-Functional Data-Parallel Language](http://arxiv.org/abs/2506.23058v1)
  > **TL;DR**: Presents a framework for verifying properties of data-parallel programs to enable optimizations. Uses index function transformations and algebraic solving to prove array properties. Eliminating dynamic checks in GPU programs yields significant speedups.
* `RL` `LoRA` `edge` [Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models](http://arxiv.org/abs/2501.14406v3)
  > **TL;DR**: Addresses challenges in federated parameter-efficient fine-tuning (FedPEFT) of PLMs: data heterogeneity and communication inefficiency. Proposes FedARA with adaptive rank allocation, dynamic module pruning, and SVD adaptation. Reduces training time by 48.9% and energy by 46.95% on edge devices.
* `kernel` `hardware` `sparse` [TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations](http://arxiv.org/abs/2506.22818v1)
  > **TL;DR**: Proposes TriADA, a hardware architecture with elastic sparse outer-product kernels for massively parallel trilinear transformations. Decouples memory and processing via mesh PEs to achieve hypercubic complexity in linear steps. Achieves energy efficiency for sparse tensors without unnecessary zero operations.
* `kernel` `training` `quantization` [MCFuser: High-Performance and Rapid Fusion of Memory-Bound Compute-Intensive Operators](http://arxiv.org/abs/2506.22169v1)
  > **TL;DR**: Proposes MCFuser, a framework for fusing memory-bound compute-intensive operator chains to improve GPU utilization. Uses high-level tiling, DAG analysis, and heuristic search to optimize kernels. Achieves up to 5.9x speedup and 70x faster tuning time compared to Ansor.
* `scaling` `serving` [Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces](http://arxiv.org/abs/2506.21467v1)
  > **TL;DR**: Proposes Discovery Space, an abstraction for efficient cloud configuration search to minimize cost and meet SLAs for workloads like LLM inference. Enables safe sharing and reuse of optimizer data across executions and similar search spaces, achieving over 90% search speed-up.
* `RAG` `serving` `storage` [BLOCKS: Blockchain-supported Cross-Silo Knowledge Sharing for Efficient LLM Services](http://arxiv.org/abs/2506.21033v1)
  > **TL;DR**: Proposes a blockchain-based framework for secure cross-silo knowledge sharing to augment LLMs. Uses knowledge distillation into prompts with reputation mechanisms and cross-validation. Achieves efficient knowledge sharing, enabling reliable RAG without compromising data privacy.
* `inference` `serving` `scaling` [SuperSONIC: Cloud-Native Infrastructure for ML Inferencing](http://arxiv.org/abs/2506.20657v1)
  > **TL;DR**: Proposes SuperSONIC, a cloud-native server infrastructure for scalable ML inference using remote coprocessors and Kubernetes. Utilizes NVIDIA Triton to decouple clients and servers, optimizing throughput and load balancing. Deployed on scientific experiments, demonstrating enhanced efficiency for accelerator-based inference.
* `training` `networking` [PAT: a new algorithm for all-gather and reduce-scatter operations at scale](http://arxiv.org/abs/2506.20252v1)
  > **TL;DR**: Proposes PAT, a parallel aggregated trees algorithm for efficient all-gather and reduce-scatter operations in distributed training. Features logarithmic network transfers and buffers, minimizing long-distance communication. Improves NCCL performance for small operations at scale, reducing latency where ring algorithms underperform.
* `training` `RAG` `kernel` [MegaFold: System-Level Optimizations for Accelerating Protein Structure Prediction Models](http://arxiv.org/abs/2506.20686v1)
  > **TL;DR**: Accelerates AlphaFold3 training via system optimizations: ahead-of-time caching for retrieval-augmented pipelines, custom Triton kernels for memory-efficient attention, and operator fusion. Achieves up to 1.73× faster iteration times and 1.35× longer sequence lengths on GPUs.
* `networking` `scaling` [MAIZX: A Carbon-Aware Framework for Optimizing Cloud Computing Emissions](http://arxiv.org/abs/2506.19972v1)
  > **TL;DR**: Introduces MAIZX, a carbon-aware framework for cloud computing emissions optimization. Dynamically ranks resources (data centers, edge nodes, multi-cloud) based on carbon intensity, PUE, and energy. Reduces CO2 emissions by 85.68% compared to baseline hypervisor operations.
* `training` `RL` `networking` [GradualDiff-Fed: A Federated Learning Specialized Framework for Large Language Model](http://arxiv.org/abs/2506.19164v1)
  > **TL;DR**: Proposes GradualDiff-Fed, a federated learning framework for efficient LLM fine-tuning that transmits weight differences to reduce communication costs. Achieves performance equivalent to centralized training while cutting communication overhead.
* `quantization` `offloading` `edge` [NestQuant: Post-Training Integer-Nesting Quantization for On-Device DNN](http://arxiv.org/abs/2506.17870v1)
  > **TL;DR**: Proposes NestQuant, a post-training integer-nesting quantization method for on-device DNNs. Uses integer weight decomposition and nesting to enable adaptive bitwidth switching with one stored model. Achieves 78.1% reduction in switching overheads while maintaining high accuracy.
* `edge` `serving` `kernel` [ConsumerBench: Benchmarking Generative AI Applications on End-User Devices](http://arxiv.org/abs/2506.17538v1)
  > **TL;DR**: Presents ConsumerBench, a benchmarking framework for evaluating GenAI application efficiency on end-user devices. Simulates multi-application scenarios on constrained hardware and captures system metrics. Reveals inefficiencies in resource sharing and shows custom kernels improve performance on consumer GPUs.
* `training` `networking` `quantization` [NetSenseML: Network-Adaptive Compression for Efficient Distributed Machine Learning](http://arxiv.org/abs/2506.16235v1)
  > **TL;DR**: Proposes NetSenseML, a network-adaptive framework that dynamically adjusts gradient compression based on real-time network conditions to balance payload reduction and accuracy. Achieves 1.55× to 9.84× higher training throughput in bandwidth-constrained distributed deep learning.
* `hardware` `kernel` `training` [HetGPU: The pursuit of making binary compatibility towards GPUs](http://arxiv.org/abs/2506.15993v1)
  > **TL;DR**: Proposes hetGPU, a system enabling single GPU binary execution across diverse vendor hardware (NVIDIA, AMD, etc.) through an architecture-agnostic IR and runtime translation. Addresses SIMT vs. MIMD divergence and state serialization challenges. Reduces migration overhead, achieving vendor-agnostic compatibility with minimal performance impact.
* `hardware` `training` `scaling` [A System Level Compiler for Massively-Parallel, Spatial, Dataflow Architectures](http://arxiv.org/abs/2506.15875v1)
  > **TL;DR**: Proposes MACH, a compiler for efficient execution on massively-parallel spatial dataflow architectures like wafer-scale engines. It introduces a Virtual Machine model, domain-specific language, and lowering flow to generate optimized code for tensor computations. Demonstrates applicability to dense tensor operations with potential for reduced execution time on specialized hardware.
* `serving` `MoE` `quantization` [Utility-Driven Speculative Decoding for Mixture-of-Experts](http://arxiv.org/abs/2506.20675v1)
  > **TL;DR**: Addresses inefficiency of speculative decoding in MoE LLMs due to increased verification cost. Proposes Cascade, a utility-driven framework that dynamically enables speculation and tunes token count K to maximize token gain per cost. Achieves 7-14% higher throughput and limits slowdown to 5% vs. up to 1.5x slowdown.
* `quantization` `offloading` `training` [ILVES: Accurate and efficient bond length and angle constraints in molecular dynamics](http://arxiv.org/abs/2503.13075v3)
  > **TL;DR**: Introduces ILVES, a parallel algorithm for molecular dynamics simulations that accurately solves bond length and angular constraints with better convergence. Achieves a 1.65x increase in simulated time using the same resources and wall-clock time compared to state-of-the-art methods.
* `video` `serving` `scaling` [DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving](http://arxiv.org/abs/2506.13497v1)
  > **TL;DR**: Proposes DDiT for efficient serving of diffusion transformer-based text-to-video models. Integrates inter-phase and intra-phase optimizations with dynamic resource allocation. Achieves up to 1.44x lower p99 latency and 1.43x lower average latency compared to baselines.
* `serving` `edge` `scaling` [QoS-aware Scheduling of Periodic Real-time Task Graphs on Heterogeneous Pre-occupied MECs](http://arxiv.org/abs/2506.12415v1)
  > **TL;DR**: Proposes QoS-aware scheduling for periodic real-time task graphs on heterogeneous MECs. Uses modified HEFT algorithm to exploit residual capacity by dynamically mapping tasks to idle VM intervals. Improves load balancing and resource utilization by 30% compared to baselines while meeting timing constraints.
* `storage` `training` `inference` [Efficient Unified Caching for Accelerating Heterogeneous AI Workloads](http://arxiv.org/abs/2506.12370v1)
  > **TL;DR**: Proposes IGTCache, a unified caching framework for AI clusters that uses hierarchical access abstraction (AccessStreamTree) to detect heterogeneous data access patterns. Tailors prefetching, eviction, and allocation strategies dynamically, improving cache hit ratio by 55.6% and reducing job completion time by 52.2%.
* `hardware` `training` `inference` [Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units](http://arxiv.org/abs/2506.11446v1)
  > **TL;DR**: Proposes vNPU for topology-aware virtualization of inter-core connected NPUs. Techniques include NPU route virtualization, memory virtualization, and best-effort topology mapping to balance resource utilization and performance. Achieves up to 2x performance improvement with 2% hardware overhead.
* `serving` `offloading` `RAG` [WindVE: Collaborative CPU-NPU Vector Embedding](http://arxiv.org/abs/2504.14941v4)
  > **TL;DR**: Optimizes vector embedding concurrency in RAG for LLM inference cost-performance. Designs WindVE: CPU-NPU queue manager with linear regression for dynamic offloading. Achieves 22.3% higher concurrency than state-of-the-art without offloading.
* `hardware` `quantization` `kernel` [Proteus: Enabling High-Performance Processing-Using-DRAM with Dynamic Bit-Precision, Adaptive Data Representation, and Flexible Arithmetic](http://arxiv.org/abs/2501.17466v2)
  > **TL;DR**: Proposes Proteus, a hardware framework to reduce latency in processing-using-DRAM (PUD) operations. Uses dynamic bit-precision reduction, parallel execution across arrays, and adaptive arithmetic representation. Achieves up to 2.2× speedup against existing PUD approaches.
* `edge` `scaling` [Automating Multi-Tenancy Performance Evaluation on Edge Compute Nodes](http://arxiv.org/abs/2506.10461v1)
  > **TL;DR**: Proposes an auto-benchmarking framework for multi-tenancy performance evaluation on edge compute nodes, integrating monitoring and diverse workloads. Achieves streamlined analysis of resource contention impact, providing hardware-specific insights for service co-location optimization.
* `scaling` `edge` [Multi-dimensional Autoscaling of Processing Services: A Comparison of Agent-based Methods](http://arxiv.org/abs/2506.10420v1)
  > **TL;DR**: Introduces an agent-based autoscaling framework for edge computing that adjusts hardware and service configurations. Compares performance of four agents (Active Inference, DQN, ASK, Deep Active Inference) for multi-dimensional scaling. Achieves acceptable SLO performance on YOLOv8 and OpenCV services.
* `training` [PerfTracker: Online Performance Troubleshooting for Large-scale Model Training in Production](http://arxiv.org/abs/2506.08528v3)
  > **TL;DR**: Proposes PerfTracker, an online troubleshooting system using fine-grained profiling to diagnose performance issues in large-scale model training. Analyzes hardware and software interactions for efficiency. Deployed on clusters with 10,000 GPUs.
* `training` `quantization` `offloading` [Low-resource domain adaptation while minimizing energy and hardware resource consumption](http://arxiv.org/abs/2506.08433v2)
  > **TL;DR**: Investigates reducing computation cost for LLM domain adaptation in low-resource settings. Evaluates numerical precision formats and data parallelization to optimize energy efficiency and hardware consumption. Achieves comparable accuracy with FP16 to FP32 at lower resource use.
* `training` `quantization` [TTrace: Lightweight Error Checking and Diagnosis for Distributed Training](http://arxiv.org/abs/2506.09280v1)
  > **TL;DR**: Addresses silent bug detection in distributed LLM training. Proposes TTrace, which compares intermediate tensors against a single-device reference with threshold analysis for floating-point errors. Detects 14 bugs in Megatron-LM with fewer than 10 code changes, including in BF16 and FP8 training.
* `training` `scaling` [A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO](http://arxiv.org/abs/2506.09275v1)
  > **TL;DR**: Surveys distributed DNN training simulators for efficient system co-design. Focuses on workload representation, simulation infrastructure, and TCO/carbon emission modeling. Highlights tools for reducing the cost and complexity of scaling large-scale training systems.
* `scaling` `training` [Balancing Fixed Number of Nodes Among Multiple Fixed Clusters](http://arxiv.org/abs/2506.08715v1)
  > **TL;DR**: Addresses underutilization of fixed nodes across multiple clusters by proposing a Node Balancing Cluster Group (NBCG) system for dynamic rebalancing based on real-time thresholds. Achieves optimized resource utilization without additional costs.
* `serving` `scaling` [DeepServe: Serverless Large Language Model Serving at Scale](http://arxiv.org/abs/2501.14417v3)
  > **TL;DR**: Proposes DeepServe, a serverless platform for scalable LLM serving. Key features include a serverless abstraction, NPU-optimized serving engine, scheduling for disaggregated/colocated instances, and optimizations like pre-warmed pods. Achieves scaling to 64 instances in seconds to handle cold starts.
* `kernel` `sparse` `training` [GPU-Parallelizable Randomized Sketch-and-Precondition for Linear Regression using Sparse Sign Sketches](http://arxiv.org/abs/2506.03070v2)
  > **TL;DR**: Proposes GPU-parallelizable randomized sketch-and-precondition method for large linear regression using sparse sign sketches. Introduces rejection-sampling based sparse sign sketch generation optimized for single/multi-GPU systems. Improves computational efficiency on GPUs, suitable for black-box least-squares solvers.
* `training` `RL` `LoRA` [Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning](http://arxiv.org/abs/2506.05977v1)
  > **TL;DR**: Addresses catastrophic forgetting in federated LLM fine-tuning. Proposes FedBE with adaptive transformer block expansion and dynamic block allocation to separate new knowledge from pre-trained representations. Achieves 12-74% higher general-task accuracy retention and 1.9-3.1x faster convergence.
* `serving` `disaggregation` `scaling` [BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures](http://arxiv.org/abs/2506.05871v1)
  > **TL;DR**: Addresses the labor-intensive process of finding optimal LLM serving strategies. Proposes BestServe, a lightweight inference simulator based on an adapted roofline model to predict goodput. Achieves optimal strategy selection in minutes with predictions within 20% error margin.
* `training` `kernel` `networking` [Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler](http://arxiv.org/abs/2504.19442v3)
  > **TL;DR**: Proposes Triton-distributed compiler extension for efficient distributed AI workloads, supporting overlapping of computation and communication via OpenSHMEM. Demonstrates up to 64 devices utilization, outperforming hand-optimized code with reduced development effort.
* `offloading` `training` `scaling` [Analysis of Server Throughput For Managed Big Data Analytics Frameworks](http://arxiv.org/abs/2506.03854v1)
  > **TL;DR**: Examines how to reduce garbage collection and serialization overhead in big data frameworks to improve server throughput. Proposes TeraHeap, which offloads objects to a secondary heap on fast storage, and studies DRAM distribution. Achieves higher throughput by co-locating memory-bound instances and optimizing memory utilization.
* `edge` `scaling` `offloading` [LRScheduler: A Layer-aware and Resource-adaptive Container Scheduler in Edge Computing](http://arxiv.org/abs/2506.03694v1)
  > **TL;DR**: Proposes LRScheduler, a layer-aware and resource-adaptive container scheduler for edge computing. Utilizes container image layers to reduce deployment costs via node scoring and resource-adaptive mechanisms for load balancing. Achieves up to 30% reduced container deployment cost compared to default Kubernetes scheduler.
* `kernel` `serving` `quantization` [FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs](http://arxiv.org/abs/2506.01969v2)
  > **TL;DR**: Proposes FlashMLA-ETAP to optimize transpose attention for MLA inference by aligning KV context with WGMMA operations, reducing redundancy. Achieves 2.78x speedup over FlashMLA at batch 16 with 64K sequence length while maintaining accuracy.
* `training` `LoRA` `edge` [Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices](http://arxiv.org/abs/2506.02940v1)
  > **TL;DR**: Proposes a split federated learning framework for memory-efficient LLM fine-tuning on heterogeneous mobile devices. Uses device-specific LoRA on lower layers and server-side scheduling of sequential fine-tuning. Reduces memory by 79% and training time by 6% compared to baselines.
* `training` `scaling` [Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity](http://arxiv.org/abs/2501.16168v3)
  > **TL;DR**: Proposes Ringmaster ASGD, an asynchronous SGD method that achieves optimal time complexity under heterogeneous worker computation times. The design dynamically manages straggler effects. Demonstrated optimal convergence matching theoretical lower bounds in scaling workers.
* `training` `scaling` [Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization](http://arxiv.org/abs/2506.02787v1)
  > **TL;DR**: Proposes an automatic parallel planning framework for LLM training on heterogeneous nodes with dynamic network changes. Uses simulation-based strategies and pruning for optimal workload allocation. Achieves up to 5× faster search speed and competitive training performance in cloud environments.
* `RL` `training` `quantization` [Reconciling Hessian-Informed Acceleration and Scalar-Only Communication for Efficient Federated Zeroth-Order Fine-Tuning](http://arxiv.org/abs/2506.02370v1)
  > **TL;DR**: Proposes HiSo, a federated zeroth-order fine-tuning method using Hessian-informed optimization and scalar-only communication to accelerate LLM fine-tuning. It transmits only scalars per round with curvature-based acceleration. Achieves up to 4.6× faster convergence than DeComFL on RoBERTa-large.
* `edge` `serving` `kernel` [Scheduling Techniques of AI Models on Modern Heterogeneous Edge GPU -- A Critical Review](http://arxiv.org/abs/2506.01377v1)
  > **TL;DR**: Reviews scheduling techniques for AI models on heterogeneous edge GPUs like NVIDIA Jetson. Analyzes schedulers that optimize resource utilization across CPU, GPU, and accelerators to meet demands of modern AI workloads. Highlights performance improvements in throughput and latency for DNNs on resource-constrained devices.
* `training` `LoRA` `RL` [FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA](http://arxiv.org/abs/2506.01194v1)
  > **TL;DR**: FedRPCA enhances federated aggregation for LoRA fine-tuning by decomposing updates with Robust-PCA: averaging common components and scaled averaging sparse client-specific ones. Achieves faster convergence and higher accuracy across vision/language tasks.
* `serving` `kernel` `networking` [SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models](http://arxiv.org/abs/2502.20727v4)
  > **TL;DR**: Proposes Sync-Point Drop (SPD) to reduce tensor parallelism communication overheads in LLM inference. Selectively drops attention synchronization via block design and sensitivity-based strategies. Achieves 20% latency reduction for LLaMA2-70B on 8 GPUs with <1% accuracy loss.
* `serving` `quantization` `offloading` [TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference](http://arxiv.org/abs/2501.16007v2)
  > **TL;DR**: Proposes TOPLOC, a locality-sensitive hashing scheme for verifiable LLM inference with polynomial encoding. Ensures model integrity by detecting unauthorized changes to model, prompts, or precision with 100% accuracy. Reduces memory overhead by 1000x, requiring 258 bytes instead of 262 KB per 32 tokens.
* `scaling` `training` `RL` [GrapheonRL: A Graph Neural Network and Reinforcement Learning Framework for Constraint and Data-Aware Workflow Mapping and Scheduling in Heterogeneous HPC Systems](http://arxiv.org/abs/2506.00260v1)
  > **TL;DR**: Proposes a GNN and RL framework for constraint and data-aware workflow scheduling in heterogeneous HPC systems. Uses GNN for dependencies and RL for policy-based scheduling to adapt to dynamic workflows. Achieves 76% faster execution than ILP with near-optimal solutions.
* `edge` `serving` `networking` [Smaller, Smarter, Closer: The Edge of Collaborative Generative AI](http://arxiv.org/abs/2505.16499v2)
  > **TL;DR**: Proposes collaborative inference systems for GenAI deploying smaller models at edge integrated with cloud resources to reduce latency, cost, and privacy issues. Designs cooperation strategies across edge-cloud continuum. Achieves reduced end-to-end latency and optimized resource utilization.
* `training` `scaling` `hardware` [DOPPLER: Dual-Policy Learning for Device Assignment in Asynchronous Dataflow Graphs](http://arxiv.org/abs/2505.23131v1)
  > **TL;DR**: Addresses device assignment for minimizing execution time in asynchronous dataflow graphs of ML workloads. Proposes Doppler, a dual-policy network with operation selection and placement stages. Reduces system execution time by 15-40% compared to baselines across tasks.
* `MoE` `serving` `disaggregation` [Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony](http://arxiv.org/abs/2505.08944v2)
  > **TL;DR**: Addresses inefficiencies in serving Mixture-of-Experts models due to load skew and synchronization. Proposes Asynchronous Expert Parallelism (AEP) with μ-queuing and adaptive re-batching. Achieves up to 2.7× higher throughput than baselines.
* `offline` `storage` `kernel` [Speeding up Model Loading with fastsafetensors](http://arxiv.org/abs/2505.23072v1)
  > **TL;DR**: Addresses slow loading of large safetensors models by optimizing deserialization with direct tensor instantiation in device memory. Proposes fastsafetensors library with parallel I/O and GPU offloading. Achieves 4.8x–7.5x speedup for models up to 176B parameters.
* `training` `LoRA` `quantization` [Profiling and optimization of multi-card GPU machine learning jobs](http://arxiv.org/abs/2505.22905v1)
  > **TL;DR**: Investigates optimization techniques for multi-GPU LLM fine-tuning. Analyzes DPO, LoRA, QLoRA, and QAT strategies for memory efficiency and computational cost. Achieves reduced VRAM utilization and iteration times on NVIDIA H100 GPUs.
* `training` `scaling` `RL` [Incentivizing Permissionless Distributed Learning of LLMs](http://arxiv.org/abs/2505.21684v1)
  > **TL;DR**: Proposes Gauntlet, an incentive system for distributed training of LLMs via permissionless contributions. Uses two-stage filtering, loss estimation, OpenSkill rating, and unique computation to reward pseudo-gradient contributions. Achieved competitive 1.2B model with real token payouts in deployed blockchain system.
* `kernel` `compiler` `hardware` [KPerfIR: Towards an Open and Compiler-centric Ecosystem for GPU Kernel Performance Tooling on Modern AI Workloads](http://arxiv.org/abs/2505.21661v1)
  > **TL;DR**: Proposes KPerfIR, a compiler-integrated profiling infrastructure for GPU kernel performance analysis in AI workloads. Enables customizable profiling via compiler passes within Triton, providing fine-grained insights for optimization. Achieves 8.2% overhead and 2% measurement error.
* `serving` `edge` `offloading` [Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits](http://arxiv.org/abs/2505.21594v1)
  > **TL;DR**: Proposes an edge-cloud speculative decoding framework with early exits to reduce LLM inference latency and cost. Combines on-device drafting with server verification and preemptive token generation, leveraging idle time for parallelism. Achieves up to 35% latency reduction and 21% speedup in real robot deployment.
* `training` `networking` `scaling` [DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks](http://arxiv.org/abs/2505.15536v2)
  > **TL;DR**: Proposes DeepCEE, a geo-distributed training system for heterogeneous GPUs and unstable networks. Uses device grouping, pipeline parallelism with zero bubble, and dynamic adaptation to network fluctuations. Achieves 1.3-2.8x higher throughput versus state-of-the-art systems.
* `edge` `training` `offloading` [ECC-SNN: Cost-Effective Edge-Cloud Collaboration for Spiking Neural Networks](http://arxiv.org/abs/2505.20835v1)
  > **TL;DR**: Proposes ECC-SNN, an edge-cloud collaboration framework using spiking neural networks to offload compute from cloud to edge. Employs joint training of ANN and SNN for enhanced performance and on-device incremental learning. Reduces energy by 79.4% and latency by 39.1%.
* `serving` `kernel` `quantization` [InstGenIE: Generative Image Editing Made Efficient with Mask-aware Caching and Scheduling](http://arxiv.org/abs/2505.20600v1)
  > **TL;DR**: Proposes InstGenIE for efficient diffusion-based image editing serving. Utilizes mask-aware caching to skip redundant computations, bubble-free pipeline, continuous batching, and load balancing. Achieves up to 3x higher throughput and 14.7x lower latency.
* `training` `sparse` `networking` [PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning](http://arxiv.org/abs/2505.18563v1)
  > **TL;DR**: Proposes PacTrain, combining pruning and adaptive sparse gradient compression to reduce gradient aggregation overhead in distributed DNN training. Achieves near-optimal compression compatible with all-reduce, improving throughput 1.25-8.72x under bandwidth constraints.
* `training` `scaling` [ATA: Adaptive Task Allocation for Efficient Resource Management in Distributed Machine Learning](http://arxiv.org/abs/2502.00775v2)
  > **TL;DR**: Addresses inefficient resource utilization in distributed ML training due to heterogeneous worker speeds. Proposes ATA, an adaptive task allocation method that dynamically assigns tasks without prior computation time knowledge. Achieves significant cost reduction versus greedy approaches while maintaining speed, without assuming known distributions.
* `training` `scaling` `scheduling` [Resource Heterogeneity-Aware and Utilization-Enhanced Scheduling for Deep Learning Clusters](http://arxiv.org/abs/2503.10918v2)
  > **TL;DR**: Proposes Hadar and HadarE schedulers for resource efficiency in deep learning clusters. Designs task-level heterogeneity-aware optimization and job forking to boost GPU utilization. Reduces training duration by 50-80% and improves utilization by 1.45x versus state-of-the-art schedulers.
* `kernel` `hardware` [Parallel Scan on Ascend AI Accelerators](http://arxiv.org/abs/2505.15112v1)
  > **TL;DR**: Proposes parallel prefix scan algorithms optimized for Ascend AI accelerators by leveraging matrix multiplication units. Implements scan-based operators like sorting and sampling with matrix multiplications and accumulations. Achieves 5×-9.6× speedups over vector-only baselines and up to 37.5% of theoretical memory bandwidth.
* `training` `scaling` [COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems](http://arxiv.org/abs/2505.15020v1)
  > **TL;DR**: Proposes COSMIC, a full-stack co-design environment for distributed ML systems using end-to-end simulation and abstraction for cross-stack optimization. Introduces Parameter Set Architecture to unify configurations. Achieves up to 48.41x higher performance in transformer models up to 175B parameters.
* `edge` `networking` `serving` [SkyMemory: A LEO Edge Cache for Transformer Inference Optimization and Scale Out](http://arxiv.org/abs/2505.14427v1)
  > **TL;DR**: Proposes SkyMemory, a LEO satellite edge cache system for transformer inference, enhancing speed via distributed key-value caching and inter-satellite links. Demonstrated in a 19x5 testbed, improves cache hits and inference latency by reducing ground data center dependencies.
* `training` `networking` `scaling` [Prime Collective Communications Library -- Technical Report](http://arxiv.org/abs/2505.14065v1)
  > **TL;DR**: Presents PCCL, a fault-tolerant collective communication library for distributed ML training over public internet. Supports dynamic peer joining/failure recovery and efficient all-reduce with techniques like multi-connection dispatch and quantization. Achieves up to 45 Gbit/s bandwidth across Europe, reduces communication frequency and bandwidth.
* `training` `scaling` `quantization` [A Study on Distributed Strategies for Deep Learning Applications in GPU Clusters](http://arxiv.org/abs/2505.12832v1)
  > **TL;DR**: Investigates distributed training strategies for scalable deep learning on GPU clusters. Empirically evaluates DDP, FSDP, and PS models, analyzing memory usage, training time, and accuracy. FSDP reduces GPU memory by over 60% but increases training time up to 6x; PS improves throughput with potential accuracy trade-offs.
* `training` `scaling` `storage` [OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training](http://arxiv.org/abs/2504.09844v2)
  > **TL;DR**: Addresses workload imbalance and redundant memory in multi-source data loading for large foundation models. Proposes Omniload with disaggregated preprocessing, centralized data plane, auto-partitioning, and fault recovery. Achieves 4.5x throughput improvement and 13.5x memory reduction at scale.
* `RL` `training` [SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment](http://arxiv.org/abs/2505.12435v1)
  > **TL;DR**: Proposes Self-Guided Direct Preference Optimization (SGDPO) for LLM alignment, adding a pilot term to steer gradient flow and control chosen/rejected reward updates. Improves preference response generation by up to 9.19% in benchmark scores.
* `scaling` `networking` [Workflow-Driven Modeling for the Compute Continuum: An Optimization Approach to Automated System and Workload Scheduling](http://arxiv.org/abs/2505.12184v1)
  > **TL;DR**: Proposes a framework for automated scheduling of workloads across HPC-Cloud continuum to minimize latency and communication overhead. Uses MILP for optimal small-scale scheduling and heuristics for large-scale with <10% deviation from optimal. Heuristics achieve 99% faster estimates with 5-10% deviation from optimal makespan.
* `offline` `offloading` `quantization` [Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission](http://arxiv.org/abs/2505.11788v1)
  > **TL;DR**: Reduces communication in hybrid language models by transmitting truncated vocabulary distributions only when uncertainty is high. Proposes CU-HLM with optimal uncertainty thresholds and truncation strategies. Achieves 206× higher token throughput with 97.4% vocabulary compression while maintaining 97.4% accuracy.
* `serving` `RL` `RAG` [Cloud-Based AI Systems: Leveraging Large Language Models for Intelligent Fault Detection and Autonomous Self-Healing](http://arxiv.org/abs/2505.11743v1)
  > **TL;DR**: Proposes an LLM system for intelligent fault detection and autonomous healing in cloud systems. Integrates LLM natural language processing with ML algorithms and a multi-level architecture for real-time anomaly detection and healing. Achieves higher accuracy and up to 50% downtime reduction over traditional systems.
* `serving` `offloading` `edge` [SpecMemo: Speculative Decoding is in Your Pocket](http://arxiv.org/abs/2506.01986v1)
  > **TL;DR**: Proposes SpecMemo, a device-aware inference engine enabling speculative decoding on memory-constrained devices. It models memory footprint to minimize allocations while retaining throughput, reducing generation-memory by 65% on mobile GPUs. Achieves 96% speculative decoding throughput on MT-Bench and 8x throughput increase with batched decoding on multiple GPUs.
* `training` `hardware` `quantization` [Assessing the Performance of Analog Training for Transfer Learning](http://arxiv.org/abs/2505.11067v1)
  > **TL;DR**: Proposes chopped TTv2 algorithm for analog in-memory computing to overcome device non-linearity and variation challenges in deep learning training. Enables energy-efficient transfer learning on Swin-ViT with CIFAR100. Achieves robustness against weight noise and asymmetry while maintaining accuracy.
* `training` `networking` `scaling` [KAITIAN: A Unified Communication Framework for Enabling Efficient Collaboration Across Heterogeneous Accelerators in Embodied AI Systems](http://arxiv.org/abs/2505.10183v1)
  > **TL;DR**: Proposes KAITIAN, a communication framework enabling efficient collaboration in heterogeneous accelerators for distributed AI training. Features unified abstraction and load-adaptive scheduling. Accelerates training by up to 42% with minimal overhead (2.8--4.3%).
* `inference` `networking` `offline` [AI Greenferencing: Routing AI Inferencing to Green Modular Data Centers with Heron](http://arxiv.org/abs/2505.09989v1)
  > **TL;DR**: Proposes Heron, a cross-site router to dispatch AI inferencing workloads to wind-powered modular data centers during power availability. Optimizes routing using power and workload traces for cost-efficient green computing. Achieves 80% higher aggregate goodput than SOTA.
* `offline` `storage` `scaling` [Efficient Graph Embedding at Scale: Optimizing CPU-GPU-SSD Integration](http://arxiv.org/abs/2505.09258v2)
  > **TL;DR**: Proposes Legend, a graph embedding system optimizing CPU-GPU-SSD integration for billion-scale datasets. Features efficient data placement, direct GPU-SSD access, and parallel execution to minimize I/O overhead. Achieves 4.8x speedup and matches 4-GPU performance using one GPU.
* `serving` `hardware` `quantization` [On the Partitioning of GPU Power among Multi-Instances](http://arxiv.org/abs/2501.17752v2)
  > **TL;DR**: Proposes ML-based models for GPU power estimation in multi-tenant environments using NVIDIA MIG technology. Utilizes partition-level utilization metrics to dynamically predict power consumption per instance. Achieves accurate power estimation across diverse workloads including LLM inference, aiding fair carbon reporting.
* `video` `networking` `offline` [Toward Accessible and Safe Live Streaming Using Distributed Content Filtering with MoQ](http://arxiv.org/abs/2505.08990v1)
  > **TL;DR**: Addresses real-time content moderation for live video streams with strict latency constraints. Extends Media Over QUIC protocol to distribute analysis tasks to clients and remove only objectionable segments. Increases client latency by only one group-of-pictures duration.
* `networking` `scaling` [ATLAHS: An Application-centric Network Simulator Toolchain for AI, HPC, and Distributed Storage](http://arxiv.org/abs/2505.08936v1)
  > **TL;DR**: Proposes ATLAHS, a network simulator toolchain for AI/HPC/storage workloads, using the GOAL format for tracing and simulating multi-job scenarios. Validated with <5% error, outperforming AstraSim in simulation runtime and trace efficiency, and applied to congestion control and job placement case studies.
* `scaling` [Local Constant Approximation for Dominating Set on Graphs Excluding Large Minors](http://arxiv.org/abs/2504.01091v3)
  > **TL;DR**: Investigates constant-factor approximation algorithms for Minimum Dominating Set in graphs excluding K_{2,t} minors. Introduces a distributed algorithm leveraging asymptotic dimension analysis, achieving a 50-approximation in f(t) rounds independent of graph size.
* `edge` `scaling` [Benchmarking of CPU-intensive Stream Data Processing in The Edge Computing Systems](http://arxiv.org/abs/2505.07755v1)
  > **TL;DR**: Investigates optimizing resource utilization in edge computing for CPU-intensive stream data processing. Uses a microbenchmark to profile performance-power relations under varied workloads and CPU frequencies. Achieves optimized edge resource usage by balancing computational efficiency with energy savings.
* `serving` `edge` `offloading` [LA-IMR: Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling for Tail-Latency-Sensitive Cloud Robotics](http://arxiv.org/abs/2505.07417v1)
  > **TL;DR**: Proposes LA-IMR, a control layer combining predictive edge-to-cloud offloading and proactive autoscaling to reduce tail latency for cloud-edge inference. Uses a utilization-driven latency model to steer QoS-aware routing and replica scaling, achieving 20.7% lower P99 latency under bursty workload.
* `training` `RL` `networking` [INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning](http://arxiv.org/abs/2505.07291v1)
  > **TL;DR**: Proposes globally distributed RL training for a 32B-parameter model. Develops PRIME-RL framework with TOPLOC verifier and SHARDCAST broadcast system for asynchronous training on heterogeneous compute. Achieves state-of-the-art reasoning results over centralized baseline (QwQ-32B).
* `storage` `offline` `serving` [TierBase: A Workload-Driven Cost-Optimized Key-Value Store](http://arxiv.org/abs/2505.06556v1)
  > **TL;DR**: Proposes TierBase, a distributed key-value store with workload-driven cost optimization. Integrates pre-trained data compression, elastic threading, and persistent memory to balance performance and storage costs. Achieves up to 62% cost reduction in real-world online data serving scenarios.
* `training` `networking` `scaling` [On Optimal Batch Size in Coded Computing](http://arxiv.org/abs/2505.06199v1)
  > **TL;DR**: Investigates optimal batch size and redundancy in coded computing for parallel job execution. Proposes joint optimization of redundancy level and batch size to minimize expected job completion time. Simulations show improvements in execution time for two service-time distributions.
* `networking` `serving` `RL` [Efficient Information Updates in Compute-First Networking via Reinforcement Learning with Joint AoI and VoI](http://arxiv.org/abs/2505.06025v1)
  > **TL;DR**: Proposes an Age-and-Value-Aware (AVA) metric for compute-first networking to optimize service information updates. Uses reinforcement learning to selectively transmit updates, reducing update frequency by over 90% on average without degrading task execution accuracy.
* `training` `hardware` `scaling` [Toward Heterogeneous, Distributed, and Energy-Efficient Computing with SYCL](http://arxiv.org/abs/2505.06022v1)
  > **TL;DR**: Proposes SYCL extensions for distributed heterogeneous computing. Introduces Celerity for workload distribution and SYnergy for energy efficiency on accelerators. Achieves scalable performance and up to 30% energy reduction in experiments.
* `training` `MoE` [DawnPiper: A Memory-scablable Pipeline Parallel Training Framework](http://arxiv.org/abs/2505.05856v1)
  > **TL;DR**: Addresses memory imbalance and inefficiency in pipeline parallelism for large model training. Proposes DawnPiper framework with fine-grained computation graph partitioning, a memory optimization algorithm, and automatic code generation. Achieves up to 4x larger batch size and 1.5x speedup over vPipe.
* `kernel` `hardware` [PUDTune: Multi-Level Charging for High-Precision Calibration in Processing-Using-DRAM](http://arxiv.org/abs/2505.05266v1)
  > **TL;DR**: Addresses precision calibration for Processing-Using-DRAM (PUD) to mitigate error-prone columns. Proposes PUDTune, a multi-level charging technique applying column-specific offsets to enhance error-free columns. Increases error-free columns by 1.81×, boosting arithmetic throughput by 1.88–1.89×.
* `training` `networking` `offline` [An Asynchronous Distributed-Memory Parallel Algorithm for k-mer Counting](http://arxiv.org/abs/2505.04431v1)
  > **TL;DR**: Presents an asynchronous distributed algorithm for k-mer counting to reduce global communication. Uses fine-grained messaging and custom aggregation to avoid Many-To-Many collectives. Achieves 9x faster performance than state-of-the-art and scales to 256 nodes.
* `training` `RL` `networking` [Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence](http://arxiv.org/abs/2505.03736v1)
  > **TL;DR**: Proposes GT-NSGDm, a decentralized optimization method using gradient tracking and normalization, to handle heavy-tailed noise in nonconvex stochastic optimization. Achieves optimal convergence rate O(1/T^{(p-1)/(3p-2)}), and shows robustness in training language models.
* `offloading` `hardware` [EPOCH: Enabling Preemption Operation for Context Saving in Heterogeneous FPGA Systems](http://arxiv.org/abs/2501.16205v3)
  > **TL;DR**: Addresses context preservation challenges when preempting FPGA tasks in multi-tenant systems. Proposes EPOCH, a framework for state snapshot capture and restoration at any clock cycle. Achieves context save and restore in 62.2μs and 67.4μs per frame on ZynQ-XC7Z020.
* `thinking` `multi-modal` `RL` [A Hashgraph-Inspired Consensus Mechanism for Reliable Multi-Model Reasoning](http://arxiv.org/abs/2505.03553v1)
  > **TL;DR**: Proposes a Hashgraph-inspired consensus mechanism for multi-model reasoning systems to reduce hallucinations and inconsistencies. Treats each reasoning model as a black-box peer, using gossip-about-gossip communication and virtual voting for iterative convergence. Achieves higher accuracy and confidence by incorporating cross-verification across models.
* `serving` `scaling` ["Two-Stagification": Job Dispatching in Large-Scale Clusters via a Two-Stage Architecture](http://arxiv.org/abs/2505.03032v1)
  > **TL;DR**: Investigates how to optimize job dispatching for improved response times in large-scale clusters. Proposes a two-stage architecture with workload-type separation and classical dispatching policies. Achieves better mean response times than single-stage policies, approaching state-aware methods.
* `serving` `quantization` [PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding](http://arxiv.org/abs/2505.01572v1)
  > **TL;DR**: Proposes PipeSpec, a hierarchical pipeline speculation framework for asynchronous token generation and verification in LLM inference. Breaks sequential dependencies with pipe-staged execution and verifier for rollback. Achieves up to 2.54× speedup over baselines on LLaMA models with increasing gains at pipeline depth.
* `training` `quantization` `sparse` [Nesterov Method for Asynchronous Pipeline Parallel Optimization](http://arxiv.org/abs/2505.01099v1)
  > **TL;DR**: Addresses staleness in asynchronous pipeline training due to delayed gradients. Proposes a modified Nesterov Accelerated Gradient that adjusts the look-ahead step to handle gradient delays. Achieves up to 1.28× faster convergence than synchronous baselines on 1B parameter models.
* `agentic` `RL` `serving` [UserCentrix: An Agentic Memory-augmented AI Framework for Smart Spaces](http://arxiv.org/abs/2505.00472v1)
  > **TL;DR**: Proposes UserCentrix, an agentic memory-augmented framework for smart spaces with personalized LLM agents and hybrid control. Features proactive scaling, VoI-based decisions, and multi-agent coordination. Reduces computational resources by up to 45% while improving response accuracy by 30% in evaluations.
* `RAG` `offline` `edge` [Distributed Retrieval-Augmented Generation](http://arxiv.org/abs/2505.00443v1)
  > **TL;DR**: Proposes Distributed RAG (DRAG) to address privacy and scalability in centralized RAG. Uses Topic-Aware Random Walk for peer-to-peer knowledge retrieval in edge environments. Achieves near-centralized performance with half the messages of flooding.
* `hardware` `offline` [Exploration of Cryptocurrency Mining-Specific GPUs in AI Applications: A Case Study of CMP 170HX](http://arxiv.org/abs/2505.03782v1)
  > **TL;DR**: Explores repurposing mining-specific GPUs for AI inference via instruction set modifications. Tests CUDA modifications on NVIDIA CMP 170HX to restore FP32 performance, achieving >15× FP32 gain and >3× LLM inference speedup. Evaluates energy efficiency for cost-effective edge computing.
* `hardware` `inference` `offline` [Comparative Analysis of FPGA and GPU Performance for Machine Learning-Based Track Reconstruction at LHCb](http://arxiv.org/abs/2502.02304v4)
  > **TL;DR**: Compares FPGA and GPU performance for ML inference in LHCb track reconstruction. Focuses on MLP deployment via HLS4ML for FPGAs against GPU baseline, showing FPGAs achieve higher throughput and lower latency at significantly reduced power consumption.
* `training` `scaling` `kernel` [Galvatron: An Automatic Distributed System for Efficient Foundation Model Training](http://arxiv.org/abs/2504.21411v1)
  > **TL;DR**: Galvatron automates selection of hybrid parallelism strategies for efficient foundation model training. It uses a profiler, decision-tree/dynamic-programming search engine, and runtime to optimize across data, tensor, pipeline, and sequence parallelism. Achieves superior throughput over existing frameworks on various clusters.
* `networking` `scaling` `storage` [Raptr: Prefix Consensus for Robust High-Performance BFT](http://arxiv.org/abs/2504.18649v2)
  > **TL;DR**: Raptr is a BFT SMR protocol using Prefix Consensus to achieve robust low-latency high-throughput distributed consensus. It optimizes data dissemination to maintain performance under attacks, handling 260,000 TPS with sub-second latency and minimal degradation at 1% message drops.
* `networking` `training` [Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learning](http://arxiv.org/abs/2504.20854v1)
  > **TL;DR**: Proposes Genie, a framework for testing network infrastructure impact on ML workloads without GPUs. Uses CPU-emulated GPU communication and adapted ASTRA-sim for network-workload interaction modeling. Achieves accurate emulation of network conditions for large-scale ML systems.
* `offloading` `edge` `networking` [Intelligent Task Offloading in VANETs: A Hybrid AI-Driven Approach for Low-Latency and Energy Efficiency](http://arxiv.org/abs/2504.20735v1)
  > **TL;DR**: Addresses high latency and energy inefficiency in vehicular task offloading. Proposes hybrid AI framework with supervised learning, RL, and PSO for dynamic offloading decisions and resource allocation. Reduces latency by 30% and energy consumption by 25% in simulations.
* `hardware` `scaling` `training` [Good things come in small packages: Should we build AI clusters with Lite-GPUs?](http://arxiv.org/abs/2501.10187v2)
  > **TL;DR**: Proposes building AI clusters with numerous small-scale Lite-GPUs instead of large monolithic GPUs for improved scalability and efficiency. Leverages advancements in co-packaged optics for high-bandwidth communication. Achieves reduced manufacturing costs, improved power efficiency, yield, and reduced blast radius compared to traditional GPU clusters.
* `training` `scaling` [Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations](http://arxiv.org/abs/2504.20490v1)
  > **TL;DR**: Addresses inefficient DL training under workload heterogeneity from mixed hardware or uneven data. Proposes HSPMD, extending SPMD with asymmetric sharding, hierarchical communication, and dynamic graph switching for spatial/temporal adaptation. Achieves performance matching or exceeding specialized systems on heterogeneous clusters and elastic training.
* `scaling` `training` `edge` [Electricity Cost Minimization for Multi-Workflow Allocation in Geo-Distributed Data Centers](http://arxiv.org/abs/2504.20105v1)
  > **TL;DR**: Proposes an electricity-cost-aware scheduling algorithm (ECMWS) for multi-workflow allocation in geo-distributed data centers. Uses graph embedding and policy network to solve MDP for deadline-constrained resource scheduling. Achieves over 15% cost reduction compared to state-of-the-art methods.
* `training` `networking` `scaling` [The Big Send-off: High Performance Collectives on GPU-based Supercomputers](http://arxiv.org/abs/2504.18658v1)
  > **TL;DR**: Addresses inefficiencies in collective communication for LLM training on large GPU clusters. Introduces PCCL, an optimized communication library for all-gather/reduce-scatter operations that maximizes network/compute utilization. Delivers 6-33x speedups over RCCL and up to 60% end-to-end training speedup for GPT-3-style models.
* `serving` `edge` `scaling` [Adaptive Heuristics for Scheduling DNN Inferencing on Edge and Cloud for Personalized UAV Fleets](http://arxiv.org/abs/2412.20860v2)
  > **TL;DR**: Proposes deadline-driven heuristics (DEMS-A and GEMS) for scheduling DNN inferencing tasks from UAV fleets on edge and cloud. Uses strategies like task dropping, work stealing, migration, and adaptation to cloud variability to maximize QoS and QoE. Achieves up to 88% task completion rate and 2.7x higher QoS utility.
* `edge` `quantization` `offloading` [EPSILON: Adaptive Fault Mitigation in Approximate Deep Neural Network using Statistical Signatures](http://arxiv.org/abs/2504.20074v1)
  > **TL;DR**: Proposes EPSILON, a lightweight framework for adaptive fault mitigation in approximate deep neural networks using statistical signatures and layer-wise metrics. Achieves 80.05% model accuracy with 22% faster inference and 28% improved energy efficiency on edge devices under faults.
* `training` `networking` `scaling` [Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation](http://arxiv.org/abs/2504.17672v1)
  > **TL;DR**: Addresses inefficiency in cross-region LLM training due to high latency. Proposes CoCoDC with Delay Compensation via Taylor expansion and Adaptive Transmission to optimize synchronization. Reduces training steps by up to 21.0% compared to baselines to reach target perplexity.
* `training` `networking` `sparse` [GRANITE : a Byzantine-Resilient Dynamic Gossip Learning Framework](http://arxiv.org/abs/2504.17471v1)
  > **TL;DR**: Proposes GRANITE, a Byzantine-resilient gossip learning framework using a history-aware peer sampling protocol and adaptive probabilistic threshold for model aggregation. Ensures convergence with 30% Byzantine nodes and operates over 9× sparser graphs than current theory allows.
* `offline` `kernel` `training` [Scalable and Performant Data Loading](http://arxiv.org/abs/2504.20067v1)
  > **TL;DR**: Addresses the data loading bottleneck in AI training. Proposes SPDL, a library using parallel preprocessing with GIL release for efficient array data transfer to GPU. Achieves 74% faster ImageNet iteration and 38% lower CPU usage versus PyTorch DataLoader.
* `edge` `networking` `serving` [6G EdgeAI: Performance Evaluation and Analysis](http://arxiv.org/abs/2504.16529v1)
  > **TL;DR**: Investigates integrated communication-computing for low-latency edge LLM services. Proposes a framework embedding compute in RAN nodes with joint resource management. Achieves 60% higher service capacity vs 5G MEC under transformer workloads.
* `training` `storage` `scaling` [Deep RC: A Scalable Data Engineering and Deep Learning Pipeline](http://arxiv.org/abs/2502.20724v2)
  > **TL;DR**: Addresses scalable data preprocessing and training pipelines for deep learning in HPC environments. Proposes Deep RC, a heterogeneous pipeline integrating data frameworks and deep learning with MPI/GLOO/NCCL support. Reduces pipeline time by 3.28 and 75.9 seconds for different models under identical resources.
* `training` `kernel` `offloading` [Scaling Neural-Network-Based Molecular Dynamics with Long-Range Electrostatic Interactions to 51 Nanoseconds per Day](http://arxiv.org/abs/2504.15508v1)
  > **TL;DR**: Optimizes neural-network-based molecular dynamics with long-range electrostatics for high-efficiency simulations. Introduces hardware-offloaded FFT, computation overlapping, and ring-based load balancing to accelerate DPLR. Achieves 37x speedup, reaching 51 ns/day simulation speed on Fugaku supercomputer.
* `kernel` `serving` `sparse` [ClusterViG: Efficient Globally Aware Vision GNNs via Image Partitioning](http://arxiv.org/abs/2501.10640v2)
  > **TL;DR**: Addresses computational bottlenecks in Vision GNNs via dynamic efficient graph convolution (DEGC), partitioning images for parallel graph construction and global feature learning. ClusterViG architecture achieves 5× inference latency reduction with similar parameter count.
* `training` `MoE` `networking` [Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design](http://arxiv.org/abs/2504.01337v2)
  > **TL;DR**: Addresses inefficiencies in MoE model parallelism due to imbalanced expert activation and communication overhead. Proposes Collaboration-Constrained Routing (C2R) to specialize expert groups via group-constrained routing. Reduces all2all communication costs by 20%-30% over MegaBlocks while improving accuracy.
* `training` `RL` `RAG` [GENE-FL: Gene-Driven Parameter-Efficient Dynamic Federated Learning](http://arxiv.org/abs/2504.14628v1)
  > **TL;DR**: Addresses communication inefficiency and model initialization in dynamic federated learning. Proposes GENE-FL framework using Learngene to compress models via Fisher-based parameter constraints and sensitivity analysis. Reduces communication costs by 4x with 9.04 MB initialization size.
* `training` `multi-modal` `scaling` [PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline](http://arxiv.org/abs/2504.14145v1)
  > **TL;DR**: Proposes PipeWeaver, a dynamic pipeline scheduler for efficient multimodal model training. Uses adaptive modality-aware partitioning and hierarchical schedule search with SEMU simulator. Improves training efficiency by up to 97.3% over SOTA systems.
* `edge` `training` `RL` [Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions](http://arxiv.org/abs/2504.15300v1)
  > **TL;DR**: Surveys collaborative learning between on-device small models and cloud-based large models to address latency, cost, and privacy. Categorizes collaboration algorithms into data, feature, and parameter-based frameworks. Reviews real-world deployments like recommender systems and mobile livestreaming.
* `edge` `diffusion` `quantization` [Diffusion Models on the Edge: Challenges, Optimizations, and Applications](http://arxiv.org/abs/2504.15298v1)
  > **TL;DR**: Addresses deployment challenges of diffusion models on edge devices. Surveys model compression, efficient sampling, and hardware-software co-design optimizations. Highlights methods achieving up to 10x latency reduction and energy savings while maintaining output quality.
* `edge` `RAG` `offloading` [Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performance](http://arxiv.org/abs/2504.11197v2)
  > **TL;DR**: Proposes DRAGON, a distributed RAG framework for on-device SLMs that decomposes multi-document retrieval into parallel token generation with speculative aggregation and network-aware scheduling. Achieves 1.9x performance gain over standalone SLM and reduces per-token latency substantially with negligible TTFT overhead.
* `kernel` `training` `quantization` [TurboFNO: High-Performance Fourier Neural Operator with Fused FFT-GEMM-iFFT on GPU](http://arxiv.org/abs/2504.11681v1)
  > **TL;DR**: Addresses inefficiency in Fourier Neural Operator (FNO) training by fusing FFT-GEMM-iFFT stages into a single GPU kernel. Proposes TurboFNO with custom FFT/GEMM kernels and shared memory optimizations to reduce global memory traffic. Achieves 150% speedup over cuBLAS/cuFFT on A100.
* `serving` `offline` `scaling` [Transformer-Based Model for Cold Start Mitigation in FaaS Architecture](http://arxiv.org/abs/2504.11338v1)
  > **TL;DR**: Addresses cold start latency in serverless FaaS architectures using Transformer models to predict and prewarm functions. Achieves up to 79% reduction in cold start times compared to baselines.
* `training` `storage` [Morphing-based Compression for Data-centric ML Pipelines](http://arxiv.org/abs/2504.11067v1)
  > **TL;DR**: Proposes BWARE, a workload-aware lossless matrix compression method that leverages data transformations and morphing for efficient compressed operations. Reduces training time for data-centric ML pipelines from days to hours by minimizing I/O and improving parallelism.
* `edge` `offloading` `networking` [High-Efficiency Split Computing for Cooperative Edge Systems: A Novel Compressed Sensing Bottleneck](http://arxiv.org/abs/2504.15295v1)
  > **TL;DR**: Proposes HECS-B, a compressed sensing-based split computing architecture for edge-cloud systems. Integrates a compressed sensing autoencoder at a DNN bottleneck layer to reduce bandwidth usage and accelerate processing. Achieves 50% bandwidth reduction and 60% speed-up while maintaining accuracy.
* `training` `kernel` `storage` [Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE](http://arxiv.org/abs/2504.10700v1)
  > **TL;DR**: Optimizes data distribution and kernel performance for efficient training of chemistry foundation models (CFMs). Addresses load balancing via multi-objective bin packing and optimizes symmetric tensor contraction kernel. Achieves 6× speedup, reducing per-epoch time from 12 to 2 minutes on 740 GPUs.
* `serving` `networking` `scaling` [Load Balancing with Network Latencies via Distributed Gradient Descent](http://arxiv.org/abs/2504.10693v1)
  > **TL;DR**: Proposes Distributed Gradient Descent Load Balancing (DGD-LB) for globally distributed LLM serving to minimize latency. Uses decentralized gradient descent with delayed feedback on backend states. Achieves substantial latency reduction compared to baselines, especially under high network delays.
* `training` `LoRA` `MoE` [DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training](http://arxiv.org/abs/2504.09983v1)
  > **TL;DR**: Proposes DeepCompile, a compiler-driven framework that optimizes distributed training via dynamic memory-aware scheduling of operations. Applies profiling-guided optimization passes to improve communication-computation overlap and coordinate optimizations like prefetching and offloading. Achieves up to 1.54x speedup over FSDP for Llama 3 70B.
* `kernel` `sparse` [A Nonlinear Hash-based Optimization Method for SpMV on GPUs](http://arxiv.org/abs/2504.08860v1)
  > **TL;DR**: Proposes a hash-based partition (HBP) format for sparse matrix-vector multiplication (SpMV) on GPUs. Introduces competitive load balancing and hash transformation to accelerate pre-processing. Achieves up to 3.5x speedup in pre-processing and 3.3x speedup in SpMV execution compared to baselines.
* `training` `storage` `scaling` [A Hybrid Cloud Management Plane for Data Processing Pipelines](http://arxiv.org/abs/2504.08225v1)
  > **TL;DR**: Proposes Titchener, a hybrid-cloud management plane for data processing pipelines, enabling seamless deployment across cloud environments. Uses Kubernetes-based control planes for global discovery and connectivity. Reduces deployment complexity by 60% in real-world workflows.
* `agentic` `RAG` `serving` [Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI](http://arxiv.org/abs/2504.08148v1)
  > **TL;DR**: Proposes a blueprint architecture for compound AI systems using streams to orchestrate agents and data for enterprise applications. Features agent registry, data registry, and planners to optimize for quality of service (cost, accuracy, latency). Demonstrates an implementation for HR with optimized QoS compliance.
* `training` `quantization` `offloading` [GPT Carry-On: Training Foundation Model for Customization Could Be Simple, Scalable and Affordable](http://arxiv.org/abs/2504.07513v1)
  > **TL;DR**: Proposes GPT Carry-On, a lightweight adapter method to customize large language models by training compact modules on pretrained embeddings. Enables outsourcing computations to inference nodes, reducing GPU memory usage to <1GB for a 100M parameter module on 30B LLMs. Achieves faster convergence and efficient task specialization with small computation.
* `scaling` `networking` `serving` [ICPS: Real-Time Resource Configuration for Cloud Serverless Functions Considering Affinity](http://arxiv.org/abs/2504.06512v1)
  > **TL;DR**: Addresses resource allocation inefficiency in serverless workflows with branching structures. Proposes ICPS, which predicts concurrency with LSTM and uses affinity-based function co-location to reduce network delays. Experimental results show consistent performance improvement over existing approaches.
* `quantization` `serving` `kernel` [MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization](http://arxiv.org/abs/2504.03661v2)
  > **TL;DR**: Proposes MILLION, a KV cache quantization framework using outlier-immunized product quantization to enhance inference efficiency. Key techniques include non-uniform quantization via product quantization and optimized GPU kernels with sparse computation. Achieves 4-bit quantization with minimal accuracy loss and 2.09x speedup at 32K context.
* `training` `sparse` `networking` [TAGC: Optimizing Gradient Communication in Distributed Transformer Training](http://arxiv.org/abs/2504.05638v1)
  > **TL;DR**: Addresses gradient synchronization bottleneck in distributed transformer training. Proposes TAGC, a transformer-aware gradient compression algorithm with layer-selective compression and dynamic sparsification. Achieves up to 15% training acceleration compared to standard FSDP with minimal accuracy loss.
* `offloading` `hardware` `storage` [dpBento: Benchmarking DPUs for Data Processing](http://arxiv.org/abs/2504.05536v1)
  > **TL;DR**: Presents dpBento, a benchmark suite to evaluate DPU offloading for data processing tasks. Measures performance of various data operations and systems across DPU resources, highlighting offloading benefits for compute, memory, and storage efficiency.
* `kernel` `hardware` `training` [oneDAL Optimization for ARM Scalable Vector Extension: Maximizing Efficiency for High-Performance Data Science](http://arxiv.org/abs/2504.04241v1)
  > **TL;DR**: Optimizes oneDAL for ARM SVE to enhance ML performance. Ports library to ARM with OpenBLAS, introduces vectorized routines and SVE-optimized SVM. Achieves up to 200x speedup vs. scikit-learn on ARM and cost parity with x86 systems.
* `storage` `quantization` `sparse` [IPComp: Interpolation Based Progressive Lossy Compression for Scientific Applications](http://arxiv.org/abs/2502.04093v3)
  > **TL;DR**: Proposes IPComp, an interpolation-based progressive lossy compression with multi-level bitplane and predictive coding for scientific data. Achieves up to 487% higher compression ratios and 698% faster speed than state-of-the-art progressive compressors.
* `training` [Accurate GPU Memory Prediction for Deep Learning Jobs through Dynamic Analysis](http://arxiv.org/abs/2504.03887v1)
  > **TL;DR**: Addresses GPU memory prediction for deep learning training to prevent OOM errors. Proposes VeritasEst, a CPU-based dynamic analysis tool for offline peak memory estimation. Reduces relative error by 84% and failure probability by 73% compared to baselines.
* `training` `kernel` [TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives](http://arxiv.org/abs/2503.20313v3)
  > **TL;DR**: Proposes TileLink, a framework for generating compute-communication overlapped kernels via tile-centric primitives and backend integration. Achieves 1.17× to 20.76× speedup over non-overlapping baselines.
* `inference` `hardware` [Exploring energy consumption of AI frameworks on a 64-core RV64 Server CPU](http://arxiv.org/abs/2504.03774v1)
  > **TL;DR**: Analyzes energy consumption of ML inference frameworks on RISC-V hardware. Benchmarks PyTorch, ONNX Runtime, and TensorFlow on 64-core RV64 CPU, finding XNNPACK-backed frameworks (ONNX, TF) consume less energy than OpenBLAS-backed PyTorch. Quantifies up to lower energy usage in specific setups.
* `training` `edge` `storage` [GraphGen+: Advancing Distributed Subgraph Generation and Graph Learning On Industrial Graphs](http://arxiv.org/abs/2503.06212v2)
  > **TL;DR**: Addresses the inefficiency of distributed subgraph generation and high storage overhead in graph learning. Proposes GraphGen+, which synchronizes distributed subgraph generation with in-memory learning. Achieves up to 27× subgraph generation speedup and supports 1 million nodes per iteration.
* `quantization` `offloading` `kernel` [A Pilot Study on Tunable Precision Emulation via Automatic BLAS Offloading](http://arxiv.org/abs/2503.22875v2)
  > **TL;DR**: Explores precision emulation for HPC workloads via automatic BLAS offloading and INT8 quantization. Uses low-bitwidth integers and unified memory to emulate double-precision operations without code modifications. Achieves improved accuracy and performance simultaneously in MuST application.
* `edge` `training` `RAG` [Satellite Edge Artificial Intelligence with Large Models: Architectures and Technologies](http://arxiv.org/abs/2504.01676v1)
  > **TL;DR**: Proposes architectures for efficient large AI model deployment on satellites. Designs federated fine-tuning over space-ground networks and microservice-based lightweight partitioning for inference. Addresses resource constraints to enable real-time downstream tasks with reduced latency for applications like extreme weather nowcasting.
* `scaling` `networking` `storage` [Carbon and Reliability-Aware Computing for Heterogeneous Data Centers](http://arxiv.org/abs/2504.00518v1)
  > **TL;DR**: Proposes a carbon and reliability-aware optimization framework for workload migration across distributed data centers. Formulates a mixed-integer problem minimizing operational/embodied carbon while meeting SLAs, with server dispatch and backup allocation. Reduces total carbon emissions by up to 21%.
* `scaling` `hardware` [Rack Position Optimization in Large-Scale Heterogeneous Data Centers](http://arxiv.org/abs/2504.00277v1)
  > **TL;DR**: Addresses rack positioning optimization in large data centers to balance operational efficiency and fault tolerance. Proposes a two-tier framework with DRL-guided heuristic for scalable rack placement. Achieves 30% better objective than MIP solvers and 100% success rate in just 2 minutes for 100k-position scale.
* `networking` `training` [GPU-centric Communication Schemes for HPC and ML Applications](http://arxiv.org/abs/2503.24230v1)
  > **TL;DR**: Surveys GPU-centric communication schemes to move the control path from CPU to GPU for reducing overheads in distributed HPC and ML workloads. Leverages GPU and NIC capabilities for direct data transfer without host staging. Discusses techniques addressing bottlenecks in parallel execution.
* `serving` `offline` [Deep Learning Model Deployment in Multiple Cloud Providers: an Exploratory Study Using Low Computing Power Environments](http://arxiv.org/abs/2503.23988v1)
  > **TL;DR**: Explores cost-effective deployment of deep learning models in low-power cloud environments for inference. Evaluates AWS, Google Cloud, and Azure across 7 execution environments using CPU vs GPU tradeoffs. Achieves 50% cost reduction with CPUs leveraging cache optimization, while GPUs cost 300% more on average.
* `training` `offloading` `networking` [FeedSign: Robust Full-parameter Federated Fine-tuning of Large Models with Extremely Low Communication Overhead of One Bit](http://arxiv.org/abs/2501.17610v2)
  > **TL;DR**: Addresses communication overhead in federated fine-tuning of large models. Proposes FeedSign, a method using zeroth-order optimization and shared PRNGs to reduce upload/download to 1 bit per step. Achieves convergence rate O(e^{-t}) with orders-of-magnitude lower communication compared to baselines.
* `training` `scaling` [Optimizing Distributed Training Approaches for Scaling Neural Networks](http://arxiv.org/abs/2503.23186v1)
  > **TL;DR**: Compares distributed training strategies for neural networks. Proposes adaptive scheduling algorithm to switch between data/model parallelism based on resources. Achieves 3.2x speedup with hybrid parallelism and 18% efficiency gain via adaptive scheduling.
* `hardware` `serving` `RAG` [PilotANN: Memory-Bounded GPU Acceleration for Vector Search](http://arxiv.org/abs/2503.21206v1)
  > **TL;DR**: Proposes PilotANN, a hybrid CPU-GPU system for vector search to accelerate ANNS under GPU memory constraints. Decomposes graph traversal into GPU subgraph processing with SVD and CPU refinement. Achieves 3.9-5.4× throughput speedup on 100M datasets and handles 12× larger datasets.
* `scaling` `training` `offline` [Cloud Resource Allocation with Convex Optimization](http://arxiv.org/abs/2503.21096v1)
  > **TL;DR**: Proposes a convex optimization resource allocation framework for Kubernetes clusters to dynamically select node types minimizing cost and fragmentation. Uses logarithmic approximation and interior-point methods to transform discrete scaling into convex problem. Reduces costs by 20% and improves resource utilization over baseline autoscalers.
* `serving` `edge` `offloading` [Solving AI Foundational Model Latency with Telco Infrastructure](http://arxiv.org/abs/2504.03708v1)
  > **TL;DR**: Proposes using telco infrastructure as hierarchical AI edges for caching and partial inference to reduce LLM latency. Introduces tiered caching strategies and split-inference architectures. Achieves significant latency reduction and compute cost savings for real-time applications.
* `networking` `scaling` `storage` [INDIGO: Page Migration for Hardware Memory Disaggregation Across a Network](http://arxiv.org/abs/2503.18140v1)
  > **TL;DR**: Addresses performance degradation in hardware-memory-disaggregated systems due to remote memory access. Proposes INDIGO, a network-aware page migration framework using page telemetry and learning-based adaptation. Achieves 50-70% application performance improvement and reduces network traffic by 2x.
* `edge` `storage` `serving` [DEEP: Edge-based Dataflow Processing with Hybrid Docker Hub and Regional Registries](http://arxiv.org/abs/2504.08741v1)
  > **TL;DR**: Proposes DEEP, a hybrid Docker registry system using regional registries to optimize edge deployment of ML microservices. Compares deployments from Docker Hub, regional registries, and hybrid approach, showing 0.34% energy reduction for text processing microservices via hybrid deployment.
* `training` `scaling` `networking` [Using a Market Economy to Provision Compute Resources Across Planet-wide Clusters](http://arxiv.org/abs/2503.17691v1)
  > **TL;DR**: Proposes a market-based framework for resource provisioning across heterogeneous clusters to balance supply and demand. Implements simulated clock auctions with dynamic reserve prices to influence user bids. Achieves transition of users to under-utilized resource pools, reducing shortages/surpluses.
* `hardware` `kernel` [Debunking the CUDA Myth Towards GPU-based AI Systems](http://arxiv.org/abs/2501.00210v2)
  > **TL;DR**: Evaluates Intel Gaudi NPUs as alternative to NVIDIA GPUs for AI systems. Benchmarks performance in AI operations, workloads, and software optimization for FBGEMM and vLLM operators. Shows Gaudi-2 achieves comparable energy efficiency to A100 but requires software maturity improvements.
* `edge` `inference` `hardware` [Achieving Dependability of AI Execution with Radiation Hardened Processors](http://arxiv.org/abs/2504.03680v1)
  > **TL;DR**: Proposes using a radiation-hardened co-processor (HPDP) with optimized AI runtime (Klepsydra) for dependable, low-latency AI inference in harsh environments. Achieves high-throughput streaming data processing without increased power consumption. Demonstrates suitability for space, nuclear, and medical applications.
* `RL` `serving` `quantization` [GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generation](http://arxiv.org/abs/2501.11006v2)
  > **TL;DR**: Proposes GREEN-CODE, an RL-based framework for energy-efficient LLM inference in code generation via dynamic early exits. Achieves 23-50% energy reduction on JavaCorpus and PY150 datasets without significant accuracy loss.
* `training` `networking` `scaling` [DeFT: Mitigating Data Dependencies for Flexible Communication Scheduling in Distributed Training](http://arxiv.org/abs/2503.16815v1)
  > **TL;DR**: Proposes DeFT, a communication scheduling scheme for distributed training that mitigates data dependencies via delayed updates and reformulates scheduling as knapsack problems. Achieves speedups of 29% to 115% on 16 A100 GPUs with no accuracy loss.
* `networking` [Contemplating a Lightweight Communication Interface for Asynchronous Many-Task Systems](http://arxiv.org/abs/2503.15400v1)
  > **TL;DR**: Proposes a lightweight communication interface (LCI) for efficient networking in Asynchronous Many-Task systems. Introduces objectized flexible functions for expressive primitives and fine-grained resource mapping targeting AMT concurrency needs. Achieves optimized communication throughput compared to existing libraries.
* `edge` `networking` `offloading` [Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks](http://arxiv.org/abs/2503.14882v1)
  > **TL;DR**: Proposes a distributed on-device LLM inference framework using tensor parallelism and over-the-air computation for communication-efficient collaboration among edge devices. Jointly optimizes model assignment and transceiver to minimize transmission error, achieving 5x inference speed acceleration.
* `kernel` `hardware` `serving` [Fake Runs, Real Fixes -- Analyzing xPU Performance Through Simulation](http://arxiv.org/abs/2503.14781v1)
  > **TL;DR**: Proposes xPU-Shark, a fine-grained ML accelerator performance analysis method using hardware simulation. Captures traces from production LLM deployments and replays in a microarchitecture simulator to identify inefficiencies. Optimizes a collective by 15% and reduces token latency by 4.1%.
* `training` `RL` `RAG` [Towards Resource-Efficient Compound AI Systems](http://arxiv.org/abs/2501.16634v3)
  > **TL;DR**: Proposes a declarative workflow programming model and adaptive runtime system for resource-efficient Compound AI Systems. Murakkab prototype decouples application logic from execution, enabling dynamic scheduling and resource-aware decisions. Achieves ∼3.4× faster workflow completion and ∼4.5× higher energy efficiency.
* `networking` `offloading` [Optimizing Data Transfer Performance and Energy Efficiency with Deep Reinforcement Learning](http://arxiv.org/abs/2503.13662v1)
  > **TL;DR**: Proposes a reinforcement learning framework for efficient large-scale data transfers. The method dynamically adjusts transfer settings and pauses threads during heavy network use to balance throughput and energy efficiency. Achieves 25% higher throughput and 40% less energy usage compared to baselines.
* `MoE` `edge` `networking` [Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge](http://arxiv.org/abs/2503.13421v1)
  > **TL;DR**: Proposes DMoE protocol for distributed Mixture-of-Experts at wireless edge, addressing joint expert selection and subcarrier allocation. Introduces Dynamic Expert Selection algorithm with linear relaxation and tunable optimization for AI-channel tradeoff. Achieves high performance with reduced communication cost.
* `video` `sparse` `training` [DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training](http://arxiv.org/abs/2502.07590v3)
  > **TL;DR**: Proposes DSV, a dynamic sparsity exploitation method for accelerating video DiT training. Utilizes two-stage low-rank approximation and custom kernels for sparse attention, plus sparsity-aware hybrid parallelism. Achieves 3.02x higher throughput at 128 GPUs with 520k token lengths.
* `edge` `offloading` `scaling` [CCRSat: A Collaborative Computation Reuse Framework for Satellite Edge Computing Networks](http://arxiv.org/abs/2503.11946v1)
  > **TL;DR**: Proposes CCRSat, a collaborative computation reuse framework for satellite edge networks using local reuse assessment and inter-satellite data similarity sharing. Reduces task completion time by up to 62.1% and computational resource consumption by up to 28.8%.
* `serving` `offloading` `training` [Alchemist: Towards the Design of Efficient Online Continual Learning System](http://arxiv.org/abs/2503.01066v2)
  > **TL;DR**: Addresses redundant computation in online continual learning for LLMs by reusing serving activations. Proposes storing prefill-phase activations/KV cache and smart offloading techniques. Achieves up to 1.72x training throughput increase with 47% memory reduction during training.
* `serving` `hardware` `scaling` [ARCAS: Adaptive Runtime System for Chiplet-Aware Scheduling](http://arxiv.org/abs/2503.11460v1)
  > **TL;DR**: Addresses memory contention in chiplet-based CPUs for parallel applications. Proposes ARCAS runtime with chiplet-aware task scheduling, NUMA-aware memory allocation, and performance monitoring. Achieves significant performance improvements in memory-intensive applications through adaptive task migration and optimization.
* `training` `kernel` `scaling` [FastCHGNet: Training one Universal Interatomic Potential to 1.5 Hours with 32 GPUs](http://arxiv.org/abs/2412.20796v2)
  > **TL;DR**: Proposes FastCHGNet to accelerate training of graph neural network interatomic potentials. Introduces Force/Stress readout modules and GPU optimizations (kernel fusion, load balancing) for multi-GPU scaling. Achieves 3.59× memory reduction and 1.53-hour training time on 32 GPUs (vs 8.3 days on single GPU).
* `training` `scaling` `networking` [Power-Aware Scheduling for Multi-Center HPC Electricity Cost Optimization](http://arxiv.org/abs/2503.11011v1)
  > **TL;DR**: Proposes TARDIS, a power-aware scheduler for multi-center HPC systems, using GNN-based job power prediction and spatio-temporal scheduling to minimize electricity costs. Achieves 10-20% cost reduction by shifting workloads across time and locations using variable electricity pricing.
* `training` `scaling` `sparse` [Galvatron: Automatic Distributed Training for Large Transformer Models](http://arxiv.org/abs/2504.03662v1)
  > **TL;DR**: Proposes Galvatron, a framework for automatically combining data, tensor, and pipeline parallelism to optimize large transformer training. It dynamically selects and adjusts strategies based on model and hardware. Achieves higher throughput compared to static frameworks.
* `serving` `training` `edge` [Concurrent Scheduling of High-Level Parallel Programs on Multi-GPU Systems](http://arxiv.org/abs/2503.10516v1)
  > **TL;DR**: Presents a runtime system for concurrent scheduling of GPU-accelerated distributed programs. Introduces an instruction-graph IR to decouple dependency analysis from execution, using lookahead for memory optimization. Achieves strong scaling on 128 GPUs with reduced runtime delays.
* `RL` `training` `edge` [Efficient Federated Fine-Tuning of Large Language Models with Layer Dropout](http://arxiv.org/abs/2503.10217v1)
  > **TL;DR**: Addresses inefficiency in federated fine-tuning of LLMs on resource-constrained devices. Proposes DropPEFT with adaptive layer dropout to reduce computational and memory overhead. Achieves 1.3-6.3× convergence speedup and 40%-67% memory reduction.
* `training` `scaling` [Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo](http://arxiv.org/abs/2503.09799v1)
  > **TL;DR**: Analyzes scaling laws for DiLoCo, a communication-efficient training method for LLMs. Shows that DiLoCo predictably scales better than data-parallel training, improving batch sizes, generalization, and evaluation loss with model size. Achieves superior performance even at small scales under fixed compute.
* `edge` `serving` [Evaluating Multi-Instance DNN Inferencing on Multiple Accelerators of an Edge Device](http://arxiv.org/abs/2503.09546v1)
  > **TL;DR**: Examines multi-instance DNN inferencing on edge accelerators (GPU CUDA, Tensor Cores, DLA). Evaluates throughput and latency with varied batch sizes and hardware combinations. Identifies contention issues and achieves up to 20% throughput gain by balancing GPU resources.
* `training` `scaling` [Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach](http://arxiv.org/abs/2503.09357v1)
  > **TL;DR**: Proposes automatic operator-level parallelism planning using mixed-integer programming to optimize distributed deep learning for complex models. Formulates as a scheduling problem with bi-level optimization to balance optimality and efficiency. Reduces computational bubbles by half compared to expert strategies like DualPipe under same memory constraints.
* `hardware` `training` `storage` [FpgaHub: Fpga-centric Hyper-heterogeneous Computing Platform for Big Data Analytics](http://arxiv.org/abs/2503.09318v1)
  > **TL;DR**: Proposes FpgaHub, an FPGA-centric hyper-heterogeneous computing platform for big data analytics. Uses FPGA as a hub for data movement, scheduling, and pre-processing to exploit hardware heterogeneity. Improves performance by leveraging reconfigurable computing and rich IO interfaces.
* `edge` `scaling` `storage` [GMB-ECC: Guided Measuring and Benchmarking of the Edge Cloud Continuum](http://arxiv.org/abs/2503.07183v2)
  > **TL;DR**: Proposes GMB-ECC to optimize energy efficiency across edge-cloud computing layers. Introduces a precision parameter for adaptable energy measurement in heterogeneous systems. Uses parameter tuning to achieve optimal energy savings without performance loss.
* `hardware` `serving` `scaling` [VersaSlot: Efficient Fine-grained FPGA Sharing with Big.Little Slots and Live Migration in FPGA Cluster](http://arxiv.org/abs/2503.05930v2)
  > **TL;DR**: Proposes VersaSlot, a FPGA sharing system with Big.Little slots and live migration to resolve reconfiguration contention and task blocking in clusters. Achieves up to 13.66x lower average response time and improves resource utilization by 35% (LUT) and 29% (FF).
* `serving` `scaling` `offline` [Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity](http://arxiv.org/abs/2503.05130v1)
  > **TL;DR**: Addresses GPU resource fragmentation in serverless DL serving. Proposes Dilu, a system with introspective elasticity for fine-grained two-dimensional co-scaling and dynamic GPU allocation. Achieves up to 1.8× inference throughput improvement and 10%-46% GPU defragmentation.
* `RL` `edge` `offloading` [Incentivizing Multi-Tenant Split Federated Learning for Foundation Models at the Network Edge](http://arxiv.org/abs/2503.04971v1)
  > **TL;DR**: Proposes PRINCE, a price-incentive mechanism for multi-tenant split federated learning (SFL) at the network edge to coordinate device participation in FM fine-tuning. Includes bias-resilient aggregation, convergence-bound-based contribution evaluation, and Stackelberg-game-based strategy optimization. Accelerates fine-tuning by up to 3.07x vs baselines.
* `offline` `parallelization` `optimization` [Parallel-in-Time Kalman Smoothing Using Orthogonal Transformations](http://arxiv.org/abs/2502.11686v2)
  > **TL;DR**: Proposes a parallel-in-time linear Kalman smoother using novel QR factorization and selective inversion for state estimation. Implements with TBB, achieving up to 47x speedup on 64 cores despite higher serial arithmetic (1.8-2.5x slower on single core). Outperforms existing parallel smoothers.
* `edge` `offline` `quantization` [Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers](http://arxiv.org/abs/2503.04148v1)
  > **TL;DR**: Addresses sustainable edge inference for multi-tenant DNNs, proposing a framework that dynamically adjusts system power threshold and employs mixed-quality models with an estimator. Achieves 30% lower carbon emissions and 25% reduced carbon delay product versus state-of-the-art.
* `edge` `networking` `scaling` [Benchmarking Dynamic SLO Compliance in Distributed Computing Continuum Systems](http://arxiv.org/abs/2503.03274v1)
  > **TL;DR**: Benchmarks Active Inference against RL algorithms for dynamic SLO compliance in edge-based video streaming applications. Continuously adjusts streaming parameters (e.g., resolution, frame rate) under shifting workloads and constraints. Active Inference reduces memory usage by 20% and stabilizes CPU while ensuring SLOs.
* `MoE` `training` `offline` [Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts](http://arxiv.org/abs/2502.19811v3)
  > **TL;DR**: Addresses communication overhead in distributed training of MoE models. Proposes COMET with fine-grained computation-communication overlapping via dependency analysis and adaptive workload assignment. Achieves 1.96× per-layer and 1.71× end-to-end speedup, saving millions of GPU hours.
* `training` `scaling` [Memory and Bandwidth are All You Need for Fully Sharded Data Parallel](http://arxiv.org/abs/2504.03655v1)
  > **TL;DR**: Investigates how hardware constraints limit training efficiency for large transformers under FSDP. Analyzes computational, memory, and network demands via simulations and tests to find hardware-optimal configurations. Shows interconnection bandwidth and GPU memory thresholds significantly influence training throughput scalability.
* `sparse` `kernel` [NM-SpMM: Accelerating Matrix Multiplication Using N:M Sparsity with GPGPU](http://arxiv.org/abs/2503.01253v2)
  > **TL;DR**: Proposes NM-SpMM, an N:M sparse matrix multiplication kernel for GPUs, using hierarchical blocking, memory access optimization, and pipelining. Achieves 2.1x speedup over prior work and up to 6.3x over dense cuBLAS GEMM, approaching theoretical peak for sparsity levels.
* `edge` `offline` `kernel` [PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators](http://arxiv.org/abs/2504.03654v1)
  > **TL;DR**: Proposes PointSplit, a 3D object detection framework for heterogeneous edge accelerators (GPU/NPU). Key designs include semantics-aware point sampling, parallel feature extraction, and role-based quantization. Achieves 24.7× speedup versus GPU-only implementation while maintaining accuracy on RGB-D datasets.
* `hardware` `edge` `quantization` [PVU: Design and Implementation of a Posit Vector Arithmetic Unit (PVU) for Enhanced Floating-Point Computing in Edge and AI Applications](http://arxiv.org/abs/2503.01313v1)
  > **TL;DR**: Proposes a Posit Vector Arithmetic Unit (PVU) for efficient floating-point computation in edge/AI. Implements vector operations in Chisel with RISC-V extension, targeting reduced hardware consumption. Achieves 100% accuracy in most operations with only 65,407 LUTs.
* `hardware` `edge` `scaling` [SAF: Scalable Acceleration Framework for dynamic and flexible scaling of FPGAs](http://arxiv.org/abs/2503.00974v1)
  > **TL;DR**: Proposes SAF, an Ethernet-based framework enabling flexible scaling and simultaneous reconfiguration of FPGAs for acceleration. Features hot-plug capability and custom protocols, reducing reconfiguration time by 13X, hardware costs by 38%, and energy consumption by 27% in edge/cloud environments.
* `training` `RL` `LoRA` [HLoRA: Efficient Federated Learning System for LLM Heterogeneous Fine-Tuning](http://arxiv.org/abs/2503.00813v1)
  > **TL;DR**: Proposes HLoRA for efficient federated fine-tuning of LLMs with heterogeneous client resources. Uses modified heterogeneous LoRA to address bias in parameter aggregation. Improves convergence speed by 1.7× and reduces communication overhead by 1.5× compared to vanilla LoRA.
* `serving` `scaling` `offloading` [Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving](http://arxiv.org/abs/2504.03651v1)
  > **TL;DR**: Addresses resource underutilization in hybrid LLM serving by co-scheduling online and offline tasks. Introduces Echo with a scheduler, KV cache manager, and estimators for optimized KV reuse and batch prediction. Achieves 3.3× higher offline throughput while meeting online SLOs.
* `kernel` `edge` `training` [WgPy: GPU-accelerated NumPy-like array library for web browsers](http://arxiv.org/abs/2503.00279v1)
  > **TL;DR**: Addresses slow deep learning in web browsers by introducing WgPy, a GPU-accelerated NumPy-like array library using WebGL/WebGPU for custom kernels and JavaScript-Python synchronization. Achieves 95x speedup in CNN training compared to CPU execution.
* `scaling` `serving` [AARC: Automated Affinity-aware Resource Configuration for Serverless Workflows](http://arxiv.org/abs/2502.20846v1)
  > **TL;DR**: Proposes AARC, an automated framework for serverless workflows that decouples CPU/memory via critical path identification and priority scheduling. Reduces total search time by up to 89.6% and costs by up to 61.7%, while ensuring SLO compliance.
* `kernel` `hardware` `training` [Methodology for GPU Frequency Switching Latency Measurement](http://arxiv.org/abs/2502.20075v1)
  > **TL;DR**: Proposes a methodology to measure GPU frequency switching latency for energy optimization in HPC and AI systems. Uses an artificial iterative workload and statistical analysis across different GPUs. Reveals significant latency variations critical for dynamic tuning.
* `training` `edge` `offloading` [RingAda: Pipelining Large Model Fine-Tuning on Edge Devices with Scheduled Layer Unfreezing](http://arxiv.org/abs/2502.19864v1)
  > **TL;DR**: Proposes RingAda for efficient on-edge fine-tuning of large transformer models via pipeline parallelism and scheduled layer unfreezing. Utilizes ring topology with adapters on devices, enabling early backpropagation termination. Reduces fine-tuning time by up to 2.4× and memory usage by 25% compared to baselines.
* `hardware` `kernel` [Can Tensor Cores Benefit Memory-Bound Kernels? (No!)](http://arxiv.org/abs/2502.16851v2)
  > **TL;DR**: Theoretical and empirical analysis challenges the claim that tensor cores benefit memory-bound kernels, showing maximum speedup of 1.33x over CUDA cores is unsound for STREAM Scale, SpMV, and stencil on V100, A100, and H100 GPUs.
* `serving` `hardware` `kernel` [PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System](http://arxiv.org/abs/2502.15470v2)
  > **TL;DR**: Addresses suboptimal static mapping of dynamic LLM decoding kernels. Proposes PAPI, a PIM-enabled architecture with online kernel characterization and heterogeneous hardware orchestration. Achieves up to 11.1× speedup over state-of-the-art PIM-only accelerators.
* `training` `scaling` `hardware` [Static task mapping for heterogeneous systems based on series-parallel decompositions](http://arxiv.org/abs/2502.19745v1)
  > **TL;DR**: Addresses task mapping for heterogeneous systems (CPUs, GPUs, FPGAs etc.) to optimize performance. Proposes a decomposition-based algorithm using series-parallel DAGs and model-based evaluation. Achieves substantially better makespan (up to orders of magnitude faster than baselines) while handling complex dependencies.
* `training` `MoE` [HDEE: Heterogeneous Domain Expert Ensemble](http://arxiv.org/abs/2502.19385v1)
  > **TL;DR**: Proposes HDEE, a heterogeneous mixture-of-experts approach allowing variable expert sizes/training steps per domain. Trains ensembles with domain-adapted experts under fixed compute budget. Reduces perplexity in 20/21 domains compared to homogeneous baselines.
* `RAG` `multi-modal` `edge` [Efficient Federated Search for Retrieval-Augmented Generation](http://arxiv.org/abs/2502.19280v1)
  > **TL;DR**: Proposes RAGRoute, a federated retrieval mechanism for RAG systems using a lightweight classifier to dynamically select data sources, reducing queries by up to 77.5% and communication volume by 76.2%.
* `edge` `hardware` [A Reliable, Time-Predictable Heterogeneous SoC for AI-Enhanced Mixed-Criticality Edge Applications](http://arxiv.org/abs/2502.18953v1)
  > **TL;DR**: Proposes a heterogeneous SoC with configurable hardware IPs for reliable, time-predictable AI-enhanced workloads on edge devices. Integrates accelerators for mixed-precision AI and floating-point workloads, achieving 304.9 GOPS at 1.6 TOPS/W within a 1.2W power envelope.
* `training` `kernel` [Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale](http://arxiv.org/abs/2503.01868v1)
  > **TL;DR**: Proposes convolutional multi-hybrid architectures optimized for token manipulation tasks. Designs hardware-aware algorithms with overlap-add blocked kernels and context parallelism. Trains 1.2-2.9x faster than Transformers and achieves 2x throughput improvement on GPUs.
* `training` `networking` `scaling` [ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression](http://arxiv.org/abs/2502.18554v1)
  > **TL;DR**: Proposes ZCCL, an error-bounded lossy compression framework for MPI collectives to reduce communication costs in distributed training. Integrates customized fZ-light compressor and optimizes collective operations. Achieves 1.9–8.9x speedup over original MPI collectives.
* `kernel` `training` `inference` [Kitsune: Enabling Dataflow Execution on GPUs](http://arxiv.org/abs/2502.18403v1)
  > **TL;DR**: Proposes Kitsune, a set of primitives and compiler for dataflow execution on GPUs to overcome limitations of bulk-synchronous execution. Achieves 1.3x-2.4x performance improvement and 16%-98% off-chip traffic reduction for both training and inference.
* `recommendation` `offline` `edge` [Collaboration of Large Language Models and Small Recommendation Models for Device-Cloud Recommendation](http://arxiv.org/abs/2501.05647v2)
  > **TL;DR**: Proposes LSC4Rec, a device-cloud collaborative framework combining large language models (LLMs) and small recommendation models for efficient real-time recommendation. Uses candidate list generation, dynamic request strategies, and edge deployment to reduce inference costs. Achieves up to 38% higher effectiveness than baselines with minimal resource overhead.
* `training` `disaggregation` `scaling` [Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks](http://arxiv.org/abs/2502.17846v1)
  > **TL;DR**: Addresses memory inefficiency in distributed training for billion-scale graph neural networks. Introduces Armada with GREM partitioning for minimal edge cuts and disaggregated architecture for resource optimization. Achieves 8-65x less memory, 8-46x faster partitioning, and up to 4.5x runtime improvement.
* `training` `offloading` `storage` [CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads](http://arxiv.org/abs/2502.16631v1)
  > **TL;DR**: Addresses checkpointing overhead in GPU-accelerated training workloads. Proposes CRIUgpu, leveraging driver capabilities for transparent GPU snapshots without steady-state overhead. Reduces recovery time with zero performance penalty during execution compared to API interception methods.
* `serving` `LoRA` `offloading` [AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure](http://arxiv.org/abs/2504.03648v1)
  > **TL;DR**: Proposes AIBrix, a co-designed cloud-native framework for scalable LLM inference. Key components include dynamic LoRA management, distributed KV cache, SLO-driven GPU optimizer, and hybrid orchestration. Achieves 50% higher throughput and 70% lower latency via token reuse.
* `serving` `edge` `scaling` [Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models](http://arxiv.org/abs/2502.15964v1)
  > **TL;DR**: Proposes MinionS, a cost-efficient local-remote LM collaboration system where the cloud LM decomposes tasks into subtasks executed by on-device LMs. Achieves 5.7x cloud cost reduction while recovering 97.9% performance vs remote-only inference.
* `serving` `sparse` [Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding](http://arxiv.org/abs/2502.11517v2)
  > **TL;DR**: Proposes PASTA, a learning-based system to enable asynchronous parallel decoding for LLMs via semantic independence annotations. Trains LLMs to express parallelizable chunks using PASTA-LANG language, interpreted for on-the-fly parallel decoding. Achieves 1.21x-1.93x decoding speedup with minor quality changes on AlpacaEval.
* `hardware` `networking` [Next-Gen Computing Systems with Compute Express Link: a Comprehensive Survey](http://arxiv.org/abs/2412.20249v2)
  > **TL;DR**: Explores Compute Express Link (CXL) to overcome interconnect bottlenecks in computing systems. Proposes memory expansion and unified memory for single-machine systems, and disaggregated architectures for distributed systems using CXL's low-latency coherence. Achieves scalable interconnection with significant latency reduction and enhanced resource sharing.
* `serving` `scaling` `offloading` [It Takes Two to Tango: Serverless Workflow Serving via Bilaterally Engaged Resource Adaptation](http://arxiv.org/abs/2502.14320v1)
  > **TL;DR**: Addresses resource inefficiency in serverless workflows due to worst-case function sizing. Proposes Janus, a framework enabling dynamic resource adaptation via developer-provided hints and runtime adjustments. Achieves up to 34.7% higher resource efficiency.
* `edge` `quantization` `offloading` [GenAI at the Edge: Comprehensive Survey on Empowering Edge Devices](http://arxiv.org/abs/2502.15816v1)
  > **TL;DR**: Surveys techniques for deploying generative AI models on edge devices. Covers software/hardware optimizations and frameworks to address constraints like model size and computation. Discusses quantization, pruning, offloading, and specialized hardware achieving over 10× speedup/footprint reduction.
* `training` `scaling` `MoE` [Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs](http://arxiv.org/abs/2502.13480v1)
  > **TL;DR**: Proposes Astra, an automatic parallel strategy search framework for transformer models on heterogeneous GPUs. Uses multi-objective optimization to find efficiency-optimal and cost-saving configurations in GPU types and parallel parameters. Achieves better throughput than expert strategies with search times under 1.35 minutes at 95% accuracy.
* `serving` `offline` `scaling` [Connecting Large Language Model Agent to High Performance Computing Resource](http://arxiv.org/abs/2502.12280v1)
  > **TL;DR**: Addresses how to connect LLM agents to HPC resources for parallel execution of scientific tasks. Integrates Parsl with LangChain/LangGraph to enable concurrent tool execution on HPC systems. Demonstrates efficient resource utilization for molecular dynamics simulations on Polaris/ALCF.
* `kernel` `hardware` [Comparison of Vectorization Capabilities of Different Compilers for X86 and ARM CPUs](http://arxiv.org/abs/2502.11906v1)
  > **TL;DR**: Evaluates compiler auto-vectorization performance on x86 and ARM CPUs using the TSVC2 suite. Analyzes loop vectorization rates and speedups across GCC, ICX, Clang, and ACFL compilers. Measures 46-56% vectorization rates with mixed performance impact.
* `edge` `offloading` [InTec: integrated things-edge computing: a framework for distributing machine learning pipelines in edge AI systems](http://arxiv.org/abs/2502.11644v1)
  > **TL;DR**: Proposes InTec, a framework for strategically distributing machine learning tasks across Things, Edge, and Cloud layers to reduce latency and optimize resources. Achieves 81.56% reduction in response time and 21.86% reduction in edge energy consumption.
* `recommendation` `serving` `hardware` [GPU-accelerated Multi-relational Parallel Graph Retrieval for Web-scale Recommendations](http://arxiv.org/abs/2502.11490v1)
  > **TL;DR**: Proposes a GPU-accelerated multi-relational parallel graph retrieval framework for web-scale recommendations. Integrates multi-relational metric learning with hierarchical parallel graph-based ANNS to optimize retrieval efficiency. Achieves throughput of over 100 million requests per second while improving accuracy.
* `kernel` `training` `edge` [Gensor: A Graph-based Construction Tensor Compilation Method for Deep Learning](http://arxiv.org/abs/2502.11407v1)
  > **TL;DR**: Proposes Gensor, a graph-based tensor compilation method that models construction space exploration as graph traversal with Markov analysis for efficient kernel generation. Achieves average performance improvements of 18% (up to 30%) and reduces optimization time to seconds.
* `edge` `serving` `hardware` [JExplore: Design Space Exploration Tool for Nvidia Jetson Boards](http://arxiv.org/abs/2502.15773v1)
  > **TL;DR**: Addresses the need for optimal hardware configurations on Nvidia Jetson boards for AI workloads at the edge. Introduces JExplore, a design space exploration tool that automates hardware tuning and integrates with search algorithms. Accelerates configuration search for edge deployments, improving efficiency.
* `training` `offline` `kernel` [Image Pre-Processing Framework for Time-Domain Astronomy in the Artificial Intelligence Era](http://arxiv.org/abs/2502.10783v1)
  > **TL;DR**: Addresses slow image pre-processing for AI training in astronomy. Proposes a GPU-accelerated framework with Eager and Pipeline modes for real-time tuning and batch processing. Achieves significantly faster pre-processing speed while matching CPU accuracy.
* `edge` `offloading` `serving` [Janus: Collaborative Vision Transformer Under Dynamic Network Environment](http://arxiv.org/abs/2502.10047v1)
  > **TL;DR**: Proposes Janus for low-latency cloud-device collaborative inference of Vision Transformers. Combines token pruning with dynamic model splitting and pruning policies to balance accuracy and latency. Reduces latency violation ratios by up to 98.7% and increases throughput by 5.15× under dynamic networks.
* `quantization` `edge` `offloading` [EmbBERT-Q: Breaking Memory Barriers in Embedded NLP](http://arxiv.org/abs/2502.10001v1)
  > **TL;DR**: Proposes EmbBERT-Q, a quantized tiny language model for memory-constrained embedded devices. Combines architectural innovations and 8-bit quantization to reduce memory footprint to 781 kB (25x smaller than SotA) while maintaining competitive NLP accuracy on TinyNLP and GLUE benchmarks.
* `edge` `offline` [SmartEdge: Smart Healthcare End-to-End Integrated Edge and Cloud Computing System for Diabetes Prediction Enabled by Ensemble Machine Learning](http://arxiv.org/abs/2502.15762v1)
  > **TL;DR**: Proposes SmartEdge, an edge-cloud system for low-latency diabetes prediction using ensemble ML. Deploys models across edge nodes and cloud servers to process IoMT data. Achieves 15% lower latency at edge versus cloud and 5% higher accuracy than single-model prediction.
* `kernel` `scaling` `serving` [Measuring GPU utilization one level deeper](http://arxiv.org/abs/2501.16909v2)
  > **TL;DR**: Proposes profiling GPU kernel resource interference to enable predictable colocation for AI applications. Develops methodology to measure contention on schedulers, caches, and memory bandwidth. Aims to build GPU schedulers improving utilization with performance guarantees.
* `training` `scaling` `offline` [General Coded Computing: Adversarial Settings](http://arxiv.org/abs/2502.08058v1)
  > **TL;DR**: Proposes a general coded computing scheme for distributed systems to handle adversarial servers during computations. The method enables robustness for diverse computations, achieving optimal adversarial tolerance with error decaying at O(N^{6/5(a-1)}). Validated with DNN inference, tolerating O(N^a) adversarial nodes.
* `serving` `scaling` [Data-aware Dynamic Execution of Irregular Workloads on Heterogeneous Systems](http://arxiv.org/abs/2502.06304v1)
  > **TL;DR**: Proposes DyPe, a scheduling framework for heterogeneous systems that dynamically analyzes input data characteristics to optimize workload partitioning. Achieves 1.53x throughput improvement and 1.09x energy efficiency over static scheduling.
* `hardware` `training` [Exploring Uncore Frequency Scaling for Heterogeneous Computing](http://arxiv.org/abs/2502.03796v1)
  > **TL;DR**: Proposes MAGUS, a runtime for energy-efficient uncore frequency scaling in heterogeneous CPU-GPU systems. It dynamically detects application phases impacting uncore utilization, predicting memory throughput with vendor power management. Achieves up to 27% energy savings and 26% EDP reduction with <5% performance loss.
* `RAG` `serving` `quantization` [Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation](http://arxiv.org/abs/2502.15734v1)
  > **TL;DR**: Addresses redundant KV computations in RAG systems by reusing chunk-caches with selective recomputation. Proposes Cache-Craft for managing reusable chunk caches via identification, fixing, and storage optimization. Reduces redundant computation by 51%, achieving 1.6× throughput improvement and 2× latency reduction.
* `training` `MoE` `scaling` [Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism](http://arxiv.org/abs/2502.02581v1)
  > **TL;DR**: Addresses straggler effects and load imbalance in Mixture-of-Experts (MoE) model training. Proposes Fully Sharded Sparse Data Parallelism (FSSDP) with sparse collectives and materialization techniques. Achieves up to 3.54× speedup over state-of-the-art MoE training systems.
* `scaling` `training` [WaterWise: Co-optimizing Carbon- and Water-Footprint Toward Environmentally Sustainable Cloud Computing](http://arxiv.org/abs/2501.17944v2)
  > **TL;DR**: Addresses conflict between carbon and water sustainability in cloud computing. Proposes WaterWise, a scheduler for parallel workloads that co-optimizes carbon and water footprints in geo-distributed data centers. Achieves reduced environmental impact with explicit trade-offs evaluation.
* `training` [Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning](http://arxiv.org/abs/2501.04266v2)
  > **TL;DR**: Addresses high communication overhead in large-scale LLM training on supercomputers. Proposes a 3-level hierarchical partitioning strategy tailored to the Frontier cluster's bandwidth hierarchy. Achieves 1.71x higher TFLOPS per GPU vs ZeRO++ and 0.94 scaling efficiency at 384 GCDs.
* `training` `hardware` [Ilargi: a GPU Compatible Factorized ML Model Training Framework](http://arxiv.org/abs/2502.01985v1)
  > **TL;DR**: Proposes Ilargi, a factorized ML training framework that uses matrix-represented metadata for automatic factorization on CPU/GPU, avoiding materialization costs. Features an ML cost estimator to choose between factorization and materialization. Achieves up to 8.9x GPU speedups and >20% batch training acceleration.
* `scaling` `hardware` [A Multi-Objective Framework for Optimizing GPU-Enabled VM Placement in Cloud Data Centers with Multi-Instance GPU Technology](http://arxiv.org/abs/2502.01909v1)
  > **TL;DR**: Optimizes GPU resource utilization for LLM serving in cloud data centers. Proposes GRMU, a multi-objective framework with migration-based defragmentation and quota baskets for workload partitioning. Increases acceptance by 22% and reduces hardware by 17%.
* `quantization` `offline` `storage` [SQUASH: Serverless and Distributed Quantization-based Attributed Vector Similarity Search](http://arxiv.org/abs/2502.01528v1)
  > **TL;DR**: Proposes SQUASH, a serverless and distributed quantization-based system for efficient hybrid vector similarity search. Uses OSQ quantization, segment-based storage, and multi-level search for pruning, eliminating redundant I/O. Achieves significant performance improvements at lower cost compared to state-of-the-art.
* `training` `sparse` [Enhancing Token Filtering Efficiency in Large Language Model Training with Collider](http://arxiv.org/abs/2502.00340v1)
  > **TL;DR**: Investigates token filtering inefficiency in LLM training. Proposes Collider, which filters inconsequential tokens across all layers and transforms sparse GEMM into dense GEMM for workload reduction. Achieves 35.1% backpropagation time reduction and 22.0% end-to-end training time improvement with 40% token filtering.
* `training` `offloading` `storage` [Uncoded Download in Lagrange-Coded Elastic Computing with Straggler Tolerance](http://arxiv.org/abs/2501.16298v2)
  > **TL;DR**: Proposes Lagrange-Coded Storage with Uncoded Download (LCSUD) to address elasticity and stragglers in cloud computing, reducing storage size, encoding complexity, and upload cost for matrix multiplications. Achieves lower storage and costs compared to prior methods.
* `scaling` `serving` `networking` [STaleX: A Spatiotemporal-Aware Adaptive Auto-scaling Framework for Microservices](http://arxiv.org/abs/2501.18734v1)
  > **TL;DR**: Proposes STaleX, a spatiotemporal-aware adaptive auto-scaling framework for microservices. Integrates PID controllers with dynamic weight adjustments based on spatial dependencies and temporal workload variations. Reduces resource usage by 26.9% compared to Kubernetes HPA.
* `edge` `agentic` `scaling` [Quantifying Energy and Cost Benefits of Hybrid Edge Cloud: Analysis of Traditional and Agentic Workloads](http://arxiv.org/abs/2501.14823v2)
  > **TL;DR**: Analyzes workload distribution inefficiencies in cloud systems for traditional and agentic AI workloads. Proposes Hybrid Edge Cloud (HEC) to mitigate bottlenecks via edge offloading. Achieves 75% energy savings and over 80% cost reductions in resource-intensive scenarios.
* `serving` `scaling` [Prompt-Aware Scheduling for Efficient Text-to-Image Inferencing System](http://arxiv.org/abs/2502.06798v1)
  > **TL;DR**: Addresses high-load inference inefficiency in text-to-image models due to prompt sensitivity. Proposes prompt-aware scheduler matching prompts to model instances at varying approximations. Achieves high image quality under fixed budget with reduced model loading overheads.
* `hardware` `offloading` `storage` [SP-IMPact: A Framework for Static Partitioning Interference Mitigation and Performance Analysis](http://arxiv.org/abs/2501.16245v1)
  > **TL;DR**: Addresses temporal interference mitigation in embedded systems with static partitioning hypervisors. Proposes SP-IMPact framework to analyze and optimize cache coloring and memory bandwidth reservation configurations. Reduces performance unpredictability by evaluating shared resource interactions on real hardware.
* `edge` `offloading` `networking` [Reinforcement Learning Controlled Adaptive PSO for Task Offloading in IIoT Edge Computing](http://arxiv.org/abs/2501.15203v1)
  > **TL;DR**: Proposes a hybrid approach combining adaptive PSO with reinforcement learning to optimize task offloading in IIoT edge computing. Achieves improved resource management and service quality with minimal latency.
* `hardware` `kernel` `scaling` [TCDM Burst Access: Breaking the Bandwidth Barrier in Shared-L1 RVV Clusters Beyond 1000 FPUs](http://arxiv.org/abs/2501.14370v1)
  > **TL;DR**: Addresses performance degradation in hierarchical network topologies for SIMD/vector cores accessing multi-banked L1 memory. Proposes TCDM Burst Access architecture with a Burst Manager for parallel word retirement. Achieves up to 77-226% higher bandwidth and 2.76x performance improvement in large clusters.
* `kernel` `training` `serving` [RadiK: Scalable and Optimized GPU-Parallel Radix Top-K Selection](http://arxiv.org/abs/2501.14336v1)
  > **TL;DR**: Develops a GPU-optimized radix top-k selection algorithm for improved scalability to large k values. Proposes a memory bandwidth and resource optimization framework with adaptive scaling. Achieves up to 4.8x speedup for batch queries on adversarial inputs over prior methods.
* `recommendation` `hardware` `storage` [Multi-Tenant SmartNICs for In-Network Preprocessing of Recommender Systems](http://arxiv.org/abs/2501.12032v2)
  > **TL;DR**: Proposes Piper, a network-attached FPGA accelerator for streaming data preprocessing in recommender systems. Introduces MiniPipe for runtime-reconfigurable multi-pipeline execution. Achieves 39–105× speedup over CPUs and 3–17× over GPUs, improving energy efficiency.
* `kernel` `hardware` [Compiler Support for Speculation in Decoupled Access/Execute Architectures](http://arxiv.org/abs/2501.13553v1)
  > **TL;DR**: Develops compiler support for speculation in decoupled access/execute architectures to maintain decoupling despite control dependencies. Speculates memory requests and poisons mis-speculations without replays. Enables DAE on previously inapplicable codes, improving efficiency for irregular workloads.
* `multi-modal` `diffusion` `networking` [Resource Allocation Driven by Large Models in Future Semantic-Aware Networks](http://arxiv.org/abs/2501.14832v1)
  > **TL;DR**: Proposes a large-model-driven semantic communication network to reduce data transmission. Uses scene graphs and multimodal pre-trained models for efficient transmission, and a diffusion model-based scheme for power allocation. Achieves improved semantic transmission quality with optimized resource utilization.
* `edge` `networking` `LoRA` [SplitLLM: Hierarchical Split Learning for Large Language Model over Wireless Network](http://arxiv.org/abs/2501.13318v1)
  > **TL;DR**: Proposes hierarchical split learning for LLM fine-tuning over wireless networks to reduce communication and memory overhead. Deploys model parts across users, edge servers, and cloud, updating only LoRA adapters. Reduces peak memory usage by 74% compared to benchmarks.
* `RAG` [Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documents](http://arxiv.org/abs/2501.13954v1)
  > **TL;DR**: Proposes Chat3GPP, an open-source RAG framework for querying 3GPP telecom documents using chunking, hybrid retrieval, and efficient indexing. Eliminates domain-specific fine-tuning while achieving superior performance on telecom datasets for protocol generation and code automation tasks.
* `scaling` `serving` `training` [It's the People, Not the Placement: Rethinking Allocations in Post-Moore Clouds](http://arxiv.org/abs/2501.11185v1)
  > **TL;DR**: Proposes dynamic resource allocation with continuous multilateral cost re-negotiation to address inefficiencies in heterogeneous neoclouds. Shifts from static allocations to improve resource utilization and reduce costs. Demonstrates significant improvements in resource efficiency and cost reduction.
* `training` `storage` `scaling` [Scalable Machine Learning Training Infrastructure for Online Ads Recommendation and Auction Scoring Modeling at Google](http://arxiv.org/abs/2501.10546v1)
  > **TL;DR**: Addresses training system bottlenecks in large-scale ads recommendation models. Proposes shared input generation, partitioning/pipelining for embeddings, and preemption handling mechanisms. Achieves 116% performance boost and 18% training cost reduction in production.
* `hardware` `inference` `memory optimization` [Managed-Retention Memory: A New Class of Memory for the AI Era](http://arxiv.org/abs/2501.09605v1)
  > **TL;DR**: Proposes Managed-Retention Memory (MRM) as a HBM alternative for AI inference workloads, optimizing for density/read bandwidth over retention/write performance. Achieves 20% higher bandwidth and 15% lower energy/bit versus HBM in simulations.
* `kernel` `training` `offline` [Boosting Performance of Iterative Applications on GPUs: Kernel Batching with CUDA Graphs](http://arxiv.org/abs/2501.09398v1)
  > **TL;DR**: Proposes kernel batching via CUDA Graphs to reduce launch overhead in iterative GPU applications. Uses iteration batch unrolling to consolidate kernels into static graphs, finding optimal batch size. Achieves up to 1.4x speed-up in skeleton app and benchmarks.
* `kernel` `training` `quantization` [Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Cores](http://arxiv.org/abs/2501.09251v1)
  > **TL;DR**: Accelerates sparse matrix-matrix multiplication (SpMM) for deep learning using GPU Tensor Cores. Proposes Acc-SpMM with data reordering, memory-efficient formats, pipelining, and load balancing. Achieves up to 5.11x speedup over cuSPARSE on RTX 4090.
* `offloading` `RL` `edge` [Split Fine-Tuning for Large Language Models in Wireless Networks](http://arxiv.org/abs/2501.09237v1)
  > **TL;DR**: Proposes Split Fine-Tuning (SFT) to enable efficient LLM fine-tuning on mobile devices by splitting the model between edge server and device, with data compression to reduce communication overhead. Achieves 80.2% lower delay and 93.6% less overhead while satisfying memory and accuracy constraints.
* `training` `serving` `scaling` [DNN-Powered MLOps Pipeline Optimization for Large Language Models: A Framework for Automated Deployment and Resource Management](http://arxiv.org/abs/2501.14802v1)
  > **TL;DR**: Proposes a DNN-based framework to optimize MLOps pipelines for LLM deployment. Uses multi-stream neural architecture and adaptive resource allocation to automate decisions. Achieves 40% higher resource utilization, 35% lower deployment latency, and 30% reduced costs.
* `serving` `scaling` [Hierarchical Autoscaling for Large Language Model Serving with Chiron](http://arxiv.org/abs/2501.08090v1)
  > **TL;DR**: Explores SLO-aware autoscaling for interactive and batch LLM inference. Proposes Chiron, a hierarchical backpressure-based autoscaler using queue metrics to manage instance count and batch sizes. Achieves up to 90% higher SLO attainment and 70% higher GPU efficiency.
* `training` `scaling` `networking` [Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimization](http://arxiv.org/abs/2501.07526v1)
  > **TL;DR**: Proposes 2D parallel SGD (HybridSGD) to optimize communication efficiency in distributed training. Combines s-step SGD and Federated SGD with Averaging for a performance trade-off, reducing communication bottlenecks. Achieves up to 121× speedup over FedAvg on a supercomputing system.
* `offline` `storage` `hardware` [COMPASS: A Compiler Framework for Resource-Constrained Crossbar-Array Based In-Memory Deep Learning Accelerators](http://arxiv.org/abs/2501.06780v1)
  > **TL;DR**: Introduces COMPASS, a compiler framework for partitioning large DNNs onto resource-constrained PIM accelerators with crossbar arrays. Optimizes layer partitioning to minimize latency and memory accesses by considering data dependencies and core utilization. Achieves 1.78× higher throughput and 1.28× lower EDP versus baselines.
* `serving` `offloading` `multi-modal` [Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management](http://arxiv.org/abs/2501.06709v1)
  > **TL;DR**: Addresses memory inefficiency in LLM serving due to imbalanced KV cache across GPUs. Proposes MELL, a system using adaptive request migration and multi-GPU scheduling for load balancing. Reduces required GPUs by 31% and increases utilization by 43% compared to existing systems.
* `training` `diffusion` `networking` [Decentralized Diffusion Models](http://arxiv.org/abs/2501.05450v2)
  > **TL;DR**: Addresses high networking costs and infrastructure constraints in diffusion model training. Proposes decentralized expert diffusion models trained in isolation and ensembled at inference. Reduces infrastructure costs and achieves FLOP-for-FLOP superior performance compared to standard diffusion models.
* `training` `scaling` `scheduling` [Prediction-Assisted Online Distributed Deep Learning Workload Scheduling in GPU Clusters](http://arxiv.org/abs/2501.05563v1)
  > **TL;DR**: Proposes A-SRPT, a prediction-assisted online scheduler for distributed deep learning training in GPU clusters. Uses random forest to predict job iterations and maps scheduling to a single-machine SRPT problem, minimizing communication overhead. Achieves competitive scheduling efficiency per theoretical proofs and evaluations.
* `validation` `GPU` `trustless` [Validation of GPU Computation in Decentralized, Trustless Networks](http://arxiv.org/abs/2501.05374v1)
  > **TL;DR**: Proposes GPU computation verification in decentralized networks. Develops probabilistic frameworks using model fingerprinting, semantic similarity, and GPU profiling. Achieves trustless verification without hardware requirements, addressing non-determinism in GPU nodes.
* `edge` `RL` `serving` [Microservice Deployment in Space Computing Power Networks via Robust Reinforcement Learning](http://arxiv.org/abs/2501.06244v1)
  > **TL;DR**: Proposes a robust reinforcement learning-based microservice deployment framework for low-latency remote sensing inference in satellite constellations. Minimizes resource utilization while meeting QoS via decomposed module optimization and handles data uncertainty. Achieves acceptable computational cost with minimized accuracy loss in simulations.
* `offloading` `edge` `networking` [Intelligent Task Offloading: Advanced MEC Task Offloading and Resource Management in 5G Networks](http://arxiv.org/abs/2501.06242v1)
  > **TL;DR**: Proposes an intelligent task offloading framework using Proximal Policy Optimization for MEC in 5G networks. Optimizes communication and computational resource allocation among UEs, reducing URLLC processing time by 4% and mMTC power consumption by 26% compared to baselines.
* `edge` `recommendation` `training` [Forward Once for All: Structural Parameterized Adaptation for Efficient Cloud-coordinated On-device Recommendation](http://arxiv.org/abs/2501.02837v1)
  > **TL;DR**: Proposes Forward-OFA for efficient on-device recommendation by dynamically constructing device-specific networks (structure and parameters) using a structure controller and mapper. Achieves adaptation in one forward pass, improving efficiency by eliminating backpropagation and adapting to varying device capabilities.
* `edge` `RL` `LoRA` [Efficient Deployment of Large Language Models on Resource-constrained Devices](http://arxiv.org/abs/2501.02438v1)
  > **TL;DR**: Proposes FedSpine, a federated learning framework combining PEFT and structured pruning to efficiently deploy LLMs on resource-constrained devices. Uses an adaptive MAB algorithm for heterogeneous pruning ratios and LoRA ranks. Achieves 1.4×-6.9× faster fine-tuning with 0.4%-4.5% higher accuracy.
* `serving` `offline` [SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services](http://arxiv.org/abs/2501.02181v1)
  > **TL;DR**: Proposes an SMDP-based dynamic batching scheme to balance latency and energy efficiency for batch services. Uses semi-Markov decision process to minimize weighted response time and power consumption, introducing abstract cost to reduce complexity. Reduces time complexity by 98% and space complexity by 63.5%.
* `edge` `disaggregation` `training` [LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networks](http://arxiv.org/abs/2501.01293v1)
  > **TL;DR**: Addresses training deep learning models on LEO satellites with intermittent connectivity. Proposes LEO-Split, a semi-supervised split learning framework with auxiliary modeling, pseudo-labeling for imbalance, and adaptive activation interpolation. Achieves superior performance over benchmarks in real satellite traces.
* `serving` [Towards Sustainable Large Language Model Serving](http://arxiv.org/abs/2501.01990v1)
  > **TL;DR**: Characterizes operational and embodied carbon emissions for LLM serving across GPU types and models. Develops analytical models for performance, energy, and carbon emissions based on hardware specs and grid regions. Insights enable optimizing sustainable LLM serving by jointly reducing both emission types.
* `edge` `serving` `multi-modal` [Distributed Mixture-of-Agents for Edge Inference with Large Language Models](http://arxiv.org/abs/2412.21200v1)
  > **TL;DR**: Proposes a distributed MoA architecture for collaborative LLM inference on edge devices using gossip algorithms. Analyzes queue stability and validates via AlpacaEval 2.0. Achieves bounded queue sizes while improving response quality over single models.
* `scaling` `training` `kernel` [TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication](http://arxiv.org/abs/2412.20501v1)
  > **TL;DR**: Addresses communication bottlenecks in long-sequence LLM parallelism. Proposes TokenRing, a fine-grained framework with bidirectional P2P communication that overlaps computation and data transmission. Achieves improved scalability and throughput compared to existing methods like Ring-Attention.
* `serving` `disaggregation` `offloading` [GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions](http://arxiv.org/abs/2412.20322v1)
  > **TL;DR**: Addresses high carbon emissions in LLM serving by reusing heterogeneous GPUs. Proposes GreenLLM, an SLO-aware framework disaggregating computations to older GPUs via offloading. Reduces carbon emissions by up to 40.6% while meeting latency SLOs for >90% requests.
* `RL` `edge` `networking` [Federated Unlearning with Gradient Descent and Conflict Mitigation](http://arxiv.org/abs/2412.20200v1)
  > **TL;DR**: Proposes Federated Unlearning with Orthogonal Steepest Descent (FedOSD) to remove client data from FL models while preserving utility. Uses orthogonal gradients to mitigate conflicts during unlearning and maintain unlearning during recovery. Achieves higher unlearning effectiveness and model utility than SOTA methods.
* `LoRA` `training` [Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices](http://arxiv.org/abs/2412.20004v1)
  > **TL;DR**: Addresses resource and heterogeneity challenges in federated fine-tuning. Proposes LEGEND, an adaptive LoRA framework that optimizes LoRA depth and rank distribution via efficient configuration algorithms. Achieves 1.5-2.8x speedup and 42.3% communication savings vs. baselines.
* `inference` `edge` `storage` [Unlocking True Elasticity for the Cloud-Native Era with Dandelion](http://arxiv.org/abs/2505.01603v2)
  > **TL;DR**: Proposes Dandelion, an elastic cloud platform with declarative DAG programming model for cloud-native applications. Uses lightweight sandboxes without guest OS to cold start in hundreds of microseconds. Achieves 96% memory reduction on Azure Functions trace compared to Firecracker.
* `serving` `scaling` [Demystifying Serverless Costs on Public Platforms: Bridging Billing, Architecture, and OS Scheduling](http://arxiv.org/abs/2506.01283v2)
  > **TL;DR**: Analyzes cost inefficiencies in public serverless platforms by characterizing billing models, request architectures, and OS scheduling. Quantifies billing overheads up to 4.35× due to wall-clock allocation, invocation fees, and rounding. Identifies architectural and scheduling factors affecting cost and performance.
* `edge` `serving` [A Task Equalization Allocation Algorithm Incorporating Blocking Estimation and Resource Similarity Analysis for Vehicle Control Real-Time Systems](http://arxiv.org/abs/2509.14086v1)
  > **TL;DR**: Addresses real-time task scheduling inefficiency in vehicle control systems. Proposes the BR-WFD algorithm that estimates blocking time and groups resource-similar tasks. Reduces processor core requirements by 11-28% and improves schedulable ratio by 15-20% under high-load scenarios.
* `edge` `training` `networking` [Ratio1 -- AI meta-OS](http://arxiv.org/abs/2509.12223v1)
  > **TL;DR**: Proposes Ratio1, a decentralized meta-OS for AI pipelines using blockchain to pool heterogeneous edge resources. Key components include dAuth, CSTORE, R1FS, EDIL, Deeploy, and OracleSync for secure execution. Claimed cost-efficiency improvements over centralized systems via Proof-of-Availability and Proof-of-AI consensus.
* `training` `offloading` `kernel` [MaLV-OS: Rethinking the Operating System Architecture for Machine Learning in Virtualized Clouds](http://arxiv.org/abs/2508.03676v1)
  > **TL;DR**: Proposes MaLV-OS, an operating system tailored for ML workloads in virtualized clouds. Features a micro-kernel that enables GPU access in kernel space and offloads system-sensitive model components to the OS. Achieves reduced execution time by optimizing resource allocation and integrating GPU virtualization.
* `edge` `kernel` `serving` [Composable OS Kernel Architectures for Autonomous Intelligence](http://arxiv.org/abs/2508.00604v1)
  > **TL;DR**: Proposes a composable OS kernel architecture for AI systems, integrating AI-oriented LKMs, built-in deep learning inference, and real-time scheduling. Enhances ML workload efficiency by enabling kernel-space processing and floating-point acceleration. Achieves adaptive scheduling for autonomous intelligence applications.
* `edge` `serving` `scaling` [ConsumerBench: Benchmarking Generative AI Applications on End-User Devices](http://arxiv.org/abs/2506.17538v1)
  > **TL;DR**: Proposes ConsumerBench, a framework for benchmarking GenAI application performance on end-user devices under multi-application contention. Evaluates system-level metrics and resource sharing inefficiencies, showing benefits of custom kernels and SLO-aware scheduling for improved latency and SLO attainment.
* `serving` `networking` `edge` [ROS 2 Agnocast: Supporting Unsized Message Types for True Zero-Copy Publish/Subscribe IPC](http://arxiv.org/abs/2506.16882v1)
  > **TL;DR**: Develops Agnocast, a true zero-copy IPC framework for ROS 2 to eliminate serialization overhead in publish/subscribe systems. Enables direct shared-memory communication for unsized message types with minimal code changes. Achieves 16–25% response time improvement in autonomous driving point cloud processing.
* `serving` `offloading` `thinking` [OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents](http://arxiv.org/abs/2506.16042v1)
  > **TL;DR**: Studies inefficiency in computer-use AI agents, identifying model call latency and step progression delays as key issues. Proposes OSWorld-Human benchmark with human-annotated trajectories for efficiency evaluation. Agents take 1.4-2.7x more steps than necessary, highlighting optimization needs.
* `training` `hardware` `kernel` [PerfTracker: Online Performance Troubleshooting for Large-scale Model Training in Production](http://arxiv.org/abs/2506.08528v3)
  > **TL;DR**: Addresses the challenge of diagnosing performance issues in large-scale model training systems. Proposes PerfTracker, an online troubleshooting system using fine-grained profiling and differential observability. Deployed on clusters up to 10,000 GPUs, diagnosing diverse performance issues with minimal impact.
* `networking` `scaling` [Söze: One Network Telemetry Is All You Need for Per-flow Weighted Bandwidth Allocation at Scale](http://arxiv.org/abs/2506.00834v2)
  > **TL;DR**: Addresses agile weighted bandwidth allocation in large-scale data centers. Proposes a decentralized system using simple network telemetry to compute and enforce flows without per-flow state. Reduces TPC-H job completion time by up to 0.59×.
* `storage` `offloading` `kernel` [Adaptive Migration Decision for Multi-Tenant Memory Systems](http://arxiv.org/abs/2505.09164v1)
  > **TL;DR**: Proposes a migration control framework for multi-tenant tiered memory systems. Introduces detection of migration friendliness using per-page ping-pong status and low-cost pattern change detection. Achieves efficient memory management in single and multi-tenant setups with improved performance.
* `edge` `serving` [Work-in-Progress: Multi-Deadline DAG Scheduling Model for Autonomous Driving Systems](http://arxiv.org/abs/2505.06780v2)
  > **TL;DR**: Proposes a multi-deadline DAG scheduling model for autonomous driving systems to decompose end-to-end latency constraints into local deadlines. Extends Global EDF for sub-DAGs to ensure data freshness without complex flow analysis. Evaluation shows improved timing guarantees for Autoware workloads.
* `hardware` `kernel` `serving` [CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs](http://arxiv.org/abs/2505.06625v1)
  > **TL;DR**: Proposes CaMDN, an architecture-scheduling co-design to enhance cache efficiency for multi-tenant DNNs on NPUs. Introduces model-exclusive cache regions and a cache-aware scheduler. Reduces memory access by 33.4% and achieves up to 2.56× model speedup.
* `kernel` [Safe and usable kernel extensions with Rex](http://arxiv.org/abs/2502.18832v2)
  > **TL;DR**: Addresses usability issues of kernel extensions via static verifiers like eBPF. Proposes Rex, a framework using safe Rust and lightweight runtime for safety properties without separate verification. Achieves comparable performance to eBPF.
* `agentic` `multi-modal` `RL` [UFO2: The Desktop AgentOS](http://arxiv.org/abs/2504.14603v2)
  > **TL;DR**: Presents UFO2, an AgentOS integrating LLM agents with desktop automation. Features OS-level integration, hybrid GUI parsing, and multi-action planning to reduce overhead. Achieves higher robustness and execution accuracy across 20+ applications over prior methods.
* `networking` `serving` [The NIC should be part of the OS](http://arxiv.org/abs/2501.10138v2)
  > **TL;DR**: Examines the split between OS and NIC for network performance. Proposes integrating NICs with OS kernels via cache-coherent interconnects to optimize RPC response latency. Achieves lower latency than kernel-bypass approaches while maintaining flexibility.
* `serving` `kernel` `training` [LithOS: An Operating System for Efficient Machine Learning on GPUs](http://arxiv.org/abs/2504.15465v1)
  > **TL;DR**: Proposes LithOS, an operating system for GPUs to efficiently manage resources during ML inference and training. Introduces TPC scheduler, kernel atomization, hardware right-sizing, and power management for improved utilization. Reduces inference tail latencies by 13x vs MPS and improves throughput by 1.6x vs SotA while saving GPU capacity and energy.
* `storage` `disaggregation` `networking` [My CXL Pool Obviates Your PCIe Switch](http://arxiv.org/abs/2503.23611v3)
  > **TL;DR**: Proposes software-based PCIe device pooling using CXL memory pools to replace hardware PCIe switches. Enables PCIe devices to use CXL pools as I/O buffers without hardware changes, improving flexibility and deployment. Achieves cost reduction and higher resource utilization compared to hardware switches.
* `kernel` `storage` `quantization` [Futureproof Static Memory Planning](http://arxiv.org/abs/2504.04874v1)
  > **TL;DR**: Addresses memory inefficiency in dynamic storage allocation for static deep neural networks. Proposes idealloc, a low-fragmentation implementation optimized for million-buffer instances. Outperforms production implementations in effectiveness/robustness on hard benchmarks.
* `serving` `offloading` [Efficient Function-as-a-Service for Large Language Models with TIDAL](http://arxiv.org/abs/2503.06421v1)
  > **TL;DR**: Addresses GPU cold start in FaaS serving for LLMs. Proposes TIDAL, which traces execution paths to generate adaptive function templates for efficient state preloading. Reduces cold start latency by 1.79x~2.11x and improves 95%-ile TTFT by 76.0%.
* `offloading` `edge` `serving` [FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference](http://arxiv.org/abs/2503.03777v1)
  > **TL;DR**: Addresses high memory demands in on-device LLM inference. Proposes FlexInfer with asynchronous prefetching, balanced memory locking, and flexible tensor preservation to enhance efficiency. Achieves 12.5× higher throughput under resource constraints compared to existing methods.
* `training` `edge` `quantization` [Accelerated Training on Low-Power Edge Devices](http://arxiv.org/abs/2502.18323v1)
  > **TL;DR**: Accelerates training on power-constrained edge devices by jointly optimizing GPU frequency and batch size. Uses a cross-layer approach combining batch size efficiency prediction and device profiling. Reduces training time by 2.4× with close to optimal energy efficiency.
* `serving` `offloading` `networking` [Taming and Controlling Performance and Energy Trade-offs Automatically in Network Applications](http://arxiv.org/abs/2502.14987v1)
  > **TL;DR**: Proposes a Bayesian optimizer for network applications to reduce energy consumption while meeting latency SLAs. Dynamically controls packet batching and processing rate (DVFS) without app changes. Achieves up to 60% energy savings across diverse hardware systems.
* `training` `kernel` `hardware` [Phoenix -- A Novel Technique for Performance-Aware Orchestration of Thread and Page Table Placement in NUMA Systems](http://arxiv.org/abs/2502.10923v2)
  > **TL;DR**: Proposes Phoenix, a kernel-level technique for coordinated thread and page table placement in NUMA systems. Integrates CPU scheduler and memory manager with on-demand page table replication and bandwidth management. Reduces CPU cycles by 2.09x and page-walk cycles by 1.58x compared to state-of-the-art.
* `networking` `serving` [Fast Userspace Networking for the Rest of Us](http://arxiv.org/abs/2502.09281v1)
  > **TL;DR**: Proposes Machnet, a userspace network stack using a 'Least Common Denominator' model for public cloud VMs. Optimized for generic NIC features without flow steering or RSS, with microkernel design using IPC for performance. Achieves comparable performance to state-of-the-art stacks despite minimal vNIC features.
* `training` `diagnostics` `scaling` [XPUTimer: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale](http://arxiv.org/abs/2502.05413v1)
  > **TL;DR**: Addresses anomaly diagnosis in large-scale LLM training clusters. Proposes XPUTimer, a real-time framework using lightweight tracing and intra-kernel diagnostics for GPU clusters. Achieved significant improvements across training stack on 6000 GPUs.
* `serving` `RAG` `quantization` [Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation](http://arxiv.org/abs/2502.15734v1)
  > **TL;DR**: Proposes Cache-Craft, a system for reusing precomputed KV caches of text chunks in RAG-based LLM inference to reduce redundant computations. It identifies reusable chunks, performs partial recomputation to maintain quality, and optimizes storage and eviction. Achieves 1.6× throughput increase and 51% less computation.
* `serving` `networking` `offloading` [GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines](http://arxiv.org/abs/2502.15712v1)
  > **TL;DR**: Investigates reducing resource overheads in complex AI inference pipelines by offloading data processing tasks to SmartNICs. Proposes leveraging network hardware packet processing pipelines for computational tasks. Explores integration to mitigate network delays and optimize serving efficiency.

### 2025-12-25
* `training` `scaling` `networking` [Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality](http://arxiv.org/abs/2512.20968v1)
  > **TL;DR**: Proposes Mesh-Attention, a distributed attention algorithm using 2D tile assignment and a greedy scheduler for communication efficiency. Reduces communication volume by up to 85.4% and achieves 3.4x speedup on 256 GPUs compared to Ring-Attention during LLM training.
* `training` `scaling` [Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictions](http://arxiv.org/abs/2512.20967v1)
  > **TL;DR**: Proposes deadline-aware scheduling for LLM fine-tuning using spot instances with predictive models. Designs mixed online algorithms with prediction and policy selection to handle price and availability uncertainties. Achieves up to 54.8% utility improvement over baselines.
* `training` `scaling` `quantization` [Diving into 3D Parallelism with Heterogeneous Spot Instance GPUs: Design and Implications](http://arxiv.org/abs/2512.20953v1)
  > **TL;DR**: Addresses efficient distributed training of LLMs across heterogeneous GPUs. Presents AutoHet, a system for automating optimal 3D parallelism plan selection and recovery upon spot preemption. Achieves up to 1.79× training throughput speedup and 4.38× faster recovery.
* `video` `training` `networking` [AirGS: Real-Time 4D Gaussian Streaming for Free-Viewpoint Video Experiences](http://arxiv.org/abs/2512.20943v1)
  > **TL;DR**: Proposes AirGS, a streaming-optimized 4D Gaussian Splatting framework for free-viewpoint video. Rearchitects training with multi-channel conversion and keyframe identification, and optimizes delivery via adaptive pruning. Reduces transmission size by ~50% and training time by 6x while maintaining PSNR above 30.
* `serving` `RL` `scaling` [RHAPSODY: Execution of Hybrid AI-HPC Workflows at Scale](http://arxiv.org/abs/2512.20795v1)
  > **TL;DR**: Enables concurrent execution of heterogeneous AI-HPC workflows (simulation, training, inference, agentic control). Proposes RHAPSODY, a multi-runtime middleware coordinating existing runtimes. Achieves near-linear scaling for high-throughput inference and minimal overhead with Dragon/vLLM on HPC systems.
* `serving` `offloading` `kernel` [PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation](http://arxiv.org/abs/2512.20687v1)
  > **TL;DR**: Addresses memory inefficiency and latency in Transformer decoding by proposing PHOTON, a hierarchical model with compressed contextual states. Reduces KV-cache traffic through multi-resolution context access. Achieves up to 1000× higher throughput per unit memory for long-context tasks.

### 2025-12-24
* `serving` `quantization` `diffusion` [Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs](http://arxiv.org/abs/2512.20573v1)
  > **TL;DR**: Proposes FailFast, a diffusion LLM-based speculative decoding framework that dynamically adapts draft length to accelerate autoregressive LLM inference. Achieves up to 4.9× speedup over vanilla decoding with lossless quality.
* `networking` `storage` [WOC: Dual-Path Weighted Object Consensus Made Efficient](http://arxiv.org/abs/2512.20485v1)
  > **TL;DR**: Addresses consensus protocols' inability to handle node heterogeneity and workload independence. Proposes WOC with dual-fast/slow paths using object-specific weighted quorums and leader coordination. Achieves 4× higher throughput than Cabinet for low-contention workloads.
* `LoRA` `serving` [Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs](http://arxiv.org/abs/2512.20210v1)
  > **TL;DR**: Addresses latency and fragmentation in serverless LLM serving with multiple LoRA adapters. Proposes proactive adapter prefetching and page-based memory management. Achieves 35% reduction in average TTFT and 1.52x throughput improvement over prior system.
* `serving` `thinking` `RL` [Reaching Agreement Among Reasoning LLM Agents](http://arxiv.org/abs/2512.20184v1)
  > **TL;DR**: Proposes Aegean, a consensus protocol for multi-agent reasoning to ensure reliable agreement while reducing latency. Implements a serving engine with incremental quorum detection for early termination. Cuts latency by 1.2-20× compared to baselines while maintaining answer quality within 2.5%.
* `sparse` `training` `kernel` [SHIRO: Near-Optimal Communication Strategies for Distributed Sparse Matrix Multiplication](http://arxiv.org/abs/2512.20178v1)
  > **TL;DR**: Addresses high communication overhead in distributed sparse matrix multiplication. Proposes fine-grained sparsity-aware communication and hierarchical strategies leveraging GPU network topologies. Achieves up to 221.5× speedup over baselines at 128-GPU scale.
* `offline` `scaling` `quantization` [FastMPS: Revisit Data Parallel in Large-scale Matrix Product State Sampling](http://arxiv.org/abs/2512.20064v1)
  > **TL;DR**: Addresses high memory and I/O overhead in large-scale Matrix Product State (MPS) sampling. Proposes Fast-MPS, combining data and tensor parallelism with compression and overlapping techniques. Achieves over 10× speedup and scales to thousands of processes, handling 8,176 sites.
* `training` `storage` `networking` [Scaling Point-based Differentiable Rendering for Large-scale Reconstruction](http://arxiv.org/abs/2512.20017v1)
  > **TL;DR**: Addresses distributed training inefficiencies in Point-based Differentiable Rendering (PBDR) for large-scale 3D reconstruction. Introduces Gaian, a system unifying PBDR APIs and optimizing data locality to reduce communication. Achieves up to 91% communication reduction and 1.50x-3.71x throughput improvement.
* `MoE` `training` `networking` [UCCL-EP: Portable Expert-Parallel Communication](http://arxiv.org/abs/2512.19849v1)
  > **TL;DR**: Addresses poor portability of expert-parallel (EP) communication systems across heterogeneous GPU/NIC platforms. Proposes UCCL-EP with GPU-CPU control channel and RDMA emulation for ordering semantics. Achieves up to 2.1× dispatch/combine throughput and 45% training throughput improvement.

### 2025-12-23
* `serving` `RL` `edge` [Q-IRIS: The Evolution of the IRIS Task-Based Runtime to Enable Classical-Quantum Workflows](http://arxiv.org/abs/2512.13931v1)
  > **TL;DR**: Proposes Q-IRIS, a hybrid runtime for classical-quantum workflows integrating IRIS task-based runtime with XACC quantum framework. Enables asynchronous scheduling of QIR programs across heterogeneous backends (including quantum simulators). Demonstrates circuit cutting to reduce per-task simulation load, improving throughput and reducing queueing behavior.
* `hardware` `training` `kernel` [Design in Tiles: Automating GEMM Deployment on Tile-Based Many-PE Accelerators](http://arxiv.org/abs/2512.13638v1)
  > **TL;DR**: Automates GEMM deployment on tile-based many-PE accelerators via DiT framework, connecting deployment toolchain with configurable executable model. Achieves 1.2-2.0x speedup over NVIDIA GH200 with higher PE utilization for diverse matrix shapes.
* `serving` `MoE` `offloading` [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](http://arxiv.org/abs/2512.13525v2)
  > **TL;DR**: Addresses inefficient resource usage in Mixture-of-Experts inference. Proposes Janus, a system that disaggregates attention and experts onto separate GPU sub-clusters with adaptive communication and lightweight scheduling. Achieves up to 3.9× higher per-GPU throughput while meeting latency targets.
* `training` `MoE` `hardware` [SIGMA: An AI-Empowered Training Stack on Early-Life Hardware](http://arxiv.org/abs/2512.13488v1)
  > **TL;DR**: SIGMA addresses reliability, stability, and efficiency challenges in large-scale training on early-life AI accelerators. It introduces the LUCIA Training Platform and Framework, optimizing clusters for high utilization. Achieved 94.45% accelerator utilization and 21.08% MFU when training a 200B MoE model on 2,048 accelerators.
* `training` `offline` `kernel` [Temporal parallelisation of continuous-time maximum-a-posteriori trajectory estimation](http://arxiv.org/abs/2512.13319v1)
  > **TL;DR**: Proposes a parallel-in-time method for continuous-time MAP trajectory estimation to accelerate computations on parallel architectures like GPUs. Uses a reformulation into an optimal control problem and associative scan algorithms for parallelization. Achieves significant speedup in GPU experiments while maintaining sequential algorithm accuracy.
* `kernel` `training` `quantization` [FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection](http://arxiv.org/abs/2512.12949v1)
  > **TL;DR**: Proposes FlashFuser, a compiler framework for kernel fusion using GPU inter-core Distributed Shared Memory. Enables fusion of large memory-intensive operators like FFN via DSM-based communication, optimized scheduling, and tiling. Achieves 3.3x kernel speedup against libraries and 58% memory access reduction.
* `serving` [PROSERVE: Unified Multi-Priority Request Scheduling for LLM Serving](http://arxiv.org/abs/2512.12928v1)
  > **TL;DR**: Addresses scheduling LLM serving requests with mixed priorities for maximizing service gain. Proposes a two-tier scheduler using SlideBatching for adaptive batching/ordering and GoRouting for distributed dispatching. Increases system gain by up to 35% and SLO attainment by up to 52%.
* `serving` `quantization` `offloading` [Fine-Grained Energy Prediction For Parallellized LLM Inference With PIE-P](http://arxiv.org/abs/2512.12801v1)
  > **TL;DR**: Addresses the challenge of accurately predicting energy consumption for multi-GPU parallelized LLM inference. Proposes PIE-P, a framework with scalable prediction via precise sampling and modeling of communication overheads. Achieves significantly higher accuracy than baselines across tensor, pipeline, and data parallelism.
* `RL` `training` [HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments](http://arxiv.org/abs/2512.12476v1)
  > **TL;DR**: Addresses inefficiency in RL training for large language models across heterogeneous GPU environments. Introduces HetRL, a distributed system with a multi-level scheduling algorithm that optimizes resource allocation. Achieves up to 9.17x higher throughput versus state-of-the-art systems.
* `training` `RL` [Reputation-Based Leader Election under Partial Synchrony: Towards a Protocol-Independent Abstraction with Enhanced Guarantees](http://arxiv.org/abs/2512.12409v2)
  > **TL;DR**: Proposes Sliding Window Leader Election (SWLE), a reputation-based abstraction for leader-based BFT protocols under partial synchrony. Enhances leader election via Byzantine-cost amplification and consensus-behavior reputation. Achieves 4.2x throughput, 75% lower latency vs state-of-the-art in geo-distributed deployment.
* `inference` `offline` `kernel` [Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates](http://arxiv.org/abs/2512.12295v2)
  > **TL;DR**: Proposes LiveUpdate, a system for near-zero-overhead freshness in recommendation systems by colocating LoRA trainers within underutilized inference nodes, using dynamic rank adaptation and NUMA-aware scheduling. Achieves <2% memory overhead and P99 latency impact <20ms, reducing update costs by 2x versus baselines.
* `edge` `offloading` `hardware` [Fast Online Digital Twinning on FPGA for Mission Critical Applications](http://arxiv.org/abs/2512.17942v1)
  > **TL;DR**: Proposes an FPGA-accelerated digital twinning framework for low-latency mission-critical applications. Offloads GRU and dense layers to reconfigurable hardware for parallel execution. Achieves 5x faster operation than human reaction time, enabling real-time edge deployment.
* `edge` `hardware` `quantization` [Accelerated Digital Twin Learning for Edge AI: A Comparison of FPGA and Mobile GPU](http://arxiv.org/abs/2512.17941v1)
  > **TL;DR**: Proposes a digital twin learning framework accelerated on FPGAs for edge AI healthcare applications. Leverages reconfigurable hardware optimizations for compute/memory efficiency. Achieves 8.8x performance-per-watt gain and 28.5x DRAM reduction over cloud GPU baselines.
* `training` `kernel` `sparse` [BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models](http://arxiv.org/abs/2512.12131v1)
  > **TL;DR**: Proposes BOOST, a training framework for low-rank bottleneck LLMs. Combines bottleneck-aware tensor parallelism, online-RMSNorm, and low-rank activation checkpointing to optimize GPU utilization and reduce communication. Achieves 1.46-1.91× speedup over full-rank baselines and up to 2.27× over naive low-rank parallelism.
* `kernel` `sparse` `hardware` [Accelerating Sparse Matrix-Matrix Multiplication on GPUs with Processing Near HBMs](http://arxiv.org/abs/2512.12036v1)
  > **TL;DR**: Optimizes SpGEMM on GPUs for irregular memory access via hardware-software co-design with HBM processing. Presents Hash-based Multi-phase approach and AIA technique, achieving up to 4.18x speedup in GNN training over cuSPARSE on large datasets.
* `video` `storage` `networking` [ECCO: Leveraging Cross-Camera Correlations for Efficient Live Video Continuous Learning](http://arxiv.org/abs/2512.11727v1)
  > **TL;DR**: Proposes ECCO, a video analytics system that reduces retraining costs for continuous learning by grouping cameras with correlated data drift and dynamically allocating GPU resources. Achieves 6.7%-18.1% higher retraining accuracy at same resource cost or supports 3.3× more cameras at same accuracy.
* `edge` `kernel` `serving` [Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems](http://arxiv.org/abs/2512.11532v1)
  > **TL;DR**: Introduces Parallax, a framework that optimizes runtime parallelization for operator fallbacks on edge devices. Employs DAG partitioning, branch-aware memory management, and adaptive scheduling to accelerate mobile DNN inference. Achieves up to 46% latency reduction and 30% energy savings on mobile devices.
* `training` `RL` `offloading` [RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training](http://arxiv.org/abs/2512.11306v2)
  > **TL;DR**: Proposes RollMux, a cluster scheduling framework for disaggregated RL post-training that reclaims dependency bubbles via co-execution groups and two-tier scheduling. Achieves 1.84x cost efficiency improvement over standard disaggregation with 100% SLO attainment.
* `kernel` `training` `hardware` [Theoretical Foundations of GPU-Native Compilation for Rapid Code Iteration](http://arxiv.org/abs/2512.11200v1)
  > **TL;DR**: Addresses latency bottlenecks in AI code generation from CPU-GPU data transfers during compilation. Proposes GPU-native compilation methods: parallel traditional, neural sequence translation with verification, and hybrid. Achieves 10-100x speedups via transfer elimination and massive parallelism.
* `serving` `offloading` `sparse` [ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp](http://arxiv.org/abs/2512.10576v1)
  > **TL;DR**: Addresses Decode-stage bottleneck in DeepSeek-V3.2-Exp due to Latent-Cache memory constraints. Proposes ESS, an offload-centric system that selectively offloads Latent-Cache to CPU to free GPU memory. Achieves 69.4% to 123% throughput improvement at 32K-128K context lengths.
* `training` `RL` [Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusters](http://arxiv.org/abs/2512.10271v1)
  > **TL;DR**: Proposes RLTune, a reinforcement learning-based scheduler for heterogeneous GPU clusters, integrating RL prioritization with MILP-based mapping to optimize job completion time and utilization. Achieves up to 70% lower job completion time and 20% higher GPU utilization without per-job profiling.
* `offline` `edge` `kernel` [Ariel-ML: Computing Parallelization with Embedded Rust for Neural Networks on Heterogeneous Multi-core Microcontrollers](http://arxiv.org/abs/2512.09800v1)
  > **TL;DR**: Aims to automate parallelization of TinyML inference on multi-core microcontrollers. Introduces Ariel-ML, an embedded Rust toolkit for parallel execution on heterogeneous cores. Achieves lower inference latency than prior work while maintaining comparable memory footprint to C/C++ toolkits.
* `training` `scheduling` [Straggler Tolerant and Resilient DL Training on Homogeneous GPUs](http://arxiv.org/abs/2512.09685v1)
  > **TL;DR**: Addresses stragglers in homogeneous GPU training due to resource imbalances. Proposes STAR, with adaptive synchronization modes and resource-aware scheduling. Achieves 48-84% lower Time-To-Accuracy than prior systems while maintaining accuracy.
* `serving` `offloading` [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](http://arxiv.org/abs/2512.09472v1)
  > **TL;DR**: Proposes WarmServe for multi-LLM serving with universal GPU workers enabling one-for-many prewarming. Uses evict-aware placement, proactive prewarming, and zero-overhead memory switching to reduce prewarming interference. Improves TTFT by up to 50.8× and serves 2.5× more requests.
* `RAG` `storage` `networking` [Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN](http://arxiv.org/abs/2512.09331v1)
  > **TL;DR**: Presents BatANN, a distributed disk-based vector search system for large datasets, optimizing throughput by sending full query state to remote machines. Achieves 2.5-6.49× higher throughput than scatter-gather baseline on 100M-1B vector datasets with mean latency below 6 ms over standard TCP.
* `offline` `kernel` `offloading` [SHARe-KAN: Holographic Vector Quantization for Memory-Bound Inference](http://arxiv.org/abs/2512.15742v1)
  > **TL;DR**: Addresses memory constraints in Kolmogorov-Arnold Networks (KANs) during inference. Proposes SHARe-KAN with Gain-Shape-Bias Vector Quantization and LUTHAM compiler for static memory planning. Achieves 88x runtime memory reduction (1.13GB to 12.91MB) while maintaining accuracy.
* `serving` `offloading` `quantization` [GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference](http://arxiv.org/abs/2512.09963v2)
  > **TL;DR**: Proposes GOODSPEED, an adaptive speculative decoding framework for distributed edge inference. Uses a gradient scheduler to dynamically allocate token verification across heterogeneous draft servers, optimizing fair goodput. Achieves near-optimal goodput with provable fairness and up to 10x lower latency.
* `MoE` `serving` `offloading` [Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens](http://arxiv.org/abs/2512.09277v1)
  > **TL;DR**: Proposes METRO, a token-routing algorithm for MoE serving that balances activated experts instead of tokens to mitigate memory-bound bottlenecks. Achieves 11-22% decode latency reduction and up to 4.11x higher decode throughput while using novel allGather optimization.
* `serving` `training` `quantization` [Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging](http://arxiv.org/abs/2512.08365v1)
  > **TL;DR**: Addresses software energy waste in ML systems via differential energy debugging. Magneton compares energy consumption at operator level to pinpoint inefficiencies in code/configurations. Applied to 9 ML systems, found 16 known and 8 new inefficiencies, with 7 confirmed.
* `training` `hardware` `kernel` [Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency](http://arxiv.org/abs/2512.08242v1)
  > **TL;DR**: Proposes Chopper, a multi-level GPU profiling tool, to analyze inefficiencies in distributed LLM training. Identifies frequency overhead (DVFS) as the primary performance gap contributor during Llama 3 training on AMD MI300X GPUs. Achieves detailed characterization leading to 10% potential performance improvement insights.
* `serving` `training` `edge` [Dora: QoE-Aware Hybrid Parallelism for Distributed Edge AI](http://arxiv.org/abs/2512.10990v1)
  > **TL;DR**: Proposes Dora for QoE-aware hybrid parallelism in distributed edge AI training and inference. It uses heterogeneity-aware partitioning, contention-aware scheduling, and runtime adaptation to balance latency and resource efficiency. Achieves 1.1-6.3× faster execution or 21-82% energy reduction while maintaining QoE.
* `training` `networking` `offloading` [Modeling the Potential of Message-Free Communication via CXL.mem](http://arxiv.org/abs/2512.08005v1)
  > **TL;DR**: Models performance benefits of using CXL.mem technology for MPI data exchange. Develops toolchain with memory trace sampling (Mitos) to identify high-potential MPI calls for replacing traditional messages with direct memory pooling. Example: Applied to HPCG benchmark with predicted communication latency reduction.
* `offline` `scheduling` [Quantifying the Carbon Reduction of DAG Workloads: A Job Shop Scheduling Perspective](http://arxiv.org/abs/2512.07799v1)
  > **TL;DR**: Quantifies carbon reduction potential for DAG-based batch workloads (e.g., offline inference) via flexible job-shop scheduling. Formulates dependency-aware scheduling to exploit carbon intensity variations. Achieves up to 25% lower emissions without increasing makespan under homogeneous servers.
* `training` `networking` [Bandwidth-Aware Network Topology Optimization for Decentralized Learning](http://arxiv.org/abs/2512.07536v1)
  > **TL;DR**: Proposes a bandwidth-aware network topology optimization framework for decentralized learning to maximize consensus speed under bandwidth constraints. Uses Mixed-Integer SDP reformulation and ADMM with conjugate gradient to solve large-scale topologies. Achieves up to 1.21× training speedup on heterogeneous bandwidth.
* `video` `training` `networking` [Communication-Efficient Serving for Video Diffusion Models with Latent Parallelism](http://arxiv.org/abs/2512.07350v1)
  > **TL;DR**: Addresses communication bottlenecks in parallel serving of video diffusion models. Proposes Latent Parallelism (LP), a dimension-rotating partition strategy with patch-aligned overlapping. Reduces communication overhead by up to 97% while maintaining generation quality.
* `edge` `RAG` `video` [Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding](http://arxiv.org/abs/2512.07344v1)
  > **TL;DR**: Proposes Venus, an edge-cloud system for efficient online video understanding with VLMs. Uses edge-based memory construction, keyframe retrieval, and progressive sampling to reduce overhead. Achieves 15x-131x speedup in latency while maintaining accuracy.
* `hardware` `kernel` `serving` [DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management](http://arxiv.org/abs/2512.07312v1)
  > **TL;DR**: Proposes DCO, a dynamic cache orchestration for LLM accelerators using predictive management. Utilizes dataflow-driven cache policies (bypass and thrashing mitigation) and dead-block prediction. Achieves up to 1.80x speedup compared to conventional cache architectures.
* `video` `serving` `kernel` [Optimizing video analytics inference pipelines: a case study](http://arxiv.org/abs/2512.07009v1)
  > **TL;DR**: Addresses high computational costs in livestock monitoring video analytics. Introduces multi-level parallelization, GPU acceleration of CPU code, vectorized clustering, and memory-efficient post-processing for a 2x pipeline speedup without accuracy loss, enabling high-throughput, low-latency inference.
* `serving` `edge` `kernel` [ELANA: A Simple Energy and Latency Analyzer for LLMs](http://arxiv.org/abs/2512.09946v1)
  > **TL;DR**: Presents ELANA, a lightweight profiler for evaluating latency metrics (TTFT, TPOT, TTLT) and energy consumption of LLMs on multi-GPU and edge platforms. Integrates with Hugging Face models and supports low-precision models. Demonstrates usage for efficient deployment analysis without specific quantitative results mentioned in abstract.
* `storage` `serving` `offline` [A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases](http://arxiv.org/abs/2512.06852v1)
  > **TL;DR**: Addresses challenges of storing and accessing large payloads exceeding NoSQL database item size limits. Proposes a chunked-object pattern that persists large entities as ordered chunks within the database for consistency. Reduces p99 cross-region time-to-consistency for 1 MB payloads by eliminating replication lag.
* `training` `MoE` `edge` [Stable-MoE: Lyapunov-based Token Routing for Distributed Mixture-of-Experts Training over Edge Networks](http://arxiv.org/abs/2512.06784v1)
  > **TL;DR**: Proposes Stable-MoE, a Lyapunov-based token routing framework for distributed MoE training over heterogeneous edge networks. Optimizes token routing and resource allocation via online Lyapunov optimization to maximize throughput and gating consistency. Achieves 40% higher throughput and 5% accuracy gains on SVHN/CIFAR-100.
* `training` `RL` `kernel` [A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation](http://arxiv.org/abs/2512.06547v1)
  > **TL;DR**: Addresses computational bottleneck in asynchronous RL training for LLMs by approximating the proximal policy without an extra forward pass. Proposes A-3PO, a staleness-aware proximal policy approximation via interpolation. Reduces training time by 18% while maintaining performance.
* `edge` `quantization` `kernel` [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices](http://arxiv.org/abs/2512.06443v1)
  > **TL;DR**: Proposes Vec-LUT, a vector lookup paradigm for ultra-low-bit LLM inference on edge devices. Introduces unified LUT across tokens and cache-aware techniques to reduce memory bandwidth underutilization during parallel inference. Achieves up to 4.2× speedup on CPUs across 5 edge devices.
* `offline` `serving` [Metronome: Differentiated Delay Scheduling for Serverless Functions](http://arxiv.org/abs/2512.05703v1)
  > **TL;DR**: Addresses inefficient scheduling for serverless functions with heterogeneous execution and locality patterns. Designs Metronome, a predictive delay scheduling framework using online Random Forest Regression to forecast execution times and optimize node selection. Reduces mean execution time by 64.88%-95.83% while maintaining SLA.
* `offloading` `quantization` [Compiler-supported reduced precision and AoS-SoA transformations for heterogeneous hardware](http://arxiv.org/abs/2512.05516v1)
  > **TL;DR**: Explores compiler-supported reduced precision and AoS-SoA layouts for particle simulations on GPUs. Proposes compiler annotations to orchestrate CPU-GPU data offloading. Achieves up to 2.6x speedup on Nvidia GPUs through optimized data transformations.
* `serving` `RAG` `agentic` [Model Gateway: Model Management Platform for Model-Driven Drug Discovery](http://arxiv.org/abs/2512.05462v1)
  > **TL;DR**: Proposes Model Gateway, a management platform for ML models in drug discovery, supporting LLM Agents for model management, execution, and results retrieval. Achieves 0% failure rate with over 10k simultaneous clients scaling.
* `offloading` `hardware` `training` [Offloading to CXL-based Computational Memory](http://arxiv.org/abs/2512.04449v1)
  > **TL;DR**: Addresses data movement costs in disaggregated memory systems using CXL. Proposes 'Asynchronous Back-Streaming' protocol and KAI system for optimized offloading. Achieves 50.4% runtime reduction and 22.11x idle time improvement.
* `training` `serving` `offline` [Reducing Fragmentation and Starvation in GPU Clusters through Dynamic Multi-Objective Scheduling](http://arxiv.org/abs/2512.10980v1)
  > **TL;DR**: Aims to improve GPU utilization and reduce fragmentation/starvation in multi-tenant clusters running heterogeneous AI workloads. Proposes three dynamic schedulers (HPS, PBS, SBS) that optimize for utilization, fairness, and throughput. Achieves up to 78.2% GPU utilization (vs 45-67% in baselines) and 25.8 jobs/hour throughput.
* `training` `kernel` `offloading` [VLCs: Managing Parallelism with Virtualized Libraries](http://arxiv.org/abs/2512.04320v1)
  > **TL;DR**: Addresses resource contention in parallel library composition for high-performance computing. Introduces Virtual Library Contexts (VLCs) to encapsulate libraries and manage resources without code modification. Achieves speedup of up to 2.85x in benchmarks with OpenMP, OpenBLAS, and LibTorch.
* `kernel` `training` `inference` [tritonBLAS: Triton-based Analytical Approach for GEMM Kernel Parameter Selection](http://arxiv.org/abs/2512.04226v1)
  > **TL;DR**: Proposes tritonBLAS, an analytical model for generating optimized GEMM kernels without autotuning. Uses architectural parameters to predict near-optimal configurations via deterministic modeling. Achieves over 95% of autotuned performance while eliminating tuning overhead.
* `MoE` `edge` `offloading` [OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference](http://arxiv.org/abs/2512.03927v1)
  > **TL;DR**: Addresses MoE inference on memory-constrained edge devices by eliminating expert caches. Proposes OD-MoE with parallelized expert loading/computation and emulative activation prediction. Achieves 99.94% prediction accuracy and 75% decoding speed of GPU-cached deployment with 1/3 GPU memory usage.
* `training` `networking` `storage` [FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management](http://arxiv.org/abs/2512.03644v1)
  > **TL;DR**: Addresses high recovery overhead in large-scale LLM training. Introduces FFTrainer, using surplus network capacity for fast failover and nearly-free state management. Reduces recovery time by up to 98% and GPU utilization loss by up to 68%.
* `kernel` `training` `inference` [Agentic Operator Generation for ML ASICs](http://arxiv.org/abs/2512.10977v1)
  > **TL;DR**: Proposes TritorX, an agentic AI system for generating correct Triton PyTorch ATen kernels for ML accelerators. Uses LLMs with linting, JIT compilation, and PyTorch OpInfo testing for broad coverage. Generates 481 verified operators passing 20,000+ tests.
* `serving` `offloading` `networking` [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](http://arxiv.org/abs/2512.03416v1)
  > **TL;DR**: Proposes TokenScale, an autoscaler for disaggregated LLM serving using Token Velocity (leading indicator of backpressure) and Convertible Decoders (dynamic resource switching). Achieves 80-96% SLO attainment (vs. 50-88% baselines) and reduces costs by 4-14%.
* `serving` `quantization` `offloading` [TokenPowerBench: Benchmarking the Power Consumption of LLM Inference](http://arxiv.org/abs/2512.03024v1)
  > **TL;DR**: Introduces TokenPowerBench, a benchmark for measuring LLM inference power consumption. It features declarative configuration, multi-level power measurement without specialized meters, and phase-aligned metrics. The benchmark quantifies efficiency, e.g., joules per token when varying batch sizes and quantization on models up to Llama3-405B.
* `offloading` `storage` `training` [Offloading Artificial Intelligence Workloads across the Computing Continuum by means of Active Storage Systems](http://arxiv.org/abs/2512.02646v1)
  > **TL;DR**: Proposes active storage for offloading AI workloads across the computing continuum to reduce data transfer. Integrates computation into storage using dataClay to optimize memory, storage, and training times. Achieves significant memory efficiency and training speed improvements while maintaining accuracy.
* `serving` `storage` `offloading` [Fantasy: Efficient Large-scale Vector Search on GPU Clusters with GPUDirect Async](http://arxiv.org/abs/2512.02278v1)
  > **TL;DR**: Addresses how to efficiently perform large-scale vector similarity search on GPU clusters. Introduces Fantasy, which pipelines search and data transfer with GPUDirect Async for overlapping computation and network communication. Achieves significantly improved throughput for large graphs and large query batches.
* `serving` `offloading` [Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity](http://arxiv.org/abs/2512.01357v1)
  > **TL;DR**: Addresses model loading latency in serverless LLMs. Proposes Tangram using GPU memory reuse via unified pool, on-demand KV cache, and affinity-aware scheduling. Achieves 6.2× faster loading and 23–55% lower TTFT.
* `edge` `serving` `offloading` [Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI](http://arxiv.org/abs/2512.01039v1)
  > **TL;DR**: Proposes dynamic partitioning and placement of foundation models for real-time edge inference. Introduces a runtime optimization framework for layer-wise assignments under latency and resource constraints. Achieves adaptive orchestration responsive to infrastructure volatility in 6G edge networks.
* `training` `RL` `MoE` [Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning](http://arxiv.org/abs/2512.00902v1)
  > **TL;DR**: Proposes SmartFed for efficient federated fine-tuning of LLMs via Mixture of Rank-Wise Experts (MoRE) and Elastic Expert Quota Allocation (EEQA). Leverages knowledge reuse in LoRA modules to reduce computation. Achieves higher accuracy with 2.3× faster training than baselines.
* `serving` `kernel` [SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving](http://arxiv.org/abs/2512.00719v1)
  > **TL;DR**: Addresses sampling bottleneck in distributed LLM serving by disaggregating to CPU with sequence-parallel sampling and speculative hot-vocab sampling. Achieves up to 96% throughput increase and 20-65% latency reduction.
* `edge` `serving` `offloading` [IslandRun: Privacy-Aware Multi-Objective Orchestration for Distributed AI Inference](http://arxiv.org/abs/2512.00595v1)
  > **TL;DR**: Proposes IslandRun, a multi-objective orchestration system for distributed AI inference across personal devices, edge servers, and cloud. Uses agent-based routing, tiered island groups, and reversible anonymization to optimize performance, privacy, and cost. Achieves up to 40% lower cost while maintaining privacy compared to federated baselines.
* `training` `offline` `kernel` [Heimdall++: Optimizing GPU Utilization and Pipeline Parallelism for Efficient Single-Pulse Detection](http://arxiv.org/abs/2512.00398v1)
  > **TL;DR**: Proposes Heimdall++, an optimized GPU parallelization framework for radio astronomy single-pulse detection pipelines. Introduces fine-grained GPU parallelization, enhanced memory management, and multi-threading to decouple CPU/GPU stages, reducing GPU stalls. Achieves up to 2.66x speedup in single-file processing.
* `diffusion` `serving` `offloading` [InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models](http://arxiv.org/abs/2512.05134v1)
  > **TL;DR**: Proposes Invariance Caching (InvarDiff), a technique to accelerate diffusion model inference by exploiting temporal and layer-wise feature invariance. Implements per-module binary cache plans with re-sampling correction. Achieves 2-3× end-to-end speedups with minimal fidelity impact on DiT and FLUX models.
* `training` `edge` `offloading` [Communication-Computation Pipeline Parallel Split Learning over Wireless Edge Networks](http://arxiv.org/abs/2511.23167v1)
  > **TL;DR**: Proposes C$^2$P$^2$SL, a pipeline parallel split learning framework for wireless edge networks. Overlaps communication and computation by splitting batches and optimizing task split/resource allocation. Achieves 38% reduction in training time while maintaining accuracy.
* `serving` `quantization` `offloading` [Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems](http://arxiv.org/abs/2511.22880v1)
  > **TL;DR**: Addresses performance skew in serving heterogeneous LoRA adapters due to rank variability. Proposes LoRAServe, a dynamic adapter placement and routing framework with remote access via GPU Direct RDMA. Achieves up to 2$	imes$ higher throughput and 9$	imes$ lower TTFT with 50% fewer GPUs under SLOs.
* `edge` `storage` `serving` [DisCEdge: Distributed Context Management for Large Language Models at the Edge](http://arxiv.org/abs/2511.22599v1)
  > **TL;DR**: Proposes DisCEdge for distributed context management in edge-deployed LLMs. Stores user context as token sequences for efficient replication and lower bandwidth. Reduces client request sizes by 90% and improves response times by up to 14.46% compared to alternatives.
* `MoE` `serving` `kernel` [OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency](http://arxiv.org/abs/2511.22481v1)
  > **TL;DR**: Proposes OmniInfer, a system-level framework optimizing LLM serving via expert placement, sparse attention, and request scheduling. Integrates load-aware MoE scheduling, attention acceleration, and disaggregation-aware coordination. Achieves 616 QPM and reduces TPOT by 36%, TTFT by 38%.
* `serving` `kernel` `inference` [PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel](http://arxiv.org/abs/2511.22333v2)
  > **TL;DR**: Accelerates LLM decoding by exploiting shared prefixes across requests. Proposes PAT, a prefix-aware attention kernel with pack-forward-merge paradigm and multi-tile execution. Achieves 53.5% average attention latency reduction compared to SOTA kernels.
* `serving` `offline` `offloading` [OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving](http://arxiv.org/abs/2511.21862v1)
  > **TL;DR**: Addresses load imbalance in co-located online-offline LLM serving. Proposes latency-disaggregated architecture with separate pools, bottleneck-based scheduler, and preemption mechanism. Achieves 3x higher offline throughput while maintaining online SLOs.
* `offloading` `hardware` `networking` [A Sustainable and Reward Incentivized High-Performance Cluster Computing for Artificial Intelligence: A Novel Bayesian-Time-Decay Trust Mechanism in Blockchain](http://arxiv.org/abs/2511.21844v1)
  > **TL;DR**: Proposes a blockchain-based incentive mechanism for distributed AI cluster computing. Integrates dynamic trust ratings and a statistical draw system to efficiently utilize diverse hardware. Achieves up to 30% improvement in resource utilization while enabling broader participation.
* `serving` `edge` `offloading` [DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving](http://arxiv.org/abs/2511.21669v2)
  > **TL;DR**: Proposes DSD, a distributed speculative decoding framework for edge-cloud LLM serving. Uses DSD-Sim simulator and an Adaptive Window Control policy to coordinate draft-target execution. Achieves up to 1.1x speedup and 9.7% higher throughput over baselines.
* `training` `MoE` `sparse` [MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training](http://arxiv.org/abs/2511.21431v1)
  > **TL;DR**: Addresses memory bottleneck in MoE training due to token imbalanc‎e. Proposes MemFine, using chunk‎ed decomposition and recomputation with a memory model for optimizer. Reduces activation memory by 48.03% and inc‎reases throughput by 4.42%.
* `serving` [Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM](http://arxiv.org/abs/2511.21413v1)
  > **TL;DR**: Proposes integrating Kubernetes, Slurm, and vLLM to dynamically scale LLM serving workloads on HPC infrastructure. Addresses challenges of synchronous user-facing AI applications on classical HPC systems. Benchmarks show ~500ms overhead scaling to 1000 concurrent requests.
* `training` [GPU Memory Prediction for Multimodal Model Training](http://arxiv.org/abs/2512.07853v1)
  > **TL;DR**: Introduces a framework for predicting peak GPU memory usage during multimodal model training to prevent OOM errors. It decomposes models into layers and applies factorization for memory estimation. Achieves ~8.7% MAPE accuracy.
* `serving` `offloading` `kernel` [LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling](http://arxiv.org/abs/2512.00083v1)
  > **TL;DR**: Proposes LLaMCAT for reducing KV cache stalls in LLM inference via LLC optimization. Combines MSHR-aware cache arbitration, load balancing, and thread throttling. Achieves up to 1.58x speedup over unoptimized baseline under limited cache scenarios.
* `networking` `hardware` [Handling of Memory Page Faults during Virtual-Address RDMA](http://arxiv.org/abs/2511.21018v1)
  > **TL;DR**: Proposes a hardware-software mechanism for handling memory page faults during RDMA communication. Integrates fault detection via ARM SMMU and resolution through retransmission, modifying Linux drivers and DMA engine. Reduces overhead and improves efficiency compared to pinning.
* `serving` `RAG` `offloading` [Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA](http://arxiv.org/abs/2512.17910v1)
  > **TL;DR**: Addresses inefficiency in multi-adapter LLM serving pipelines via Activated LoRA for cross-model KV-cache reuse. Proposes base-aligned block hashing and activation-aware masking in vLLM. Achieves up to 58x latency reduction and 100x TTFT improvement over standard LoRA baselines.
* `serving` `offloading` `quantization` [DOPO: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving](http://arxiv.org/abs/2511.20982v2)
  > **TL;DR**: Proposes DOPD, a dynamic LLM inference system that optimizes prefill-to-decoding instance ratio via real-time load monitoring and request scheduling. Achieves 1.5x higher goodput, 67.5% lower P90 TTFT, and 22.8% lower P90 TPOT vs vLLM and DistServe.
* `serving` `RL` `agentic` [Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows](http://arxiv.org/abs/2511.20975v2)
  > **TL;DR**: Addresses cost and latency in serving multi-stage agentic workflows by dynamically routing agents to LLMs. Proposes a two-step approach: offline accuracy-preserving configuration identification and runtime scheduler using system load. Increases throughput by 50-217% and reduces latency by 32.5-78.9%.
* `offloading` `serving` `hardware` [Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management](http://arxiv.org/abs/2511.20172v2)
  > **TL;DR**: Proposes Beluga, a CXL-based memory architecture for scalable KVCache management in LLM inference. Enables GPU direct access to a shared memory pool via CXL, reducing latency and complexity. Achieves 89.6% lower TTFT and 7.35x throughput gains over RDMA.
* `kernel` `training` [QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation](http://arxiv.org/abs/2511.20100v1)
  > **TL;DR**: Addresses efficient generation of optimized GPU kernels using LLMs. Proposes a hierarchical framework (Macro Thinking Micro Coding) decoupling optimization strategies (via RL-guided LLMs) from implementation details (via general LLMs). Achieves up to 7.3x speedup over LLMs and 2.2x over PyTorch kernels on KernelBench.
* `training` `RL` `networking` [ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models](http://arxiv.org/abs/2511.19959v1)
  > **TL;DR**: Proposes ParaBlock, a communication-computation parallel federated learning method for efficient LLM fine-tuning. Uses dual threads to overlap local block updates and communication. Achieves 50% communication reduction while maintaining convergence rate and performance on instruction following tasks.
* `batch_denoising` [Batch Denoising for AIGC Service Provisioning in Wireless Edge Networks](http://arxiv.org/abs/2511.19847v1)
  > **TL;DR**: Proposes a batch denoising framework for AI-generated image services in edge networks. Develops STACKING algorithm to optimize batch processing and transmission, reducing per-step delay and improving parallelism. Achieves up to 40% lower latency and higher quality within service delay constraints.
* `storage` `edge` `RAG` [AME: An Efficient Heterogeneous Agentic Memory Engine for Smartphones](http://arxiv.org/abs/2511.19192v1)
  > **TL;DR**: Proposes AME, an on-device agentic memory engine for smartphones, optimizing vector database storage with hardware-aware matrix pipelines and workload-aware scheduling. Achieves up to 1.4x higher query throughput at matched recall and 7x faster index construction.
* `training` `serving` `hardware` [An Online Fragmentation-Aware GPU Scheduler for Multi-Tenant MIG-based Clouds](http://arxiv.org/abs/2511.18906v1)
  > **TL;DR**: Addresses GPU fragmentation in multi-tenant MIG clouds for AI workloads. Proposes an online fragmentation-aware scheduler using a fragmentation metric and greedy algorithm. Achieves 10% higher workload acceptance rate under heavy load.
* `kernel` `quantization` `sparse` [Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration](http://arxiv.org/abs/2511.18674v1)
  > **TL;DR**: Proposes Low-Rank GEMM for matrix multiplication via low-rank approximations and FP8. Leverages hardware-aware decomposition selection and FP8 quantization to reduce computation complexity and memory usage. Achieves 7.8× speedup over PyTorch FP32 and 75% memory savings for large matrices (N=20480).
* `RL` `training` [ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning](http://arxiv.org/abs/2511.18291v1)
  > **TL;DR**: Proposes ADF-LoRA for decentralized federated fine-tuning, alternating updates of one low-rank matrix per round while mixing both to stabilize decentralized propagation. Achieves faster convergence and higher average accuracy on GLUE tasks vs. existing LoRA variants in decentralized FL.
* `edge` `offloading` `multi-modal` [AVERY: Adaptive VLM Split Computing through Embodied Self-Awareness for Efficient Disaster Response Systems](http://arxiv.org/abs/2511.18151v1)
  > **TL;DR**: Proposes AVERY, an adaptive split computing framework for Vision-Language Models on UAVs. Features dual-stream separation and dynamic compression selection based on network conditions. Achieves 93.98% lower energy consumption vs full-edge execution and 11.2% higher accuracy than raw compression.
* `scheduling` `training` `storage` [Simulating Dynamic Cloud Marketspaces: Modeling Spot Instance Behavior and Scheduling with CloudSim Plus](http://arxiv.org/abs/2511.18137v1)
  > **TL;DR**: Extends CloudSim Plus to model spot instance lifecycle for cost-effective workload scheduling in cloud environments. Proposes HLEM-VMP algorithm to reduce interruptions under volatility. Achieves fewer interruptions and shorter max interruption duration in simulations using Google Cluster Trace.
* `training` [Pier: Efficient Large Language Model pretraining with Relaxed Global Communication](http://arxiv.org/abs/2511.17849v2)
  > **TL;DR**: Proposes Pier, an optimizer reducing global communication bottlenecks in LLM pretraining via relaxed communication, momentum warmup/decay, and parallelization. Achieves 2.7x-3.7x GPT-2 XL speedup on 256 A100 GPUs with no loss degradation.
* `training` `hardware` `networking` [Towards a future space-based, highly scalable AI infrastructure system design](http://arxiv.org/abs/2511.19468v1)
  > **TL;DR**: Investigates space-based AI infrastructure for scalable ML compute using solar power. Proposes satellite fleets with TPUs, optical interlinks, and formation flight for low-latency communication. Radiation tests show Trillium TPUs withstand 5-year mission doses; projected launch costs drop below $200/kg by 2030s.
* `MoE` `training` `hardware` [Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design](http://arxiv.org/abs/2511.17127v2)
  > **TL;DR**: Presents large-scale MoE pretraining on AMD hardware (MI300X GPUs, Pollara networking), with cluster characterization, kernel/memory microbenchmarks, and transformer/MoE sizing rules. Optimizes for training throughput and inference latency. Achieves competitive model performance (ZAYA1-base) vs Qwen3-4B/Gemma3-12B.
* `edge` `sparse` `kernel` [SparOA: Sparse and Operator-aware Hybrid Scheduling for Edge DNN Inference](http://arxiv.org/abs/2511.19457v1)
  > **TL;DR**: Proposes SparOA, a CPU-GPU hybrid DNN inference framework for edge devices using sparsity and operator-aware scheduling. It includes a RL-based scheduler and asynchronous execution with batch optimization. Achieves 1.22-1.31x speedup and 7-16% lower energy consumption than baselines.
* `serving` `kernel` [Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems](http://arxiv.org/abs/2511.16964v1)
  > **TL;DR**: Investigates how to optimize PyTorch inference using LLM-based multi-agent systems. Proposes a framework comparing strategies with exploit-heavy approaches and error-fixing agents. Achieves 2.88x speedup on H100 GPU across tasks in KernelBench.
* `training` `MoE` `sparse` [MicroMoE: Fine-Grained Load Balancing for Mixture-of-Experts with Token Scheduling](http://arxiv.org/abs/2511.16947v1)
  > **TL;DR**: Addresses load imbalance in Mixture-of-Experts training. Proposes MicroEP, a fine-grained parallelization strategy with token scheduling, and implements it in MicroMoE system. Achieves 47.6% higher training throughput over state-of-the-art.
* `training` `RL` `sparse` [Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter](http://arxiv.org/abs/2511.16665v1)
  > **TL;DR**: Addresses long-tail execution inefficiency in RL-based LLM reasoning training. Proposes TLT with Adaptive Drafter (idle GPU-trained speculative model) and Adaptive Rollout Engine (CUDAGraph strategies). Achieves 1.7x end-to-end RL training speedup without accuracy loss.
* `training` `RL` `quantization` [Fast LLM Post-training via Decoupled and Best-of-N Speculation](http://arxiv.org/abs/2511.16193v2)
  > **TL;DR**: Accelerates LLM post-training rollout via dynamic decoupled and Best-of-N speculative decoding. Maximizes GPU efficiency for large batches and adapts drafting methods. Achieves 1.3–1.5× faster rollout than speculative decoding baselines.
* `storage` `networking` [Mitigating Shared Storage Congestion Using Control Theory](http://arxiv.org/abs/2511.16177v1)
  > **TL;DR**: Proposes a control-theoretic approach to regulate client I/O rates dynamically in HPC shared storage systems. Uses runtime load metrics to mitigate congestion, reducing total runtime by up to 20% and tail latency while ensuring stability.
* `kernel` `training` [Can Asymmetric Tile Buffering Be Beneficial?](http://arxiv.org/abs/2511.16041v1)
  > **TL;DR**: Examines tiling strategies for GEMM operations in AI workloads. Proposes asymmetric tile buffering (ATB) that decouples buffered tile dimensions of inputs and outputs. Achieves up to 4.54x speedup (24.6 TFLOPS vs 4.8 TFLOPS) on mixed-precision GEMM for AMD XDNA2 AI Engine.
* `serving` `hardware` `kernel` [A Tensor Compiler for Processing-In-Memory Architectures](http://arxiv.org/abs/2511.15503v1)
  > **TL;DR**: Proposes DCC, a data-centric compiler for PIM systems that co-optimizes data rearrangements and compute code for ML kernels on diverse PIM backends. Achieves up to 7.71x speedup in LLM inference over GPU on AttAcc PIM.
* `MoE` `training` `networking` [GPU-Initiated Networking for NCCL](http://arxiv.org/abs/2511.15076v2)
  > **TL;DR**: Proposes GPU-Initiated Networking (GIN) in NCCL to enable device-initiated RDMA communication for modern AI workloads like MoE. Integrates three operation modes and leverages DOCA GPUNetIO or proxy queues to reduce CPU overhead. Benchmarks show up to 8-16x lower latency in MoE communication.
* `kernel` `training` `serving` [PolyKAN: Efficient Fused GPU Operators for Polynomial Kolmogorov-Arnold Network Variants](http://arxiv.org/abs/2511.14852v1)
  > **TL;DR**: Proposes PolyKAN, an optimized CUDA kernel library for polynomial KAN variants to overcome low GPU utilization. Fuses forward/backward passes using lookup tables, 2D tiling, two-stage reduction, and coefficient reordering. Achieves up to 10× faster inference and 12× faster training over Triton+cuBLAS baseline.
* `training` `RL` `serving` [Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning](http://arxiv.org/abs/2511.14617v2)
  > **TL;DR**: Addresses synchronous RL rollout bottlenecks for LLMs via online context learning. Proposals include divided rollout load balancing, context-aware scheduling, and adaptive grouped speculative decoding. Seer improves rollout throughput by up to 97% and reduces long-tail latency by up to 93%.
* `serving` `offloading` `edge` [Hyperion: Hierarchical Scheduling for Parallel LLM Acceleration in Multi-tier Networks](http://arxiv.org/abs/2511.14450v2)
  > **TL;DR**: Proposes Hyperion, a hierarchical framework for efficient LLM inference in multi-tier edge networks by jointly optimizing model partitioning and request scheduling. Combines offline inter-tier partitioning (HypSplit-DP) with online intra-tier scheduling (HypSched-RT). Achieves 52.1% latency reduction vs. GPipe.
* `training` `offloading` [10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training](http://arxiv.org/abs/2511.14124v1)
  > **TL;DR**: Proposes 10Cache, a tensor caching and migration system for LLM training that optimizes memory across GPU/CPU/NVMe tiers via prefetching and buffer reuse. Achieves up to 2x training speedup and 86.6x higher GPU cache hit rate over existing offloading methods.
* `serving` `training` [FailSafe: High-performance Resilient Serving](http://arxiv.org/abs/2511.14116v1)
  > **TL;DR**: Addresses fragility in tensor-parallel LLM serving under GPU failures. Proposes FailSafe with cyclic KVCache placement, hybrid attention, and dynamic routing to balance computation and memory, plus proactive backup for fault recovery. Achieves 2× higher throughput and 100× lower recovery latency amidst multiple GPU failures.
* `MoE` `offloading` `quantization` [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](http://arxiv.org/abs/2511.14102v1)
  > **TL;DR**: Addresses I/O bottlenecks during resource-limited MoE inference due to synchronous expert offloading. Proposes speculative expert sequence prediction via a draft model and prefetching, dynamically tuned by an amortization roofline model. Achieves up to 2.34x speedup over state-of-the-art offloading frameworks.
* `training` `kernel` `networking` [ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels](http://arxiv.org/abs/2511.13940v1)
  > **TL;DR**: Proposes ParallelKittens, a minimal CUDA framework simplifying multi-GPU kernel design for communication-efficient AI workloads. Embodies principles via reusable primitives and unified template across architectures. Achieves up to 4.08× speedup for sequence-parallel workloads on Hopper/Blackwell GPUs.
* `serving` `offloading` `kernel` [Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI](http://arxiv.org/abs/2511.17593v1)
  > **TL;DR**: Empirically evaluates LLM serving systems vLLM and TGI. Compares throughput, latency, memory, and scalability using LLaMA-2 models. vLLM achieves 24× higher throughput than TGI under high concurrency via PagedAttention, while TGI offers lower tail latency for interactive use.
* `edge` `training` `offloading` [Distributed Hierarchical Machine Learning for Joint Resource Allocation and Slice Selection in In-Network Edge Systems](http://arxiv.org/abs/2511.13313v2)
  > **TL;DR**: Proposes a distributed hierarchical model (DeepSets-S) for joint resource allocation and slice selection in edge systems. Uses a DeepSets architecture with slack-aware normalization and task-specific decoders for efficient online inference. Reduces execution time by 86.1% while maintaining within 6.1% of optimal cost.
* `edge` `serving` [Pico-Cloud: Cloud Infrastructure for Tiny Edge Devices](http://arxiv.org/abs/2511.13253v1)
  > **TL;DR**: Proposes Pico-Cloud, a micro-edge cloud architecture for ultra-minimal hardware like Raspberry Pi Zero. Features container virtualization and lightweight orchestration for local low-latency workloads, including edge AI inference. Achieves cost-effective and decentralized operation for lightweight distributed applications.
* `training` `storage` `offloading` [Learning Process Energy Profiles from Node-Level Power Data](http://arxiv.org/abs/2511.13155v1)
  > **TL;DR**: Proposes a method to model per-process energy consumption using synchronized node-level power data and process-level resource metrics via regression. Enables fine-grained energy prediction, improving efficiency monitoring for AI workloads over coarse tools like RAPL.
* `sparse` `kernel` `inference` [MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity](http://arxiv.org/abs/2511.13061v1)
  > **TL;DR**: Proposes MACKO-SpMV, a GPU-optimized sparse matrix-vector multiplication format and kernel for efficient inference in pruned sparse LLMs. Achieves 1.5x memory reduction and 1.5x speedup over dense at 50% sparsity with unstructured pruning.
* `training` `kernel` `networking` [Iris: First-Class Multi-GPU Programming Experience in Triton](http://arxiv.org/abs/2511.12500v1)
  > **TL;DR**: Addresses complexity in multi-GPU programming by introducing Iris, a Python/Triton-based communication library with tile-based symmetric memory abstractions. Enables computation-communication overlap with minimal code. Achieves 1.79x speedup over PyTorch/RCCL for GEMM+All-Scatter.
* `edge` `networking` `multi-modal` [Semantic Multiplexing](http://arxiv.org/abs/2511.13779v1)
  > **TL;DR**: Proposes Semantic Multiplexing to offload parallel tasks by merging compressed task representations for edge devices. Shifts multiplexing from bits to tasks, enabling more concurrent tasks than physical channels. Reduces latency by 8× and energy consumption by 25× while maintaining accuracy within 4% drop for 8 tasks over 4×4 channel.
* `kernel` `quantization` [Guaranteed DGEMM Accuracy While Using Reduced Precision Tensor Cores Through Extensions of the Ozaki Scheme](http://arxiv.org/abs/2511.13778v1)
  > **TL;DR**: Proposes Automatic Dynamic Precision (ADP) for emulating FP64 accuracy using low-precision Tensor Cores. ESC estimator determines decomposition parameters, with exception handling and fallback to FP64. Achieves up to 13.2x speedup over native FP64 GEMM while preserving accuracy with <10% overhead.
* `serverless` `storage` `networking` [Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications](http://arxiv.org/abs/2511.12185v2)
  > **TL;DR**: Proposes Cylon, a high-performance distributed data frame solution using TCP hole punching for serverless environments. Enables direct communication between functions to speed up machine learning data pipelines. Achieves performance close to EC2/HPC clusters.
* `video` `training` `offloading` [PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling](http://arxiv.org/abs/2511.12056v1)
  > **TL;DR**: Addresses slow inference and high memory in diffusion transformer video generation. Proposes PipeDiT with pipelined sequence parallelism, decoupled VAE execution, and attention co-processing. Achieves up to 4.02x speedup over state-of-the-art frameworks on 8-GPU systems.
* `serving` `offline` `offloading` [Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding](http://arxiv.org/abs/2511.12031v1)
  > **TL;DR**: Proposes BMC, a KV cache allocation method balancing memory copy overhead and redundant compute for LLM inference. BMC allocates tensors with redundant rows periodically to avoid copies, using extra rows for speculative decoding. Achieves up to 3.2x throughput over HuggingFace baseline and 1.36x over vLLM server.
* `kernel` [Modular GPU Programming with Typed Perspectives](http://arxiv.org/abs/2511.11939v1)
  > **TL;DR**: Addresses modular GPU programming challenges for collective operations like Tensor Core instructions. Introduces Prism, a GPU language with typed perspectives tracking thread granularity control. Maintains performance parity without safety compromises for state-of-the-art kernels.
* `offloading` `edge` `kernel` [KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference](http://arxiv.org/abs/2511.11907v2)
  > **TL;DR**: Addresses on-device KV cache memory bottleneck for long-context inference. Proposes KVSwap, a disk-aware offloading framework with predictive preloading, compute-disk overlap, and I/O-optimized access patterns. Achieves higher throughput under memory constraints with no quality loss.
* `edge` `serving` `RAG` [Flash-Fusion: Enabling Expressive, Low-Latency Queries on IoT Sensor Streams with LLMs](http://arxiv.org/abs/2511.11885v1)
  > **TL;DR**: Proposes Flash-Fusion, an edge-cloud system for low-latency LLM queries on IoT sensor streams. Uses edge summarization for data reduction and cloud-based query planning with clustered data for prompt assembly. Achieves 95% latency reduction and 98% cost decrease compared to raw data feeding.
* `agentic` `edge` `networking` [UFO$^3$: Weaving the Digital Agent Galaxy](http://arxiv.org/abs/2511.11332v1)
  > **TL;DR**: Proposes UFO$^3$, a system for orchestrating LLM-powered agents across heterogeneous devices. Models user requests as mutable TaskConstellations with dynamic DAGs, asynchronous execution, and reliable Agent Interaction Protocol. Reduces end-to-end latency by 31% on a benchmark of 55 cross-device tasks.
* `training` `networking` [What happens when nanochat meets DiLoCo?](http://arxiv.org/abs/2511.13761v1)
  > **TL;DR**: Studies performance impact of DiLoCo's communication-efficient decentralized training on LLMs. Implements DiLoCo with inner-outer optimizer into nanochat baseline, reducing communication by orders of magnitude vs data-parallel training. Shows irreversible representation drift causing 5-10% accuracy drops on downstream tasks despite pretraining convergence.
* `offloading` `edge` `offline` [SemanticNN: Compressive and Error-Resilient Semantic Offloading for Extremely Weak Devices](http://arxiv.org/abs/2511.11038v1)
  > **TL;DR**: Proposes SemanticNN, an error-resilient semantic offloading system for weak IoT devices featuring BER-aware decoder and Soft Quantization encoder. Includes Feature-augmentation Learning and XAI-based Asymmetry Compensation. Reduces feature transmission volume by 56.82-344.83x while maintaining accuracy on STM32.
* `agentic` `RL` `offline` [HPCAgentTester: A Multi-Agent LLM Approach for Enhanced HPC Unit Test Generation](http://arxiv.org/abs/2511.10860v1)
  > **TL;DR**: Proposes a multi-agent LLM framework for automated HPC unit test generation. Uses specialized agents in iterative critique loops to generate tests targeting parallelism and communication patterns. Increases test compilation rates by 30% and correctness by 25% compared to standalone LLMs.
* `edge` `serving` `multi-modal` [EarthSight: A Distributed Framework for Low-Latency Satellite Intelligence](http://arxiv.org/abs/2511.10834v1)
  > **TL;DR**: Addresses high latency in satellite image analysis by introducing EarthSight, a distributed runtime for multi-task inference with shared backbones, query scheduling, and dynamic filtering. Achieves 1.9× faster compute time and reduces 90th percentile latency from 51 to 21 minutes.
* `inference` `offloading` `hardware` [FengHuang: Next-Generation Memory Orchestration for AI Inferencing](http://arxiv.org/abs/2511.10753v1)
  > **TL;DR**: Proposes FengHuang, a disaggregated infrastructure with multi-tier memory and active tensor paging to overcome GPU memory and scaling limits for LLM inference. Achieves 93% local memory reduction, 50% GPU savings, and 16x-70x faster inter-GPU communication while maintaining performance.
* `training` `RL` `storage` [STAGE: A Symbolic Tensor grAph GEnerator for distributed AI system co-design](http://arxiv.org/abs/2511.10480v2)
  > **TL;DR**: Introduces STAGE, a framework for synthesizing high-fidelity execution traces to model LLM workloads. Enables scalable exploration of parallelization strategies and system configurations for large-scale training. Generates traces spanning 32K GPUs with tensor-level accuracy in compute, memory, and communication.
* `kernel` `offloading` [On The Performance of Prefix-Sum Parallel Kalman Filters and Smoothers on GPUs](http://arxiv.org/abs/2511.10363v1)
  > **TL;DR**: Evaluates parallel scan algorithms for GPU-based Kalman filters and smoothers. Implements all-prefix-sum algorithms for temporal parallelization and proposes a novel two-filter smoother. Achieves reduced run times on CUDA/Metal GPUs through optimized kernel operations and offloading techniques.
* `edge` `serving` `RL` [Dynamic Edge Server Selection in Time-Varying Environments: A Reliability-Aware Predictive Approach](http://arxiv.org/abs/2511.10146v1)
  > **TL;DR**: Proposes MO-HAN, a lightweight edge server selection method for embedded applications. It fuses latency prediction with adaptive reliability and hysteresis-based handover to balance predicted latency and reliability. Reduces handovers by 50% and lowers mean/tail latencies compared to baselines.
* `serving` `offloading` `networking` [Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput](http://arxiv.org/abs/2511.11733v1)
  > **TL;DR**: Proposes Decentralized Speculative Decoding (DSD) to turn network latency into computation throughput for distributed LLM inference. Uses parallel verification of candidate tokens across nodes and adaptive thresholding by token importance. Achieves 2.59x speedup on GSM8K with preserved accuracy.
* `serving` `offline` `RL` [Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms](http://arxiv.org/abs/2511.11729v2)
  > **TL;DR**: Proposes Harli, a system for co-locating PEFT-based finetuning with LLM inference decode. Integrates memory reuse, latency prediction, and QoS-aware scheduling to maximize throughput. Achieves 46.2% average finetune throughput gain while maintaining QoS.
* `training` `hardware` `offloading` [Lit Silicon: A Case Where Thermal Imbalance Couples Concurrent Execution in Multiple GPUs](http://arxiv.org/abs/2511.09861v2)
  > **TL;DR**: Investigates thermal imbalance-induced performance variation in multi-GPU LLM training systems. Proposes analytical models and mitigation techniques via power management optimization. Achieves up to 6% performance gain and 4% power reduction on AMD MI300X systems.
* `training` `quantization` [MoFa: A Unified Performance Modeling Framework for LLM Pretraining](http://arxiv.org/abs/2511.09837v2)
  > **TL;DR**: Proposes MoFa, a unified performance modeling framework for LLM pretraining that integrates optimization features and fault tolerance. Uses an enhanced cost model and reliability-based fault model to optimize distributed training strategies. Achieves high prediction accuracy and reveals performance bottlenecks for pretraining configurations.
* `edge` `offloading` `serving` [ECCENTRIC: Edge-Cloud Collaboration Framework for Distributed Inference Using Knowledge Adaptation](http://arxiv.org/abs/2511.11719v1)
  > **TL;DR**: Proposes ECCENTRIC, an edge-cloud collaboration framework for distributed inference that uses knowledge adaptation to balance computation, communication, and performance. Achieves reduced costs while maintaining accuracy for classification and object detection tasks.
* `training` `MoE` `sparse` [TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training](http://arxiv.org/abs/2511.09741v1)
  > **TL;DR**: Addresses high communication overhead in long-context LLM training and intra-node bandwidth underutilization. Proposes topology-aware weight pipeline parallelism that groups devices hierarchically to reduce redundant transfers and overlaps communication with computation. Achieves up to 1.5x higher throughput and scales to 24 GPUs without saturation.
* `serving` `networking` `offline` [LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication](http://arxiv.org/abs/2511.09557v3)
  > **TL;DR**: Analyzes bottlenecks in multi-node distributed LLM inference and introduces NVRAR, a fast hierarchical all-reduce communication algorithm using NVSHMEM. Achieves up to 1.9×-3.6× lower latency for key operations and 1.72× lower batch latency for Llama 405B in decode workloads.
* `serving` `storage` `hardware` [Flex-MIG: Enabling Distributed Execution on MIG](http://arxiv.org/abs/2511.09143v2)
  > **TL;DR**: Addresses GPU underutilization in multi-tenant clusters with NVIDIA MIG. Proposes Flex-MIG, a software framework enabling distributed execution across MIG instances via host-shared-memory collectives. Achieves up to 17% makespan improvement by reducing fragmentation and eliminating drain-required reconfiguration.
* `edge` `training` `RL` [A Structure-Agnostic Co-Tuning Framework for LLMs and SLMs in Cloud-Edge Systems](http://arxiv.org/abs/2511.11678v1)
  > **TL;DR**: Proposes Co-PLMs, a co-tuning framework for collaborative training of cloud LLMs and edge SLMs via distilled proxy models, enabling structure-agnostic knowledge exchange. Achieves 5.38% Rouge-L and 4.88% EM gains over SOTA.
* `training` `edge` `hardware` [OSGym: Super-Scalable Distributed Data Engine for Generalizable Computer Agents](http://arxiv.org/abs/2511.11672v2)
  > **TL;DR**: Presents OSGym, a scalable and affordable distributed engine for training computer agents. Utilizes parallelized OS replicas for dynamic environments, optimizing resource use and cost. Generates up to 1420 trajectories/min at $0.2-0.3/day per instance.
* `offloading` `hardware` `kernel` [An MLIR pipeline for offloading Fortran to FPGAs via OpenMP](http://arxiv.org/abs/2511.08713v1)
  > **TL;DR**: Proposes an MLIR-based pipeline for offloading Fortran code to FPGAs via OpenMP directives. Combines OpenMP and HLS dialects to enable portable compilation and kernel optimization. Achieves reduced development effort while supporting manual kernel tuning for FPGA acceleration.
* `edge` `offloading` `storage` [Range Asymmetric Numeral Systems-Based Lightweight Intermediate Feature Compression for Split Computing of Deep Neural Networks](http://arxiv.org/abs/2511.11664v1)
  > **TL;DR**: Addresses communication overhead in split DNN inference by proposing a lightweight compression framework using rANS encoding, asymmetric integer quantization, and sparse tensor representation. Achieves near-baseline accuracy on vision and language models (e.g., ResNet, Llama2) with sub-millisecond encoding latency.
* `hardware` `kernel` `sparse` [LOw-cOst yet High-Performant Sparse Matrix-Matrix Multiplication on Arm SME Architectures](http://arxiv.org/abs/2511.08158v2)
  > **TL;DR**: Proposes LOOPS, a hybrid framework for sparse matrix multiplication exploiting Arm SME and SIMD resources. Combines row-wise and vector-wise layouts with adaptive parallelization, achieving up to 14.4x speedup and superior energy efficiency over baselines on Apple M4Pro CPU compared to GPUs.
* `training` `kernel` [UniFormer: Unified and Efficient Transformer for Reasoning Across General and Custom Computing](http://arxiv.org/abs/2511.08135v1)
  > **TL;DR**: Proposes UniFormer, a unified Transformer architecture for efficient deployment on both general-purpose GPUs and custom hardware. Unifies parallelism and compute-storage fusion to optimize performance. Achieves SOTA accuracy and latency on GPUs with improved FPGA adaptability.
* `edge` `serving` `offloading` [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](http://arxiv.org/abs/2511.07885v2)
  > **TL;DR**: Proposes intelligence per watt (IPW) to measure efficiency of local LLM inference on edge devices. Evaluates 20+ models across 8 accelerators using 1M queries, finding local accelerators achieve 1.4x better IPW than cloud counterparts, enabling feasible redistribution of inference demand.
* `serving` `training` `quantization` [Parallel Sampling via Autospeculation](http://arxiv.org/abs/2511.07869v1)
  > **TL;DR**: Proposes parallel sampling via autospeculation to reduce sampling time for autoregressive and diffusion models. Uses speculative rejection sampling with sequence-level speculation from the same oracle. Achieves expected sampling time of ̃O(n^{1/2}), improving over sequential O(n).
* `training` `hardware` `kernel` [HeteroSTA: A CPU-GPU Heterogeneous Static Timing Analysis Engine with Holistic Industrial Design Support](http://arxiv.org/abs/2511.11660v1)
  > **TL;DR**: Proposes HeteroSTA, a CPU-GPU heterogeneous timing analysis engine supporting various delay models and industry formats. Features end-to-end GPU acceleration for graph-based and path-based timing queries. Achieves remarkable runtime speed-up in industrial use cases.
* `thinking` `kernel` `RAG` [SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction](http://arxiv.org/abs/2511.07584v1)
  > **TL;DR**: Proposes SemanticForge for repository-level code generation by integrating semantic knowledge graphs and constraint solving. Combines graph query learning, beam search with SMT solving, and incremental graph maintenance to address logical/schematic hallucination. Achieves 73% query precision versus 51% baseline.
* `serving` `hardware` `offloading` [LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure](http://arxiv.org/abs/2511.07229v1)
  > **TL;DR**: Presents LLMServingSim2.0, a trace-driven simulator for heterogeneous hardware integration in LLM serving systems. Features operator-level latency profiling and flexible interfaces for serving techniques like request routing and cache management. Achieves 18.5x fewer LoC in TPU integration with 1.9% error in GPU serving simulation.
* `diffusion` `serving` `offloading` [Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System](http://arxiv.org/abs/2511.06724v1)
  > **TL;DR**: Focuses on high-throughput serving for text-to-image diffusion models. Proposes Argus, a system that dynamically selects approximation levels per prompt to balance quality and throughput. Achieves 40% higher throughput with 10% higher average quality.
* `training` `networking` `offloading` [DMA Collectives for Efficient ML Communication Offloads](http://arxiv.org/abs/2511.06605v1)
  > **TL;DR**: Analyses DMA offloading for ML communication collectives to improve training efficiency. Optimizes DMA collective implementations on AMD GPUs by reducing synchronization costs. Achieves up to 16% better performance and 32% lower power at large sizes, improving small-size performance by 20-30%.
* `networking` [Towards Optimal Constellation Design for Digital Over-the-Air Computation](http://arxiv.org/abs/2511.06372v1)
  > **TL;DR**: Addresses optimal constellation design for digital over-the-air computation to minimize mean-squared error. Proposes a modulation framework with non-linear equations for unique solutions and high-SNR closed-form analysis. Achieves reduced MSE under power constraints for efficient wireless function aggregation.
* `kernel` `training` [PRAGMA: A Profiling-Reasoned Multi-Agent Framework for Automatic Kernel Optimization](http://arxiv.org/abs/2511.06345v2)
  > **TL;DR**: PRAGMA proposes a profiling-reasoned multi-agent framework for automatic GPU/CPU kernel generation. It integrates hardware profiling into the LLM reasoning loop to identify bottlenecks and iteratively refine kernels. Achieves 2.81× and 2.30× speedups on CPU and GPU platforms compared to Torch.
* `serving` `sparse` `offloading` [Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism](http://arxiv.org/abs/2511.06247v2)
  > **TL;DR**: Optimizes long-context LLM serving with fine-grained sequence parallelism. Proposes Tetris system that dynamically allocates parallelism per token segments in intra-request chunks. Achieves 4.35× lower TTFT, 40.1% median TBT reduction, and 45% higher request capacity.
* `training` `hardware` `kernel` [Exploring Parallelism in FPGA-Based Accelerators for Machine Learning Applications](http://arxiv.org/abs/2511.11640v1)
  > **TL;DR**: Proposes speculative backpropagation for neural network training acceleration by overlapping forward and backward passes. Implemented using OpenMP multi-threading on CPUs and targeted for FPGA synthesis. Achieves up to 35% speedup in step execution time while maintaining comparable accuracy on MNIST.
* `networking` `offline` `storage` [Elastic Data Transfer Optimization with Hybrid Reinforcement Learning](http://arxiv.org/abs/2511.06159v2)
  > **TL;DR**: Optimizes large-scale data transfer for scientific datasets using hybrid heuristics and deep reinforcement learning. Combines parallelism, pipelining, and a lightweight simulator for fast concurrency tuning. Achieves 9.5x higher throughput compared to state-of-the-art solutions.
* `serving` `offloading` `kernel` [MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference](http://arxiv.org/abs/2511.06010v1)
  > **TL;DR**: Addresses KV cache bottlenecks in long-sequence LLM inference. Proposes MoSKA with Shared KV Attention to batch shared data processing (GEMM instead of GEMV), sparse attention pruning, and disaggregated hardware. Achieves up to 538.7x higher throughput in high-sharing scenarios.
* `serving` `kernel` [Kunlun Anomaly Troubleshooter: Enabling Kernel-Level Anomaly Detection and Causal Reasoning for Large Model Distributed Inference](http://arxiv.org/abs/2511.05978v1)
  > **TL;DR**: Presents Kunlun Anomaly Troubleshooter (KAT) for kernel-level anomaly detection in distributed LLM inference. Uses GPU trace data and domain-adapted LLM for causal reasoning. Achieves 0.884 precision and 0.936 recall in anomaly detection.
* `offloading` `RL` `networking` [DWM-RO: Decentralized World Models with Reasoning Offloading for SWIPT-enabled Satellite-Terrestrial HetNets](http://arxiv.org/abs/2511.05972v1)
  > **TL;DR**: Proposes DWM-RO framework for decentralized MARL in satellite-terrestrial hetnets with SWIPT. Uses world models for predictive representations and offloading gate for selective edge coordination. Achieves 5x faster convergence, 34.7% higher spectral efficiency, and 40% lower violations.
* `hardware` `training` `kernel` [MT4G: A Tool for Reliable Auto-Discovery of NVIDIA and AMD GPU Compute and Memory Topologies](http://arxiv.org/abs/2511.05958v1)
  > **TL;DR**: Proposes MT4G, a vendor-agnostic tool for auto-discovering GPU compute/memory topologies via microbenchmarks and statistical methods. Enables performance modeling, bottleneck analysis, and resource partitioning, demonstrated across NVIDIA/AMD GPUs to optimize HPC/AI system utilization.
* `edge` `RAG` `serving` [CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing](http://arxiv.org/abs/2511.05915v1)
  > **TL;DR**: Proposes CoEdge-RAG, a hierarchical scheduling framework for RAG-enhanced LLMs in collaborative edge environments. Uses online PPO-based query identification, dynamic inter-node workload balancing, and intra-node resource optimization to handle privacy constraints and heterogeneity. Achieves 4.23% to 91.39% performance gains over baselines.
* `training` `networking` [An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning](http://arxiv.org/abs/2511.05770v1)
  > **TL;DR**: Proposes an error-bounded lossy compressor for federated learning gradients to reduce communication overhead. The method exploits temporal correlations and convolutional kernel structures via two predictors: magnitude and sign predictors. Achieves up to 1.53x higher compression ratio than SZ3 and reduces communication time by 76.1%-96.2%.
* `edge` `offline` `hardware` [Optimal Multi-Constrained Workflow Scheduling for Cyber-Physical Systems in the Edge-Cloud Continuum](http://arxiv.org/abs/2511.07466v1)
  > **TL;DR**: Proposes an optimal multi-constrained workflow scheduler for edge-cloud continuum CPS to minimize latency. Uses continuous-time MILP formulation with heterogeneous multicore and capability constraints. Achieves 13.54% average latency reduction vs. enhanced heuristic in real-world use case.
* `kernel` `hardware` [GPU Under Pressure: Estimating Application's Stress via Telemetry and Performance Counters](http://arxiv.org/abs/2511.05067v1)
  > **TL;DR**: Studies estimating GPU stress under sustained workloads using telemetry and performance counters. Combines telemetry data with specific counters (throughput, instructions issued, stalls) to assess stress. Demonstrates effectiveness for CNN workloads, aiming to predict reliability and aging effects.
* `video` `edge` `hardware` [Accelerating HDC-CNN Hybrid Models Using Custom Instructions on RISC-V GPUs](http://arxiv.org/abs/2511.05053v1)
  > **TL;DR**: Explores optimizing hybrid HDC-CNN models for energy-efficient ML. Designs custom instructions on RISC-V GPUs for accelerated processing. Achieves 56.2x speedup in microbenchmarks.
* `edge` `serving` `quantization` [Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges](http://arxiv.org/abs/2511.11624v1)
  > **TL;DR**: This paper analyzes the power efficiency of small language models on edge devices, comparing CPU and GPU configurations. It evaluates energy-to-performance ratios across Raspberry Pi, Jetson Nano, and Jetson Orin platforms. Jetson Orin Nano with GPU achieves highest efficiency, with Llama 3.2 offering best accuracy-power balance.
* `kernel` `hardware` `networking` [Marionette: Data Structure Description and Management for Heterogeneous Computing](http://arxiv.org/abs/2511.04853v1)
  > **TL;DR**: Proposes Marionette, a C++17 library for flexible and efficient data structure management in heterogeneous computing. It decouples data layout from interface description, supports multiple memory strategies, and enables efficient data transfers across devices. Achieves minimal runtime overhead, demonstrated in a CUDA case study.
* `sparse` `quantization` `kernel` [Enabling Dynamic Sparsity in Quantized LLM Inference](http://arxiv.org/abs/2511.04477v1)
  > **TL;DR**: Explores enabling dynamic sparsity in quantized LLM inference. Proposes a zigzag-patterned quantization layout, specialized GEMV kernel, and sparse index gathering. Achieves up to 1.55x faster decoding throughput on commodity GPUs without accuracy loss.
* `serving` `offloading` `kernel` [AIvailable: A Software-Defined Architecture for LLM-as-a-Service on Heterogeneous and Legacy GPUs](http://arxiv.org/abs/2511.11621v1)
  > **TL;DR**: Proposes AIvailable, a software-defined LLM-as-a-Service platform for heterogeneous GPUs. Uses VRAM-aware dynamic allocation to run GPU-accelerated inference without CPU fallbacks on mixed NVIDIA/AMD nodes. Achieves efficient resource utilization by repurposing legacy GPUs for scalable serving.
* `training` `sparse` `networking` [Parallel Spawning Strategies for Dynamic-Aware MPI Applications](http://arxiv.org/abs/2511.04268v1)
  > **TL;DR**: Addresses the high cost of reconfiguration for malleable MPI applications. Proposes a cooperative parallel spawning strategy that reuses original processes and fully releases unneeded nodes during shrinkage. Reduces shrink operation cost by at least 20× while maintaining expansion overhead below 1.25×.
* `edge` `scaling` [Stochastic Modeling for Energy-Efficient Edge Infrastructure](http://arxiv.org/abs/2511.03941v1)
  > **TL;DR**: Addresses energy efficiency in edge computing via AI-driven predictive power scaling. Uses Markov Chains for modeling power state transitions and optimizes workload distribution to minimize unnecessary transitions. Reduces energy consumption disparities by 30% and improves system responsiveness.
* `offloading` `serving` `MoE` [AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism](http://arxiv.org/abs/2511.11617v1)
  > **TL;DR**: Proposes AnchorTP, a state-preserving elastic tensor parallelism framework for resilient LLM inference. Preserves model parameters and KV caches via a daemon and minimizes recovery time with a bandwidth-aware planner and pipelined transfers. Reduces Time to First Success by 11x and Time to Peak by 59%.
* `edge` `serving` `offloading` [UMDAM: A Unified Data Layout and DRAM Address Mapping for Heterogenous NPU-PIM](http://arxiv.org/abs/2511.03293v2)
  > **TL;DR**: Addresses memory bottlenecks in edge LLM inference with NPU-PIM co-execution. Proposes UMDAM, a unified data layout and DRAM mapping scheme optimized for compute and PIM efficiency. Achieves 3.0× TTFT reduction and 2.18× TTLT improvement for OPT models.
* `serving` `offloading` `kernel` [SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators](http://arxiv.org/abs/2511.03092v5)
  > **TL;DR**: Proposes SnapStream, a KV cache compression method for efficient long-sequence LLM inference on static-graph accelerators. Integrates sparse attention with continuous batching to reduce on-chip memory by 4× while maintaining accuracy, achieving up to 1832 tokens/sec with 128k context on real hardware deployment.
* `training` `hardware` `kernel` [Implementing Multi-GPU Scientific Computing Miniapps Across Performance Portable Frameworks](http://arxiv.org/abs/2511.02655v1)
  > **TL;DR**: Evaluates performance portability frameworks (Kokkos, OpenMP, RAJA, OCCA) for multi-GPU scientific computing miniapps. Measures time-to-solution on NVIDIA A100 GPUs for N-body and structured grid simulations. Shows OCCA achieves faster execution for small problems but lacks optimized reductions.
* `edge` `serving` `networking` [Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks](http://arxiv.org/abs/2511.02647v1)
  > **TL;DR**: Proposes Federated Attention (FedAttn), a distributed LLM inference framework for edge networks that exchanges aggregated KV matrices across Transformer blocks with privacy. Reduces communication by 3.2× and latency by 1.7× while maintaining model quality via sparse attention and adaptive aggregation.
* `serving` `kernel` `offloading` [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](http://arxiv.org/abs/2511.02248v1)
  > **TL;DR**: Proposes operator-level autoscaling for generative models by treating models as graphs of heterogeneous operators. Optimizes scaling, batching, and placement per operator. Achieves 40% fewer GPUs and 1.6x higher throughput under fixed resources while preserving SLOs.
* `hardware` `inference` `edge` [Beyond the GPU: The Strategic Role of FPGAs in the Next Wave of AI](http://arxiv.org/abs/2511.11614v1)
  > **TL;DR**: Explores FPGA-based acceleration for AI to address GPU limitations in latency and energy efficiency. Advocates for direct algorithm-to-logic mapping with parallel pipelines, deterministic timing, and local inference near sensors. Achieves up to 40% lower power consumption compared to GPUs.
* `training` `kernel` `serving` [Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs](http://arxiv.org/abs/2511.02168v1)
  > **TL;DR**: Addresses inefficiencies in distributed GPU execution for LLM workloads. Moves beyond bulk synchronous model with fine-grained programming patterns using in-kernel communication to create tile-level pipelines. Achieves 10-20% end-to-end latency speedup.
* `training` `kernel` `hardware` [Optimizing Attention on GPUs by Exploiting GPU Architectural NUMA Effects](http://arxiv.org/abs/2511.02132v1)
  > **TL;DR**: Investigates how NUMA effects degrade attention performance on modern GPUs. Proposes Swizzled Head-first Mapping, a scheduling strategy that aligns attention heads with NUMA domains to enhance cache reuse. Achieves up to 50% higher performance and 80-97% L2 cache hit rates on AMD MI300X.
* `serving` `offloading` `edge` [Why Should the Server Do It All?: A Scalable, Versatile, and Model-Agnostic Framework for Server-Light DNN Inference over Massively Distributed Clients via Training-Free Intermediate Feature Compression](http://arxiv.org/abs/2511.11608v1)
  > **TL;DR**: Proposes SLICER, a training-free framework to compress intermediate features for distributed DNN inference, reducing communication and server load. Uses asymmetric top-K filtering, magnitude-splitting, and adaptive quantization to achieve up to 10x lower uplink volume and 4.4x server GPU time reduction.
* `edge` [Boosting performance of computer vision applications through embedded GPUs on the edge](http://arxiv.org/abs/2511.01129v1)
  > **TL;DR**: Proposes using embedded GPUs on edge devices to boost computer vision application performance. Designs a system offloading intensive tasks to embedded GPUs for efficient execution. Achieves performance gains over CPU-only, improving user experience through faster processing.
* `offloading` `edge` `sparse` [Neuro-Inspired Task Offloading in Edge-IoT Networks Using Spiking Neural Networks](http://arxiv.org/abs/2511.01127v1)
  > **TL;DR**: Explores energy-efficient task offloading in edge-IoT networks. Uses spiking neural networks (SNNs) for real-time, adaptive task orchestration. Achieves 32% lower energy consumption and 25% higher success rate versus baselines under high load.
* `serving` `energy` [FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs](http://arxiv.org/abs/2511.00807v2)
  > **TL;DR**: Proposes FREESH for energy-efficient LLM serving on heterogeneous GPUs by jointly optimizing query routing, scheduling, and GPU frequency scaling. Matches GPU power-throughput characteristics with query workloads to reduce carbon and energy. Achieves 28.6% energy reduction and 45.45% emissions reduction while improving SLO attainment.
* `RL` `training` `offloading` [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](http://arxiv.org/abs/2511.00796v1)
  > **TL;DR**: Aims to maximize training throughput and cost efficiency for reinforcement learning (RL) with LLMs over heterogeneous GPUs. Proposes AReaL-Hex, a heterogeneity-aware scheduler with two-phase optimization (MILP-based assignment and graph partitioning) to balance stages. Achieves up to 1.50x higher throughput or 1.46x cost reduction.
* `edge` `serving` `offloading` [Toward Sustainability-Aware LLM Inference on Edge Clusters](http://arxiv.org/abs/2512.04088v1)
  > **TL;DR**: Explores carbon- and latency-aware routing for LLM inference on edge clusters. Proposes strategies that balance latency and carbon footprint by routing prompts to specific hardware based on benchmarking. Achieves optimal batch size of 4 for throughput and energy efficiency without memory saturation.
* `serving` `edge` `scheduling` [EPARA: Parallelizing Categorized AI Inference in Edge Clouds](http://arxiv.org/abs/2511.00603v1)
  > **TL;DR**: Proposes EPARA, an end-to-edge parallel inference framework for edge clouds. It categorizes tasks by latency/GPU sensitivity, using parallelism allocator, request handler, and state-aware scheduler. Achieves 2.1× higher goodput in production workloads versus prior frameworks.
* `offloading` `edge` `serving` [Gaia: Hybrid Hardware Acceleration for Serverless AI in the 3D Compute Continuum](http://arxiv.org/abs/2511.13728v1)
  > **TL;DR**: Proposes Gaia, a GPU-as-a-service architecture for serverless AI. Dynamically switches between CPU/GPU backends based on workload and SLOs via execution modes and runtime adjustments. Achieves up to 95% reduction in end-to-end latency.
* `training` `MoE` `multi-modal` [LongCat-Flash-Omni Technical Report](http://arxiv.org/abs/2511.00279v2)
  > **TL;DR**: Addresses efficient large-scale omni-modal model training. Proposes modality-decoupled parallelism and efficient MoE architecture with progressive curriculum. Achieves over 90% of text-only training throughput for a 560B parameter model.
* `networking` `serving` [Fix: externalizing network I/O in serverless computing](http://arxiv.org/abs/2511.00205v1)
  > **TL;DR**: Proposes a serverless computing system that externalizes network I/O, enabling the platform to manage data transfers. Uses deterministic procedures with precise data dependencies to optimize scheduling and reduce starvation. Shifts billing model to pay-for-results with improved efficiency.
* `networking` `training` `offloading` [RDMA Point-to-Point Communication for LLM Systems](http://arxiv.org/abs/2510.27656v1)
  > **TL;DR**: Addresses inflexible point-to-point communication in LLM systems requiring support for disaggregated inference, MoE, and RL fine-tuning. Introduces TransferEngine, a uniform NIC interface using one-sided WriteImm operations. Achieves 400 Gbps throughput and 1.3-second RL updates for trillion-parameter models.
* `kernel` `training` [ML-Based Optimum Sub-system Size Heuristic for the GPU Implementation of the Tridiagonal Partition Method](http://arxiv.org/abs/2510.27351v1)
  > **TL;DR**: Proposes a ML-based heuristic to find optimal sub-system size for CUDA implementation of parallel partition algorithms. Uses kNN to predict optimal sizes for solving systems of linear equations, reducing empirical tuning. Achieves acceptably good predictions validated against actual data.
* `training` `kernel` [Synergistic Tensor and Pipeline Parallelism](http://arxiv.org/abs/2510.27257v1)
  > **TL;DR**: Proposes a synergistic tensor-pipeline parallel schedule to reduce communication and synchronization overheads in LLM/MLLM training. Decouples forward/backward passes into fine-grained units braided into a composite sequence. Achieves 12-16% training throughput improvement over baselines.
* `serving` `offloading` [SERFLOW: A Cross-Service Cost Optimization Framework for SLO-Aware Dynamic ML Inference](http://arxiv.org/abs/2510.27182v1)
  > **TL;DR**: Addresses how to optimize cloud costs for dynamic ML inference requests with variable early exits. Proposes SERFLOW, which leverages FaaS and stage-specific resource provisioning with adaptive load balancing. Reduces cloud costs by over 23% while adapting to dynamic workloads.
* `serving` `training` [Glia: A Human-Inspired AI for Automated Systems Design and Optimization](http://arxiv.org/abs/2510.27176v3)
  > **TL;DR**: Proposes Glia, an AI system using multi-agent LLMs to autonomously design computer systems. It generates interpretable algorithms for distributed LLM inference, including routing, scheduling, and autoscaling. Achieves human-expert performance levels with significant time reduction.
* `hardware` `inference` `training` [Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators](http://arxiv.org/abs/2511.11601v1)
  > **TL;DR**: Investigates inconsistencies in ML model execution across heterogeneous AI accelerators. Uses automated pipeline to test 100,000+ model variants on five accelerators. Finds newer platforms support 17% fewer operators and exhibit >5% output discrepancies due to implementation differences.
* `training` `offline` `RL` [FlowMesh: A Service Fabric for Composable LLM Workflows](http://arxiv.org/abs/2510.26913v1)
  > **TL;DR**: Proposes FlowMesh, a service fabric for composable LLM workflows like RLHF and agentic pipelines. Uses fine-grained operators with lineage for deduplication and batching on heterogeneous GPUs via a global scheduler. Achieves 3.8x cost reduction and 2.0x lower energy usage.
* `MoE` `serving` `offloading` [ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference](http://arxiv.org/abs/2510.26730v1)
  > **TL;DR**: Proposes ExpertFlow, a runtime system for MoE inference that adaptively prefetches experts using runtime statistics and hybrid cross-layer prediction. Reduces model stall time to less than 0.1% of baseline by decreasing cache misses via coordinated memory management and expert scheduling.
* `sparse` `training` `networking` [An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning](http://arxiv.org/abs/2510.26709v3)
  > **TL;DR**: Addresses communication bottlenecks in distributed training via an All-Reduce compatible Top-K compressor that aligns sparsity patterns using gradient sketches. Combines contraction property with efficient index-free All-Reduce. Achieves up to 60.7% reduction in wall-clock training time.
* `RL` `training` `offline` [ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems](http://arxiv.org/abs/2510.26475v1)
  > **TL;DR**: Proposes ReSpec to address three gaps limiting speculative decoding in RL-based LLM training: diminishing speedup at scale, drafter staleness, and policy degradation. Combines dynamic SD tuning, drafter evolution via distillation, and reward-weighted updates. Achieves up to 4.5x speedup on Qwen models while preserving reward convergence.
* `training` `networking` `storage` [Detecting Anomalies in Machine Learning Infrastructure via Hardware Telemetry](http://arxiv.org/abs/2510.26008v2)
  > **TL;DR**: Proposes Reveal, an unsupervised pipeline that uses hardware telemetry to detect anomalies in ML infrastructure without workload knowledge. Achieves 5.97% acceleration for DeepSeek model by identifying network and system configuration issues.
* `quantization` `serving` `kernel` [Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms](http://arxiv.org/abs/2511.19438v1)
  > **TL;DR**: Optimizes memory and computation for 4-bit GPTQ quantized LLM inference on heterogeneous platforms. Proposes SMB-Opt, VML-Opt, and ILA-Opt in vLLM. Achieves 84.42% higher throughput and 51.35% latency reduction.
* `MoE` `hardware` `offlining` [MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference](http://arxiv.org/abs/2510.25258v1)
  > **TL;DR**: Addresses imbalanced communication and expert migration overhead in wafer-scale chips for mixture-of-expert parallelism. Proposes entwined ring mapping and non-invasive balancer to optimize layer mapping and split migration steps. Achieves up to 62% communication reduction and 54% faster MoE computation.
* `training` `networking` `quantization` [Machine Learning and CPU (Central Processing Unit) Scheduling Co-Optimization over a Network of Computing Centers](http://arxiv.org/abs/2510.25176v1)
  > **TL;DR**: Co-optimizes CPU resource allocation and distributed ML training over networked computing nodes. Combines local training with consensus-based resource scheduling and log-quantized communication. Achieves over 50% better cost optimality than existing schedulers.
* `training` `kernel` `sparse` [Multi-Resolution Model Fusion for Accelerating the Convolutional Neural Network Training](http://arxiv.org/abs/2510.25170v1)
  > **TL;DR**: Introduces Multi-Resolution Model Fusion (MRMF) to accelerate CNN training time. Trains initial models on reduced-resolution data, then fuses and refines them with original-resolution data. Achieves up to 47% training time reduction while maintaining accuracy on CosmoFlow and Neuron Inverter benchmarks.
* `edge` `training` `serving` [AeroResQ: Edge-Accelerated UAV Framework for Scalable, Resilient and Collaborative Escape Route Planning in Wildfire Scenarios](http://arxiv.org/abs/2511.00038v1)
  > **TL;DR**: Proposes AeroResQ, an edge-accelerated UAV framework for real-time escape route planning in wildfires. Uses multi-layer orchestration with service drones running DNNs on edge accelerators and coordinator drones for path planning. Achieves end-to-end latency <=500 ms and over 98% task reassignment success.
* `edge` `serving` `networking` [Bayes-Split-Edge: Bayesian Optimization for Constrained Collaborative Inference in Wireless Edge Systems](http://arxiv.org/abs/2510.23503v1)
  > **TL;DR**: Proposes Bayes-Split-Edge, a Bayesian optimization framework for constrained collaborative inference between edge devices and servers. Jointly optimizes transmission power and neural network split point to minimize energy/delay costs. Achieves up to 2.4x cost reduction vs standard BO and near-linear convergence with max 20 evaluations.
* `edge` `offloading` [Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions](http://arxiv.org/abs/2510.22909v1)
  > **TL;DR**: Surveys optimal placement of DL model partitions across edge and cloud for multi-objective optimization. Analyzes offloading techniques, model adaptations, and compression to balance latency, cost, and privacy. Highlights trade-offs between ≤40ms edge latency vs. cloud cost savings.
* `networking` `scheduling` `storage` [Learning to Schedule: A Supervised Learning Framework for Network-Aware Scheduling of Data-Intensive Workloads](http://arxiv.org/abs/2510.21419v1)
  > **TL;DR**: Addresses network-aware job scheduling for data-intensive workloads via supervised learning. Collects real-time telemetry to predict job completion times per node and ranks placements. Achieves 34-54% higher accuracy in optimal node selection versus default Kubernetes scheduler.
* `edge` `kernel` `serving` [Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution](http://arxiv.org/abs/2510.21081v1)
  > **TL;DR**: Reduces mobile inference latency via CPU-GPU co-execution. Introduces lightweight SVM-based synchronization and ML-based execution time prediction for task assignment. Achieves up to 1.89x speedup for linear layers on Pixel 5 smartphone.
* `training` `offloading` `storage` [xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads](http://arxiv.org/abs/2510.21048v1)
  > **TL;DR**: Proposes xMem, a CPU-only dynamic analysis framework for precise a priori estimation of peak GPU memory in deep learning training. Targets resource allocation optimizations without runtime GPU overhead. Achieves 91% lower median relative error and 75% reduction in OOM probability.
* `hardware` `training` `inference` [Lincoln AI Computing Survey (LAICS) and Trends](http://arxiv.org/abs/2510.20931v1)
  > **TL;DR**: Surveys AI accelerators for GenAI training and inference, plotting peak performance vs. power. Updates previous data with new market segment analysis and architectural categorizations. Highlights trends and provides performance/power metrics for commercial systems.
* `serving` `networking` [Morpheus: Lightweight RTT Prediction for Performance-Aware Load Balancing](http://arxiv.org/abs/2510.20506v1)
  > **TL;DR**: Proposes lightweight RTT predictors using time-series cluster metrics for performance-aware load balancing in distributed systems. Achieves up to 95% prediction accuracy, reducing tail latency and resource waste in Kubernetes-managed GPU clusters via proactive routing optimization.
* `serving` `offloading` [FLAS: a combination of proactive and reactive auto-scaling architecture for distributed services](http://arxiv.org/abs/2510.20388v1)
  > **TL;DR**: Proposes FLAS, an auto-scaler combining proactive forecasting of SLA trends with reactive estimation from resource metrics, enabling adaptive resource scaling. Ensures SLA compliance for distributed services with >99% adherence during worst-case scenarios.
* `training` `networking` `kernel` [Collective Communication for 100k+ GPUs](http://arxiv.org/abs/2510.20171v3)
  > **TL;DR**: Addresses collective communication bottlenecks in LLM training at 100k+ GPU scale. Proposes NCCLX framework optimizing throughput and latency for synchronous training and inference workloads. Achieves substantial communication efficiency gains on Llama4 model.
* `training` `MoE` `networking` [AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training](http://arxiv.org/abs/2510.20111v1)
  > **TL;DR**: Proposes AsyncHZP, an asynchronous hierarchical ZeRO parallelism method for scalable LLM training, featuring adaptive resharding and multi-stream scheduling to reduce communication overhead. Achieves state-of-the-art performance without complex tuning, validated on Dense and MoE models.
* `offline` `serving` `quantization` [Serverless GPU Architecture for Enterprise HR Analytics: A Production-Scale BDaaS Implementation](http://arxiv.org/abs/2510.19689v1)
  > **TL;DR**: Proposes a serverless GPU architecture for low-latency, cost-efficient TabNet inference in regulated HR analytics. Integrates serverless GPU runtime with interpretability for compliance. Achieves up to 4.5× higher throughput and 98× lower latency vs. Spark at 90% lower cost per 1K inferences.
* `MoE` `training` `networking` [HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission](http://arxiv.org/abs/2510.19470v1)
  > **TL;DR**: Proposes HybridEP, a framework optimizing Mixture-of-Experts training for cross-datacenter scalability under constrained bandwidth. Dynamically transforms expert placement via domain-based partition and parameter-efficient migration, guided by stream-based modeling. Achieves up to 5.6x speedup over existing systems.
* `networking` `training` [Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networks](http://arxiv.org/abs/2510.19322v1)
  > **TL;DR**: Addresses inefficiency in optical networks for distributed ML training collectives. Proposes SWOT framework with intra-collective reconfiguration, overlapping switch reconfigurations with transmissions. Achieves performance improvements in simulations.
* `training` `MoE` `networking` [RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training](http://arxiv.org/abs/2510.19262v2)
  > **TL;DR**: Addresses imbalanced all-to-all communication overhead in MoE training. Proposes RailS, a load-balancing framework leveraging rail topology symmetry with local LPT-based scheduling and multipath transmission. Improves bus bandwidth by 20%–78% and reduces iteration time by 18%–40% for Mixtral workloads.
* `RL` `training` `offloading` [RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs](http://arxiv.org/abs/2510.19225v2)
  > **TL;DR**: Explores cost-efficient RL training for LLMs by harvesting preemptible GPU resources. Proposes RLBoost, a hybrid architecture with adaptive offload, pull-based weight transfer, and token-level response migration. Achieves 1.51x-1.97x training throughput and 28%-49% cost savings over on-demand resources.
* `hardware` `training` [Learned Cost Model for Placement on Reconfigurable Dataflow Hardware](http://arxiv.org/abs/2511.01872v1)
  > **TL;DR**: Proposes a learned cost model to predict throughput for mapping ML dataflow graphs onto reconfigurable hardware, improving accuracy by 31-52% and achieving 5.6% faster compiled graphs compared to analytical models.
* `training` `sparse` [MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training](http://arxiv.org/abs/2510.18830v1)
  > **TL;DR**: Addresses computational imbalance in distributed training of LLMs with ultra-long contexts via dynamic sparse attention. Introduces MTraining with balanced and hierarchical sparse ring attention to reduce overhead. Achieves 6x higher training throughput while maintaining accuracy on Qwen2.5-3B.
* `serving` `offloading` `RAG` [Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](http://arxiv.org/abs/2510.18586v2)
  > **TL;DR**: Addresses KV cache inefficiency in multi-agent LLM serving with external tool calls. Proposes Tokencake framework with agent-aware dynamic memory partitioning and proactive offload/upload mechanisms. Achieves 47.06% end-to-end latency reduction and 16.9% better GPU memory utilization vs vLLM.
* `serving` `edge` `offline` [SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices](http://arxiv.org/abs/2510.18544v3)
  > **TL;DR**: Proposes SLICE, a scheduling system for LLM inference on edge devices with diverse SLO constraints. Combines utility-based scheduling and dynamic generation rate control to improve SLO attainment. Achieves 35x higher SLO attainment and 3.4x faster task completion vs. Orca/FastServe.
* `edge` `serving` `thinking` [EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs](http://arxiv.org/abs/2511.01866v1)
  > **TL;DR**: Characterizes deployment of reasoning LLMs on edge GPUs under latency constraints. Evaluates model sizes, token reduction techniques, and parallelism strategies to balance accuracy-latency. Achieves optimized configurations mapping Pareto frontiers, improving edge deployment efficiency by up to 4x.
* `training` `kernel` [Efficient Long-context Language Model Training by Core Attention Disaggregation](http://arxiv.org/abs/2510.18121v1)
  > **TL;DR**: Addresses load imbalance in long-context LLM training by disaggregating core attention computation. Proposes Core Attention Disaggregation (CAD) that schedules stateless attention tasks on dedicated devices using dynamic batching. Achieves up to 1.35x throughput improvement on 512 H200 GPUs with 512k context lengths.
* `serving` `storage` `training` [AI for Distributed Systems Design: Scalable Cloud Optimization Through Repeated LLMs Sampling And Simulators](http://arxiv.org/abs/2510.18897v1)
  > **TL;DR**: Develops AI-driven distributed scheduler design via LLM-generated policies verified in a domain-specific simulator. Proposes iterative generate-verify loop using simulator (Eudoxia) and feedback to optimize policies. Achieves throughput improvements across models in Function-as-a-Service runtime.
* `offloading` `edge` `serving` [DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones](http://arxiv.org/abs/2511.07427v1)
  > **TL;DR**: Proposes DynaKV, an adaptive key-value cache management system for efficient long-sequence LLM decoding on smartphones. Uses migration-free cluster adaptation, continuity-centric flash layout, and virtualized cache to address KVCache distribution shifts. Achieves 1.47x speedups and 1.38x accuracy gains over state-of-the-art.
* `agentic` `RL` `offline` [Network and Systems Performance Characterization of MCP-Enabled LLM Agents](http://arxiv.org/abs/2511.07426v1)
  > **TL;DR**: Characterizes performance and cost of MCP-enabled LLM agents. Measures token efficiency, cost, time and success rate across different models and MCP configurations. Achieves optimizations via parallel tool calls and task abort for cost-effective workflows.
* `training` `kernel` `thinking` [Integrating Performance Tools in Model Reasoning for GPU Kernel Optimization](http://arxiv.org/abs/2510.17158v1)
  > **TL;DR**: Proposes training LLMs to interact with performance tools during reasoning for GPU kernel optimization. Introduces a method integrating hardware/environment data via chain-of-thought reasoning. Achieves state-of-the-art results in optimizing GPU kernels, though specific metrics not detailed in abstract.
* `edge` `quantization` `offloading` [An Evaluation of LLMs Inference on Popular Single-board Computers](http://arxiv.org/abs/2511.07425v1)
  > **TL;DR**: Evaluates LLM inference performance on single-board computers (SBCs) for edge deployment. Benchmarks 25 quantized LLMs using Ollama/Llamafile runtimes on three SBCs, measuring throughput, memory, and power. Achieves 4x higher throughput and 30-40% lower power usage with Llamafile.
* `serving` `RL` [Justitia: Fair and Efficient Scheduling for LLM Applications](http://arxiv.org/abs/2510.17015v1)
  > **TL;DR**: Proposes Justitia, a fair and efficient scheduler for LLM applications to reduce head-of-line blocking and resource allocation issues. Uses memory-centric cost modeling, neural demand prediction, and virtual-time fair queuing. Achieves improved scheduling efficiency while preserving fairness.
* `networking` [Host-Side Telemetry for Performance Diagnosis in Cloud and HPC GPU Infrastructure](http://arxiv.org/abs/2510.16946v1)
  > **TL;DR**: Presents an eBPF-based telemetry system for diagnosing GPU tail latency in cloud/HPC. Correlates host-side metrics with GPU events for unified observability, achieving 81-88% diagnostic accuracy and detecting spikes in under 5 seconds with 1.21% CPU overhead at 100Hz.
* `serving` `edge` `hardware` [Exact Nearest-Neighbor Search on Energy-Efficient FPGA Devices](http://arxiv.org/abs/2510.16736v1)
  > **TL;DR**: Investigates energy-efficient exact kNN search for high-dimensional latent spaces using FPGA configurations. Proposes two FPGA solutions: one maximizing throughput via batched stream processing, another minimizing latency with in-memory processing. Achieves 16.6× higher throughput and 11.9× energy savings versus CPUs.
* `serving` `offline` [CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation](http://arxiv.org/abs/2510.18893v1)
  > **TL;DR**: Proposes CodeCRDT, an observation-driven coordination using CRDTs for multi-agent LLM code generation to avoid explicit messaging. Enables lock-free concurrent code generation with 100% convergence, achieving up to 21.1% speedup but also up to 39.4% slowdown depending on task structure.
* `training` `networking` `sparse` [Reimagining RDMA Through the Lens of ML](http://arxiv.org/abs/2510.16606v1)
  > **TL;DR**: Addresses tail latency in collective communication for distributed ML training. Proposes Celeris, a domain-specific RDMA transport that removes retransmissions and in-order delivery, exploiting ML's tolerance for partial data. Achieves up to 2.3x lower 99th-percentile latency and doubles NIC resilience.
* `edge` `serving` `offloading` [Edge-Based Speech Transcription and Synthesis for Kinyarwanda and Swahili Languages](http://arxiv.org/abs/2510.16497v1)
  > **TL;DR**: Proposes an edge-cloud cascading framework to optimize speech transcription/synthesis for under-resourced languages using Whisper and SpeechT5. Distributes inference workload between edge and cloud to reduce latency and memory usage. Achieves 9.5-14% model compression on edge devices with memory usage capped at 149MB.
* `serving` `offloading` `edge` [FourierCompress: Layer-Aware Spectral Activation Compression for Efficient and Accurate Collaborative LLM Inference](http://arxiv.org/abs/2510.16418v1)
  > **TL;DR**: Proposes FourierCompress, a layer-aware spectral activation compression method for collaborative LLM inference between edge devices and servers. Uses FFT to retain low-frequency coefficients, reducing activation size by 7.6x with under 0.3% accuracy loss and 32x faster compression than Top-k.
* `training` `offloading` `edge` [MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization](http://arxiv.org/abs/2510.16415v1)
  > **TL;DR**: Proposes MeCeFO, a fault-tolerant optimization for distributed LLM training with minimal overhead during node failures. It uses skip-connection, recomputation, and low-rank gradient approximation to offload tasks to neighboring nodes. Achieves only 4.18% throughput drop under high failure rates, 5.0–6.7× better resilience than SOTA.
* `serving` `offline` `networking` [Enhancing reliability in AI inference services: An empirical study on real production incidents](http://arxiv.org/abs/2511.07424v1)
  > **TL;DR**: Presents empirical analysis of production incidents in hyperscale LLM inference services. Develops taxonomy and methodology using real incident data, identifying failure modes and mitigation strategies (e.g., GPU-aware routing). Achieves high labeling consistency (Cohen's K ~0.89) and reduces incident impact.
* `hardware` `serving` `storage` [Funky: Cloud-Native FPGA Virtualization and Orchestration](http://arxiv.org/abs/2510.15755v1)
  > **TL;DR**: Proposes Funky, a cloud-native FPGA orchestration engine with virtualization, state management, and FPGA-aware components to enhance performance and utilization. Achieves only 7.4% performance overhead vs. native execution while enabling strong isolation and efficient scheduling in distributed FPGAs.
* `serving` `training` `hardware` [GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusters](http://arxiv.org/abs/2510.15652v1)
  > **TL;DR**: Proposes GOGH, a learning-based resource allocator for ML workloads in heterogeneous GPU clusters. Uses two NNs: one for initial performance/co-location prediction and one for online refinement. Minimizes energy while meeting SLOs, achieving 30% lower energy use and improved SLO compliance.
* `training` `kernel` `sparse` [PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training](http://arxiv.org/abs/2510.15596v1)
  > **TL;DR**: Proposes PRISM, a probabilistic performance modeling framework for large-scale distributed training that accounts for runtime variations. It uses statistical methods to provide guarantees on training time, analyzes sensitivity to hardware variation, and identifies critical kernels (e.g., AllGather, ReduceScatter) for optimization. Achieves 1.26x potential performance improvement.
* `serving` `offloading` [BeLLMan: Controlling LLM Congestion](http://arxiv.org/abs/2510.15330v1)
  > **TL;DR**: Addresses how to control LLM serving congestion to avoid latency inflation. Proposes beLLMan, a controller that signals applications to adjust output length based on system load. Achieves 8× lower latency and 25% energy reduction while serving 19% more requests.
* `serving` `offloading` `edge` [Synera: Synergistic LLM Serving across Device and Cloud at Scale](http://arxiv.org/abs/2511.07423v1)
  > **TL;DR**: Proposes Synera, a device-cloud synergistic LLM serving system with selective offloading, parallel inference, and scalable batching to balance quality and latency. Achieves 1.20-5.47x better generation quality than baselines with on-par latency and 8.2-16.5% lower cloud costs.
* `kernel` `training` `offline` [Hive Hash Table: A Warp-Cooperative, Dynamically Resizable Hash Table for GPUs](http://arxiv.org/abs/2510.15095v1)
  > **TL;DR**: Introduces Hive, a high-performance GPU hash table with warp-cooperative protocols and dynamic resizing for concurrent updates. Proposes cache-aligned buckets, warp-synchronous concurrency, and load-aware resizing. Achieves 3.5 billion updates/s and 2x higher throughput than state-of-the-art GPU hash tables.
* `video` `multi-modal` `edge` [Multi-modal video data-pipelines for machine learning with minimal human supervision](http://arxiv.org/abs/2510.14862v1)
  > **TL;DR**: Proposes a multi-modal video data-pipeline requiring minimal human supervision, using pre-trained experts and PHG-MAE for efficient modeling. Deploys a distilled model (<1M params) for real-time semantic segmentation on commodity hardware, achieving competitive results against larger models.
* `serving` `multi-modal` `offline` [xLLM Technical Report](http://arxiv.org/abs/2510.14686v1)
  > **TL;DR**: Introduces xLLM, a decoupled inference framework with service and engine layers for efficient serving. Features intelligent scheduling for multimodal inputs, workload-adaptive disaggregation policies, and engine optimizations including execution pipelines and memory management. Achieves up to 2.2x higher throughput than baselines.
* `hardware` `training` `networking` [MPI-over-CXL: Enhancing Communication Efficiency in Distributed HPC Systems](http://arxiv.org/abs/2510.14622v1)
  > **TL;DR**: Proposes MPI-over-CXL, a communication paradigm using CXL's shared memory for direct pointer-based access in MPI, eliminating data copying. Reduces latency and bandwidth usage in distributed HPC systems. Achieves substantial performance gains over traditional MPI implementations.
* `training` `networking` `hardware` [ScalePool: Hybrid XLink-CXL Fabric for Composable Resource Disaggregation in Unified Scale-up Domains](http://arxiv.org/abs/2510.14580v1)
  > **TL;DR**: Proposes ScalePool, a hybrid XLink-CXL fabric for composable resource disaggregation in accelerator clusters. Integrates low-latency XLink for intra-cluster communication and hierarchical CXL for memory sharing and tiering. Accelerates LLM training by up to 1.84× and reduces memory latency by 4.5×.
* `serving` `offline` `quantization` [FairBatching: Fairness-Aware Batch Formation for LLM Inference](http://arxiv.org/abs/2510.14392v1)
  > **TL;DR**: Addresses unfair resource allocation in LLM inference batching. Introduces FairBatching scheduler with adaptive batch capacity and dynamic resource reclamation between prefill and decode tasks. Reduces TTFT tail latency by up to 2.29x and improves cluster capacity by 54.3%.
* `serving` `offloading` `networking` [From Attention to Disaggregation: Tracing the Evolution of LLM Inference](http://arxiv.org/abs/2511.07422v1)
  > **TL;DR**: Proposes disaggregated inference architecture for LLMs, decoupling prefill and decode phases as independently scalable components to overcome GPU cluster limitations. Mitigates resource contention, optimizing Time to First Token and Inter Token Latency while reducing cost. Achieves significant improvements in latency and throughput.
* `serving` `agentic` [Cortex: Workflow-Aware Resource Pooling and Scheduling for Agentic Serving](http://arxiv.org/abs/2510.14126v1)
  > **TL;DR**: Investigates resource pooling and scheduling for agentic workflows. Proposes Cortex with stage isolation, dedicating resource pools per workflow stage to mitigate interference and improve KV cache utilization. Increases throughput by 4.2× and reduces SLO violations by 48% compared to baseline.
* `training` `RL` `edge` [FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients](http://arxiv.org/abs/2510.14054v1)
  > **TL;DR**: Proposes FedHFT for efficient federated fine-tuning of LLMs across heterogeneous edge clients. Uses masked adapters for resource heterogeneity and bi-level optimization with client clustering for non-iid data. Achieves up to 15% accuracy gain with 60% less communication overhead.
* `serving` `offloading` `edge` [Efficiently Executing High-throughput Lightweight LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management](http://arxiv.org/abs/2510.14024v1)
  > **TL;DR**: Addresses high LLM startup costs and queue delays in HPC clusters with lightweight LLM inference. Proposes pervasive context management to decouple initialization from inference, retaining context on GPUs opportunistically. Achieves 72.1% execution time reduction (from 3h to 48min) and further scales to 13min using idle GPUs.
* `serving` `offline` `RAG` [FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access](http://arxiv.org/abs/2510.13724v1)
  > **TL;DR**: Proposes FIRST, a federated framework for scalable and private LLM inference on HPC clusters. Utilizes an OpenAI-compliant API with auto-scaling, parallel backend support, and hot nodes to balance batch/interactive modes. Enables billions of tokens daily on-premises.
* `serving` `offloading` [Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference](http://arxiv.org/abs/2510.13668v1)
  > **TL;DR**: Addresses dynamic workload imbalance during LLM inference caused by output length variation. Introduces ARES with lightweight LLM-native length prediction to anticipate workloads and enable adaptive rescheduling, reducing P99 TPOT by 74.77% and achieving 2.24x higher goodput.
* `quantization` `edge` `kernel` [F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs](http://arxiv.org/abs/2510.13401v1)
  > **TL;DR**: Proposes F-BFQ, a flexible FPGA accelerator supporting dynamic switching between BFP quantization variants for LLM inference. Achieves 1.4x faster inference and 5.2 tokens/sec on AMD Kria compared to CPU baseline.
* `edge` `serving` `offline` [ACE-GNN: Adaptive GNN Co-Inference with System-Aware Scheduling in Dynamic Edge Environments](http://arxiv.org/abs/2511.11586v1)
  > **TL;DR**: Proposes ACE-GNN, an adaptive device-edge co-inference framework for GNNs in dynamic edge environments. Uses system-level abstraction and prediction for runtime optimization, combining pipeline and data parallelism. Achieves up to 12.7x speedup and 82.3% energy savings over prior methods.
* `serving` `offloading` `networking` [BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure](http://arxiv.org/abs/2510.13223v1)
  > **TL;DR**: Proposes BanaServe, a dynamic orchestration framework for disaggregated LLM serving. It uses layer/attention-level migration and global KV cache store to balance prefill/decode stages and eliminate cache-induced hotspots. Achieves 1.2x-3.9x higher throughput and 3.9%-78.4% lower processing time than vLLM.
* `kernel` `training` `hardware` [A GPU-resident Memory-Aware Algorithm for Accelerating Bidiagonalization of Banded Matrices](http://arxiv.org/abs/2510.12705v1)
  > **TL;DR**: Presents a GPU-optimized algorithm for bidiagonal reduction of banded matrices, crucial for SVD in scientific computing. Utilizes hardware-agnostic implementation via Julia's Array abstractions and KernelAbstractions. Achieves up to 100x speedup over CPU libraries for 32k x 32k matrices and linear performance scaling with bandwidth size.
* `training` `RL` `kernel` [Laminar: A Scalable Asynchronous RL Post-Training Framework](http://arxiv.org/abs/2510.12633v1)
  > **TL;DR**: Scales RL post-training for LLMs by decoupling weight synchronization via relay workers and dynamic trajectory repack. Achieves 5.48× training speedup and reduces convergence time on a 1024-GPU cluster.
* `training` `networking` [Metronome: Efficient Scheduling for Periodic Traffic Jobs with Network and Priority Awareness](http://arxiv.org/abs/2510.12274v1)
  > **TL;DR**: Addresses scheduling inefficiencies for distributed training jobs with periodic traffic. Proposes Metronome, a network-aware scheduler with time-division multiplexing and priority-based optimization. Reduces job completion time by up to 19.50% and improves bandwidth utilization by 23.20%.
* `training` `hardware` `quantization` [Deploying Atmospheric and Oceanic AI Models on Chinese Hardware and Framework: Migration Strategies, Performance Optimization and Analysis](http://arxiv.org/abs/2510.17852v1)
  > **TL;DR**: Develops migration framework for porting atmospheric/oceanic AI models (e.g., FourCastNet) to Chinese chips/MindSpore. Optimizes via software-hardware adaptation, memory optimization, and parallelism. Achieves preserved accuracy with improved operational efficiency and reduced dependencies.
* `serving` `offloading` [FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters](http://arxiv.org/abs/2510.11938v1)
  > **TL;DR**: Addresses efficient LLM serving in serverless clusters with dynamic pipeline adaptations. Introduces FlexPipe, with fine-grained model partitioning, live pipeline refactoring, and topology-aware resource placement. Achieves 8.5× better resource efficiency and 38.3% lower latency while reducing GPU reservations to 30%.
* `training` `serving` `offloading` [An Explorative Study on Distributed Computing Techniques in Training and Inference of Large Language Models](http://arxiv.org/abs/2510.11211v1)
  > **TL;DR**: Explores distributed computing for democratizing LLMs on consumer hardware and comparing serving techniques. Proposes metaheuristic-based modification for resource optimization and evaluates three serving systems. Achieves efficient resource utilization enabling LLMs on limited hardware.
* `edge` `serving` `networking` [A Decentralized Microservice Scheduling Approach Using Service Mesh in Cloud-Edge Systems](http://arxiv.org/abs/2510.11189v1)
  > **TL;DR**: Proposes a decentralized microservice scheduling approach using service mesh sidecar proxies for cloud-edge systems. Embeds lightweight autonomous schedulers in sidecars to avoid centralized control, leveraging service mesh for distributed traffic management. Initial results show improved scalability in response time and latency under varying request rates.
* `training` `hardware` `storage` [Improving AI Efficiency in Data Centres by Power Dynamic Response](http://arxiv.org/abs/2510.11119v1)
  > **TL;DR**: Proposes dynamic power management for AI data centres to improve efficiency. Introduces active/passive solutions where input power adapts to computing demand. Quantifies 12-15% energy savings and capital expenditure reduction in global hyperscalers.
* `training` [DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism](http://arxiv.org/abs/2510.10620v1)
  > **TL;DR**: Addresses inefficiency of static context parallelism for long-context training with variable sequence lengths. Proposes DCP, a dynamic framework using block-wise partitioning for adaptive computation mapping. Accelerates attention layers by up to 2.45x for causal masks and 3.77x for sparse patterns.
* `networking` [A Verified High-Performance Composable Object Library for Remote Direct Memory Access (Extended Version)](http://arxiv.org/abs/2510.10531v1)
  > **TL;DR**: Introduces LOCO, a formally verified composable object library for RDMA to simplify distributed programming. Achieves performance comparable to custom RDMA systems with verified correctness via the Mowgli verification framework. Reduces complexity while matching custom system performance.
* `MoE` `offloading` `serving` [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](http://arxiv.org/abs/2510.10302v2)
  > **TL;DR**: Addresses GPU memory and bandwidth bottlenecks in speculative decoding for MoE model inference. Proposes SP-MoE with speculative expert prefetching, cutoff-layer policy, and pipelined runtime. Achieves 1.07-3.5× TPOT speedup over SOTA methods.
* `training` `edge` `quantization` [Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge](http://arxiv.org/abs/2511.11585v1)
  > **TL;DR**: Proposes FedGen-Edge, a federated learning framework using LoRA adapters to train generative models efficiently at the edge. Reduces uplink traffic by >99% vs full FedAvg, achieves lower perplexity/FID on PTB/CIFAR-10 while personalizing client models.
* `edge` `serving` `RL` [Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization](http://arxiv.org/abs/2510.10028v1)
  > **TL;DR**: Addresses efficient vision-language inference on resource-constrained UAVs. Proposes combined resource allocation and LLM-enhanced reinforcement learning for trajectory optimization. Achieves reduced task latency and power consumption under accuracy constraints in dynamic networks.
* `hardware` `training` `edge` [Co-designing a Programmable RISC-V Accelerator for MPC-based Energy and Thermal Management of Many-Core HPC Processors](http://arxiv.org/abs/2510.09163v1)
  > **TL;DR**: Proposes a hardware-software codesign for an MPC-based energy/thermal management accelerator for many-core processors. Uses a pruned operator-splitting quadratic programming solver and scheduled parallel execution on RISC-V cores. Achieves sub-millisecond latency controlling 144 PEs with 33× lower latency than baseline.
* `RAG` `multi-modal` `sparse` [Hierarchical Scheduling for Multi-Vector Image Retrieval](http://arxiv.org/abs/2510.08976v1)
  > **TL;DR**: Proposes HiMIR, a hierarchical scheduling framework for multi-vector image retrieval to improve accuracy and efficiency in MLLM RAG systems. Uses multiple granularities and cross-hierarchy similarity with sparsity to reduce redundant matching. Reduces computation by 3.5× while improving accuracy.
* `training` `storage` `networking` [Slicing Is All You Need: Towards A Universal One-Sided Algorithm for Distributed Matrix Multiplication](http://arxiv.org/abs/2510.08874v1)
  > **TL;DR**: Proposes a universal one-sided algorithm for distributed matrix multiplication to avoid operand redistribution across partitionings. Uses slicing to compute overlapping tiles and lowers to optimized IR. Achieves competitive performance with PyTorch DTensor for varied partitionings and replication factors.
* `training` `offline` `kernel` [Maple: A Multi-agent System for Portable Deep Learning across Clusters](http://arxiv.org/abs/2510.08842v2)
  > **TL;DR**: Proposes Maple, a multi-agent system for generating correct command lines for distributed DL training across heterogeneous GPU clusters using natural language. Agents handle extraction, template retrieval, verification, and correction, achieving 92.0% accuracy across 567 test cases.
* `hardware` `serving` `offline` [SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference](http://arxiv.org/abs/2510.08544v1)
  > **TL;DR**: Proposes SPAD, specialized hardware for disaggregated LLM inference with Prefill and Decode Chips optimized for each phase. Achieves 19%-41% lower hardware cost and 2%-17% lower TDP compared to H100 clusters while maintaining performance.
* `training` `RL` [DYNAMIX: RL-based Adaptive Batch Size Optimization in Distributed Machine Learning Systems](http://arxiv.org/abs/2510.08522v1)
  > **TL;DR**: Proposes DYNAMIX, an RL-based framework for adaptive batch size optimization in distributed ML training. Uses PPO with multi-dimensional system state to dynamically adjust batch sizes without explicit modeling. Achieves 46% training time reduction and 6.3% higher accuracy.
* `edge` `training` `serving` [Distributed Resource Selection for Self-Organising Cloud-Edge Systems](http://arxiv.org/abs/2510.08228v1)
  > **TL;DR**: Addresses efficient resource allocation in cloud-edge systems for distributed applications. Proposes a distributed consensus-based mechanism using local knowledge and inter-agent collaboration for dynamic resource selection. Achieves up to 30 times faster allocation than centralized heuristics without compromising optimality.
* `hardware` `serving` `edge` [Towards Energy-Efficient Serverless Computing with Hardware Isolation](http://arxiv.org/abs/2510.08180v1)
  > **TL;DR**: Proposes hardware isolation for serverless computing by assigning individual processors per function to avoid software overhead and idle servers. Achieves 90.63% reduction in energy consumption overheads in preliminary evaluation.
* `networking` `training` [When Light Bends to the Collective Will: A Theory and Vision for Adaptive Photonic Scale-up Domains](http://arxiv.org/abs/2510.08072v1)
  > **TL;DR**: Proposes adaptive photonic interconnects to optimize collective communications in scale-up systems. Uses dynamic reconfiguration guided by operation structure and theoretical framework balancing reconfiguration delay and performance gains. Improves maximum concurrent flow for collectives (e.g., AllReduce) via BvN decomposition.
* `serving` `MoE` `offloading` [From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill](http://arxiv.org/abs/2510.08055v1)
  > **TL;DR**: Proposes layered prefill scheduling for LLM serving, replacing token-based chunking with layer-group interleaving to reduce redundant MoE weight reloads. Achieves 70% lower TTFT, 41% lower end-to-end latency, and 22% lower per-token energy versus stall-free chunked prefill.
* `serving` `networking` `hardware` [Evaluating Rapid Makespan Predictions for Heterogeneous Systems with Programmable Logic](http://arxiv.org/abs/2510.06998v1)
  > **TL;DR**: Addresses rapid makespan prediction for task scheduling on heterogeneous accelerators (CPUs, GPUs, FPGAs). Proposes an evaluation framework using task graphs to validate analytical models against real-world data transfers and congestion. Achieves practical prediction methods for mapping efficiency.
* `edge` `serving` `offloading` [Multi-Dimensional Autoscaling of Stream Processing Services on Edge Devices](http://arxiv.org/abs/2510.06882v1)
  > **TL;DR**: Proposes MUDAP for multi-dimensional autoscaling of stream processing on resource-constrained edge devices. Combines service-level (e.g., data quality, model size) and resource-level vertical scaling with a RASK agent for regression-based action optimization. Achieves 28% fewer SLO violations than Kubernetes VPA and RL baselines.
* `kernel` `serving` `quantization` [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](http://arxiv.org/abs/2510.06834v1)
  > **TL;DR**: Optimizes FlashAttention computation by vectorizing operations and introducing low-cost exponential approximation for softmax on RISC-V processors. Combines tiling strategies for memory efficiency. Achieves significant speedups over baseline.
* `hardware` `serving` `training` [On-Package Memory with Universal Chiplet Interconnect Express (UCIe): A Low Power, High Bandwidth, Low Latency and Low Cost Approach](http://arxiv.org/abs/2510.06513v1)
  > **TL;DR**: Proposes enhancing UCIe with memory semantics for on-package memory to address AI memory walls. Approaches include reusing LPDDR6/HBM via logic die and native UCIe DRAM, yielding up to 10x bandwidth density, 3x lower latency/power, and reduced cost vs. HBM4/LPDDR.
* `training` `offline` [Adaptive Protein Design Protocols and Middleware](http://arxiv.org/abs/2510.06396v1)
  > **TL;DR**: Proposes IMPRESS, an adaptive protein design system with middleware for coupling AI/ML to HPC. Implements dynamic resource allocation and asynchronous workload execution to improve throughput. Achieves increased consistency and enhanced throughput in computational protein design tasks.
* `training` `RL` `offloading` [EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models](http://arxiv.org/abs/2510.05943v1)
  > **TL;DR**: Addresses memory inflation and OOM failures in agentic RL for LLMs. Proposes EARL with dynamic parallelism selector and layout-aware data dispatcher for adaptive model parallelism and decentralized data exchange. Reduces long-context failures and achieves stable large-scale training at scale.
* `serving` `kernel` `hardware` [The Anatomy of a Triton Attention Kernel](http://arxiv.org/abs/2511.11581v1)
  > **TL;DR**: Develops a portable paged attention kernel for LLM inference using Triton to eliminate manual tuning. The kernel achieves 105.9% of state-of-the-art performance across NVIDIA and AMD GPUs through JIT compilation and auto-tuning, improving portability and efficiency.
* `agentic` `offloading` `serving` [Toward Systems Foundations for Agentic Exploration](http://arxiv.org/abs/2510.05556v1)
  > **TL;DR**: Addresses systems challenges in supporting agentic exploration with LLMs. Identifies key issues in fork semantics, external side-effects, and native forking. Benchmarks show existing snapshot/restore mechanisms are insufficient for real deployment speeds, requiring microsecond-scale cloning without bulk copying.
* `serving` `MoE` `hardware` [Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting](http://arxiv.org/abs/2510.05497v3)
  > **TL;DR**: Addresses data movement overhead in MoE LLM serving from random expert selection. Proposes profiling-driven insights for hardware redesign, with wafer-scale GPU modifications achieving up to 5.3× speedup on DeepSeek V3.
* `networking` [cMPI: Using CXL Memory Sharing for MPI One-Sided and Two-Sided Inter-Node Communications](http://arxiv.org/abs/2510.05476v2)
  > **TL;DR**: Optimizes MPI inter-node communication using CXL memory sharing to bypass network protocols. Proposes cMPI, transforming cross-node communication into CXL memory transactions. Achieves up to 72x higher bandwidth and 49x lower latency versus TCP over Ethernet and SmartNIC for small messages.
* `training` `offloading` `kernel` [OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training](http://arxiv.org/abs/2510.05186v1)
  > **TL;DR**: Optimizes pipeline parallelism scheduling for LLM training via constrained optimization modeling memory, activation reuse, and bubble minimization. Dynamically adjusts offloading tradeoffs by hardware/model structure. Reduces idle pipeline time by up to 50% under same memory limits.
* `networking` `serving` `offline` [Next-Generation Event-Driven Architectures: Performance, Scalability, and Intelligent Orchestration Across Messaging Frameworks](http://arxiv.org/abs/2510.04404v2)
  > **TL;DR**: Studies performance and scalability of event-driven architectures for AI inference pipelines. Introduces AIEO with ML-driven predictive scaling and RL-based resource allocation. Achieves 34% latency reduction, 28% resource utilization improvement across messaging systems.
* `agentic` `serving` [Speculative Actions: A Lossless Framework for Faster Agentic Systems](http://arxiv.org/abs/2510.04371v1)
  > **TL;DR**: Addresses high-latency bottlenecks in agentic systems via speculative parallel execution of likely actions. Introduces a framework with faster models to predict multiple steps ahead, verified for losslessness. Achieves up to 55% action prediction accuracy and significant end-to-end latency reduction.
* `serving` `RAG` [SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading](http://arxiv.org/abs/2510.05164v1)
  > **TL;DR**: Addresses LLM serving cost vs. accuracy tradeoffs via SATER, a dual-mode routing framework applying response optimization and confidence-aware rejection. Reduces computational costs by over 50% and cascade latency by 80%.
* `training` `networking` `sparse` [Toward Co-adapting Machine Learning Job Shape and Cluster Topology](http://arxiv.org/abs/2510.03891v1)
  > **TL;DR**: Addresses resource allocation for distributed ML jobs in torus-topology clusters. Proposes RFold, adapting job shapes and cluster topology via homomorphic shape identification and optical circuit switch reconfiguration. Achieves 57% higher utilization and up to 11× lower job completion time.
* `training` `hardware` `serving` [Datacenter Energy Optimized Power Profiles](http://arxiv.org/abs/2510.03872v2)
  > **TL;DR**: Introduces datacenter power profiles for NVIDIA Blackwell GPUs to optimize energy efficiency in AI/HPC workloads. Employs hardware-software innovations and application-aware optimization recipes. Achieves up to 15% energy savings and 13% throughput increase within power constraints.
* `training` `networking` `kernel` [Short-circuiting Rings for Low-Latency AllReduce](http://arxiv.org/abs/2510.03491v1)
  > **TL;DR**: Challenges the assumption that Recursive Doubling is superior to Ring for small AllReduce messages in GPU clusters. Proposes a heuristic for circuit-switching photonic interconnects to dynamically balance reconfiguration delays and congestion. Achieves faster completion times than static Ring AllReduce in evaluations with realistic delays.
* `diffusion` `training` `sparse` [Paris: A Decentralized Trained Open-Weight Diffusion Model](http://arxiv.org/abs/2510.03434v1)
  > **TL;DR**: Proposes Paris, a decentralized trained diffusion model with expert partitioning for text-to-image generation. Distributes training across isolated experts with data clustering and dynamic routing, avoiding synchronization. Achieves comparable quality with 14× less data and 16× less compute than prior decentralized baseline.
* `training` `quantization` `offloading` [Energy Efficiency in Cloud-Based Big Data Processing for Earth Observation: Gap Analysis and Future Directions](http://arxiv.org/abs/2510.02882v1)
  > **TL;DR**: Addresses energy inefficiency in cloud-based big data processing for Earth Observation with foundation models. Proposes energy-aware monitoring, optimization techniques, and task scheduling for distributed frameworks. Aims to reduce power consumption and carbon footprint while maintaining processing performance.
* `serving` `edge` `offloading` [Action Deviation-Aware Inference for Low-Latency Wireless Robots](http://arxiv.org/abs/2510.02851v2)
  > **TL;DR**: Proposes Action Deviation-Aware Hybrid Inference (ADAHI) to reduce latency for robot policy inference by selectively offloading drafts to a server based on action deviation. Combines lightweight on-device model with remote verification, cutting latency by 39.2% and transmission by 40%, preserving 97.2% task-success.
* `video` `serving` `offloading` [TridentServe: A Stage-level Serving System for Diffusion Pipelines](http://arxiv.org/abs/2510.02838v1)
  > **TL;DR**: Proposes TridentServe, a dynamic stage-level serving system for diffusion pipelines that co-optimizes resource allocation across model stages and requests. It automates placement and dispatch plans, achieving up to 2.5x lower average latency and 4.1x lower P95 latency over existing systems.
* `training` `networking` `sparse` [Distributed Low-Communication Training with Decoupled Momentum Optimization](http://arxiv.org/abs/2510.03371v1)
  > **TL;DR**: Reduces communication overhead in distributed LLM training by decomposing Nesterov momentum into high/low-frequency components via DCT, synchronizing only high-frequency components every H steps. Achieves 16x less communication than DiLoCo baseline while maintaining generalization across architectures.
* `hardware` `kernel` `quantization` [UPMEM Unleashed: Software Secrets for Speed](http://arxiv.org/abs/2510.15927v1)
  > **TL;DR**: Examines software inefficiencies in UPMEM PIM platforms for low-precision kernels. Proposes assembly-level optimizations, bit-serial processing INT4/INT8, and NUMA-aware allocations. Achieves 1.4-5.9x arithmetic speedups and over 3x faster INT8 GEMV vs CPU after optimizations.
* `serving` `MoE` `offloading` [ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models](http://arxiv.org/abs/2510.02613v1)
  > **TL;DR**: Addresses elastic scaling challenges for MoE model serving. Proposes ElasticMoE with decoupled execution-memory operations and zero-copy KV cache reuse for concurrent scaling. Achieves up to 9x lower scale-up latency and 2x higher throughput during scaling.
* `edge` `offloading` `serving` [Accuracy vs Performance: An abstraction model for deadline constrained offloading at the mobile-edge](http://arxiv.org/abs/2510.01885v1)
  > **TL;DR**: Proposes a deadline-aware scheduling algorithm for DNN offloading to mobile edge devices, using lightweight network state representation and dynamic bandwidth estimation. Improves task throughput under resource scarcity and reduces latency in high-volume workloads.
* `edge` `serving` `RL` [Percepta: High Performance Stream Processing at the Edge](http://arxiv.org/abs/2510.05149v1)
  > **TL;DR**: Proposes Percepta, a lightweight edge stream processing system for AI workloads, focusing on reinforcement learning. It includes reward computation, real-time data prep, and handles normalization/harmonization. Achieves reduced latency and efficient data handling for continuous decision-making at the edge.
* `training` `serving` `storage` [Semantic-Aware Scheduling for GPU Clusters with Large Language Models](http://arxiv.org/abs/2510.03334v1)
  > **TL;DR**: Proposes SchedMate, a semantic-aware GPU cluster scheduler using LLMs to analyze unstructured data (source code, logs, history). Integrates with existing schedulers to reduce profiling overhead and improve estimates. Achieves up to 1.91× reduction in average job completion time.
* `serving` `diffusion` `edge?` [TetriServe: Efficient DiT Serving for Heterogeneous Image Generation](http://arxiv.org/abs/2510.01565v2)
  > **TL;DR**: Proposes TetriServe, a DiT serving system using step-level sequence parallelism to adapt parallelism per request based on deadlines. Implements round-based scheduling to minimize GPU hours and late completions. Achieves 32% higher SLO attainment.
* `agentic` `thinking` `serving` [FlashResearch: Real-time Agent Orchestration for Efficient Deep Research](http://arxiv.org/abs/2510.05145v1)
  > **TL;DR**: Addresses inefficiency in deep research agents via runtime parallelization. Introduces adaptive query breakdown, orchestration pruning, and multidimension concurrency. Achieves 5x speedup while maintaining report quality under fixed time budget.
* `training` `networking` `kernel` [An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters](http://arxiv.org/abs/2510.00991v1)
  > **TL;DR**: Addresses inefficiency, reliability, and observability in collective communication for large-scale LLM training. Proposes ICCL, a library that offloads P2P communication to CPU, adds NIC failure tolerance, and introduces fine-grained monitoring. Achieves 23.4%/28.5% better P2P throughput/latency and 6.02% higher training throughput versus NCCL.
* `training` `offloading` `kernel` [ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training](http://arxiv.org/abs/2510.00606v3)
  > **TL;DR**: Addresses fault tolerance and elasticity in large-scale LLM training. Proposes ElasWave with multi-dimensional scheduling, online pipeline resharding, and DVFS to maintain consistency and reduce recovery time. Achieves up to 1.60x higher throughput and 51% lower migration MTTR.
* `edge` `serving` `RAG` [PolyLink: A Blockchain Based Decentralized Edge AI Platform for LLM Inference](http://arxiv.org/abs/2510.02395v1)
  > **TL;DR**: Proposes PolyLink, a blockchain-based decentralized platform for LLM inference at the edge. Uses crowdsourced devices, TIQE protocol for integrity, and token incentives. Achieves low verification latency and resists security attacks in geo-distributed deployment.
* `MoE` `training` `offloading` [FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training](http://arxiv.org/abs/2510.00207v2)
  > **TL;DR**: Addresses inefficiencies in distributed MoE training pipelines. Proposes FlowMoE, a unified scheduling framework integrating MHA, gating, expert computing, and communications via tensor-chunk prioritization. Reduces training time by 13%-57%, energy by 10%-39%, and memory by 7%-32%.
* `training` `kernel` `quantization` [LoRAFusion: Efficient LoRA Fine-Tuning for LLMs](http://arxiv.org/abs/2510.00206v1)
  > **TL;DR**: Presents LoRAFusion, optimizing LoRA fine-tuning via fused kernels and adaptive batching. Addresses redundant memory access and concurrent multi-job scheduling. Achieves up to 1.96x end-to-end speedup over Megatron-LM.
* `networking` `training` `edge` [Lattica: A Decentralized Cross-NAT Communication Framework for Scalable AI Inference and Training](http://arxiv.org/abs/2510.00183v2)
  > **TL;DR**: Proposes Lattica, a decentralized cross-NAT communication framework using P2P mesh, CRDTs, and DHTs for scalable AI inference and training in heterogeneous environments. Achieves sovereign operation and efficient model synchronization.
* `training` `sparse` `kernel` [TASP: Topology-aware Sequence Parallelism](http://arxiv.org/abs/2509.26541v2)
  > **TL;DR**: Proposes topology-aware sequence parallelism for long-context LLMs to improve communication efficiency. Utilizes Hamiltonian decomposition to map Ring AllGather to AlltoAll topology via concurrent ring datapaths. Achieves 3.58x speedup over Ring Attention.
* `offline` `hardware` `training` [Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework](http://arxiv.org/abs/2509.26534v1)
  > **TL;DR**: Proposes a holistic TCO-driven framework for AI datacenter lifecycle management across building, hardware refresh, and operation stages. Coordinates design choices in power, cooling, networking, refresh strategies, and software optimizations. Achieves up to 40% reduction in total cost of ownership.
* `serving` `offloading` `networking` [Parallax: Efficient LLM Inference Service over Decentralized Environment](http://arxiv.org/abs/2509.26182v1)
  > **TL;DR**: Addresses efficient LLM inference in decentralized environments with heterogeneous GPU resources. Proposes Parallax, a two-phase scheduler that optimizes model allocation and request-time GPU pipeline selection. Reduces latency and increases throughput over decentralized baselines in real volunteer node deployments.
* `serving` `offline` `edge` [Accelerating LLM Inference with Precomputed Query Storage](http://arxiv.org/abs/2509.25919v1)
  > **TL;DR**: Proposes StorInfer, which precomputes and stores query-response pairs to bypass GPU inference for matched queries. Uses adaptive LLM-driven query generation and disk-backed vector DB for fast retrieval. Achieves 17.3% latency reduction with 150K pairs using 830 MB storage.
* `agentic` `edge` `offloading` [Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey](http://arxiv.org/abs/2510.00078v1)
  > **TL;DR**: Surveys adaptive and resource-efficient agentic AI systems for mobile/embedded devices, covering techniques like elastic inference and test-time adaptation. Focuses on balancing model complexity with constraints like memory, energy, and latency in edge deployments.
* `kernel` `hardware` `sparse` [LAPIS: A Performance Portable, High Productivity Compiler Framework](http://arxiv.org/abs/2509.25605v1)
  > **TL;DR**: Proposes LAPIS, an MLIR-based compiler framework for high performance portability across architectures. Focuses on automatic lowering of sparse/dense linear algebra kernels from scientific and AI use cases, facilitating integration of PyTorch and Kokkos. Achieves comparable kernel performance to MLIR on diverse architectures.
* `training` `security` `edge` [Enhancing Split Learning with Sharded and Blockchain-Enabled SplitFed Approaches](http://arxiv.org/abs/2509.25555v1)
  > **TL;DR**: Proposes Sharded SplitFed Learning (SSFL) and Blockchain-enabled SplitFed (BSFL) to address scalability, performance, and security in federated settings. SSFL distributes server workload across shards; BSFL adds blockchain consensus for integrity. Achieves 85.2% scalability improvement and 62.7% higher attack resilience.
* `edge` `serving` `kernel` [Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units](http://arxiv.org/abs/2509.25155v2)
  > **TL;DR**: Analyzes performance bottlenecks of causal inference operators on NPUs for edge LLM deployment. Benchmarks quadratic attention vs. sub-quadratic alternatives under architectural constraints. Finds attention becomes memory-bound with cache inefficiency; achieves profiling insights for 4-10× optimization in co-design approaches.
* `hardware` `kernel` `offloading` [Accelerating Dynamic Image Graph Construction on FPGA for Vision GNNs](http://arxiv.org/abs/2509.25121v1)
  > **TL;DR**: Addresses the bottleneck of dynamic image graph construction in Vision GNNs. Proposes a streaming FPGA accelerator with on-chip buffers and parallel sorting to minimize memory traffic. Achieves up to 16.6x speedup over CPU baselines.
* `training` `kernel` `storage` [A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration](http://arxiv.org/abs/2509.25044v1)
  > **TL;DR**: Proposes FFDP with fused kernels and distributed framework for large-scale image registration. Optimizes non-GEMM bottlenecks with IO-aware kernels and convolution-aware sharding. Achieves 7x speedup and 59% peak memory reduction in multimodal brain registration.
* `MoE` `serving` `offloading` [GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference](http://arxiv.org/abs/2509.25041v2)
  > **TL;DR**: Addresses communication overhead and computational load imbalance in distributed MoE inference. Proposes GRACE-MoE: expert grouping/replication to reduce cross-device transfers and locality-aware routing with load prediction. Achieves up to 3.79x inference speedup over SOTA systems.
* `MoE` `serving` `networking` [From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing](http://arxiv.org/abs/2510.03293v1)
  > **TL;DR**: Proposes LASER, a plug-and-play routing algorithm for MoE inference that adapts routing based on gate score distributions to balance expert load. Achieves better load balancing, reducing latency by 1.2-2.1x and increasing throughput by 1.6-2.3x while maintaining accuracy across datasets.
* `training` `MoE` [HAPT: Heterogeneity-Aware Automated Parallel Training on Heterogeneous Clusters](http://arxiv.org/abs/2509.24859v1)
  > **TL;DR**: Presents Hapt, an automated fine-grained inter-operator parallel training framework for heterogeneous clusters. It includes a planner for load balancing and a scheduler for optimized computation-communication overlap. Achieves 1.3x-1.6x higher performance vs. state-of-the-art frameworks.
* `RAG` `storage` `serving` [Intent-Driven Storage Systems: From Low-Level Tuning to High-Level Understanding](http://arxiv.org/abs/2510.15917v1)
  > **TL;DR**: Proposes Intent-Driven Storage Systems (IDSS) using LLMs to infer workload intent for adaptive storage parameter reconfiguration. Integrates LLMs into control loops with policy guardrails, generating configurations for components like caching. Achieves up to 2.45x IOPS improvement on FileBench workloads.
* `serving` `offloading` `sparse` [SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving](http://arxiv.org/abs/2509.24626v1)
  > **TL;DR**: Reduces costs of serving long-context LLMs by optimizing dynamic sparse attention and hierarchical KV cache management. Introduces fragmentation-aware transfers, working-set batch control, and layer-segmented prefill to minimize HBM constraints and avoid thrashing. Achieves 9.26x lower TTFT latency and 3.14x higher throughput compared to state-of-the-art systems.
* `multi-modal` `serving` [RServe: Overlapping Encoding and Prefill for Efficient LMM Inference](http://arxiv.org/abs/2509.24381v1)
  > **TL;DR**: Addresses inefficiency in serving large multimodal models (LMMs) by proposing REDServe, which overlaps multimodal encoding with prefill computations and balances loads via token budgeting. Achieves up to 66% lower latency and 109% higher throughput than existing systems.
* `training` `RL` `networking` [Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning](http://arxiv.org/abs/2509.24305v1)
  > **TL;DR**: Proposes asynchronous policy gradient aggregation algorithms (Rennala/Malenia NIGT) for efficient distributed RL. Addresses heterogeneous computations and communication bottlenecks, with improved theoretical complexity and experiments showing significant speedup over prior methods.
* `training` `RL` `offloading` [RL in the Wild: Characterizing RLVR Training in LLM Deployment](http://arxiv.org/abs/2509.25279v2)
  > **TL;DR**: Characterizes system challenges in RLVR training for LLMs, identifying GPU idling from skewed sequence lengths, inefficient parallelism, and load imbalance. Proposes the PolyTrace benchmark suite for realistic evaluation, achieving 94.7% accuracy in validation.
* `edge` `serving` `training` [MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment](http://arxiv.org/abs/2510.03283v1)
  > **TL;DR**: Proposes MACE, an edge LLM serving system with colocated SLO-aware continuous retraining. Uses iteration-level fine-tuning scheduling and memory management to balance inference latency and model accuracy. Reduces inference latency by up to 63% while maintaining throughput and high GPU utilization.
* `training` `MoE` [AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models](http://arxiv.org/abs/2509.23722v1)
  > **TL;DR**: Addresses pipeline bubbles in LLM training on heterogeneous models. Proposes AdaPtis, which jointly optimizes model partition, placement and scheduling using a performance model. Achieves an average 1.42x speedup over Megatron-LM.
* `serving` `offline` `edge` [A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving](http://arxiv.org/abs/2509.23384v3)
  > **TL;DR**: Proposes NexusSched, a two-layer scheduling framework for LLM inference serving. Uses a performance model for per-step latency prediction to enable adaptive engine scheduling and predictive cluster routing. Achieves 43% higher SLO attainment and up to 3x throughput speedup in heterogeneous scenarios.
* `edge` `quantization` `serving` [Scaling LLM Test-Time Compute with Mobile NPU on Smartphones](http://arxiv.org/abs/2509.23324v1)
  > **TL;DR**: Addresses inefficient LLM inference on mobile NPUs by proposing a hardware-aware tile quantization and LUT-based operators for test-time scaling. Achieves up to 19.0x speedup for GEMM and enables smaller models to match larger model accuracy with lower cost.
* `training` `sparse` [A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training](http://arxiv.org/abs/2510.05112v2)
  > **TL;DR**: Proposes FlexPipe, a programmable pipeline parallelism framework with a DSL and automated scheduler for efficient DNN training. Enables automated exploration of diverse pipeline schedules and customizations. Achieves up to 2.28X speedup over Megatron-LM.
* `MoE` `training` `offloading` [Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression](http://arxiv.org/abs/2510.02345v1)
  > **TL;DR**: Addresses load imbalance, redundancy, and communication in MoE LLMs via dynamic expert clustering and structured compression. Introduces online clustering with routing-based regrouping and low-rank residual adapters, enabling hierarchical routing and heterogeneous precision. Achieves 80% parameter reduction, 10-20% higher throughput, and 3x lower load variance.
* `training` [Memory Efficient and Staleness Free Pipeline Parallel DNN Training Framework with Improved Convergence Speed](http://arxiv.org/abs/2509.23241v1)
  > **TL;DR**: Proposes V-TiMePReSt and I-TiMePReSt frameworks for pipeline-parallel DNN training to eliminate weight staleness and improve memory efficiency. V-TiMePReSt ensures staleness-free training with latest weights, while I-TiMePReSt uses intermediate weights to balance memory and convergence. Achieves improved GPU memory efficiency and optimal convergence speed.
* `training` `kernel` `hardware` [Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM](http://arxiv.org/abs/2509.22832v1)
  > **TL;DR**: Proposes a fine-grained GPU performance modeling system for distributed LLM training. Combines operator-level decomposition, lightweight hardware-aware prediction, and end-to-end integration for parallelism. Achieves 4.98% prediction error on A100 and 9.38% on GH200 at 20B/128GPU scale without cluster testing.
* `serving` `storage` `hardware` [Agora: Bridging the GPU Cloud Resource-Price Disconnect](http://arxiv.org/abs/2510.05111v1)
  > **TL;DR**: Addresses market inefficiency in cloud GPU pricing for bandwidth-bound workloads. Proposes Agora, a feature-based pricing framework linked to resource consumption like memory bandwidth. Implementation with 10us sampling shows only 2.4% revenue loss compared to ideal sampling.
* `offloading` `training` `hardware` [The AI_INFN Platform: Artificial Intelligence Development in the Cloud](http://arxiv.org/abs/2509.22117v2)
  > **TL;DR**: Presents AI_INFN, a Kubernetes-based platform for efficient GPU resource sharing in AI workflows. Integrates offloading via Virtual Kubelet and InterLink API to span resources across distributed providers. Demonstrated scalable execution on heterogeneous resources, including WLCG sites and HPC centers.
* `training` `networking` `sparse` [Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training](http://arxiv.org/abs/2509.21841v2)
  > **TL;DR**: Addresses load imbalance in large-scale data-parallel training of LLMs with variable sequence lengths. Proposes Zeppelin with hierarchical sequence partitioning, dynamic routing for NIC utilization, and layout remapping. Achieves 2.80x average speedup over SOTA methods.
* `multi-modal` `edge` `offloading` [Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices](http://arxiv.org/abs/2510.05109v3)
  > **TL;DR**: Addresses high latency and inefficiency in monolithic execution of LMMs on heterogeneous battery-powered devices. Proposes NanoMind: hardware-software co-design with modular offloading to accelerators and token-aware memory management. Achieves 42.3% lower energy consumption and 20.8-hour runtime on one charge.
* `training` `sparse` `offloading` [Data-Centric Elastic Pipeline Parallelism for Efficient Long-Context LLM Training](http://arxiv.org/abs/2509.21275v2)
  > **TL;DR**: Proposes Elastic Pipeline Parallelism (EPP) and InfiniPipe system to optimize pipeline parallelism granularity for long-context LLM training. Adaptively combines token-level and batch-level PP with sequence packing and adaptive checkpointing. Achieves 1.69x speedup over state-of-the-art systems.
* `training` `offloading` `hardware` [SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips](http://arxiv.org/abs/2509.21271v1)
  > **TL;DR**: Proposes SuperOffload, a Superchip-centric offloading system for large-scale LLM training. Introduces adaptive weight offloading, bucket repartitioning, and optimized Adam for Grace CPUs. Achieves 2.5x throughput improvement, enabling 25B model training on a single NVIDIA GH200 Superchip.
* `training` `offloading` `networking` [Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models](http://arxiv.org/abs/2509.21221v1)
  > **TL;DR**: Proposes GWTF, a churn-tolerant decentralized framework for collaborative LLM training on volunteer clients. Introduces a flow algorithm to optimize routing of microbatches amid node churn and network instability. Reduces training time by up to 45% in heterogeneous, high-churn environments across 10 locations.
* `hardware` `offloading` `training` [From GPUs to RRAMs: Distributed In-Memory Primal-Dual Hybrid Gradient Method for Solving Large-Scale Linear Optimization Problem](http://arxiv.org/abs/2509.21137v3)
  > **TL;DR**: Proposes a distributed in-memory PDHG method for large-scale linear optimization using RRAM arrays to reduce energy and latency. Minimizes write cycles and unifies operations across crossbars; achieves up to 1000× energy and latency reductions versus GPUs while maintaining accuracy.
* `kernel` `hardware` [Mojo: MLIR-Based Performance-Portable HPC Science Kernels on GPUs for the Python Ecosystem](http://arxiv.org/abs/2509.21039v1)
  > **TL;DR**: Explores Mojo, an MLIR-based language for efficient science kernels on GPUs. Combines Python compatibility with low-level optimizations for cross-platform GPU portability. Achieves competitive performance to CUDA/HIP on memory-bound kernels, reducing development time while maintaining efficiency.
* `RL` `training` `sparse` [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](http://arxiv.org/abs/2509.21009v1)
  > **TL;DR**: Addresses GPU underutilization in synchronous RL post-training for LLMs. Proposes RollPacker with tail batching to consolidate long-tail responses into designated steps. Achieves 2.03x-2.56x end-to-end training speedup on Qwen2.5 models.
* `edge` `RAG` [IoT-MCP: Bridging LLMs and IoT Systems Through Model Context Protocol](http://arxiv.org/abs/2510.01260v1)
  > **TL;DR**: Proposes IoT-MCP, a framework using Model Context Protocol for standardized communication between LLMs and IoT systems via edge servers. Achieves 100% task success rate and 205ms avg response time on complex tasks with only 74KB peak memory footprint on microcontrollers.
* `serving` [Prompt-Aware Scheduling for Low-Latency LLM Serving](http://arxiv.org/abs/2510.03243v2)
  > **TL;DR**: Tackles Head-of-Line blocking in LLM inference scheduling by introducing Prompt-Aware Scheduler (PARS), which approximates shortest-job-first via pairwise ranking. Integrated into vLLM, PARS predicts response lengths to reduce latency. Experiments show significant latency reduction with minimal overhead across various LLMs.
* `training` `inference` `edge` [Integrating and Characterizing HPC Task Runtime Systems for hybrid AI-HPC workloads](http://arxiv.org/abs/2509.20819v1)
  > **TL;DR**: Studies how to manage hybrid AI-HPC workloads combining MPI, training, and inference. Integrates RADICAL-Pilot with Flux and Dragon for hierarchical resource management. Achieves over 1,500 tasks/s with 99.6% utilization, reducing makespan by 30-60% and increasing throughput 4x on Frontier.
* `training` `serving` `storage` [Kant: An Efficient Unified Scheduling System for Large-Scale AI Clusters](http://arxiv.org/abs/2510.01256v1)
  > **TL;DR**: Proposes Kant, a unified scheduler for co-scheduling LLM training and inference jobs in large-scale AI clusters. Utilizes Backfill and E-Binpack strategies to optimize resource utilization and reduce fragmentation. Achieves higher GPU Allocation Ratio and Scheduling Occupancy Rate.
* `serving` `RAG` [Experience Deploying Containerized GenAI Services at an HPC Center](http://arxiv.org/abs/2509.20603v2)
  > **TL;DR**: Explores deployment of GenAI services, including LLM inference servers, at an HPC center. Proposes a converged HPC and Kubernetes architecture for containerized workloads using vLLM for inference. Achieves reproducible multi-platform deployment across different container runtimes.
* `training` `offline` `storage` [FZModules: A Heterogeneous Computing Framework for Customizable Scientific Data Compression Pipelines](http://arxiv.org/abs/2509.20563v1)
  > **TL;DR**: Addresses high data volume challenges in scientific computing with customizable compression pipelines. Proposes FZModules, a heterogeneous framework with asynchronous task execution and dependency management for modular lossy compression. Achieves significant end-to-end speedup comparable to GPU compressors with improved rate-distortion fidelity.
* `scheduling` `RL` `training` [Adaptive Approach to Enhance Machine Learning Scheduling Algorithms During Runtime Using Reinforcement Learning in Metascheduling Applications](http://arxiv.org/abs/2509.20520v1)
  > **TL;DR**: Proposes an online reinforcement learning unit for metaschedulers to adapt AI-based scheduling during runtime. Integrates RL models to discover new scheduling solutions and optimize existing ones under dynamic events. Achieves continuous refinement for robustness in safety-critical environments.
* `RL` `edge` `networking` [Reconstruction-Based Adaptive Scheduling Using AI Inferences in Safety-Critical Systems](http://arxiv.org/abs/2509.20513v1)
  > **TL;DR**: Proposes a reconstruction framework to dynamically validate and assemble schedules in time-triggered safety-critical systems. Transforms AI-generated or heuristic scheduling priorities into executable schedules with safety checks and recovery mechanisms. Improves system adaptability and runtime performance, achieving significant enhancement in operational integrity.
* `serving` `quantization` `hardware` [Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute](http://arxiv.org/abs/2509.20241v1)
  > **TL;DR**: Estimates per-query energy use for large-scale LLM inference with token throughput methodology. Analyzes efficiency gains via model-level optimizations, hardware improvements, and serving platform enhancements. Achieves up to 20x reduction in energy per query with combined interventions.
* `training` `inference` `edge` [Fulcrum: Optimizing Concurrent DNN Training and Inferencing on Edge Accelerators](http://arxiv.org/abs/2509.20205v1)
  > **TL;DR**: Addresses optimizing concurrent DNN training and inference on edge accelerators under power and latency constraints. Proposes Fulcrum scheduler with gradient descent (GMD) and active learning (ALS) techniques to interleave workloads and select power modes. Satisfies latency/power budgets in 97% of runs, achieving within 7% of optimal throughput.
* `edge` `serving` `hardware` [Pagoda: An Energy and Time Roofline Study for DNN Workloads on Edge Accelerators](http://arxiv.org/abs/2509.20189v1)
  > **TL;DR**: Studies power-performance trade-offs for DNN workloads on Nvidia Jetson edge accelerators. Develops energy and time roofline models coupled with analytical workload analysis. Achieves up to 15% energy reduction with minimal latency degradation by tuning power modes.
* `training` `sparse` `kernel` [BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens](http://arxiv.org/abs/2509.19836v1)
  > **TL;DR**: Proposes BurstEngine, a distributed framework with topology-aware communication (BurstAttention), fine-grained overlap, selective checkpointing, and workload balance for efficient training on sequences >1M tokens. Achieves 1.2× speedup with lower memory overhead than SOTA baselines.
* `serving` `offloading` `kernel` [Gyges: Dynamic Cross-Instance Parallelism Transformation for Efficient LLM Inference](http://arxiv.org/abs/2509.19729v1)
  > **TL;DR**: Addresses throughput degradation due to context length variance in LLM serving. Proposes Gyges, a system featuring KV cache layout optimizations, weight padding, and transformation-aware scheduling for dynamic parallelism adjustments. Achieves 1.75x-6.57x higher throughput versus state-of-the-art solutions.
* `kernel` `serving` `training` [gpu_ext: Extensible OS Policies for GPUs via eBPF](http://arxiv.org/abs/2512.12615v2)
  > **TL;DR**: Proposes gpu_ext, an eBPF-based runtime for extensible GPU resource management policies in drivers and devices. Enables safe programmability and fine-grained control for events like memory placement and scheduling. Achieves up to 4.8x throughput improvement and 2x tail latency reduction for inference and training workloads.
* `kernel` `serving` `offloading` [Principled Performance Tunability in Operating System Kernels](http://arxiv.org/abs/2512.12530v1)
  > **TL;DR**: Addresses inefficient in-situ tuning of performance-critical constants (perf-consts) in Linux kernels. Introduces KernelX using Scoped Indirect Execution (SIE) to safely transform perf-consts into runtime-tunable knobs. Achieves millisecond-scale policy updates with significant performance improvements in kernel subsystems.
* `training` `kernel` [VLCs: Managing Parallelism with Virtualized Libraries](http://arxiv.org/abs/2512.04320v1)
  > **TL;DR**: Addresses library contention in parallel computing by proposing Virtual Library Contexts (VLCs) that encapsulate and isolate library resources without code changes. Enables resource partitioning and parallel execution. Achieves up to 2.85x speedup in benchmarks with OpenMP, OpenBLAS, and LibTorch.
* `edge` `offloading` `storage` [SARA: A Stall-Aware Memory Allocation Strategy for Mixed-Criticality Systems](http://arxiv.org/abs/2511.19991v1)
  > **TL;DR**: Proposes SARA, a stall-aware real-time memory allocator for mixed-criticality edge systems. It balances memory allocation between soft RT tasks and non-RT applications, minimizes soft RT memory usage by modeling latency effects, and mitigates stalls via proactive job dropping. Achieves 97.13% deadline hit ratio and up to 22.32x throughput improvement under memory constraints.
* `training` `storage` [Crash-Consistent Checkpointing for AI Training on macOS/APFS](http://arxiv.org/abs/2511.18323v1)
  > **TL;DR**: Investigates crash-consistent checkpointing protocols for AI training on macOS/APFS. Implements three write modes with increasing durability and a SHA-256-based integrity guard for auto-rollback. Detects 99.8-100% corruptions with zero false positives at 56.5-570.6% overhead versus unsafe baseline.
* `kernel` `edge` `storage` [eBPF-PATROL: Protective Agent for Threat Recognition and Overreach Limitation using eBPF in Containerized and Virtualized Environments](http://arxiv.org/abs/2511.18155v1)
  > **TL;DR**: Introduces eBPF-PATROL, a lightweight runtime security agent using eBPF to monitor system calls in containerized/virtualized environments. It enforces user-defined policies with context awareness, achieving <2.5% overhead and high detection accuracy against real-world attacks.
* `offloading` `hardware` `kernel` [Taiji: A DPU Memory Elasticity Solution for In-production Cloud Environments](http://arxiv.org/abs/2511.09936v2)
  > **TL;DR**: Proposes Taiji, a memory elasticity solution for DPUs using hybrid virtualization and parallel memory swapping to enable memory overcommitment. Achieves over 50% memory expansion with 5% virtualization overhead and 90% swap-ins under 10μs.
* `training` `offloading` `storage` [GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training](http://arxiv.org/abs/2511.07035v1)
  > **TL;DR**: Addresses checkpoint saving overhead in LLM training by overlapping multi-step transfers with gradient data. Proposes GoCkpt for partial checkpoint updates with GPU-CPU optimization. Achieves 38.4% higher throughput and 86.7% less interruption time versus traditional methods.
* `hardware` `storage` `kernel` [Guidelines for Building Indexes on Partially Cache-Coherent CXL Shared Memory](http://arxiv.org/abs/2511.06460v1)
  > **TL;DR**: Investigates efficient indexing on Partial Cache-Coherence (PCC) platforms like CXL. Proposes SP and P$^3$ guidelines to ensure correctness and mitigate overhead. Achieves up to 16× throughput gain versus non-optimized designs.
* `serving` `offloading` `agentic` [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](http://arxiv.org/abs/2511.02230v2)
  > **TL;DR**: Addresses inefficient KV cache eviction in LLM serving for multi-turn agentic workloads. Proposes Continuum with KV cache time-to-live mechanism to selectively retain KV cache during tool call pauses. Reduces average job completion time by up to 32% on real-world agentic benchmarks.
* `offloading` [Jenga: Responsive Tiered Memory Management without Thrashing](http://arxiv.org/abs/2510.22869v1)
  > **TL;DR**: Addresses tiered memory performance issues due to hot/cold object mixing and hotness fluctuation. Proposes Jenga with context-based page allocation and stable hotness measurement for timely migration without thrashing. Runs applications 28% faster with fast tier at working set size.
* `storage` `kernel` `serving` [LatticeHashForest: An Efficient Data Structure for Repetitive Data and Operations](http://arxiv.org/abs/2510.18496v2)
  > **TL;DR**: Presents LatticeHashForest (LHF), a data structure for efficient deduplication and computation in repetitive operations, such as compiler optimizations. LHF reduces memory usage and speeds up operations by enabling immediate deduplication and nested construction. In pointer analysis, it achieves over 4x speedup and negligible memory for 10 million inputs.
* `hardware` `training` `offline` [Funky: Cloud-Native FPGA Virtualization and Orchestration](http://arxiv.org/abs/2510.15755v1)
  > **TL;DR**: Addresses FPGA virtualization and orchestration challenges in cloud-native environments. Proposes Funky, a full-stack FPGA-aware orchestration engine with virtualization, state management, and standard-compliant components. Achieves 7.4% performance overhead vs native with strong isolation and 28.7x smaller images.
* `agentic` `offloading` `serving` [Toward Systems Foundations for Agentic Exploration](http://arxiv.org/abs/2510.05556v1)
  > **TL;DR**: Addresses the lack of systems support for agentic exploration in LLMs. Identifies limitations of snapshot/restore mechanisms and proposes fundamental challenges: fork semantics, side-effect management, and native forking. Benchmarks show generic tools fail in real deployments.
* `storage` `kernel` [An Early Exploration of Deep-Learning-Driven Prefetching for Far Memory](http://arxiv.org/abs/2510.04360v1)
  > **TL;DR**: Explores deep-learning-driven prefetching for far-memory systems to reduce on-demand data fetches. Proposes Memix, co-designing deep learning with system architecture to predict memory accesses via application semantics and runtime context. Achieves up to 42% fewer far-memory accesses than state-of-the-art.
* `RL` `agentic` `offloading` [Secure and Efficient Access Control for Computer-Use Agents via Context Space](http://arxiv.org/abs/2509.22256v2)
  > **TL;DR**: Addresses security risks in LLM-based computer-use agents by developing CSAgent, a system-level access control framework with intent- and context-aware policies enforced via an optimized OS service. Achieves 99.36% attack defense rate with 6.83% overhead.
* `serving` `offloading` `multi-modal` [Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization](http://arxiv.org/abs/2509.21301v1)
  > **TL;DR**: Proposes Nova, a real-time scheduling framework for serving agentic vision-language models on a single GPU. Utilizes adaptive cross-stage parallelization with elastic GPU partitioning and Pareto-optimal resource calibration, plus vision encoder weight offloading. Reduces maximum latency by up to 23.3% while maintaining competitive throughput.
* `training` `storage` [Preparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca](http://arxiv.org/abs/2511.13724v1)
  > **TL;DR**: Addresses input data preprocessing bottlenecks in concurrent ML training. Proposes Seneca, a data loading system featuring cache partitioning for encoded/decoded/augmented data and opportunistic cached data sampling. Reduces makespan by 45.23% and increases throughput by 3.45x over baselines.

### 2025-12-20
* `training` `serving` `networking` [RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference](http://arxiv.org/abs/2512.19606v1)
  > **TL;DR**: Proposes RAPID-LLM, a unified performance modeling framework for LLM training and inference on GPU clusters. Combines DeepFlow-based frontend and Astra-Sim backend to simulate hardware-aware execution with network faults. Predicts latency within 10.4% of measurements, enabling configuration sweeps and resilience analysis.
* `serving` `offline` `networking` [Faster Distributed Inference-Only Recommender Systems via Bounded Lag Synchronous Collectives](http://arxiv.org/abs/2512.19342v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed recommender system inference. Proposes bounded lag synchronous (BLS) alltoallv collective with adjustable lag bounds to mask process delays. Achieves improved latency and throughput in unbalanced scenarios, masking delays entirely in best cases.
* `serving` `offloading` `scheduling` [L4: Low-Latency and Load-Balanced LLM Serving via Length-Aware Scheduling](http://arxiv.org/abs/2512.19179v1)
  > **TL;DR**: Addresses GPU underutilization and latency in LLM serving due to request-length heterogeneity. Proposes L4, a runtime system for dynamic request rescheduling and instance partitioning based on length groups. Achieves up to 69% lower tail latency and 2.89× throughput improvement over state-of-the-art schedulers.
* `edge` `offline` `RL` [Evidential Trust-Aware Model Personalization in Decentralized Federated Learning for Wearable IoT](http://arxiv.org/abs/2512.19131v1)
  > **TL;DR**: Proposes Murmura, a trust-aware model personalization framework for decentralized federated learning on edge devices. Uses evidential deep learning to compute compatibility scores via cross-evaluation and adaptive aggregation. Achieves 7.4× faster convergence vs IID degradation in wearable IoT datasets.
* `edge` `networking` `serving` [QoS-Aware Load Balancing in the Computing Continuum via Multi-Player Bandits](http://arxiv.org/abs/2512.18915v1)
  > **TL;DR**: Proposes QEdgeProxy, a decentralized QoS-aware load balancer for edge computing, modeled as a Multi-Player MAB with kernel density estimation for per-client QoS. Kubernetes implementation achieves improved per-client QoS satisfaction in latency-sensitive workloads.
* `MoE` `offloading` `serving` [Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing](http://arxiv.org/abs/2512.18674v1)
  > **TL;DR**: Addresses high cost and memory overhead in MoE inference for serverless computing. Proposes Remoe, a heterogeneous system that assigns non-expert modules to GPUs and experts to CPUs, with SPS for activation prediction and MMP for SLO compliance. Reduces cost by 57% and cold start latency by 47%.
* `serving` `offloading` `kernel` [Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systems](http://arxiv.org/abs/2512.18318v1)
  > **TL;DR**: Proposes an asynchronous pipeline-parallel Transformer framework for real-time multilingual lip synchronization. Employs message-queue decoupling, low-level graph compilation, mixed-precision quantization, and kernel fusion to reduce latency. Achieves up to 3.1× lower end-to-end latency than sequential approaches.
* `serving` `offloading` `networking` [TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale](http://arxiv.org/abs/2512.18194v1)
  > **TL;DR**: Proposes TraCT, a rack-scale LLM serving system using CXL shared memory for KV cache, avoiding RDMA networks. Addresses synchronization and consistency via software solutions like two-tier synchronization. Achieves 9.8x lower TTFT, 6.2x lower P99 latency, and 1.6x higher throughput.
* `training` `networking` `sparse` [ACE-Sync: An Adaptive Cloud-Edge Synchronization Framework for Communication-Efficient Large-Scale Distributed Model Training](http://arxiv.org/abs/2512.18127v1)
  > **TL;DR**: Proposes ACE-Sync, an adaptive cloud-edge synchronization framework with attention-based gradient importance prediction, differentiated compression, and hierarchical coordination to reduce communication in distributed training. Reduces communication cost from 112.5 GB to 44.7 GB (60% reduction) while maintaining model accuracy within 0.3%.
* `serving` `training` `multi-modal` [Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing](http://arxiv.org/abs/2512.17574v1)
  > **TL;DR**: Addresses bottlenecks in multi-stage MLLM inference pipelines. Proposes FlashCodec for GPU-accelerated video decoding and UnifiedServe for resource sharing and decoupled execution. Achieves up to 4.4× higher throughput and serves 3.0× more requests compared to SOTA systems.
* `edge` `sparse` `networking` [Adaptive Graph Pruning with Sudden-Events Evaluation for Traffic Prediction using Online Semi-Decentralized ST-GNNs](http://arxiv.org/abs/2512.17352v1)
  > **TL;DR**: Proposes adaptive graph pruning to reduce communication overhead in online semi-decentralized ST-GNNs for traffic prediction. Dynamically filters redundant neighbor features based on model performance and event responsiveness. Reduces communication cost by 20-40% while maintaining accuracy via novel SEPA metric.
* `storage` `offline` `serving` [Scalable Distributed Vector Search via Accuracy Preserving Index Construction](http://arxiv.org/abs/2512.17264v1)
  > **TL;DR**: Addresses scalable distributed vector search for ANNS with accuracy-latency-throughput tradeoffs. Proposes SPIRE via balanced partition granularity and accuracy-preserving recursive index construction. Achieves 9.64x higher throughput vs state-of-the-art at 8B vector scale.
* `serving` `diffusion` `sparse` [Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving](http://arxiv.org/abs/2512.17077v1)
  > **TL;DR**: Addresses the memory footprint crisis in serving diffusion LLMs. Proposes dLLM-Serve with Logit-Aware Activation Budgeting, Phase-Multiplexed Scheduler, and Head-Centric Sparse Attention. Achieves up to 1.81× higher throughput and 4× lower tail latency versus baselines.
* `multi-modal` [LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation](http://arxiv.org/abs/2512.17023v1)
* `serving` `offloading` `MoE` [Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems](http://arxiv.org/abs/2512.16473v1)
  > **TL;DR**: Proposes CPU-GPU collaborative inference for MoE-based LLMs to overcome GPU memory limits. Introduces GPU expert caching to reduce transfers, offloads cache misses to optimized CPU threads. On consumer hardware, achieves up to 5.6x speedup over full offloading with minimal accuracy loss.
* `kernel` `inference` [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](http://arxiv.org/abs/2512.16391v1)
  > **TL;DR**: Proposes Kascade, a training-free sparse attention method for accelerating long-context LLM inference. Reuses Top-k indices from anchor layers through algorithmic layer selection and head-aware reuse, optimized for tile-level GPU operations. Achieves 4.1× decode attention speedup over FlashAttention-3 with matching accuracy on benchmarks.
* `offloading` `storage` `networking` [FlexKV: Flexible Index Offloading for Memory-Disaggregated Key-Value Store](http://arxiv.org/abs/2512.16148v1)
  > **TL;DR**: Proposes FlexKV, a memory-disaggregated key-value store with index proxying to address poor performance. Dynamically offloads index to compute nodes with load balancing, memory optimization, and RPC-aggregated cache coherence. Achieves up to 2.94× higher throughput and 85.2% lower latency.
* `serving` `storage` `networking` [Lotus: Optimizing Disaggregated Transactions with Disaggregated Locks](http://arxiv.org/abs/2512.16136v1)
  > **TL;DR**: Proposes Lotus, a distributed transaction system for disaggregated memory that moves lock management to compute nodes. Introduces lock-first protocol and application-aware lock partitioning to reduce RDMA bottlenecks. Achieves 1.5× higher throughput and 49.4% lower latency than state-of-the-art systems.
* `serving` `offloading` [Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference](http://arxiv.org/abs/2512.16134v1)
  > **TL;DR**: Identifies queuing bubbles from immediate scheduling in distributed DP+EP LLM serving systems. Proposes Staggered Batch Scheduling that buffers and batches requests to eliminate queuing. Reduces TTFT by 30-40% and improves throughput by 15-20% on Deepseek-V3 serving.
* `serving` `offloading` `kernel` [An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs](http://arxiv.org/abs/2512.16099v1)
  > **TL;DR**: Addresses GPU fragmentation and resource contention in Multi-Instance GPUs for efficient sharing. Proposes an online scheduler with dynamic partitioning, job migration, and load balancing to minimize contention and combat fragmentation. Achieves up to 35% makespan improvement.
* `serving` `offloading` `kernel` [MultiPath Transfer Engine: Breaking GPU and Host-Memory Bandwidth Bottlenecks in LLM Services](http://arxiv.org/abs/2512.16056v1)
  > **TL;DR**: Proposes Multipath Memory Access (MMA) to overcome PCIe bandwidth bottlenecks in GPU-host data transfer for LLM serving. Uses multipath data transfer via dynamic library injection to increase bandwidth. Achieves peak bandwidth of 245 GB/s (4.62x speedup) and reduces TTFT by up to 2.38x.
* `training` `storage` `offline` [LOG.io: Unified Rollback Recovery and Data Lineage Capture for Distributed Data Pipelines](http://arxiv.org/abs/2512.16038v1)
  > **TL;DR**: Introduces LOG.io, a log-based system for rollback recovery and data lineage in distributed data pipelines. It supports non-deterministic operators and dynamic scaling, with non-blocking recovery. Achieves marginal overhead (≤1.5%) for lineage capture and outperforms ABS in straggler scenarios.
* `serving` `offline` [Dynamic Rebatching for Efficient Early-Exit Inference with DREX](http://arxiv.org/abs/2512.15705v1)
  > **TL;DR**: Addresses inefficient batching for Early-Exit LLM inference. Proposes Dynamic Rebatching via DREX, featuring copy-free buffering and EE/SLA-aware scheduling. Achieves 2-12% higher throughput while eliminating involuntary exits.
* `offline` `agentic` `serving` [Optimizing Agentic Language Model Inference via Speculative Tool Calls](http://arxiv.org/abs/2512.15834v1)
  > **TL;DR**: Addresses performance bottlenecks in tool-using agentic LMs via speculative tool calls and sequence residency. Optimizations include speculative execution and tool caching to reduce inference overheads. Achieves hundreds of tokens per second throughput improvement over baselines.
* `kernel` `storage` `hardware` [Optimizing Bloom Filters for Modern GPU Architectures](http://arxiv.org/abs/2512.15595v1)
  > **TL;DR**: Explores Bloom filter optimization on GPUs for high-throughput approximate membership queries. Proposes a GPU design with vectorization, thread cooperation, and compute latency tuning. Achieves 11.35× faster lookups and above 92% of practical speed limit at iso error rate on a B200 GPU.
* `training` `offloading` `quantization` [LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs](http://arxiv.org/abs/2512.15306v1)
  > **TL;DR**: Introduces LLMQ, an efficient CUDA/C++ system for LLM training on consumer GPUs with low memory. Combines 8-bit quantization, activation checkpointing, weight offloading, and optimized collectives. Achieves 50% FLOP utilization when training a 7B model on a single 16GB GPU.
* `training` `sparse` `networking` [PruneX: A Hierarchical Communication-Efficient System for Distributed CNN Training with Structured Pruning](http://arxiv.org/abs/2512.14628v1)
  > **TL;DR**: Addresses high communication overhead in distributed CNN training. Proposes hierarchical structured pruning with buffer compaction for reduced inter-node transmissions. Reduces inter-node communication volume by 60% and achieves 6.75x speedup on ResNet at 64 GPUs.
* `serving` `multi-modal` [Cornserve: Efficiently Serving Any-to-Any Multimodal Models](http://arxiv.org/abs/2512.14098v2)
  > **TL;DR**: Proposes Cornserve, a serving system for Any-to-Any multimodal models that optimizes deployment by disaggregating components. Introduces a planner and runtime for efficient handling of heterogeneous computations. Achieves up to 3.81× higher throughput and 5.79× lower tail latency.
* `serving` `offloading` `RAG` [Trustworthy and Controllable Professional Knowledge Utilization in Large Language Models with TEE-GPU Execution](http://arxiv.org/abs/2512.16238v2)
  > **TL;DR**: Proposes PKUS, a system for trustworthy professional knowledge integration in LLMs, using TEE-GPU co-execution with separable adapters. Implements hardware-rooted protocols and split-execution scheduling. Achieves 8.1-11.9x speedup over CPU-only TEE inference with comparable accuracy.
* `serving` `offloading` `compression` [EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving](http://arxiv.org/abs/2512.14946v1)
  > **TL;DR**: Addresses KV-cache management inefficiencies in LLM inference. Proposes EVICPRESS, which jointly optimizes lossy compression and adaptive eviction across storage tiers via a unified utility function. Achieves up to 2.19x faster time-to-first-token while maintaining generation quality.

### 2025-12-19
* `serving` `multi-modal` `video` [Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing](http://arxiv.org/abs/2512.17574v1)
  > **TL;DR**: Addresses latency and throughput bottlenecks in multi-stage multimodal LLM (MLLM) serving. Proposes FlashCodec for GPU-accelerated video decoding and UnifiedServe for resource sharing and inter-stage optimization. Achieves up to 4.4× higher throughput and 3.0× more requests served vs. SOTA.
* `offline` `network` `edge` [Adaptive Graph Pruning with Sudden-Events Evaluation for Traffic Prediction using Online Semi-Decentralized ST-GNNs](http://arxiv.org/abs/2512.17352v1)
  > **TL;DR**: Proposes adaptive graph pruning to reduce communication in distributed ST-GNNs for traffic prediction. Dynamically adjusts pruning based on performance and introduces SEPA metric. Achieves reduced communication costs by 27% while maintaining accuracy in semi-decentralized edge settings.
* `storage` `offline` `networking` [Scalable Distributed Vector Search via Accuracy Preserving Index Construction](http://arxiv.org/abs/2512.17264v1)
  > **TL;DR**: Scaling distributed ANN search for billion-scale vectors. Proposes SPIRE with balanced partition granularity and recursive accuracy-preserving index construction. Achieves up to 9.64x higher throughput vs. SOTA.

### 2025-12-18
* `edge` `offloading` `training` [Delay-Aware Multi-Stage Edge Server Upgrade with Budget Constraint](http://arxiv.org/abs/2512.16792v1)
  > **TL;DR**: Proposes M-ESU algorithm for multi-stage edge server upgrade and task offloading under budget constraints. Optimizes deployment/upgrade decisions and offloading to maximize tasks meeting delay requirements. Achieves 21.57% higher task satisfaction with efficient heuristic for large networks.
* `MoE` `offloading` `serving` [Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems](http://arxiv.org/abs/2512.16473v1)
  > **TL;DR**: Proposes a CPU-GPU collaborative inference framework for memory-limited systems running MoE-based LLMs. Uses expert caching on GPU and CPU offloading with multithreading to minimize data transfer. Achieves faster inference with reduced latency compared to traditional offloading methods.
* `training` `offline` `storage` [AI4EOSC: a Federated Cloud Platform for Artificial Intelligence in Scientific Research](http://arxiv.org/abs/2512.16455v1)
  > **TL;DR**: Presents a federated cloud platform for AI in science, offering integrated tools for the ML lifecycle including distributed GPU training, model deployment, and storage resources. Achieves reproducible deployments across distributed infrastructures with unified service catalog.
* `serving` `kernel` `sparse` [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](http://arxiv.org/abs/2512.16391v1)
  > **TL;DR**: Proposes Kascade, a training-free sparse attention method for long-context LLM inference. It reuses top-k indices across anchor and reuse layers with head-aware selection, optimizing tile-level operations. Achieves up to 4.1x decode attention speedup over FlashAttention-3 on H100 GPUs with minimal accuracy loss.
* `storage` `networking` `hardware` [Lotus: Optimizing Disaggregated Transactions with Disaggregated Locks](http://arxiv.org/abs/2512.16136v1)
  > **TL;DR**: Addresses the bottleneck of RDMA NICs in disaggregated memory transaction systems by disaggregating locks to compute nodes. Introduces Lotus with application-aware lock sharding, lock-first protocol, and lock-rebuild-free recovery. Achieves 2.1× higher throughput and 49.4% lower latency.
* `serving` `networking` [Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference](http://arxiv.org/abs/2512.16134v1)
  > **TL;DR**: Optimizes scheduling in P/D-separated LLM inference systems to reduce Time-to-First-Token (TTFT) and boost throughput. Proposes Staggered Batch Scheduling (SBS) with Load-Aware Global Allocation to eliminate queuing bubbles and balance DP load. Achieves 30-40% TTFT reduction and 15-20% throughput gain.
* `serving` `offline` `networking` [An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs](http://arxiv.org/abs/2512.16099v1)
  > **TL;DR**: Addresses GPU fragmentation and resource contention in MIG-enabled clusters. Proposes an online scheduler integrating load balancing, dynamic partitioning, and migration for efficient GPU-sharing. Achieves up to 35% makespan improvement.
* `serving` `offloading` `kernel` [MultiPath Transfer Engine: Breaking GPU and Host-Memory Bandwidth Bottlenecks in LLM Services](http://arxiv.org/abs/2512.16056v1)
  > **TL;DR**: Proposes Multipath Memory Access (MMA) to overcome PCIe bandwidth bottlenecks for GPU-host data transfer in LLM serving. Utilizes dynamic library injection for deployment transparency. Achieves up to 4.62x higher bandwidth and reduces TTFT by up to 2.38x in vLLM.
* `serving` `offloading` `RAG` [Trustworthy and Controllable Professional Knowledge Utilization in Large Language Models with TEE-GPU Execution](http://arxiv.org/abs/2512.16238v1)
  > **TL;DR**: Proposes PKUS, a system for trustworthy and controllable utilization of professional knowledge adapters in LLM serving. Uses TEE-GPU split execution with attested adapters, hardware-rooted protocols, and scheduling to isolate knowledge. Achieves 8.1-11.9x speedup over CPU-only TEE inference while matching fine-tuning accuracy.

### 2025-12-17
* `serving` `offloading` [Dynamic Rebatching for Efficient Early-Exit Inference with DREX](http://arxiv.org/abs/2512.15705v1)
  > **TL;DR**: Addresses inefficiency in Early-Exit LLM inference due to static batching. Proposes DREX system with copy-free dynamic rebatching and scheduler to optimally regroup requests at exit points. Achieves 2-12% higher throughput while eliminating involuntary exits and preserving quality.
* `serving` `kernel` `offloading` [Optimizing Bloom Filters for Modern GPU Architectures](http://arxiv.org/abs/2512.15595v1)
  > **TL;DR**: Explores GPU-optimized Bloom filters to accelerate approximate membership queries. Introduces designs leveraging vectorization, thread cooperation, and compute latency optimization. Achieves up to 15.4× faster construction and 92% of practical speed-of-light throughput on B200 GPU.
* `training` `offloading` `quantization` [LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs](http://arxiv.org/abs/2512.15306v1)
  > **TL;DR**: Proposes LLMQ, a CUDA/C++ system for efficient 8-bit LLM training on consumer GPUs with limited memory. Uses activation checkpointing, weight offloading, and copy-engine collectives to handle memory and communication bottlenecks. Trains a 7B model on a single 16GB GPU and maintains 50% FLOP utilization.
* `hardware` `storage` `networking` [Reexamining Paradigms of End-to-End Data Movement](http://arxiv.org/abs/2512.15028v1)
  > **TL;DR**: Examines end-to-end data movement bottlenecks beyond raw bandwidth. Proposes holistic hardware-software co-design addressing host CPU, virtualization, and congestion control. Achieves consistent performance across 1-100 Gbps links, reducing latency and throughput disparities in edge-to-core transfers.

### 2025-12-16
* `training` `sparse` `offloading` [Performance and Stability of Barrier Mode Parallel Systems with Heterogeneous and Redundant Jobs](http://arxiv.org/abs/2512.14445v1)
  > **TL;DR**: Analyzes stability and performance overhead of barrier synchronization in parallel ML training systems. Models (s,k,l) barrier systems that allow partial task completion and hybrid workloads. Validated against Apache Spark, showing overhead from dual event/polling mechanisms with quantified performance bounds.
* `edge` `serving` [A Hybrid Reactive-Proactive Auto-scaling Algorithm for SLA-Constrained Edge Computing](http://arxiv.org/abs/2512.14290v1)
  > **TL;DR**: Proposes a hybrid reactive-proactive auto-scaling algorithm for SLA-constrained edge computing. Combines ML-based demand forecasting with reactive resource adjustment, integrated into Kubernetes. Reduces SLA violation rate from 23% to 6% compared to baselines.
* `serving` `multi-modal` `kernel` [Cornserve: Efficiently Serving Any-to-Any Multimodal Models](http://arxiv.org/abs/2512.14098v1)
  > **TL;DR**: Proposes Cornserve, a system for efficient online serving of multimodal Any-to-Any models. Automatically optimizes deployment plans by disaggregating models and handling heterogeneous components. Achieves up to 3.81× throughput improvement and 5.79× tail latency reduction over existing solutions.

### 2025-12-15
* `MoE` `serving` `networking` [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](http://arxiv.org/abs/2512.13525v1)
  > **TL;DR**: Proposes Janus, a scalable MoE inference system that disaggregates attention and experts across GPU sub-clusters. Key designs include adaptive communication, lightweight scheduling, and dynamic resource scaling. Achieves up to 3.9× higher per-GPU throughput with latency guarantees.
* `training` `MoE` `storage` [SIGMA: An AI-Empowered Training Stack on Early-Life Hardware](http://arxiv.org/abs/2512.13488v1)
  > **TL;DR**: Proposes SIGMA, a training stack for large-scale distributed training on early-life AI accelerators. Combines LTP platform for reliability and LTF framework for efficient MoE model training. Achieves 94.45% accelerator utilization and trains 200B MoE model with 21.08% MFU.
* `kernel` `quantization` `hardware` [FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection](http://arxiv.org/abs/2512.12949v1)
  > **TL;DR**: Proposes FlashFuser, a compiler framework for kernel fusion using GPU inter-core connections to overcome memory limits. Utilizes Distributed Shared Memory (DSM) for complex operators, optimizing data movement and tile selection. Achieves 58% less memory access, 3.3x kernel speedup against tuned libraries.
* `serving` [PROSERVE: Unified Multi-Priority Request Scheduling for LLM Serving](http://arxiv.org/abs/2512.12928v1)
  > **TL;DR**: Addresses multi-priority request scheduling in LLM serving to maximize service gain balancing SLO attainment and client priority. Proposes PROSERVE, a two-tier scheduler with SlideBatching for batch formation and GoRouting for request dispatching. Achieves up to 35% higher system gain and 52% better SLO attainment.

### 2025-12-14
* `serving` `offline` `training` [Fine-Grained Energy Prediction For Parallellized LLM Inference With PIE-P](http://arxiv.org/abs/2512.12801v1)
  > **TL;DR**: Proposes PIE-P, a framework for fine-grained energy prediction in multi-GPU parallellized LLM inference. Uses precise sampling and modeling of inter-GPU communication to account for parallelism overhead. Achieves accurate energy predictions, significantly outperforming baselines in parallelized settings.
* `training` `networking` `quantization` [SPARK: Igniting Communication-Efficient Decentralized Learning via Stage-wise Projected NTK and Accelerated Regularization](http://arxiv.org/abs/2512.12737v1)
  > **TL;DR**: Addresses communication overhead in decentralized federated learning with statistical heterogeneity. Proposes SPARK, integrating Jacobian compression via random projection, stage-wise distillation, and Nesterov momentum. Reduces communication by 98.7% compared to NTK-DFL with 3× faster convergence.
* `edge` `networking` `storage` [Strategic Server Deployment under Uncertainty in Mobile Edge Computing](http://arxiv.org/abs/2512.12532v1)
  > **TL;DR**: Addresses strategic server deployment in mobile edge computing under uncertainty. Formulates as stochastic bilevel optimization and uses submodular approximation with greedy algorithms. Achieves up to 55% improvement over alternatives in real-world evaluations.
* `training` `serving` `kernel` [gpu_ext: Extensible OS Policies for GPUs via eBPF](http://arxiv.org/abs/2512.12615v1)
  > **TL;DR**: Addresses inflexible GPU resource management policies for diverse workloads. Proposes gpu_ext, an eBPF-based runtime enabling programmable GPU driver/device policies. Achieves up to 4.8x higher throughput and 2x lower tail latency for inference/training workloads.
* `kernel` `serving` [Principled Performance Tunability in Operating System Kernels](http://arxiv.org/abs/2512.12530v1)
  > **TL;DR**: Addresses the problem of safely tuning Linux kernel performance constants (perf-consts). Proposes KernelX, using Scoped Indirect Execution (SIE) to enable live updates without rebuilds. Achieves millisecond-scale updates and significant performance improvements in case studies.

### 2025-12-13
* `training` `RL` `networking` [HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments](http://arxiv.org/abs/2512.12476v1)
  > **TL;DR**: Addresses efficient RL training for LLMs in heterogeneous GPU environments. Proposes HetRL system with a constrained joint optimization formulation and a multi-level search scheduling algorithm. Achieves up to 9.17x higher throughput over state-of-the-art systems.
* `edge` `RL` `serving` [A Conflict-Aware Resource Management Framework for the Computing Continuum](http://arxiv.org/abs/2512.12299v1)
  > **TL;DR**: Proposes a DRL-based framework for conflict-aware resource orchestration across edge, fog, and cloud. Integrates real-time performance feedback and historical data to mediate resource conflicts. Achieves efficient resource reallocation and adaptive learning in dynamic Kubernetes environments.
* `inference-side model updates` `resource efficiency` `latency reduction` [Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates](http://arxiv.org/abs/2512.12295v1)
  > **TL;DR**: LiveUpdate introduces inference-side model updates for recommendation systems using Low-Rank Adaptation, reducing synchronization costs and improving freshness with minimal latency impact (<20ms P99) and higher accuracy (0.04-0.24% gain).
* `training` `sparse` [BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models](http://arxiv.org/abs/2512.12131v1)
  > **TL;DR**: Proposes BOOST, a training framework for low-rank LLMs with bottleneck-aware tensor parallelism and other optimizations. Addresses poor scalability of low-rank architectures by reducing communication and improving GPU utilization. Achieves 1.46-1.91× speedup over full-rank baselines.

### 2025-12-12
* `video` `training` `RAG` [ECCO: Leveraging Cross-Camera Correlations for Efficient Live Video Continuous Learning](http://arxiv.org/abs/2512.11727v1)
  > **TL;DR**: Introduces ECCO, a framework for efficient continuous learning in multi-camera systems by grouping cameras with correlated data drift for shared model retraining. ECCO includes dynamic grouping, GPU allocation, and transmission control. Achieves 6.7%-18.1% higher accuracy or supports 3.3× more cameras at same resource.
* `serving` `edge` `kernel` [Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems](http://arxiv.org/abs/2512.11532v1)
  > **TL;DR**: Addresses inefficient CPU fallbacks for unsupported DNN operators on edge devices. Proposes Parallax, a framework with DAG partitioning, branch-aware memory management, and an adaptive scheduler. Achieves up to 46% latency reduction compared to state-of-the-art frameworks.
* `training` `RL` `offloading` [RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training](http://arxiv.org/abs/2512.11306v1)
  > **TL;DR**: Proposes RollMux, a cluster scheduling framework for disaggregated RL post-training to reclaim dependency bubbles through cross-cluster orchestration. Uses co-execution groups for two-tier scheduling and warm caching. Achieves 1.84x higher cost efficiency than standard disaggregation on a 656-GPU testbed.
* `kernel` `hardware` `training` [Theoretical Foundations of GPU-Native Compilation for Rapid Code Iteration](http://arxiv.org/abs/2512.11200v1)
  > **TL;DR**: Proposes GPU-native compilation to eliminate CPU-GPU data transfer bottlenecks. Introduces parallel traditional, neural, and hybrid compilation strategies with probabilistic verification. Achieves 10-100x speedups in code iteration cycles through massive parallelism and transfer elimination.

### 2025-12-11
* `serving` `offloading` [ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp](http://arxiv.org/abs/2512.10576v1)
  > **TL;DR**: Addresses GPU memory bottleneck in DeepSeek-V3.2-Exp's decode stage caused by linear growth of latent-cache. Proposes ESS, an offload-centric system that selectively moves latent-cache to CPU memory to enable larger batch sizes. Achieves up to 123% throughput improvement at 128K context length.
* `training` `RL` `offline` [Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusters](http://arxiv.org/abs/2512.10271v1)
  > **TL;DR**: Proposes RLTune, an RL-based scheduling framework with MILP job mapping for DL workloads on heterogeneous GPU clusters. Aims to optimize GPU utilization, queueing delay, and JCT without per-job profiling. Achieves up to 20% higher GPU utilization, 81% lower queueing delay, and 70% shorter JCT.
* `training` `networking` `kernel` [Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap](http://arxiv.org/abs/2512.10236v1)
  > **TL;DR**: Proposes FiCCO, a finer-grain compute-communication overlap method using DMA offloading for distributed ML training. Introduces heuristics to select optimal schedules by characterizing inefficiencies, achieving up to 1.6× speedup and 81% accuracy in unseen scenarios.

### 2025-12-10
* `edge` `kernel` [Ariel-ML: Computing Parallelization with Embedded Rust for Neural Networks on Heterogeneous Multi-core Microcontrollers](http://arxiv.org/abs/2512.09800v1)
  > **TL;DR**: Presents Ariel-ML, a Rust toolkit for parallelized ANN inference on multi-core microcontrollers. Combines a generic TinyML pipeline with embedded Rust to utilize multi-core capabilities. Achieves lower inference latency than prior art while maintaining comparable memory footprint to C/C++ toolkits.
* `training` `optimization` `scheduling` [Straggler Tolerant and Resilient DL Training on Homogeneous GPUs](http://arxiv.org/abs/2512.09685v1)
  > **TL;DR**: Investigate stragglers in distributed training on GPU clusters. Propose STAR with adaptive synchronization modes and resource reallocation to mitigate stragglers caused by CPU/bandwidth imbalance. Reduces Time-To-Accuracy by 70% compared to state-of-the-art.
* `serving` `offloading` [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](http://arxiv.org/abs/2512.09472v1)
  > **TL;DR**: Addresses performance degradation in multi-LLM GPU serving due to cold starts. Proposes WarmServe with universal GPU workers, evict-aware placement, proactive prewarming, and zero-overhead memory switching. Achieves 50.8× TTFT improvement and 2.5× request capacity.
* `RAG` `storage` `networking` [Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN](http://arxiv.org/abs/2512.09331v1)
  > **TL;DR**: Presents BatANN, a distributed disk-based ANN system for scalable vector search using batched query state handoff between servers to maintain locality. Achieves 2.5-6.49x higher throughput over baseline while keeping mean latency below 6ms on billion-point datasets.
* `edge` `offloading` `multi-modal` [A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge](http://arxiv.org/abs/2512.09309v1)
  > **TL;DR**: Proposes a distributed framework for privacy-enhanced Vision Transformers on edge devices. Uses hierarchical offloading to partition visual data across untrusted clouds and aggregates results locally on a trusted edge device. Reduces reconstruction risk while maintaining near-baseline segmentation performance in SAM case study.
* `training` `MoE` `offloading` [Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens](http://arxiv.org/abs/2512.09277v1)
  > **TL;DR**: Proposes METRO for efficient MoE serving in memory-bound regimes; balances activated experts per GPU instead of tokens to reduce memory pressure and improve performance. Achieves up to 22% lower decode latency and 4.11x higher throughput.

### 2025-12-09
* `training` `serving` `kernel` [Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging](http://arxiv.org/abs/2512.08365v1)
  > **TL;DR**: Proposes differential energy debugging to identify software-caused energy waste in ML systems. Magneton compares energy use at operator level across similar systems to pinpoint inefficient code/configuration. Applied to LLM inference, reduces energy consumption by up to 47% in diagnosed cases.
* `training` `kernel` [Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency](http://arxiv.org/abs/2512.08242v1)
  > **TL;DR**: Introduces Chopper, a multi-level GPU profiling tool for LLM training analysis. Collects and aligns kernel traces and hardware performance counters across granularities to identify bottlenecks in FSDP. Identifies frequency overhead (DVFS) as largest inefficiency source (exceeding MFMA utilization loss etc.) in Llama 3 8B training.

### 2025-12-08
* `offline` `training` `sparse` [Quantifying the Carbon Reduction of DAG Workloads: A Job Shop Scheduling Perspective](http://arxiv.org/abs/2512.07799v1)
  > **TL;DR**: Quantifies carbon reduction for DAG workloads (e.g., video encoding, offline inference) by modeling as job-shop scheduling. Uses offline solver to compute upper bounds, achieving 25% lower carbon emissions without increased makespan; doubling makespan nearly doubles savings.
* `training` `serving` `networking` [Designing Co-operation in Systems of Hierarchical, Multi-objective Schedulers for Stream Processing](http://arxiv.org/abs/2512.07792v1)
  > **TL;DR**: Investigates scheduling co-operation to optimize resource allocation in hierarchical stream processing systems at Meta. Proposes integration of new schedulers into existing hierarchies for effective load balancing across compute resources. Enables processing terabytes of data within seconds.
* `training` `networking` [Bandwidth-Aware Network Topology Optimization for Decentralized Learning](http://arxiv.org/abs/2512.07536v1)
  > **TL;DR**: Proposes bandwidth-aware network topology optimization for decentralized learning to maximize consensus speed under edge constraints. Uses Mixed-Integer SDP reformulation and ADMM with conjugate gradient for scalability. Reduces training time by 1.21× for heterogeneous bandwidth settings.
* `video` `diffusion` `serving` [Communication-Efficient Serving for Video Diffusion Models with Latent Parallelism](http://arxiv.org/abs/2512.07350v1)
  > **TL;DR**: Addresses communication bottlenecks in video diffusion model serving. Proposes Latent Parallelism (LP), which dynamically rotates partitioning dimensions in latent space to reduce transfers. Achieves up to 97% communication overhead reduction while maintaining generation quality.
* `edge` `RAG` `video` [Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding](http://arxiv.org/abs/2512.07344v1)
  > **TL;DR**: Addresses high deployment overhead for VLM-based online video understanding. Proposes Venus, an edge-cloud disaggregated system with hierarchical memory construction and threshold-based sampling for adaptive cost-accuracy tradeoff. Achieves 15x-131x latency speedup while maintaining accuracy.
* `hardware` `offloading` `kernel` [DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management](http://arxiv.org/abs/2512.07312v1)
  > **TL;DR**: Proposes DCO, a shared system-level cache with predictive management for LLM accelerators, using dataflow-guided cache replacement and thrashing mitigation. Achieves up to 1.80x speedup over conventional cache architectures and is validated with RTL implementation at 2 GHz.

### 2025-12-07
* `video` `serving` `offline` [Optimizing video analytics inference pipelines: a case study](http://arxiv.org/abs/2512.07009v1)
  > **TL;DR**: Optimizes video analytics inference pipelines for livestock monitoring. Introduces multi-level parallelization, GPU acceleration, vectorized clustering, and memory-efficient post-processing. Achieves 2x speedup across pipelines without accuracy loss.
* `storage` `serving` [A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases](http://arxiv.org/abs/2512.06852v1)
  > **TL;DR**: Proposes a 'chunked-object' pattern for storing large payloads exceeding NoSQL item size limits using chunking within the database. Eliminates replication lag risks by avoiding offloading to object storage, reducing p99 cross-region consistency latency for 1 MB payloads by keeping data in a single consistency domain.
* `MoE` `edge` `training` [Stable-MoE: Lyapunov-based Token Routing for Distributed Mixture-of-Experts Training over Edge Networks](http://arxiv.org/abs/2512.06784v1)
  > **TL;DR**: Proposes Lyapunov-based token routing for distributed MoE training on edge networks with heterogeneous resources. Formulates stochastic optimization for throughput and gating consistency via online routing/resource allocation. Gains 40% throughput and 5% accuracy on SVHN/CIFAR-100.

### 2025-12-06
* `training` `RL` `sparse` [A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation](http://arxiv.org/abs/2512.06547v1)
  > **TL;DR**: Addresses computational bottleneck in asynchronous RL training for LLMs caused by proximal policy. Proposes A-3PO, which approximates the proximal policy via interpolation instead of extra forward pass. Reduces training time by 18%.
* `offline` `quantization` `edge` [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices](http://arxiv.org/abs/2512.06443v1)
  > **TL;DR**: Proposes Vec-LUT for efficient parallel ultra-low-bit LLM inference. Replaces scalar LUTs with a vector lookup to reduce bandwidth underutilization via tensor layout and cache-aware techniques. On 5 edge devices, achieves up to 4.2× speedup over baselines.

### 2025-12-05
* `serving` `offloading` `networking` [Metronome: Differentiated Delay Scheduling for Serverless Functions](http://arxiv.org/abs/2512.05703v1)
  > **TL;DR**: Proposes differentiated delay scheduling for serverless FaaS to optimize locality-aware execution. Metronome uses online Random Forest Regression to predict function times and select optimal nodes. Achieves 64.88%-95.83% reduction in mean execution time over baselines.
* `serving` `RL` `RAG` [Model Gateway: Model Management Platform for Model-Driven Drug Discovery](http://arxiv.org/abs/2512.05462v1)
  > **TL;DR**: Proposes Model Gateway, a management platform for ML models in drug discovery. It integrates LLM Agents for dynamic consensus model, model registration, asynchronous execution, and result retrieval. Achieves 0% failure rate with 10k simultaneous clients.

### 2025-12-04
* `offloading` `hardware` `kernel` [Offloading to CXL-based Computational Memory](http://arxiv.org/abs/2512.04449v1)
  > **TL;DR**: Proposes KAI, a system using Asynchronous Back-Streaming protocol to offload operations to CXL-based Computational Memory. Utilizes layered data/control transfers for async data movement and pipelining, reducing end-to-end runtime by up to 50.4% and idle times by 22.11x/3.85x.

### 2025-12-03
* `MoE` `edge` `offloading` [OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference](http://arxiv.org/abs/2512.03927v1)
  > **TL;DR**: Proposes OD-MoE, a distributed MoE inference framework for edge devices that eliminates expert caching via on-demand expert loading and an emulative predictor. Achieves 99.94% expert prediction accuracy and 75% decoding speed of fully GPU-cached deployment with only 1/3 GPU memory, enabling sub-1GB deployments.
* `training` `storage` `networking` [FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management](http://arxiv.org/abs/2512.03644v1)
  > **TL;DR**: Addresses recovery inefficiencies and storage overhead in large-scale LLM training. FFTrainer uses surplus network bandwidth for fast state management and failure rollback reduction. Achieves up to 98% faster recovery time and 68% higher GPU utilization preservation.
* `serving` `autoscaling` `resource management` [TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity](http://arxiv.org/abs/2512.03416v1)
  > **TL;DR**: Proposes TokenScale for autoscaling disaggregated LLM serving using Token Velocity metric and convertible decoders to handle bursty workloads. Achieves SLO attainment up to 96% and reduces costs by 4-14%.

### 2025-12-02
* `serving` `kernel` `inference` [TokenPowerBench: Benchmarking the Power Consumption of LLM Inference](http://arxiv.org/abs/2512.03024v1)
  > **TL;DR**: Investigators tackle the lack of power consumption benchmarks for LLM inference by introducing TokenPowerBench, a tool enabling configuration of model, prompt, and engine; capturing multi-level power metrics and attributing energy per request phase. It quantifies joules per token across varying settings, achieving measurable energy efficiency assessment without specialized hardware.
* `offloading` `training` `storage` [Offloading Artificial Intelligence Workloads across the Computing Continuum by means of Active Storage Systems](http://arxiv.org/abs/2512.02646v1)
  > **TL;DR**: Proposes an active storage architecture for offloading AI workloads across heterogeneous devices. Embeds computation in storage to reduce data transfer and uses dataClay platform. Achieves improved memory efficiency and training speeds while maintaining accuracy.

### 2025-12-01
* `serving` `offloading` `storage` [Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity](http://arxiv.org/abs/2512.01357v1)
  > **TL;DR**: Tackles high cold-start latency in serverless LLM serving via GPU memory reuse. Proposes Tangram with unified GPU memory pool, on-demand KV cache allocation, and affinity-aware scheduling. Achieves 6.2x faster model loading and 23-55% lower Time-To-First-Token.

### 2025-11-29
* `serving` `edge` `offloading` [IslandRun: Privacy-Aware Multi-Objective Orchestration for Distributed AI Inference](http://arxiv.org/abs/2512.00595v1)
  > **TL;DR**: Addresses the challenge of orchestrating LLM inference across heterogeneous devices (edge, cloud) with conflicting goals of performance, privacy, and cost. Proposes IslandRun, a system using agent-based routing and reversible anonymization to route compute to data. Achieves improved privacy and multi-objective optimization.

### 2025-11-28
* `training` `offloading` `edge` [Communication-Computation Pipeline Parallel Split Learning over Wireless Edge Networks](http://arxiv.org/abs/2511.23167v1)
  > **TL;DR**: Proposes a pipeline-parallel split learning method to overlap communication and computation in distributed training over wireless edge networks. Uses micro-batching and jointly optimizes task split and resource allocation. Reduces total training time by over 38% while maintaining accuracy.
* `serving` `RL` `networking` [Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems](http://arxiv.org/abs/2511.22880v1)
  > **TL;DR**: Addresses performance skew from serving heterogeneous LoRA adapters. Proposes LoRAServe, a dynamic adapter placement and routing framework that rebalances adapters across GPUs using RDMA for remote access. Achieves up to 2× higher throughput and 50% fewer GPUs under SLOs compared to SOTA.

### 2025-11-27
* `serving` `edge` `networking` [DisCEdge: Distributed Context Management for Large Language Models at the Edge](http://arxiv.org/abs/2511.22599v1)
  > **TL;DR**: Proposes DisCEdge, a distributed context management system for edge LLM serving. It stores/replicates user context as token sequences instead of raw text to reduce synchronization overhead. Improves median response time by 14.46% and reduces client request size by 90%.
* `serving` `MoE` `sparse` [OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency](http://arxiv.org/abs/2511.22481v1)
  > **TL;DR**: Proposes OmniInfer, a unified LLM serving system that optimizes throughput and latency via Mixture-of-Experts scheduling and sparse attention acceleration. Built on vLLM, it reduces TPOT by 36% and TTFT by 38% in cluster evaluations.
* `kernel` `serving` [PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel](http://arxiv.org/abs/2511.22333v1)
  > **TL;DR**: Proposes PAT, a prefix-aware attention kernel for LLM decoding that packs queries by shared prefix and uses a multi-tile design to minimize KV cache memory accesses. Achieves 67.4% average attention latency reduction over state-of-the-art kernels.

### 2025-11-26
* `serving` `edge` `offloading` [DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving](http://arxiv.org/abs/2511.21669v1)
  > **TL;DR**: Addresses the challenge of speculative decoding for LLM inference in distributed edge-cloud environments. Introduces DSD, a framework for coordinated draft-target execution across devices, evaluated via a custom simulator and adaptive window control. Achieves up to 1.1x speedup and 9.7% higher throughput.
* `training` `MoE` `offloading` [MemFine: Memory-Aware Fine-Grained Scheduling for MoE Training](http://arxiv.org/abs/2511.21431v1)
  > **TL;DR**: Proposes MemFine, a memory-aware scheduling framework to solve the memory bottleneck in MoE training caused by dynamic token routing. It uses chunk-level decomposition and optimized recomputation guided by a memory model. Reduces activation memory by 48.03% and increases throughput by 4.42% versus full recomputation.
* `serving` `edge` [Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM](http://arxiv.org/abs/2511.21413v1)
  > **TL;DR**: Proposes a system to dynamically scale LLM inference on HPC infrastructure by integrating vLLM, Slurm, and Kubernetes to handle synchronous user-facing workloads. Achieves efficient scaling for up to 1000 concurrent requests with only ~500 ms end-to-end latency overhead.
* `serving` [A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving](http://arxiv.org/abs/2511.20982v1)
  > **TL;DR**: Proposes DOPD, a dynamic LLM inference system that adjusts prefill-to-decoding instance allocations based on real-time load to resolve workload imbalance. Achieves up to 1.5x higher goodput and reduces P90 TTFT by 67.5% compared to vLLM and DistServe.
* `serving` `agentic` `offline` [Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows](http://arxiv.org/abs/2511.20975v1)
  > **TL;DR**: Proposes Aragog, a system for cost-aware LLM serving of agentic workflows using just-in-time model routing. It decouples configuration into an offline accuracy-preserving step and an online per-stage scheduler. At peak load, achieves 50-217% higher throughput and 32.5-78.9% lower median latency.

### 2025-11-25
* `serving` `offloading` `hardware` [Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management](http://arxiv.org/abs/2511.20172v1)
  > **TL;DR**: Introduces Beluga, a CXL-based memory architecture for efficient KVCache management in LLM serving. It enables GPUs to natively access a shared memory pool via CXL switches, reducing access latency. Achieves 89.6% lower TTFT and 7.35x higher throughput vs. RDMA-based solutions.
* `training` `kernel` [QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation](http://arxiv.org/abs/2511.20100v1)
  > **TL;DR**: Proposes Macro Thinking Micro Coding (MTMC), a hierarchical LLM framework for generating high-performance GPU kernels. Decouples optimization strategy (reinforcement learning) from implementation (LLM coding). Achieves up to 7.3x speedup over LLMs and 2.2x over expert-optimized PyTorch kernels.
* `training` `networking` `sparse` [ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models](http://arxiv.org/abs/2511.19959v1)
  > **TL;DR**: Addresses high communication latency in federated fine-tuning of large LLM blocks. Proposes ParaBlock, a scheme with parallel communication and computation threads to overlap these phases. Reduces communication costs by up to 42% while maintaining model convergence and performance.
* `diffusion` `edge` [Batch Denoising for AIGC Service Provisioning in Wireless Edge Networks](http://arxiv.org/abs/2511.19847v1)
  > **TL;DR**: Optimizes image generation service quality in wireless edge networks under latency constraints. Proposes STACKING, an algorithm for batch denoising that exploits step importance for parallelism. Achieves lower per-step delay and higher service quality compared to baseline methods.

### 2025-11-24
* `edge` `storage` `RAG` [AME: An Efficient Heterogeneous Agentic Memory Engine for Smartphones](http://arxiv.org/abs/2511.19192v1)
  > **TL;DR**: Proposes AME, a hardware-aware vector database engine for on-device agents on smartphones. It co-designs an efficient matrix pipeline and workload scheduler with mobile SoC constraints. Achieves 1.4x higher query throughput and 7x faster index construction.
* `kernel` [Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration](http://arxiv.org/abs/2511.18674v1)
  > **TL;DR**: Proposes Low-Rank GEMM, a matrix multiplication method using low-rank approximations and FP8 precision to reduce complexity and improve hardware efficiency. Achieves a 7.8x speedup and 75% memory savings over PyTorch FP32 on large matrices.

### 2025-11-22
* `edge` `multi-modal` `offloading` [AVERY: Adaptive VLM Split Computing through Embodied Self-Awareness for Efficient Disaster Response Systems](http://arxiv.org/abs/2511.18151v1)
  > **TL;DR**: Presents AVERY, an adaptive split-computing framework for Vision-Language Models on UAVs. It uses a dual-stream split (context/insight) and a self-aware controller to dynamically offload processing. Outperforms baselines with 93.98% lower energy consumption versus full-edge execution.
* `training` `networking` [Pier: Efficient Large Language Model pretraining with Relaxed Global Communication](http://arxiv.org/abs/2511.17849v1)
  > **TL;DR**: Addresses the high cost of global communication (e.g., all-reduce) in LLM pretraining. Proposes Pier, an optimizer with relaxed communication via momentum warmup/decay. Achieves a 2.7x-3.7x speedup in GPT-2 XL training on 256 A100s without performance loss.

### 2025-11-21
* `training` `MoE` `hardware` [Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design](http://arxiv.org/abs/2511.17127v1)
  > **TL;DR**: Presents a full-stack system design for training MoE foundation models on AMD MI300X GPUs with Pollara interconnect. Characterizes cluster networking, memory bandwidth, and introduces hardware-aware transformer sizing rules. Achieves competitive model performance with optimized throughput on pure AMD hardware.
* `kernel` `serving` [Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems](http://arxiv.org/abs/2511.16964v1)
  > **TL;DR**: Investigates multi-agent LLM systems for optimizing PyTorch inference. Proposes a framework to compare different agent strategies, finding exploit-heavy strategies with error-fixing agents work best. Achieves an average 2.88x speedup on an H100 GPU across the KernelBench suite.
* `training` `MoE` [MicroMoE: Fine-Grained Load Balancing for Mixture-of-Experts with Token Scheduling](http://arxiv.org/abs/2511.16947v1)
  > **TL;DR**: Addresses load imbalance in MoE model training. Proposes MicroMoE, a distributed system with MicroEP, a fine-grained parallelization strategy using token scheduling for load balancing. Improves end-to-end training throughput by up to 47.6%.

### 2025-11-20
* `training` `RL` `kernel` [Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter](http://arxiv.org/abs/2511.16665v1)
  > **TL;DR**: Proposes TLT, a system to accelerate reasoning RL training by using adaptive speculative decoding to overcome the bottleneck of long-tail response generation. Achieves over 1.7x end-to-end training speedup while preserving model accuracy.
* `RL` `serving` [Fast LLM Post-training via Decoupled and Best-of-N Speculation](http://arxiv.org/abs/2511.16193v1)
  > **TL;DR**: Addresses the slow rollout phase in LLM post-training (e.g., RLHF). Introduces SpecActor, which uses dynamic decoupled speculation and Best-of-N drafting to maximize GPU efficiency and draft accuracy. Achieves 1.3–1.7× speedup over common post-training baselines.
* `kernel` `hardware` [Can Asymmetric Tile Buffering Be Beneficial?](http://arxiv.org/abs/2511.16041v1)
  > **TL;DR**: Proposes Asymmetric Tile Buffering (ATB) to decouple input and output operand tile dimensions in GEMM, increasing arithmetic intensity. A performance model balances ATB benefits against kernel switching overhead. On AMD XDNA2 AI Engine, achieves 4.54x speedup (4.8 to 24.6 TFLOPS) for BFP16-BF16 GEMM.
* `hardware` `serving` `offloading` [A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference](http://arxiv.org/abs/2511.15950v1)
  > **TL;DR**: Presents a vertically integrated system with NorthPole accelerators for scalable, low-latency LLM inference. Combines custom hardware, a runtime stack, and containerized pipeline. Delivers 2.8 ms per-user inter-token latency for an 8B model while consuming only 30 kW.

### 2025-11-19
* `kernel` `hardware` `serving` [A Tensor Compiler for Processing-In-Memory Architectures](http://arxiv.org/abs/2511.15503v1)
  > **TL;DR**: Proposes DCC, a data-centric compiler optimizing data layouts and compute code jointly for ML kernels on PIM architectures. It uses a unified tuning process with performance prediction. Achieves up to 7.71x speedup for LLM inference over GPU-only execution.
* `networking` `MoE` `training` [GPU-Initiated Networking for NCCL](http://arxiv.org/abs/2511.15076v1)
  > **TL;DR**: Investigates how to reduce GPU communication latency for AI workloads like MoE models. Proposes GPU-Initiated Networking (GIN), an NCCL extension enabling direct, CPU-bypass GPU-to-GPU communication via RDMA. Achieves lower latency device-initiated operations within NCCL's collective runtime.

### 2025-11-18
* `RL` `serving` [Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning](http://arxiv.org/abs/2511.14617v1)
  > **TL;DR**: Addresses performance bottlenecks in synchronous LLM reinforcement learning. Proposes Seer, a system using online context learning with dynamic load balancing and grouped speculative decoding to exploit prompt similarities. Achieves up to 97% higher rollout throughput and 93% lower long-tail latency.
* `serving` `edge` `networking` [Hyperion: Hierarchical Scheduling for Parallel LLM Acceleration in Multi-tier Networks](http://arxiv.org/abs/2511.14450v1)
  > **TL;DR**: This paper tackles minimizing end-to-end latency for LLM inference in multi-tier networks. Hyperion uses a two-stage framework that jointly optimizes offline model partitioning and online request scheduling. It reduces latency by up to 52.1% compared to GPipe while increasing GPU utilization.
* `training` `offloading` `storage` [10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training](http://arxiv.org/abs/2511.14124v1)
  > **TL;DR**: Addresses memory bottlenecks and tensor migration latency in LLM training. Proposes 10Cache, a system for intelligent tensor caching/migration across GPU, CPU, and NVMe tiers using prefetching and buffer reuse. Achieves up to 2x training speedup and an 86.6x higher GPU cache hit rate.
* `serving` `networking` [FailSafe: High-performance Resilient Serving](http://arxiv.org/abs/2511.14116v1)
  > **TL;DR**: Addresses how to sustain LLM serving performance under GPU failures in tensor-parallel settings. Proposes FailSafe with cyclic KVCache placement, hybrid attention, and dynamic routing, plus proactive backup and recovery. Achieves 2x higher throughput and 100x lower recovery latency vs. standard fault handling.
* `MoE` `offloading` `serving` [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](http://arxiv.org/abs/2511.14102v1)
  > **TL;DR**: Addresses the I/O bottleneck of expert offloading in MoE model inference. Proposes MoE-SpeQ, a system that uses a draft model for speculative expert prefetching and an adaptive governor to overlap I/O with computation. Achieves a 2.34x speedup over the state-of-the-art offloading framework.

### 2025-11-17
* `kernel` `inference` `sparse` [MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity](http://arxiv.org/abs/2511.13061v1)
  > **TL;DR**: Proposes MACKO-SpMV, a GPU-optimized kernel and storage format for efficient sparse matrix-vector multiplication at low (30-90%) sparsity levels common in pruned LLMs. Applied to Llama2-7B, it achieves 1.5x memory reduction and 1.5x faster inference at 50% sparsity compared to a dense baseline.

### 2025-11-16
* `kernel` `training` [Iris: First-Class Multi-GPU Programming Experience in Triton](http://arxiv.org/abs/2511.12500v1)
  > **TL;DR**: Presents Iris, a multi-GPU communication library implemented in Python/Triton to simplify and optimize compute-communication overlap. Introduces tile-based symmetric memory abstractions for single-source kernels. Achieves up to 1.79x speedup over PyTorch and RCCL in GEMM+All-Scatter workloads.

### 2025-11-15
* `video` `serving` [PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decoupling](http://arxiv.org/abs/2511.12056v1)
  > **TL;DR**: Proposes PipeDiT, a pipelining framework to reduce inference latency and memory consumption for DiT-based video generation. Key innovations include pipeline sequence parallelism, decoupling diffusion/VAE modules, and attention co-processing. Achieves 1.06x-4.02x speedups over baseline frameworks.
* `serving` `kernel` `offloading` [Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding](http://arxiv.org/abs/2511.12031v1)
  > **TL;DR**: Proposes BMC, a KV cache allocation method balancing memory copies and compute redundancy for LLM inference. It pre-allocates tensors with extra rows for in-place updates, repurposing redundancy for speculative decoding. Achieves up to 3.2x throughput acceleration over baseline HuggingFace.

### 2025-11-14
* `agentic` `edge` `serving` [UFO$^3$: Weaving the Digital Agent Galaxy](http://arxiv.org/abs/2511.11332v1)
  > **TL;DR**: Presents UFO3, a system for orchestrating LLM-powered agents across heterogeneous devices. Models user requests as mutable distributed DAGs with asynchronous execution and dynamic optimization. Achieves 31% lower end-to-end latency compared to a sequential baseline.
* `edge` `offloading` `networking` [SemanticNN: Compressive and Error-Resilient Semantic Offloading for Extremely Weak Devices](http://arxiv.org/abs/2511.11038v1)
  > **TL;DR**: Addresses how to perform accurate AI inference offloading for resource-constrained IoT devices over unreliable networks. Proposes SemanticNN, a semantic codec with a BER-aware decoder and soft quantization, that is resilient to bit errors. Reduces feature transmission volume by 56.82-344.83x while maintaining accuracy.

### 2025-11-13
* `training` `kernel` [Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs](http://arxiv.org/abs/2511.10480v1)
  > **TL;DR**: Presents STAGE, a framework to synthesize high-fidelity execution traces for modeling distributed LLM training workloads using symbolic tensor graphs. It enables systematic exploration of parallelization strategies and scales to model configurations spanning over 32K GPUs with tensor-level accuracy.
* `training` `hardware` [Lit Silicon: A Case Where Thermal Imbalance Couples Concurrent Execution in Multiple GPUs](http://arxiv.org/abs/2511.09861v1)
  > **TL;DR**: Identifies Lit Silicon, a thermal imbalance causing straggler GPUs in multi-GPU LLM training systems. Proposes detection/mitigation techniques and power management models, including GPU/CPU power optimization. Achieves up to 6% performance and 4% power improvements on two LLM training frameworks.
* `training` [MoFa: A Unified Performance Modeling Framework for LLM Pretraining](http://arxiv.org/abs/2511.09837v1)
  > **TL;DR**: Presents MoFa, a performance modeling framework for distributed LLM pretraining that integrates optimization features and fault tolerance overhead. It uses an enhanced cost model and fault tolerance analysis to guide system tuning. Achieves high prediction accuracy across various scenarios.
* `hardware` `offloading` [Taiji: A DPU Memory Elasticity Solution for In-production Cloud Environments](http://arxiv.org/abs/2511.09936v1)
  > **TL;DR**: Proposes Taiji, a resource-elasticity architecture for Data Processing Units (DPUs) using hybrid virtualization and parallel memory swapping to enable memory overcommitment. Achieves over 50% memory expansion with ~5% virtualization overhead and 90% of swap-ins under 10 microseconds.

### 2025-11-12
* `training` `networking` [TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training](http://arxiv.org/abs/2511.09741v1)
  > **TL;DR**: Proposes TawPipe, a topology-aware weight pipeline parallelism method for LLM training that optimizes communication hierarchy to reduce cross-node traffic. Groups devices by topology, assigns fixed weight shards, and overlaps communication. Achieves 1.56x higher throughput than WeiPipe on 24 GPUs with long sequences.
* `serving` `networking` [LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication](http://arxiv.org/abs/2511.09557v2)
  > **TL;DR**: Analyzes bottlenecks in multi-node LLM inference. Proposes NVRAR, a low-latency hierarchical all-reduce algorithm using NVSHMEM and recursive doubling. Reduces communication latency by up to 3.6x and end-to-end batch latency by 1.72x for large models like Llama 3.1 405B.
* `training` `kernel` [No Cords Attached: Coordination-Free Concurrent Lock-Free Queues](http://arxiv.org/abs/2511.09410v1)
  > **TL;DR**: Addresses high coordination overhead in concurrent queues for massively parallel AI workloads. Proposes Cyclic Memory Protection (CMP), a coordination-free lock-free queue using bounded protection windows. Outperforms state-of-the-art queues by 1.72-4x under high contention with hundreds of threads.
* `hardware` `serving` [Flex-MIG: Enabling Distributed Execution on MIG](http://arxiv.org/abs/2511.09143v2)
  > **TL;DR**: Addresses GPU underutilization from rigid NVIDIA MIG partitioning. Proposes Flex-MIG, a software framework enabling one-to-many allocation and shared-memory collectives for distributed execution across MIG instances. Improves cluster makespan by up to 17% on diverse traces.

### 2025-11-11
* `kernel` `sparse` [\uline{LO}w-c\uline{O}st yet High-\uline{P}erformant \uline{S}parse Matrix-Matrix Multiplication on Arm SME Architectures](http://arxiv.org/abs/2511.08158v1)
  > **TL;DR**: Proposes LOOPS, a hybrid framework for efficient SpMM on Arm SME architectures. Combines row-wise and vector-wise layouts to co-utilize NEON and SME units via an adaptive parallelization scheme. Achieves up to 14.4x (FP64) speedup over the baseline TACO on CPU.

### 2025-11-06
* `quantization` `sparse` `edge` [Enabling Dynamic Sparsity in Quantized LLM Inference](http://arxiv.org/abs/2511.04477v1)
  > **TL;DR**: Proposes techniques to enable dynamic sparsity in quantized LLM inference for edge devices. Key designs include a zigzag-patterned quantization layout and a specialized GEMV kernel for commodity GPUs. Achieves up to 1.55x faster decoding throughput while maintaining accuracy.

### 2025-11-05
* `edge` `serving` `kernel` [UMDAM: A Unified Data Layout and DRAM Address Mapping for Heterogenous NPU-PIM](http://arxiv.org/abs/2511.03293v1)
  > **TL;DR**: Proposes UMDAM, a unified data layout and DRAM mapping scheme for NPU-PIM co-execution to optimize LLM inference on edge devices. Uses a column-major, tile-based layout to eliminate bandwidth loss and redundant storage. Reduces TTFT by up to 3.0x and TTLT by 2.18x.
* `serving` `offloading` [SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators](http://arxiv.org/abs/2511.03092v1)
  > **TL;DR**: Proposes SnapStream, a KV cache compression method for efficient long-sequence LLM inference with continuous batching. It integrates sparsification into static-graph dataflow accelerators. Achieves 4× improved on-chip memory usage while maintaining throughput of 1832 tokens/sec for DeepSeek-671B at 128k context length.

### 2025-11-04
* `serving` `edge` `networking` [Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks](http://arxiv.org/abs/2511.02647v1)
  > **TL;DR**: Proposes Federated Attention (FedAttn), a distributed LLM inference framework for edge networks that performs local self-attention and periodic KV matrix aggregation to preserve privacy and reduce communication. Achieves a 3.2x reduction in communication cost while maintaining response quality.
* `serving` `kernel` [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](http://arxiv.org/abs/2511.02248v1)
  > **TL;DR**: Proposes operator-level autoscaling for large generative models to replace inefficient model-level scaling. The framework profiles individual model operators and independently scales resources for each. Achieves up to 40% fewer GPUs while preserving SLOs, or 1.6x higher throughput under fixed resources.
* `training` `kernel` `networking` [Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs](http://arxiv.org/abs/2511.02168v1)
  > **TL;DR**: Addresses performance inefficiencies in distributed LLM execution across multiple GPUs due to the Bulk Synchronous Parallel (BSP) model. Proposes fine-grained programming patterns using in-kernel communication primitives to replace global barriers with dataflow synchronization. Achieves a 10-20% speedup in end-to-end latency.
* `serving` `offloading` `agentic` [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](http://arxiv.org/abs/2511.02230v1)
  > **TL;DR**: Proposes Continuum, a serving system optimizing multi-turn LLM agent job completion time by predicting tool call durations to set KV cache time-to-live and enable program-level scheduling. Reduces average job completion time and prevents scheduling bubbles for real-world agent workloads.

### 2025-11-03
* `kernel` `training` `serving` [Optimizing Attention on GPUs by Exploiting GPU Architectural NUMA Effects](http://arxiv.org/abs/2511.02132v1)
  > **TL;DR**: Addresses performance degradation of attention kernels on disaggregated GPUs due to NUMA effects. Proposes Swizzled Head-first Mapping, a scheduling strategy aligning attention heads with NUMA domains for cache reuse. Achieves 50% higher performance on AMD MI300X with L2 cache hit rates of 80-97%.

### 2025-11-02
* `serving` `networking` [FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs](http://arxiv.org/abs/2511.00807v1)
  > **TL;DR**: Explores fair, energy/carbon-efficient LLM serving on distributed, heterogeneous GPUs. Proposes FREESH, a joint routing and scheduling system that optimizes query placement and GPU frequency based on spatiotemporal power/carbon variability. Achieves 45.45% emissions reduction and 28.6% energy saving while improving SLO attainment.
* `RL` `training` [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](http://arxiv.org/abs/2511.00796v1)
  > **TL;DR**: Introduces AReaL-Hex, a heterogeneity-aware system for asynchronous RL training on heterogeneous GPUs. It uses a two-phase scheduler (MILP-based planning and graph partitioning) to assign stages optimally. On reasoning tasks, it achieves 1.50x higher throughput or 1.46x cost reduction versus homogeneous deployments.

### 2025-11-01
* `serving` `edge` `multi-modal` [EPARA: Parallelizing Categorized AI Inference in Edge Clouds](http://arxiv.org/abs/2511.00603v1)
  > **TL;DR**: Proposes EPARA, an end-to-end parallel inference framework for edge clouds that categorizes AI tasks by latency sensitivity and GPU needs. It uses a task allocator, distributed handler, and state-aware scheduler to improve resource allocation. Achieves up to 2.1x higher goodput compared to prior frameworks.

### 2025-10-31
* `networking` `serving` `RL` [RDMA Point-to-Point Communication for LLM Systems](http://arxiv.org/abs/2510.27656v1)
  > **TL;DR**: Addresses the need for flexible point-to-point communication in LLM systems using diverse NICs. Proposes TransferEngine, a uniform interface that exposes one-sided WriteImm operations with completion primitives. Achieves 400 Gbps throughput and enables 1.3-second RL weight updates for trillion-parameter models.
* `training` `multi-modal` [Synergistic Tensor and Pipeline Parallelism](http://arxiv.org/abs/2510.27257v1)
  > **TL;DR**: Proposes a synergistic schedule for tensor and pipeline parallelism to reduce communication and synchronization bubbles during LLM/MLLM training. Decouples forward/backward passes into fine-grained units and braids them into a composite sequence. Improves training throughput by up to 16% over baselines.
* `serving` `offloading` [SERFLOW: A Cross-Service Cost Optimization Framework for SLO-Aware Dynamic ML Inference](http://arxiv.org/abs/2510.27182v1)
  > **TL;DR**: Addresses cost-efficient, SLO-aware serving for dynamic ML inference where requests may exit early. Proposes SERFLOW, a framework that offloads model stages between IaaS VMs and FaaS functions using stage-specific provisioning and adaptive load balancing. Reduces cloud costs by over 23%.
* `serving` `RL` [Glia: A Human-Inspired AI for Automated Systems Design and Optimization](http://arxiv.org/abs/2510.27176v1)
  > **TL;DR**: Presents Glia, an automated AI system for designing computer systems using a multi-agent LLM workflow with reasoning and experimentation. Applied to distributed GPU clusters for LLM inference, it generates novel algorithms for routing and scheduling, achieving human-expert level performance in less time.

### 2025-10-30
* `training` `agentic` `RL` [FlowMesh: A Service Fabric for Composable LLM Workflows](http://arxiv.org/abs/2510.26913v1)
  > **TL;DR**: Proposes FlowMesh, a service fabric for fine-grained execution of composite LLM workflows (e.g., RLHF, agent workflows). It decomposes workflows into operators for cross-user deduplication, batching, and global scheduling. Achieves up to 3.8x cost reduction compared to baseline pipelines.
* `serving` `MoE` `offloading` [ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference](http://arxiv.org/abs/2510.26730v1)
  > **TL;DR**: Proposes ExpertFlow, a runtime system for MoE inference that uses adaptive expert prefetching and cache-aware routing to reduce latency from parameter transfers. It dynamically predicts expert needs to minimize cache misses, reducing model stall time to <0.1% of the baseline.
* `training` `sparse` `networking` [An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning](http://arxiv.org/abs/2510.26709v2)
  > **TL;DR**: Proposes ARC-Top-K, an All-Reduce-compatible gradient compressor for distributed training. It aligns sparsity patterns via a lightweight sketch to enable index-free communication. Achieves up to 60.7% reduction in wall-clock training time while matching Top-K accuracy.
* `RL` `training` `serving` [ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems](http://arxiv.org/abs/2510.26475v1)
  > **TL;DR**: Adapts speculative decoding to accelerate generation in RL training systems. Proposes ReSpec with dynamic SD tuning, drafter distillation, and reward-weighted updates to mitigate staleness and policy drift. Achieves up to 4.5x speedup on Qwen models (3B-14B) while preserving reward convergence.

### 2025-10-29
* `MoE` `serving` `networking` [MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference](http://arxiv.org/abs/2510.25258v1)
  > **TL;DR**: Proposes ER-Mapping and NI-Balancer to optimize MoE inference on wafer-scale chips by co-designing expert and attention layer mappings and overlapping expert migration with idle network links, achieving 62% communication reduction and 39% higher per-device performance than NVL72.

### 2025-10-27
* `edge` `offloading` `serving` [Bayes-Split-Edge: Bayesian Optimization for Constrained Collaborative Inference in Wireless Edge Systems](http://arxiv.org/abs/2510.23503v1)
  > **TL;DR**: Proposes Bayes-Split-Edge, a Bayesian optimization framework for jointly optimizing neural network split points and transmission power in edge-served inference to meet energy and latency constraints. Achieves 2.4x reduction in evaluation cost and near-linear convergence with as few as 20 function evaluations.
* `offloading` `edge` `serving` [Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions](http://arxiv.org/abs/2510.22909v1)
  > **TL;DR**: Addresses optimal placement of DL model inference across edge and cloud to balance latency, cost, and privacy. Proposes a multi-objective optimization framework for partitioning and offloading, enabling tailored deployment for latency-sensitive applications like chatbots.

### 2025-10-23
* `training` `networking` `hardware` [Collective Communication for 100k+ GPUs](http://arxiv.org/abs/2510.20171v1)
  > **TL;DR**: Addresses communication bottlenecks in training LLMs at 100k+ GPU scales. Proposes NCCLX, a collective communication framework optimizing all-reduce and all-gather for ultra-scale clusters, achieving significant throughput and latency improvements over standard NCCL on Llama4 training.
* `training` `MoE` `offloading` [AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training](http://arxiv.org/abs/2510.20111v1)
  > **TL;DR**: Proposes AsyncHZP, an asynchronous hierarchical ZeRO variant that reduces communication overhead and improves memory efficiency in large-scale LLM training. By adaptively resharding and overlapping communication with computation via multi-stream scheduling, it achieves superior scalability over ND parallelism on both dense and MoE models.

### 2025-10-22
* `MoE` `training` `networking` [HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission](http://arxiv.org/abs/2510.19470v1)
  > **TL;DR**: Addresses scalability limits of MoE training across datacenters due to low cross-DC bandwidth. Proposes HybridEP, a framework that dynamically optimizes expert and data communication patterns via modeling-guided hybrid transmission, achieving up to 5.6x speedup under constrained bandwidth.
* `training` `MoE` `networking` [RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training](http://arxiv.org/abs/2510.19262v2)
  > **TL;DR**: Addresses all-to-all communication bottlenecks in distributed MoE training by leveraging Rail topology symmetry to enable local, topology-aware load balancing. RailS uses multipath spraying and LPT scheduling to reduce iteration time by up to 40% and boost bus bandwidth by up to 78%.
* `RL` `serving` `offloading` [RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs](http://arxiv.org/abs/2510.19225v1)
  > **TL;DR**: RLBoost improves cost-efficiency of LLM reinforcement learning by harvesting preemptible GPUs for rollout stages, using adaptive offloading, pull-based weight transfer, and token-level response migration. It achieves 1.51x–1.97x higher throughput and 28%–49% better cost efficiency than on-demand-only setups.

### 2025-10-21
* `training` `sparse` `distributed` [MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training](http://arxiv.org/abs/2510.18830v1)
  > **TL;DR**: Addresses inefficient training of LLMs with ultra-long contexts due to computational imbalance. Proposes MTraining, a distributed method with dynamic sparse attention and balanced ring attention, achieving 6× higher training throughput while scaling context from 32K to 512K tokens on 32 A100 GPUs.
* `serving` `offloading` [Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](http://arxiv.org/abs/2510.18586v1)
  > **TL;DR**: Addresses KV-cache inefficiencies in LLM-based multi-agent serving due to memory contention and idle stalls. Tokencake introduces agent-aware space and time schedulers with proactive KV-cache offloading and predictive upload, reducing end-to-end latency by 47.06% and boosting GPU memory utilization by 16.9% over vLLM.
* `serving` `edge` `thinking` [SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices](http://arxiv.org/abs/2510.18544v1)
  > **TL;DR**: Addresses SLO-aware LLM inference scheduling on edge devices by optimizing TTFT, TPOT, and end-to-end latency. Proposes SLICE, a utility-driven scheduler with dynamic generation rate control, achieving up to 35× higher SLO attainment than Orca and FastServe.

### 2025-10-20
* `training` `kernel` `sparse` [Efficient Long-context Language Model Training by Core Attention Disaggregation](http://arxiv.org/abs/2510.18121v1)
  > **TL;DR**: Proposes Core Attention Disaggregation (CAD) to eliminate load imbalance in long-context LLM training by offloading attention computation to dedicated devices. Leverages stateless, composable attention kernels with dynamic rebatching, achieving up to 1.35x training throughput on 512k context lengths.

### 2025-10-19
* `serving` `offloading` `thinking` [Justitia: Fair and Efficient Scheduling for LLM Applications](http://arxiv.org/abs/2510.17015v1)
  > **TL;DR**: Addresses unfair and inefficient scheduling of LLM applications in shared GPU environments. Proposes Justitia, a memory-centric scheduler using neural demand prediction and virtual-time fair queuing to balance efficiency and worst-case guarantees, reducing completion time by up to 3.2× while ensuring fairness compared to vLLM.

### 2025-10-18
* `networking` `training` [Reimagining RDMA Through the Lens of ML](http://arxiv.org/abs/2510.16606v1)
  > **TL;DR**: Celeris redesigns RDMA transport for distributed ML training by eliminating strict reliability and in-order delivery, leveraging ML's fault tolerance to reduce tail latency. By removing retransmissions and using software-level recovery (e.g., Hadamard Transform), it cuts 99th-percentile latency by 2.3x and reduces BRAM usage by 67%.
* `serving` `offloading` `quantization` [FourierCompress: Layer-Aware Spectral Activation Compression for Efficient and Accurate Collaborative LLM Inference](http://arxiv.org/abs/2510.16418v1)
  > **TL;DR**: Addresses communication bottlenecks in collaborative LLM inference by compressing intermediate activations across edge-client and server boundaries. Proposes FourierCompress, a layer-aware FFT-based method retaining low-frequency coefficients, achieving 7.6x compression with <0.3% accuracy loss and 32x faster compression than Top-k.
* `training` `offloading` `sparse` [MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization](http://arxiv.org/abs/2510.16415v1)
  > **TL;DR**: Proposes MeCeFO, a fault-tolerant LLM training system that transfers failed node workloads to neighbors with minimal overhead using skip-connections, recomputation, and low-rank gradient approximation. Achieves 5.0–6.7× higher resilience than SOTA with only 4.18% throughput drop under failures.

### 2025-10-17
* `training` `serving` `hardware` [GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusters](http://arxiv.org/abs/2510.15652v1)
  > **TL;DR**: Proposes GOGH, a learning-based system for adaptive GPU resource orchestration in heterogeneous clusters to minimize energy and meet performance targets. Uses two neural networks to predict model-hardware compatibility and co-location effects, improving allocation over time. Reduces energy consumption by up to 27% while maintaining SLOs.
* `training` `kernel` `sparse` [PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training](http://arxiv.org/abs/2510.15596v1)
  > **TL;DR**: PRISM models stochastic performance variability in large-scale distributed training, offering probabilistic guarantees on training time. It identifies communication kernels (e.g., AllGather, ReduceScatter) as primary variability sources and enables 1.26x performance gains via placement-aware optimization, with 20.8% KS distance in prediction accuracy.
* `serving` `thinking` [BeLLMan: Controlling LLM Congestion](http://arxiv.org/abs/2510.15330v1)
  > **TL;DR**: Addresses LLM inference congestion due to uncontrolled autoregressive token generation. Proposes beLLMan, a feedback controller that dynamically adjusts output length based on system load, reducing end-to-end latency by up to 8× and energy use by 25% while increasing request throughput by 19%.

### 2025-10-16
* `serving` `multi-modal` `offloading` [xLLM Technical Report](http://arxiv.org/abs/2510.14686v1)
  > **TL;DR**: Designs xLLM, a high-performance LLM serving framework with decoupled service-engine architecture for multimodal inference, featuring dynamic PD/EPD disaggregation and global KV cache management. Achieves up to 2.2x higher throughput than vLLM-Ascend under identical TPOT constraints on Qwen models.
* `training` `offloading` `hardware` [ScalePool: Hybrid XLink-CXL Fabric for Composable Resource Disaggregation in Unified Scale-up Domains](http://arxiv.org/abs/2510.14580v1)
  > **TL;DR**: Proposes ScalePool, a hybrid XLink-CXL fabric for disaggregated memory and accelerator interconnection in LLM training clusters. Achieves 1.22x average speedup and 4.5x lower latency for memory-intensive workloads by enabling coherent, tiered memory pooling beyond RDMA.
* `serving` `offline` [FairBatching: Fairness-Aware Batch Formation for LLM Inference](http://arxiv.org/abs/2510.14392v1)
  > **TL;DR**: Addresses computational unfairness in LLM inference batching by redesigning batch formation to balance prefill and decode tasks. Introduces adaptive budgeting and dynamic scheduling to reduce TTFT tail latency by up to 2.29x while improving single-node and cluster capacity by 20.0% and 54.3%, respectively.

### 2025-10-15
* `serving` `offloading` `networking` [FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access](http://arxiv.org/abs/2510.13724v1)
  > **TL;DR**: FIRST enables federated LLM inference across distributed HPC clusters using a cloud-like API, auto-scaling resources and maintaining hot nodes for low-latency serving. It achieves scalable, on-premises generation of billions of tokens daily without commercial cloud reliance.
* `serving` `offloading` `thinking` [Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference](http://arxiv.org/abs/2510.13668v1)
  > **TL;DR**: Addresses workload imbalance in disaggregated LLM inference due to unpredictable decode lengths. Proposes ARES, a system using LLM-internal state to predict output length and dynamically reschedule decode tasks, reducing P99 TPOT by 74.77% and improving goodput by 2.24×.
* `quantization` `hardware` `edge` [F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs](http://arxiv.org/abs/2510.13401v1)
  > **TL;DR**: Designs a hardware accelerator (F-BFQ) to efficiently execute mixed block-floating-point quantized LLMs on edge devices, dynamically switching between BFP variants without reconfiguration. Achieves 1.4x faster inference than Arm NEON CPU and 5.2 tokens/s on AMD Kria.
* `serving` `offloading` `RAG` [BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure](http://arxiv.org/abs/2510.13223v1)
  > **TL;DR**: Addresses load imbalance in disaggregated LLM serving by dynamically migrating KV cache and model layers between prefill and decode nodes. Enables cache-agnostic scheduling with overlapped transmission, achieving 1.2x-3.9x higher throughput and up to 78.4% lower latency than vLLM.

### 2025-10-14
* `RL` `training` `offloading` [Laminar: A Scalable Asynchronous RL Post-Training Framework](http://arxiv.org/abs/2510.12633v1)
  > **TL;DR**: Addresses scalability bottlenecks in RL post-training of LLMs caused by trajectory latency skew. Proposes Laminar, a fully decoupled architecture with tiered relay workers for asynchronous parameter updates and dynamic trajectory repackaging. Achieves up to 5.48× training throughput speedup on a 1024-GPU cluster.

### 2025-10-13
* `training` `serving` `offloading` [An Explorative Study on Distributed Computing Techniques in Training and Inference of Large Language Models](http://arxiv.org/abs/2510.11211v1)
  > **TL;DR**: Explores distributed computing techniques for training and serving LLMs, including system modifications to enable consumer-grade deployment and a comparative analysis of three serving frameworks. Implements a metaheuristic-based offloading method that reduces memory usage by up to 40% on consumer hardware.

### 2025-10-12
* `training` `sparse` `offloading` [DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism](http://arxiv.org/abs/2510.10620v1)
  > **TL;DR**: Addresses dynamic sequence length and attention pattern variability in long-context LLM training. Proposes DCP, a fine-grained dynamic context parallelism framework that adapts data and computation blocks to device resources, achieving up to 1.46x end-to-end training speed-up over static methods.

### 2025-10-11
* `serving` `offloading` `MoE` [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](http://arxiv.org/abs/2510.10302v1)
  > **TL;DR**: Proposes SP-MoE, an SD-aware offloading framework for MoE-based LLM inference that prefetches experts using draft-target model correspondence and pipelines I/O to reduce latency. Achieves 1.07–3.5× speedup in tokens-per-token (TPOT) over state-of-the-art methods.

### 2025-10-10
* `serving` `hardware` [SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference](http://arxiv.org/abs/2510.08544v1)
  > Designs specialized hardware chips for disaggregated LLM inference, tailoring prefill (compute-heavy) and decode (memory-heavy) stages to their distinct workloads. SPAD achieves 19%-41% lower hardware cost and 2%-17% lower TDP than GPU-based systems while maintaining performance.

### 2025-10-09
* `serving` `MoE` `offloading` [From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill](http://arxiv.org/abs/2510.08055v1)
  > Addresses high energy and latency in MoE LLM serving due to redundant expert weight loads during chunked prefill. Proposes layered prefill, which schedules by layer groups instead of tokens to eliminate reloads. Reduces TTFT by up to 70% and per-token energy by 22% while maintaining stall-free decoding.

### 2025-10-08
* `kernel` [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](http://arxiv.org/abs/2510.06834v1)
  > **TL;DR**: Vectorizes FlashAttention on RISC-V vector processors using low-cost exponential approximations and tiling to optimize attention kernels. Achieves significant performance gains without custom ISA extensions.
* `RL` `training` [EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models](http://arxiv.org/abs/2510.05943v1)
  > **TL;DR**: Addresses scalability bottlenecks in agentic RL training for LLMs by dynamically adapting parallelism and decentralizing intermediate data exchange. Achieves 2.1× higher throughput and eliminates OOM failures during long-context RL training.

### 2025-10-01 ~ 2025-10-07
* `agentic` `offloading` `serving` [Toward Systems Foundations for Agentic Exploration](http://arxiv.org/abs/2510.05556v1)
  > **TL;DR**: Addresses the need for efficient execution branching in LLM agents. Proposes foundational support for fork semantics, side-effect isolation, and microsecond-scale forking. Enables scalable agentic exploration with order-of-magnitude faster snapshot/restore than existing tools.
* `MoE` `serving` [Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting](http://arxiv.org/abs/2510.05497v1)
  > **TL;DR**: Addresses data movement bottlenecks in large-scale MoE LLM serving by forecasting expert selection patterns via massive profiling, enabling architectural optimizations that achieve up to 6.3X speedup on DeepSeek V3.
* `training` `offloading` [OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training](http://arxiv.org/abs/2510.05186v1)
  > **TL;DR**: Optimizes pipeline parallelism for LLM training by formulating scheduling as a constrained optimization problem that balances memory, computation, and bubble minimization. Dynamically adjusts activation offloading and scheduling to reduce idle time by up to 50% under fixed memory constraints.
* `agentic` `serving` `thinking` [Speculative Actions: A Lossless Framework for Faster Agentic Systems](http://arxiv.org/abs/2510.04371v1)
  > **TL;DR**: Proposes speculative actions to accelerate agentic systems by predicting future actions with faster models, enabling parallel execution. Achieves up to 55% accuracy in action prediction and significantly reduces end-to-end latency in real-world environments.
* `serving` `thinking` `MoE` [SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading](http://arxiv.org/abs/2510.05164v1)
  > **TL;DR**: How to reduce cost and latency of LLM inference while maintaining performance? SATER introduces a self-aware routing and cascade framework with confidence-aware rejection and preference optimization, cutting computational cost by >50% and cascade latency by >80%.
* `diffusion` `MoE` [Paris: A Decentralized Trained Open-Weight Diffusion Model](http://arxiv.org/abs/2510.03434v1)
  > **TL;DR**: Can high-quality diffusion models be trained decentralized without gradient synchronization? Paris uses 8 isolated expert diffusion models with a router for inference, achieving comparable quality to centralized baselines using 16× less compute and 14× less data.
* `serving` `diffusion` [TridentServe: A Stage-level Serving System for Diffusion Pipelines](http://arxiv.org/abs/2510.02838v1)
  > **TL;DR**: Addresses inefficient static serving of diffusion pipelines by introducing dynamic stage-level resource allocation. TridentServe co-optimizes stage placement and request routing, achieving up to 4.1x lower P95 latency while improving SLO attainment.
* `MoE` `serving` [ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models](http://arxiv.org/abs/2510.02613v1)
  > **TL;DR**: Enables fine-grained, zero-downtime scaling of MoE LLMs during inference by decoupling execution from memory operations and using zero-copy remapping. Achieves up to 9× lower scale-up latency and 2× higher throughput during scaling.
* `serving` `diffusion` [TetriServe: Efficient DiT Serving for Heterogeneous Image Generation](http://arxiv.org/abs/2510.01565v1)
  > **TL;DR**: Addresses efficient serving of DiT models under heterogeneous SLOs by introducing step-level sequence parallelism and round-based scheduling. TetriServe dynamically adjusts parallelism per request step, achieving up to 32% higher SLO attainment while maintaining image quality.
* `agentic` `serving` [FlashResearch: Real-time Agent Orchestration for Efficient Deep Research](http://arxiv.org/abs/2510.05145v1)
  > **TL;DR**: How to accelerate deep research agents by parallelizing sequential reasoning? FlashResearch dynamically decomposes queries into tree-structured tasks and orchestrates parallel execution across breadth and depth, achieving 5x speedup while maintaining report quality.
* `training` `networking` [An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters](http://arxiv.org/abs/2510.00991v1)
  > **TL;DR**: Designs ICCL, a collective communication library for large-scale LLM training, to improve P2P efficiency, tolerate NIC failures, and enable microsecond-level anomaly observability, achieving 28.5% lower latency and 6.02% higher training throughput than NCCL.
* `training` `offloading` [ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training](http://arxiv.org/abs/2510.00606v3)
  > **TL;DR**: Designs an elastic-native LLM training system that maintains parameter consistency and low recovery time during dynamic scaling. Introduces multi-dimensional scheduling, online resharding, and asynchronous migration with in-memory snapshots. Achieves 1.6× higher throughput and 51% lower MTTR than baselines.
* `MoE` `training` [FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training](http://arxiv.org/abs/2510.00207v2)
  > **TL;DR**: FlowMoE develops a unified pipeline scheduling framework for distributed MoE training by integrating MHA, gating, expert computation, and communication. It uses tensor chunk-based priority scheduling to overlap all-reduce with computing, reducing training time by up to 57%.
* `training` `kernel` [LoRAFusion: Efficient LoRA Fine-Tuning for LLMs](http://arxiv.org/abs/2510.00206v1)
  > **TL;DR**: Improves LoRA fine-tuning efficiency by fusing memory-bound ops via kernel optimization and scheduling multiple LoRA adapters with adaptive batching. Achieves up to 1.96× speedup over Megatron-LM and 1.46× over mLoRA.
* `networking` `training` `serving` [Lattica: A Decentralized Cross-NAT Communication Framework for Scalable AI Inference and Training](http://arxiv.org/abs/2510.00183v2)
  > **TL;DR**: Designs a decentralized cross-NAT framework to enable scalable AI training and inference without centralized infrastructure. Uses NAT traversal, CRDTs, and DHT-based discovery for peer-to-peer model synchronization. Achieves reliable communication in permissionless environments with low latency and high throughput.
* `serving` [TASP: Topology-aware Sequence Parallelism](http://arxiv.org/abs/2509.26541v2)
  > **TL;DR**: Addresses inefficient communication in sequence parallelism for long-context LLMs by decomposing Ring AllGather into topology-aware concurrent ring paths. TASP exploits AlltoAll accelerator topology to boost communication efficiency, achieving up to 3.58x speedup over Ring Attention.
* `serving` `offloading` [Parallax: Efficient LLM Inference Service over Decentralized Environment](http://arxiv.org/abs/2509.26182v1)
  > **TL;DR**: How to efficiently serve LLMs over decentralized, heterogeneous GPU pools? Parallax uses a two-phase scheduler for layer-wise model allocation and dynamic pipeline construction, reducing latency by up to 40% and improving throughput versus decentralized baselines.

### 2025-09-16 ~ 2025-09-30
* `serving` `storage` [Accelerating LLM Inference with Precomputed Query Storage](http://arxiv.org/abs/2509.25919v1)
  > **TL;DR**: Reduces LLM inference latency by precomputing and storing response pairs for predictable queries. Uses LLM-driven query generation and disk-backed vector indexing for efficient retrieval. Achieves up to 17.3% latency reduction without compromising response quality.
* `edge` `hardware` `kernel` [Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units](http://arxiv.org/abs/2509.25155v1)
  > **TL;DR**: How to enable efficient long-context LLM inference on edge NPUs? Analyzes quadratic vs. sub-quadratic attention operators on NPUs, identifying memory-bound vs. compute-bound bottlenecks, and proposes hardware-aware model co-design. Achieves up to 95% reduction in pipeline stalls for long contexts.
* `MoE` `serving` [GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference](http://arxiv.org/abs/2509.25041v1)
  > **TL;DR**: Addresses high communication overhead and load imbalance in distributed MoE inference. Proposes GRACE-MoE with expert grouping, dynamic replication, and locality-aware routing to co-optimize communication and computation. Achieves up to 3.79x latency reduction over SOTA systems.
* `MoE` `serving` `load-balancing` [From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing](http://arxiv.org/abs/2510.03293v1)
  > **TL;DR**: How to improve MoE inference efficiency without retraining? LASER dynamically balances expert load using gate score distributions, routing tokens to least-loaded experts when scores are ambiguous. Achieves up to 30% lower latency and higher throughput on Mixtral and DeepSeek-MoE with near-zero accuracy drop.
* `training` [HAPT: Heterogeneity-Aware Automated Parallel Training on Heterogeneous Clusters](http://arxiv.org/abs/2509.24859v1)
  > **TL;DR**: HAPT automates parallel training on heterogeneous GPU clusters by optimizing inter-operator parallel strategies and adaptive 1F1B scheduling to maximize computation-communication overlap, achieving 1.3x-1.6x higher training throughput than existing frameworks.
* `serving` `offloading` [SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving](http://arxiv.org/abs/2509.24626v1)
  > **TL;DR**: How to enable efficient serving of long-context LLMs using dynamic sparse attention? SparseServe introduces hierarchical HBM-DRAM KV cache management with fragmentation-aware transfers, working-set-aware batching, and layer-segmented prefill, achieving up to 9.26x lower TTFT and 3.14x higher throughput.
* `serving` `multi-modal` [RServe: Overlapping Encoding and Prefill for Efficient LMM Inference](http://arxiv.org/abs/2509.24381v1)
  > **TL;DR**: Addressing high latency in LMM inference, REDServe overlaps multimodal encoding with language model prefill via disaggregation and fine-grained scheduling, achieving up to 66% lower latency and 109% higher throughput.
* `RL` `training` [RL in the Wild: Characterizing RLVR Training in LLM Deployment](http://arxiv.org/abs/2509.25279v1)
  > **TL;DR**: Characterizes system challenges in RL with verifiable rewards (RLVR) for LLM training, identifying issues like GPU idling and load imbalance due to skewed workloads; proposes PolyTrace benchmark achieving 94.7% accuracy in workload simulation.
* `serving` `training` `edge` [MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment](http://arxiv.org/abs/2510.03283v1)
  > **TL;DR**: How to jointly serve LLM inference and fine-tuning on edge devices without violating SLOs? MACE collocates inference and iteration-level retraining with dynamic GPU resource allocation, achieving up to 63% lower latency while maintaining >85% GPU utilization on NVIDIA AGX Orin.
* `training` `sparse` [AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models](http://arxiv.org/abs/2509.23722v1)
  > **TL;DR**: Addresses pipeline bubbles in LLM training caused by model heterogeneity by jointly optimizing partition, placement, and scheduling. AdaPtis uses a performance model to guide adaptive pipeline parallelism, achieving up to 2.14x speedup over Megatron-LM.
* `serving` [A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving](http://arxiv.org/abs/2509.23384v3)
  > **TL;DR**: Addresses inefficient two-layer LLM serving by introducing predictive, synergistic scheduling to bridge cluster- and engine-layer information gaps. Uses a performancemodel for adaptive batching and state-driven routing, improving SLO attainment by 43% and throughput by 3x.
* `serving` `quantization` `edge` [Scaling LLM Test-Time Compute with Mobile NPU on Smartphones](http://arxiv.org/abs/2509.23324v1)
  > **TL;DR**: Can smaller LLMs match larger models' accuracy on smartphones by leveraging underused NPU compute via test-time scaling? Proposes hardware-aware quantization and LUT optimizations for NPU-efficient inference, achieving up to 2.2× speedup and matching larger model accuracy.
* `training` [A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training](http://arxiv.org/abs/2510.05112v2)
  > **TL;DR**: Designs FlexPipe, a programmable framework for automated and customizable pipeline parallelism in DNN training, using a DSL and scheduler to explore efficient schedules; achieves up to 2.28× speedup over Megtron-LM.
* `MoE` `quantization` `offloading` [Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression](http://arxiv.org/abs/2510.02345v1)
  > **TL;DR**: Addresses the MoE trilemma by dynamically clustering experts and applying structured compression with hierarchical routing, mixed-precision storage (FP16/INT4), and dynamic offloading, reducing parameters by 80% while improving throughput by 10–20% and load balance by 3×.
* `training` `hardware` [Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM](http://arxiv.org/abs/2509.22832v1)
  > **TL;DR**: Predicts end-to-end LLM training time across distributed GPUs by decomposing models into primitives and using lightweight hardware-aware models. Achieves <10% prediction error on 20B models across 128 GPUs while running entirely on CPU.
* `training` `sparse` [Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training](http://arxiv.org/abs/2509.21841v2)
  > **TL;DR**: Addresses load imbalance in data-parallel LLM training due to variable sequence lengths. Introduces hierarchical sequence partitioning, dynamic NIC routing, and module-aware remapping to balance computation and communication. Achieves 2.80x speedup over state-of-the-art.
* `multi-modal` `serving` `edge` `kernel` [Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices](http://arxiv.org/abs/2510.05109v1)
  > **TL;DR**: Designs a software-hardware co-designed system to efficiently run multimodal models on battery-powered edge devices by modularly scheduling vision, audio, and language components across heterogeneous accelerators. Uses optimized low-bit kernels and token-aware buffering, reducing energy consumption by 42.3% and enabling all-day LMM inference on-device.
* `training` `sparse` [Data-Centric Elastic Pipeline Parallelism for Efficient Long-Context LLM Training](http://arxiv.org/abs/2509.21275v1)
  > **TL;DR**: Addresses inefficient pipeline parallelism in long-context LLM training by adaptively switching between token- and batch-level partitioning. InfiniPipe uses workload-aware sequence packing and stage-aware checkpointing to balance load and reduce memory. Achieves 1.69x speedup over SOTA.
* `training` `offloading` [SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips](http://arxiv.org/abs/2509.21271v1)
  > **TL;DR**: How to optimize large-scale LLM training on Superchips using offloading? SuperOffload introduces Superchip-aware techniques like adaptive weight offloading and CPU-optimized Adam, achieving 2.5x higher throughput and enabling 25B model training on a single GH200.
* `training` `networking` [Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models](http://arxiv.org/abs/2509.21221v1)
  > **TL;DR**: Proposes GWTF, a decentralized framework for LLM training tolerant to node churn and network instability. Uses a novel flow-based routing algorithm to optimize microbatch scheduling across heterogeneous clients. Reduces training time by up to 45% under high churn.
* `RL` `serving` [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](http://arxiv.org/abs/2509.21009v1)
  > **TL;DR**: Addresses GPU underutilization in synchronous RL post-training caused by long-tail response delays. Introduces tail batching to schedule long responses separately, enabling balanced workloads across training stages. Achieves up to 2.56x faster training time on 128 H800 GPUs.
* `serving` [PARS: Low-Latency LLM Serving via Pairwise Learning-to-Rank](http://arxiv.org/abs/2510.03243v1)
  > **TL;DR**: How to reduce LLM serving latency caused by Head-of-Line blocking? PARS uses pairwise learning-to-rank to predict optimal task ordering by response length, integrated into vLLM. Achieves up to 35% lower average latency without sacrificing throughput.
* `training` `serving` [Kant: An Efficient Unified Scheduling System for Large-Scale AI Clusters](http://arxiv.org/abs/2510.01256v1)
  > **TL;DR**: Designs Kant, a unified scheduler for co-scheduling LLM training and inference in large AI clusters. Uses Backfill and E-Binpack to improve GPU utilization and reduce fragmentation. Achieves up to 30% higher GPU Allocation Ratio (GAR) compared to baseline schedulers.
* `serving` `hardware` `offline` [Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute](http://arxiv.org/abs/2509.20241v1)
  > **TL;DR**: Estimates energy use per LLM inference query at scale, accounting for real-world GPU utilization and PUE. Proposes a bottom-up methodology to quantify efficiency gains at model, platform, and hardware levels, achieving up to 20x reduction in energy per query for 1B queries/day.
* `training` `offloading` [BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens](http://arxiv.org/abs/2509.19836v1)
  > **TL;DR**: Designs BurstEngine to efficiently train LLMs on sequences >1M tokens by introducing topology-aware BurstAttention, selective checkpointing, and fused loss computation, achieving 1.2× speedup and reduced memory overhead compared to state-of-the-art methods.
* `serving` `offloading` [Gyges: Dynamic Cross-Instance Parallelism Transformation for Efficient LLM Inference](http://arxiv.org/abs/2509.19729v1)
  > **TL;DR**: Addresses dynamic context length variance in LLM serving by adaptively transforming parallelism strategies across instances. Proposes a header-centric KV cache layout, weight padding, and transformation-aware scheduler, achieving up to 6.57× higher throughput than state-of-the-art systems.
* `RAG` [On The Reproducibility Limitations of RAG Systems](http://arxiv.org/abs/2509.18869v1)
  > **TL;DR**: Addresses the reproducibility limitations of RAG systems by introducing ReproRAG, a benchmarking framework that quantifies non-determinism across embedding models, retrieval algorithms, and hardware. Evaluates trade-offs using metrics like Exact Match Rate and Jaccard Similarity.
* `MoE` `serving` [Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving](http://arxiv.org/abs/2509.17863v1)
  > **TL;DR**: Addresses efficient serving of large-scale MoE models by disaggregating experts into stateless services. Uses peer-to-peer communication for low-overhead routing and dynamic resource scaling. Achieves 37.5% resource savings with <2% throughput loss under failures.
* `serving` `hardware` [Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs](http://arxiv.org/abs/2509.17542v2)
  > **TL;DR**: Designs a disaggregated LLM inference system using heterogeneous GPUs to separate prefill and decoding stages, enabling cost-efficient deployment. Introduces a heterogeneous-compatible transmission module and joint optimization for parallelism and instance allocation. Achieves 38% higher resource utilization compared to homogeneous setups.
* `agentic` `serving` `RAG` [Asteria: Semantic-Aware Cross-Region Caching for Agentic LLM Tool Access](http://arxiv.org/abs/2509.17360v1)
  > **TL;DR**: Asteria improves agentic LLM performance by introducing semantic-aware cross-region caching for tool access. It uses semantic embeddings and a lightweight LLM judger for precise retrieval, achieving 3.6× higher throughput with 85%+ cache hit rates while maintaining accuracy.
* `serving` `heterogeneous` [Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill](http://arxiv.org/abs/2509.17357v1)
  > **TL;DR**: Cronus improves LLM inference throughput on heterogeneous GPU clusters by partially disaggregating prefill across low- and high-end GPUs, overlapping stages to balance load. It reduces P99 TTFT and TBT by up to 40% while maintaining high throughput.
* `multi-modal` `offloading` `edge` [MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference](http://arxiv.org/abs/2509.16995v1)
  > **TL;DR**: How to efficiently infer multimodal LLMs on edge devices? MoA-Off proposes a modality-aware offloading framework that dynamically splits computation between edge and cloud based on input complexity, reducing latency by 30% and resource overhead by 30%-65% while preserving accuracy.
* `serving` `offloading` `hardware` [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](http://arxiv.org/abs/2509.16857v1)
  > **TL;DR**: Addresses KV cache fetch interference in distributed prefix caching for LLM serving. Proposes ShadowServe, a SmartNIC-offloaded system with chunked pipelining and minimal-copy memory management. Achieves up to 2.2x lower TPOT and 1.38x lower TTFT in low-bandwidth scenarios.
* `serving` [Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads](http://arxiv.org/abs/2509.16495v1)
  > **TL;DR**: Addresses the latency-throughput tradeoff in LLM serving by introducing Shift Parallelism, which dynamically switches between tensor and sequence parallelism to leverage KV cache invariance. Achieves 1.51× lower latency in interactive workloads and 50% higher throughput in batch workloads than tensor parallelism alone.
* `training` [Robust LLM Training Infrastructure at ByteDance](http://arxiv.org/abs/2509.16293v2)
  > **TL;DR**: How to ensure stable large-scale LLM training amid frequent failures? ByteRobust introduces data-driven fault detection and recovery tailored to LLM parallelism, achieving 97% effective training time ratio (ETTR) over 9,600 GPUs.
* `RL` `training` [RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation](http://arxiv.org/abs/2509.15965v1)
  > **TL;DR**: Addresses low hardware utilization in RL training by introducing M2Flow, a macro-to-micro flow transformation that optimizes workflow execution via context switching and elastic pipelining. Achieves 1.1x–2.13x speedup in end-to-end training throughput.
* `networking` `training` [Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs](http://arxiv.org/abs/2509.15940v1)
  > **TL;DR**: Addresses communication inefficiencies in large-scale LLM training by aligning communication patterns with data center topology. Proposes Arnold, a scheduling system that reduces communication spread and improves end-to-end training throughput by 10.6% on 9600+ GPUs.
* `hardware` `networking` [PCCL: Photonic circuit-switched collective communication for distributed ML](http://arxiv.org/abs/2509.15450v1)
  > **TL;DR**: Addresses communication bottlenecks in distributed ML training by reconfiguring photonic networks to eliminate congestion. Proposes PCCL, a hardware-agnostic system that creates direct circuits for collective operations, achieving up to 3X faster communication and 1.3X end-to-end training speedup.
* `RAG` `agentic` [LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology](http://arxiv.org/abs/2509.13978v2)
  > **TL;DR**: Explores using LLM agents to interpret and query scientific workflow provenance via natural language. Combines metadata-driven design and RAG to translate prompts into structured queries, achieving high accuracy on real-world chemistry workflows.
* `serving` `kernel` `hardware` [FLAME: A Serving System Optimized for Large-Scale Generative Recommendation with Efficiency](http://arxiv.org/abs/2509.22681v1)
  > **TL;DR**: Designs a production-grade serving system for large-scale generative recommendation models by decoupling pre-processing and computation, optimizing memory with PDA, and accelerating inference via TensorRT-based fused kernels. Achieves up to 6.3x throughput gain and 2.3x latency reduction.
* `diffusion` `hardware` [AERIS: Argonne Earth Systems Model for Reliable and Skillful Predictions](http://arxiv.org/abs/2509.13523v1)
  > **TL;DR**: Designs AERIS, a billion-parameter Swin diffusion transformer for high-resolution weather prediction, using SWiPe to enable efficient window, sequence, and pipeline parallelism; achieves 10.21 ExaFLOPS on Aurora with 95.5% weak scaling efficiency.
* `serving` `offloading` [Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management](http://arxiv.org/abs/2509.13201v1)
  > **TL;DR**: Addresses how to improve throughput of non-latency-sensitive LLM inference on opportunistic GPU clusters. Introduces pervasive context management to reuse computational context across dynamic resources, enabling 98.1% reduction in execution time.
* `agentic` `serving` `offloading` [Toward Systems Foundations for Agentic Exploration](http://arxiv.org/abs/2510.05556v1)
  > **TL;DR**: Addresses the lack of efficient system support for LLM agentic exploration. Proposes fundamental challenges in fork semantics, side-effect handling, and microsecond-level native forking. Achieves scalable branching with minimal overhead in multi-agent deployments.
* `agentic` `serving` `offloading` [Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization](http://arxiv.org/abs/2509.21301v1)
  > **TL;DR**: Nova enables efficient real-time serving of agentic vision-language models by adaptively partitioning GPU resources across vision, prefill, and decode stages, plus lightweight vision encoder offloading. It achieves up to 23.3% lower max latency while maintaining high throughput.

### 2025-09-15

* [Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective](https://arxiv.org/abs/2509.10371)
* [Ordered Consensus with Equal Opportunity](https://arxiv.org/abs/2509.09868)
* [The (R)evolution of Scientific Workflows in the Agentic AI Era: Towards Autonomous Science](https://arxiv.org/abs/2509.09915)
* [DBOS Network Sensing: A Web Services Approach to Collaborative Awareness](https://arxiv.org/abs/2509.09898)
* [XBOF: A Cost-Efficient CXL JBOF with Inter-SSD Compute Resource Sharing](https://arxiv.org/abs/2509.10251)
* [DBOS Network Sensing: A Web Services Approach to Collaborative Awareness](https://arxiv.org/abs/2509.09898)

### 2025-09-12

* [TrEnv: Transparently Share Serverless Execution Environments Across Different Functions and Nodes](https://arxiv.org/abs/2509.09525)
* [Weaker Assumptions for Asymmetric Trust](https://arxiv.org/abs/2509.09493)
* [Barycentric Coded Distributed Computing with Flexible Recovery Threshold for Collaborative Mobile Edge Computing](https://arxiv.org/abs/2509.09435)
* [WebAssembly and Unikernels: A Comparative Study for Serverless at the Edge](https://arxiv.org/abs/2509.09400)
* [Coherence-Aware Task Graph Modeling for Realistic Application](https://arxiv.org/abs/2509.09094)
* [Optimizing the Variant Calling Pipeline Execution on Human Genomes Using GPU-Enabled Machines](https://arxiv.org/abs/2509.09058)
* [A Comparative Analysis of Identifier Schemes: UUIDv4, UUIDv7, and ULID for Distributed Systems](https://arxiv.org/abs/2509.08969)
* [Towards A High-Performance Quantum Data Center Network Architecture](https://arxiv.org/abs/2509.09653)
* [HARD: A Performance Portable Radiation Hydrodynamics Code based on FleCSI Framework](https://arxiv.org/abs/2509.08971)
* [μFork: Supporting POSIX fork Within a Single-Address-Space OS](https://arxiv.org/abs/2509.09439)
* [TrEnv: Transparently Share Serverless Execution Environments Across Different Functions and Nodes](https://arxiv.org/abs/2509.09525)

### 2025-09-11

* [Reconfigurable Holographic Surfaces and Near Field Communication for Non-Terrestrial Networks: Potential and Challenges](https://arxiv.org/abs/2509.08770)
* [A 410GFLOP/s, 64 RISC-V Cores, 204.8GBps Shared-Memory Cluster in 12nm FinFET with Systolic Execution Support for Efficient B5G/6G AI-Enhanced O-RAN](https://arxiv.org/abs/2509.08608)
* [An HPC Benchmark Survey and Taxonomy for Characterization](https://arxiv.org/abs/2509.08347)
* [Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism](https://arxiv.org/abs/2509.08309)
* [Design and Implementation of Code Completion System Based on LLM and CodeBERT Hybrid Subsystem](https://arxiv.org/abs/2509.08215)
* [Aurora: Architecting Argonne's First Exascale Supercomputer for Accelerated Scientific Discovery](https://arxiv.org/abs/2509.08207)
* [Towards Scalable Proteomics: Opportunistic SMC Samplers on HTCondor](https://arxiv.org/abs/2509.08020)

### 2025-09-10

* [Scaling atomic ordering in shared memory](https://arxiv.org/abs/2509.07781)
* [AgentX: Towards Orchestrating Robust Agentic Workflow Patterns with FaaS-hosted MCP Services](https://arxiv.org/abs/2509.07595)
* [Navigating Energy Doldrums: Modeling the Impact of Energy Price Volatility on HPC Cost of Ownership](https://arxiv.org/abs/2509.07567)
* [Astra: A Multi-Agent System for GPU Kernel Performance Optimization](https://arxiv.org/abs/2509.07506)
* [DREAMS: Decentralized Resource Allocation and Service Management across the Compute Continuum Using Service Affinity](https://arxiv.org/abs/2509.07497)
* [Dependency-Aware Execution Mechanism in Hyperledger Fabric Architecture](https://arxiv.org/abs/2509.07425)
* [DuoServe-MoE: Dual-Phase Expert Prefetch and Cache Scheduling for Efficient MoE LLM Inference](https://arxiv.org/abs/2509.07379)
* [Optimizing Task Scheduling in Fog Computing with Deadline Awareness](https://arxiv.org/abs/2509.07378)
* [A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows](https://arxiv.org/abs/2509.07199)
* [Bodega: Serving Linearizable Reads Locally from Anywhere at Anytime via Roster Leases](https://arxiv.org/abs/2509.07158)
* [Crossword: Adaptive Consensus for Dynamic Data-Heavy Workloads](https://arxiv.org/abs/2509.07157)
* [MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?](https://arxiv.org/abs/2509.07727)
* [HYLU: Hybrid Parallel Sparse LU Factorization](https://arxiv.org/abs/2509.07690)
* [veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD](https://arxiv.org/abs/2509.07003)

### 2025-09-09

* [IM-PIR: In-Memory Private Information Retrieval](https://arxiv.org/abs/2509.06514)
* [MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS](https://arxiv.org/abs/2509.06362)
* [FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving](https://arxiv.org/abs/2509.06261)
* [20 Years in Life of a Smart Building: A retrospective](https://arxiv.org/abs/2509.06229)
* [Gathering in Non-Vertex-Transitive Graphs Under Round Robin](https://arxiv.org/abs/2509.06064)
* [DISTRIBUTEDANN: Efficient Scaling of a Single DISKANN Graph Across Thousands of Computers](https://arxiv.org/abs/2509.06046)
* [A Simple and Robust Protocol for Distributed Counting](https://arxiv.org/abs/2509.05870)
* [Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats](https://arxiv.org/abs/2509.05303)
* [Distributed Automatic Generation Control subject to Ramp-Rate-Limits: Anytime Feasibility and Uniform Network-Connectivity](https://arxiv.org/abs/2509.06588)
* [Tackling Device Data Distribution Real-time Shift via Prototype-based Parameter Editing](https://arxiv.org/abs/2509.06552)
* [Several Performance Bounds on Decentralized Online Optimization are Highly Conservative and Potentially Misleading](https://arxiv.org/abs/2509.06466)
* [Social Dynamics of DAOs: Power, Onboarding, and Inclusivity](https://arxiv.org/abs/2509.06163)
* [Introduction to Number Theoretic Transform](https://arxiv.org/abs/2509.05884)
* [Tiga: Accelerating Geo-Distributed Transactions with Synchronized Clocks [Technical Report]](https://arxiv.org/abs/2509.05759)
* [Distributed Deep Learning using Stochastic Gradient Staleness](https://arxiv.org/abs/2509.05679)
* [Workflow for High-Fidelity Dynamic Analysis of Structures with Pile Foundation](https://arxiv.org/abs/2509.05675)
* [Efficient Fault Localization in a Cloud Stack Using End-to-End Application Service Topology](https://arxiv.org/abs/2509.05511)
* [MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs](https://arxiv.org/abs/2509.05488)

### 2025-09-08

* [Scaling Performance of Large Language Model Pretraining](https://arxiv.org/abs/2509.05258)
* [Dynamic reconfiguration for malleable applications using RMA](https://arxiv.org/abs/2509.05248)
* [Toward Distributed 3D Gaussian Splatting for High-Resolution Isosurface Visualization](https://arxiv.org/abs/2509.05216)
* [VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving](https://arxiv.org/abs/2509.04827)
* [STADI: Fine-Grained Step-Patch Diffusion Parallelism for Heterogeneous GPUs](https://arxiv.org/abs/2509.04719)

### 2025-09-05

* [On the impact of unlimited computational power in OBLOT: consequences for synchronous robots on graphs](https://arxiv.org/abs/2509.04383)
* [Trustworthy Second-hand Marketplace for Built Environment](https://arxiv.org/abs/2509.04085)
* [LowDiff: Efficient Frequent Checkpointing via Low-Cost Differential for High-Performance Distributed Training Systems](https://arxiv.org/abs/2509.04084)
* [Counterfactual simulations for large scale systems with burnout variables](https://arxiv.org/abs/2509.04038)
* [Gathering of asynchronous robots on circle with limited visibility using finite communication](https://arxiv.org/abs/2509.04004)
* [Distributed Download from an External Data Source in Asynchronous Faulty Settings](https://arxiv.org/abs/2509.03755)
* [Combining Performance and Productivity: Accelerating the Network Sensing Graph Challenge with GPUs and Commodity Data Science Software](https://arxiv.org/abs/2509.03653)
* [Massively-Parallel Implementation of Inextensible Elastic Rods Using Inter-block GPU Synchronization](https://arxiv.org/abs/2509.04277)
* [Cloud-Assisted Remote Control for Aerial Robots: From Theory to Proof-of-Concept Implementation](https://arxiv.org/abs/2509.04095)
* [Prob-GParareal: A Probabilistic Numerical Parallel-in-Time Solver for Differential Equations](https://arxiv.org/abs/2509.03945)
* [Towards Deterministic Sub-0.5 us Response on Linux through Interrupt Isolation](https://arxiv.org/abs/2509.03855)

### 2025-09-04

* [CloudFormer: An Attention-based Performance Prediction for Public Clouds with Unknown Workload](https://arxiv.org/abs/2509.03394)
* [Efficient and Secure Sleepy Model for BFT Consensus](https://arxiv.org/abs/2509.03145)
* [The High Cost of Keeping Warm: Characterizing Overhead in Serverless Autoscaling Policies](https://arxiv.org/abs/2509.03104)
* [FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs](https://arxiv.org/abs/2509.03047)
* [Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training](https://arxiv.org/abs/2509.03018)
* [A Novel IaaS Tax Model as Leverage Towards Green Cloud Computing](https://arxiv.org/abs/2509.02767)
* [DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling](https://arxiv.org/abs/2509.03472)
* [A description of the radio astronomy data processing tool DDF Pipeline](https://arxiv.org/abs/2509.03075)
* [Treasure Hunt in Anonymous Graphs with Quantum Pebbles by Oblivious Agents](https://arxiv.org/abs/2509.02909)
* [\textit{In Silico} Benchmarking of Detectable Byzantine Agreement in Noisy Quantum Networks](https://arxiv.org/abs/2509.02629)
* [On the Optimization of Methods for Establishing Well-Connected Communities](https://arxiv.org/abs/2509.02590)
* [Safe Sharing of Fast Kernel-Bypass I/O Among Nontrusting Applications](https://arxiv.org/abs/2509.02899)

### 2025-09-03

* [Energy-Efficient Split Learning for Resource-Constrained Environments: A Smart Farming Solution](https://arxiv.org/abs/2509.02549)
* [MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall](https://arxiv.org/abs/2509.02480)
* [Safe Memory Reclamation Techniques](https://arxiv.org/abs/2509.02457)
* [KubeIntellect: A Modular LLM-Orchestrated Agent Framework for End-to-End Kubernetes Management](https://arxiv.org/abs/2509.02449)
* [An Efficient and Adaptive Watermark Detection System with Tile-based Error Correction](https://arxiv.org/abs/2509.02447)
* [Efficient Pyramidal Analysis of Gigapixel Images on a Decentralized Modest Computer Cluster](https://arxiv.org/abs/2509.02440)
* [A Continuous Energy Ising Machine Leveraging Difference-of-Convex Programming](https://arxiv.org/abs/2509.01928)
* [Optimal Parallel Scheduling under Concave Speedup Functions](https://arxiv.org/abs/2509.01811)
* [STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data](https://arxiv.org/abs/2509.01626)
* [HiCR, an Abstract Model for Distributed Heterogeneous Programming](https://arxiv.org/abs/2509.01425)
* [LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving](https://arxiv.org/abs/2509.01229)
* [LobRA: Multi-tenant Fine-tuning over Heterogeneous Data](https://arxiv.org/abs/2509.01193)
* [DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving](https://arxiv.org/abs/2509.01083)
* [Parallelizing Drug Discovery: HPC Pipelines for Alzheimer's Molecular Docking and Simulation](https://arxiv.org/abs/2509.00937)
* [Accelerating Latency-Critical Applications with AI-Powered Semi-Automatic Fine-Grained Parallelization on SMT Processors](https://arxiv.org/abs/2509.00883)
* [HADIS: Hybrid Adaptive Diffusion Model Serving for Efficient Text-to-Image Generation](https://arxiv.org/abs/2509.00642)
* [KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache](https://arxiv.org/abs/2509.00579)
* [HydroGAT: Distributed Heterogeneous Graph Attention Transformer for Spatiotemporal Flood Prediction](https://arxiv.org/abs/2509.02481)
* [Online Identification of IT Systems through Active Causal Learning](https://arxiv.org/abs/2509.02130)
* [Batch Query Processing and Optimization for Agentic Workflows](https://arxiv.org/abs/2509.02121)
* [OASIS: Object-based Analytics Storage for Intelligent SQL Query Offloading in Scientific Tabular Workloads](https://arxiv.org/abs/2509.01966)
* [AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving](https://arxiv.org/abs/2509.00105)
* [Towards Agentic OS: An LLM Agent Framework for Linux Schedulers](https://arxiv.org/abs/2509.01245)

### 2025-09-01

* [Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding](https://arxiv.org/abs/2508.21706)
* [Odyssey: Adaptive Policy Selection for Resilient Distributed Training](https://arxiv.org/abs/2508.21613)
* [Unpacking Maximum Extractable Value on Polygon: A Study on Atomic Arbitrage](https://arxiv.org/abs/2508.21473)
* [Addressing Reproducibility Challenges in HPC with Continuous Integration](https://arxiv.org/abs/2508.21289)
* [Fast and Scalable Mixed Precision Euclidean Distance Calculations Using GPU Tensor Cores](https://arxiv.org/abs/2508.21230)
* [An Optimistic Gradient Tracking Method for Distributed Minimax Optimization](https://arxiv.org/abs/2508.21431)

### 2025-08-29

* [Collaborative Evolution of Intelligent Agents in Large-Scale Microservice Systems](https://arxiv.org/abs/2508.20508)
* [pdGRASS: A Fast Parallel Density-Aware Algorithm for Graph Spectral Sparsification](https://arxiv.org/abs/2508.20403)
* [CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference](https://arxiv.org/abs/2508.20375)
* [Predictable LLM Serving on GPU Clusters](https://arxiv.org/abs/2508.20274)
* [SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization](https://arxiv.org/abs/2508.20258)
* [SpeedMalloc: Improving Multi-threaded Applications via a Lightweight Core for Memory Allocation](https://arxiv.org/abs/2508.20253)
* [A Hybrid Stochastic Gradient Tracking Method for Distributed Online Optimization Over Time-Varying Directed Networks](https://arxiv.org/abs/2508.20645)
* [High performance visualization for Astronomy and Cosmology: the VisIVO's pathway toward Exascale systems](https://arxiv.org/abs/2508.20603)
* [Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs](https://arxiv.org/abs/2508.20333)

### 2025-08-28

* [HPC Digital Twins for Evaluating Scheduling Policies, Incentive Structures and their Impact on Power and Cooling](https://arxiv.org/abs/2508.20016)
* [Separation of Three or More Autonomous Mobile Models under Hierarchical Schedulers](https://arxiv.org/abs/2508.19805)
* [Beyond the Bermuda Triangle of Contention: IOMMU Interference in Mixed Criticality Systems](https://arxiv.org/abs/2508.19670)
* [Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference](https://arxiv.org/abs/2508.19559)
* [Formal Modeling and Verification of the Algorand Consensus Protocol in CADP](https://arxiv.org/abs/2508.19452)
* [HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference](https://arxiv.org/abs/2508.19373)
* [New Tools, Programming Models, and System Support for Processing-in-Memory Architectures](https://arxiv.org/abs/2508.19868)
* [Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents](https://arxiv.org/abs/2508.19504)

### 2025-08-27

* [Ab-initio Quantum Transport with the GW Approximation, 42,240 Atoms, and Sustained Exascale Performance](https://arxiv.org/abs/2508.19138)
* [CARMA: Collocation-Aware Resource Manager with GPU Memory Estimator](https://arxiv.org/abs/2508.19073)
* [Deep Learning-Enabled Supercritical Flame Simulation at Detailed Chemistry and Real-Fluid Accuracy Towards Trillion-Cell Scale](https://arxiv.org/abs/2508.18969)
* [SIREN: Software Identification and Recognition in HPC Systems](https://arxiv.org/abs/2508.18950)
* [ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive](https://arxiv.org/abs/2508.18850)
* [Examining MPI and its Extensions for Asynchronous Multithreaded Communication](https://arxiv.org/abs/2508.18667)
* [Strata: Hierarchical Context Caching for Long Context Language Model Serving](https://arxiv.org/abs/2508.18572)
* [Managing Multi Instance GPUs for High Throughput and Energy Savings](https://arxiv.org/abs/2508.18556)
* [Experiences with Model Context Protocol Servers for Science and High Performance Computing](https://arxiv.org/abs/2508.18489)
* [Architecting Distributed Quantum Computers: Design Insights from Resource Estimation](https://arxiv.org/abs/2508.19160)
* [History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL](https://arxiv.org/abs/2508.18588)
* [DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction](https://arxiv.org/abs/2508.18376)

### 2025-08-26

* [Flash Sparse Attention: An Alternative Efficient Implementation of Native Sparse Attention Kernel](https://arxiv.org/abs/2508.18224)
* [Practical GPU Choices for Earth Observation: ResNet-50 Training Throughput on Integrated, Laptop, and Cloud Accelerators](https://arxiv.org/abs/2508.18206)
* [Wait-free Replicated Data Types and Fair Reconciliation](https://arxiv.org/abs/2508.18193)
* [Scalable Engine and the Performance of Different LLM Models in a SLURM based HPC architecture](https://arxiv.org/abs/2508.17814)
* [ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale](https://arxiv.org/abs/2508.17624)
* [Zen-Attention: A Compiler Framework for Dynamic Attention Folding on AMD NPUs](https://arxiv.org/abs/2508.17593)
* [Easy Acceleration with Distributed Arrays](https://arxiv.org/abs/2508.17493)
* [Bine Trees: Enhancing Collective Operations by Optimizing Communication Locality](https://arxiv.org/abs/2508.17311)
* [TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving](https://arxiv.org/abs/2508.17219)
* [PICO: Performance Insights for Collective Operations](https://arxiv.org/abs/2508.16809)
* [Neuromorphic Simulation of Drosophila Melanogaster Brain Connectome on Loihi 2](https://arxiv.org/abs/2508.16792)
* [Equinox: Holistic Fair Scheduling in Serving Large Language Models](https://arxiv.org/abs/2508.16646)
* [GPU Acceleration for Faster Evolutionary Spatial Cyclic Game Systems](https://arxiv.org/abs/2508.16639)
* [Performance measurements of modern Fortran MPI applications with Score-P](https://arxiv.org/abs/2508.16592)
* [Views: A Hardware-friendly Graph Database Model For Storing Semantic Information](https://arxiv.org/abs/2508.18123)
* [Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective](https://arxiv.org/abs/2508.16712)
* [GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model](https://arxiv.org/abs/2508.16700)
* [Scalable Hybrid quantum Monte Carlo simulation of U(1) gauge field coupled to fermions on GPU](https://arxiv.org/abs/2508.16298)
* [Iridescent: A Framework Enabling Online System Implementation Specialization](https://arxiv.org/abs/2508.16690)
* [Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors](https://arxiv.org/abs/2508.17764)

### 2025-08-25

* [Generalizing Brooks' theorem via Partial Coloring is Hard Classically and Locally](https://arxiv.org/abs/2508.16308)
* [HyperFlexis: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling](https://arxiv.org/abs/2508.15919)
* [On the Duality of Task and Actor Programming Models](https://arxiv.org/abs/2508.16522)
* [Hybrid Classical-Quantum Supercomputing: A demonstration of a multi-user, multi-QPU and multi-GPU environment](https://arxiv.org/abs/2508.16297)
* [Self-Healing Network of Interconnected Edge Devices Empowered by Infrastructure-as-Code and LoRa Communication](https://arxiv.org/abs/2508.16268)
* [Towards Integrated Energy-Communication-Transportation Hub: A Base-Station-Centric Design in 5G and Beyond](https://arxiv.org/abs/2508.15833)
* [CXLAimPod: CXL Memory is all you need in AI era](https://arxiv.org/abs/2508.15980)

### 2025-08-22

* [CausalMesh: A Formally Verified Causal Cache for Stateful Serverless Computing](https://arxiv.org/abs/2508.15647)
* [Efficient Mixed-Precision Large Language Model Inference with TurboMind](https://arxiv.org/abs/2508.15601)
* [Lower Bounds for $k$-Set Agreement in Fault-Prone Networks](https://arxiv.org/abs/2508.15562)
* [Universal Dancing by Luminous Robots under Sequential Schedulers](https://arxiv.org/abs/2508.15484)
* [Databelt: A Continuous Data Path for Serverless Workflows in the 3D Compute Continuum](https://arxiv.org/abs/2508.15351)
* [Declarative Data Pipeline for Large Scale ML Services](https://arxiv.org/abs/2508.15105)
* [Mitigating context switching in densely packed Linux clusters with Latency-Aware Group Scheduling](https://arxiv.org/abs/2508.15703)
* [On the Effectiveness of Graph Reordering for Accelerating Approximate Nearest Neighbor Search on GPU](https://arxiv.org/abs/2508.15436)
* [Optimizing Compilation for Distributed Quantum Computing via Clustering and Annealing](https://arxiv.org/abs/2508.15267)
* [Reliable Multi-view 3D Reconstruction for `Just-in-time' Edge Environments](https://arxiv.org/abs/2508.15158)
* [TOAST: Fast and scalable auto-partitioning based on principled static analysis](https://arxiv.org/abs/2508.15010)
* [Scalable FPGA Framework for Real-Time Denoising in High-Throughput Imaging: A DRAM-Optimized Pipeline using High-Level Synthesis](https://arxiv.org/abs/2508.14917)
* [Mitigating context switching in densely packed Linux clusters with Latency-Aware Group Scheduling](https://arxiv.org/abs/2508.15703)

### 2025-08-21

* [The Cost Advantage of Virtual Machine Migrations: Empirical Insights into Amazon's EC2 Marketspace](https://arxiv.org/abs/2508.14883)
* [Leveraging Hardware-Aware Computation in Mixed-Precision Matrix Multiply: A Tile-Centric Approach](https://arxiv.org/abs/2508.14848)
* [MOHAF: A Multi-Objective Hierarchical Auction Framework for Scalable and Fair Resource Allocation in IoT Ecosystems](https://arxiv.org/abs/2508.14830)
* [DAG it off: Latency Prefers No Common Coins](https://arxiv.org/abs/2508.14716)
* [A Systematic Evaluation of the Potential of Carbon-Aware Execution for Scientific Workflows](https://arxiv.org/abs/2508.14625)
* [Boosting Payment Channel Network Liquidity with Topology Optimization and Transaction Selection](https://arxiv.org/abs/2508.14524)
* [Auditable Shared Objects: From Registers to Synchronization Primitives](https://arxiv.org/abs/2508.14506)
* [SSSP-Del: Fully Dynamic Distributed Algorithm for Single-Source Shortest Path](https://arxiv.org/abs/2508.14319)
* [Pure Data Spaces](https://arxiv.org/abs/2508.14271)
* [Time-optimal Asynchronous Minimal Vertex Covering by Myopic Robots](https://arxiv.org/abs/2508.14247)
* [Cooperative SGD with Dynamic Mixing Matrices](https://arxiv.org/abs/2508.14565)
* [Lagrangian Simulation Volume-Based Contour Tree Simplification](https://arxiv.org/abs/2508.14339)
* [Power Stabilization for AI Training Datacenters](https://arxiv.org/abs/2508.14318)
* [A High Performance GPU CountSketch Implementation and Its Application to Multisketching and Least Squares Problems](https://arxiv.org/abs/2508.14209)

### 2025-08-20

* [Is RISC-V ready for High Performance Computing? An evaluation of the Sophon SG2044](https://arxiv.org/abs/2508.13840)
* [Estimating CO$_2$ emissions of distributed applications and platforms with SimGrid/Batsim](https://arxiv.org/abs/2508.13693)
* [LUNDIsim: model meshes for flow simulation and scientific data compression benchmarks](https://arxiv.org/abs/2508.13636)
* [LAMMPS-KOKKOS: Performance Portable Molecular Dynamics Across Exascale Architectures](https://arxiv.org/abs/2508.13523)
* [DDoS Attacks in Cloud Computing: Detection and Prevention](https://arxiv.org/abs/2508.13522)
* [Optimizing Allreduce Operations for Heterogeneous Architectures with Multiple Processes per GPU](https://arxiv.org/abs/2508.13397)
* [OrbitChain: Orchestrating In-orbit Real-time Analytics of Earth Observation Data](https://arxiv.org/abs/2508.13374)
* [Persistent and Partitioned MPI for Stencil Communication](https://arxiv.org/abs/2508.13370)
* [Harnessing the Full Potential of RRAMs through Scalable and Distributed In-Memory Computing with Integrated Error Correction](https://arxiv.org/abs/2508.13298)
* [Analog computation with transcriptional networks](https://arxiv.org/abs/2508.14017)
* [PennyLane-Lightning MPI: A massively scalable quantum circuit simulator based on distributed computing in CPU clusters](https://arxiv.org/abs/2508.13615)
* [X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms](https://arxiv.org/abs/2508.13337)
* [Sustainable AI Training via Hardware-Software Co-Design on NVIDIA, AMD, and Emerging GPU Architectures](https://arxiv.org/abs/2508.13163)
* [Towards Timing Isolation for Mixed-Criticality Communication in Software-Defined Vehicles](https://arxiv.org/abs/2508.13652)

### 2025-08-19

* [Team Formation and Applications](https://arxiv.org/abs/2508.13084)
* [Congested Clique Counting for Local Gibbs Distributions](https://arxiv.org/abs/2508.13083)
* [WANify: Gauging and Balancing Runtime WAN Bandwidth for Geo-distributed Data Analytics](https://arxiv.org/abs/2508.12961)
* [Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement](https://arxiv.org/abs/2508.12851)
* [Dissecting CPU-GPU Unified Physical Memory on AMD MI300A APUs](https://arxiv.org/abs/2508.12743)
* [DIT: Dimension Reduction View on Optimal NFT Rarity Meters](https://arxiv.org/abs/2508.12671)
* [Proceedings 18th Interaction and Concurrency Experience](https://arxiv.org/abs/2508.12308)
* [Data-driven Trust Bootstrapping for Mobile Edge Computing-based Industrial IoT Services](https://arxiv.org/abs/2508.12560)
* [Attack Graph Generation on HPC Clusters](https://arxiv.org/abs/2508.12161)
* [OS-R1: Agentic Operating System Kernel Tuning with Reinforcement Learning](https://arxiv.org/abs/2508.12551)

### 2025-08-18

* [Efficient GPU-Centered Singular Value Decomposition Using the Divide-and-Conquer Method](https://arxiv.org/abs/2508.11467)
* [Time, Fences and the Ordering of Events in TSO](https://arxiv.org/abs/2508.11415)
* [Space-efficient population protocols for exact majority in general graphs](https://arxiv.org/abs/2508.11384)
* [Inter-APU Communication on AMD MI300A Systems via Infinity Fabric: a Deep Dive](https://arxiv.org/abs/2508.11298)
* [Element and Everything Tokens: Two-Tier Architecture for Mobilizing Alternative Assets](https://arxiv.org/abs/2508.11266)
* [EMLIO: Minimizing I/O Latency and Energy Consumption for Large-Scale AI Training](https://arxiv.org/abs/2508.11035)
* [OpenCXD: An Open Real-Device-Guided Hybrid Evaluation Framework for CXL-SSDs](https://arxiv.org/abs/2508.11477)

### 2025-08-15

* [Minimmit: Fast Finality with Even Faster Blocks](https://arxiv.org/abs/2508.10862)
* [Introducing CQ: A C-like API for Quantum Accelerated HPC](https://arxiv.org/abs/2508.10854)
* [Dalek: An Unconventional and Energy-Aware Heterogeneous Cluster](https://arxiv.org/abs/2508.10481)
* [GPZ: GPU-Accelerated Lossy Compressor for Particle Data](https://arxiv.org/abs/2508.10305)
* [Mixed-Precision Performance Portability of FFT-Based GPU-Accelerated Algorithms for Block-Triangular Toeplitz Matrices](https://arxiv.org/abs/2508.10202)
* [Hard Shell, Reliable Core: Improving Resilience in Replicated Systems with Selective Hybridization](https://arxiv.org/abs/2508.10141)
* [Leveraging OS-Level Primitives for Robotic Action Management](https://arxiv.org/abs/2508.10259)

### 2025-08-14

* [Closing the HPC-Cloud Convergence Gap: Multi-Tenant Slingshot RDMA for Kubernetes](https://arxiv.org/abs/2508.09663)
* [HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap](https://arxiv.org/abs/2508.09591)
* [Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference](https://arxiv.org/abs/2508.09505)
* [Distributed Diamond Formation of Sliding Squares](https://arxiv.org/abs/2508.09638)
* [Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference](https://arxiv.org/abs/2508.09229)
* [Semantic-Aware LLM Orchestration for Proactive Resource Management in Predictive Digital Twin Vehicular Networks](https://arxiv.org/abs/2508.09149)
* [Holistic Heterogeneous Scheduling for Autonomous Applications using Fine-grained, Multi-XPU Abstraction](https://arxiv.org/abs/2508.09503)
* [A Limits Study of Memory-side Tiering Telemetry](https://arxiv.org/abs/2508.09351)

### 2025-08-13

* [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://arxiv.org/abs/2508.09035)
* [A Reinforcement Learning-Driven Task Scheduling Algorithm for Multi-Tenant Distributed Systems](https://arxiv.org/abs/2508.08525)
* [Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson -- Extended](https://arxiv.org/abs/2508.08430)
* [Ultra Ethernet's Design Principles and Architectural Innovations](https://arxiv.org/abs/2508.08906)
* [Scalable Graph Indexing using GPUs for Approximate Nearest Neighbor Search](https://arxiv.org/abs/2508.08744)
* [Two for One, One for All: Deterministic LDC-based Robust Computation in Congested Clique](https://arxiv.org/abs/2508.08740)
* [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://arxiv.org/abs/2508.08712)
* [Vector-Centric Machine Learning Systems: A Cross-Stack Approach](https://arxiv.org/abs/2508.08469)
* [Towards Efficient and Practical GPU Multitasking in the Era of LLM](https://arxiv.org/abs/2508.08448)
* [Extremely Scalable Distributed Computation of Contour Trees via Pre-Simplification](https://arxiv.org/abs/2508.08433)
* [XDMA: A Distributed, Extensible DMA Architecture for Layout-Flexible Data Movements in Heterogeneous Multi-Accelerator SoCs](https://arxiv.org/abs/2508.08396)
* [Towards Efficient and Practical GPU Multitasking in the Era of LLM](https://arxiv.org/abs/2508.08448)
* [Ultra Ethernet's Design Principles and Architectural Innovations](https://arxiv.org/abs/2508.08906)
* [Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference](https://arxiv.org/abs/2508.08438)

### 2025-08-12

* [On the Operational Resilience of CBDC: Threats and Prospects of Formal Validation for Offline Payments](https://arxiv.org/abs/2508.08064)
* [Performance Evaluation of Brokerless Messaging Libraries](https://arxiv.org/abs/2508.07934)
* [Towards Lock Modularization for Heterogeneous Environments](https://arxiv.org/abs/2508.07756)
* [Over-the-Top Resource Broker System for Split Computing: An Approach to Distribute Cloud Computing Infrastructure](https://arxiv.org/abs/2508.07744)
* [Perpetual exploration in anonymous synchronous networks with a Byzantine black hole](https://arxiv.org/abs/2508.07703)
* [Taming Cold Starts: Proactive Serverless Scheduling with Model Predictive Control](https://arxiv.org/abs/2508.07640)
* [Coordinated Power Management on Heterogeneous Systems](https://arxiv.org/abs/2508.07605)
* [An Experimental Exploration of In-Memory Computing for Multi-Layer Perceptrons](https://arxiv.org/abs/2508.07317)
* [FlashMP: Fast Discrete Transform-Based Solver for Preconditioning Maxwell's Equations on GPUs](https://arxiv.org/abs/2508.07193)
* [The Fused Kernel Library: A C++ API to Develop Highly-Efficient GPU Libraries](https://arxiv.org/abs/2508.07071)
* [Convergence Sans Synchronization](https://arxiv.org/abs/2508.06949)
* [Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud](https://arxiv.org/abs/2508.06948)
* [PiKV: KV Cache Management System for Mixture of Experts](https://arxiv.org/abs/2508.06526)
* [Fully-Fluctuating Participation in Sleepy Consensus](https://arxiv.org/abs/2508.08068)
* [GPU-Accelerated Syndrome Decoding for Quantum LDPC Codes below the 63 $μ$s Latency Threshold](https://arxiv.org/abs/2508.07879)
* [Enhancing Privacy in Decentralized Min-Max Optimization: A Differentially Private Approach](https://arxiv.org/abs/2508.07505)
* [Real-Time Analysis of Unstructured Data with Machine Learning on Heterogeneous Architectures](https://arxiv.org/abs/2508.07423)
* [DSperse: A Framework for Targeted Verification in Zero-Knowledge Machine Learning](https://arxiv.org/abs/2508.06972)
* [A Portable Multi-GPU Solver for Collisional Plasmas with Coulombic Interactions](https://arxiv.org/abs/2508.06771)
* [PANAMA: A Network-Aware MARL Framework for Multi-Agent Path Finding in Digital Twin Ecosystems](https://arxiv.org/abs/2508.06767)

### 2025-08-11

* [Performant Unified GPU Kernels for Portable Singular Value Computation Across Hardware and Precision](https://arxiv.org/abs/2508.06339)
* [KV Cache Compression for Inference Efficiency in LLMs: A Review](https://arxiv.org/abs/2508.06297)
* [EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference](https://arxiv.org/abs/2508.06024)
* [KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training](https://arxiv.org/abs/2508.06001)
* [Snowpark: Performant, Secure, User-Friendly Data Engineering and AI/ML Next To Your Data](https://arxiv.org/abs/2508.05904)
* [A Dynamic Approach to Load Balancing in Cloud Infrastructure: Enhancing Energy Efficiency and Resource Utilization](https://arxiv.org/abs/2508.05821)
* [Accelerating Data Chunking in Deduplication Systems using Vector Instructions](https://arxiv.org/abs/2508.05797)
* [Voting-Based Semi-Parallel Proof-of-Work Protocol](https://arxiv.org/abs/2508.06489)

### 2025-08-08

* [Simulating LLM training workloads for heterogeneous compute and network infrastructure](https://arxiv.org/abs/2508.05370)

### 2025-08-07

* [S2M3: Split-and-Share Multi-Modal Models for Distributed Multi-Task Inference on the Edge](https://arxiv.org/abs/2508.04271)

### 2025-08-06

* [Block: Balancing Load in LLM Serving with Context, Knowledge and Predictive Scheduling](https://arxiv.org/abs/2508.03611)
* [Frontier: Simulating the Next Generation of LLM Inference Systems](https://arxiv.org/abs/2508.03148)

### 2025-08-05

* [PUSHtap: PIM-based In-Memory HTAP with Unified Data Storage Format](https://arxiv.org/abs/2508.02309)
* [Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving](https://arxiv.org/abs/2508.01989)

### 2025-08-04

* [SwarnRaft: Leveraging Consensus for Robust Drone Swarm Coordination in GNSS-Degraded Environments](https://arxiv.org/abs/2508.00622)
* [Adacc: Adaptive Compression and Activation Checkpointing for LLM Memory Management](https://arxiv.org/abs/2508.00806)
* [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://arxiv.org/abs/2508.00234)

### 2025-07-31

* [DSPE: Profit Maximization in Edge-Cloud Storage System using Dynamic Space Partitioning with Erasure Code](https://arxiv.org/abs/2507.22801)
* [Leveraging Caliper and Benchpark to Analyze MPI Communication Patterns: Insights from AMG2023, Kripke, and Laghos](https://arxiv.org/abs/2507.22372)

### 2025-07-30

* [LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems](https://arxiv.org/abs/2507.21276)
* [Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communications](https://arxiv.org/abs/2507.21199)

### 2025-07-29

* [MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training](https://arxiv.org/abs/2507.19845)

### 2025-07-28

* [RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems](https://arxiv.org/abs/2507.18889)

### 2025-07-25

* [Cloud Native System for LLM Inference Serving](https://arxiv.org/abs/2507.18007)
* [Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling](https://arxiv.org/abs/2507.18006)
* [Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving](https://arxiv.org/abs/2507.18454)

### 2025-07-24

* [BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs](https://arxiv.org/abs/2507.17133)
* [BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving](https://arxiv.org/abs/2507.17120)

### 2025-07-23

* [Cooling Matters: Benchmarking Large Language Models and Vision-Language Models on Liquid-Cooled Versus Air-Cooled H100 GPU Systems](https://arxiv.org/abs/2507.16781)
* [Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges](https://arxiv.org/abs/2507.16731)
* [Reducing GPU Memory Fragmentation via Spatio-Temporal Planning for Efficient Large-Scale Model Training](https://arxiv.org/abs/2507.16274)

### 2025-07-22

* [Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing](https://arxiv.org/abs/2507.15553)
* [GALE: Leveraging Heterogeneous Systems for Efficient Unstructured Mesh Data Analysis](https://arxiv.org/abs/2507.15230)
* [Byzantine-Robust Decentralized Coordination of LLM Agents](https://arxiv.org/abs/2507.14928)
* [Characterizing Communication Patterns in Distributed Large Language Model Inference](https://arxiv.org/abs/2507.14392)
* [IDSS, a Novel P2P Relational Data Storage Service](https://arxiv.org/abs/2507.14682)
* [A Sparsity Predicting Approach for Large Language Models via Activation Pattern Clustering](https://arxiv.org/abs/2507.14179)

### 2025-07-21

* [DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training](https://arxiv.org/abs/2507.13833)
* [Leveraging Multi-Instance GPUs through moldable task scheduling](https://arxiv.org/abs/2507.13601)
* [An End-to-End DNN Inference Framework for the SpiNNaker2 Neuromorphic MPSoC](https://arxiv.org/abs/2507.13736)

### 2025-07-18

* [BootSeer: Analyzing and Mitigating Initialization Bottlenecks in Large-Scale LLM Training](https://arxiv.org/abs/2507.12619)

### 2025-07-17

* [Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storage](https://arxiv.org/abs/2507.12205)
* [Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI](https://arxiv.org/abs/2507.11830)

### 2025-07-16

* [Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations](https://arxiv.org/abs/2507.11417)
* [MIRAGE: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving](https://arxiv.org/abs/2507.11507)

### 2025-07-15

* [Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters](https://arxiv.org/abs/2507.10392)
* [Cross-Timeslot Optimization for Distributed GPU Inference Using Reinforcement Learning](https://arxiv.org/abs/2507.10259)
* [Past-Future Scheduler for LLM Serving under SLA Guarantees](https://arxiv.org/abs/2507.10150)
* [ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism](https://arxiv.org/abs/2507.10069)
* [EAT: QoS-Aware Edge-Collaborative AIGC Task Scheduling via Attention-Guided Diffusion Reinforcement Learning](https://arxiv.org/abs/2507.10026)
* [Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference](https://arxiv.org/abs/2507.09942)
* [SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding](https://arxiv.org/abs/2507.09201)
* [On Evaluating Performance of LLM Inference Serving Systems](https://arxiv.org/abs/2507.09019)

### 2025-07-11

* [KIS-S: A GPU-Aware Kubernetes Inference Simulator with RL-Based Auto-Scaling](https://arxiv.org/abs/2507.07932)
* [KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows](https://arxiv.org/abs/2507.07400)
* [Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding](https://arxiv.org/abs/2507.07120)
* [Analysing semantic data storage in Distributed Ledger Technologies for Data Spaces](https://arxiv.org/abs/2507.07116)

### 2025-07-10

* [Nexus: Taming Throughput-Latency Tradeoff in LLM Serving via Efficient GPU Sharing](https://arxiv.org/abs/2507.06608)
* [SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference](https://arxiv.org/abs/2507.06567)

### 2025-07-08

* [On Fault Tolerance of Data Storage Systems: A Holistic Perspective](https://arxiv.org/abs/2507.03849)
* [Analysis and Optimized CXL-Attached Memory Allocation for Long-Context LLM Fine-Tuning](https://arxiv.org/abs/2507.03305)
* [Symbiosis: Multi-Adapter Inference and Fine-Tuning](https://arxiv.org/abs/2507.03220)
* [ZettaLith: An Architectural Exploration of Extreme-Scale AI Inference Acceleration](https://arxiv.org/abs/2507.02871)
* [Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation](https://arxiv.org/abs/2507.04697)

### 2025-07-04

* [FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference](https://arxiv.org/abs/2507.02620)
* [Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency](https://arxiv.org/abs/2507.02135)

### 2025-07-03

* [Deep Recommender Models Inference: Automatic Asymmetric Data Flow Optimization](https://arxiv.org/abs/2507.01676)
* [EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices](https://arxiv.org/abs/2507.01438)

### 2025-07-02

* [Accelerating Loading WebGraphs in ParaGrapher](https://arxiv.org/abs/2507.00716)
* [DynoStore: A wide-area distribution system for the management of data over heterogeneous storage](https://arxiv.org/abs/2507.00576)
* [LLM-Mesh: Enabling Elastic Sharing for Serverless LLM Inference](https://arxiv.org/abs/2507.00507)
* [Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and High-Performance GPUs](https://arxiv.org/abs/2507.00418)
* [Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration](https://arxiv.org/abs/2507.00672)
* [HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism](https://arxiv.org/abs/2507.00394)

### 2025-07-01

* [Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC](https://arxiv.org/abs/2506.24045)
* [QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inference](https://arxiv.org/abs/2506.23934)
* [Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model](https://arxiv.org/abs/2506.23635)

### 2025-06-30

* [MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism](https://arxiv.org/abs/2506.22175)
* [SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference](https://arxiv.org/abs/2506.22033)

### 2025-06-27

* [ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks](https://arxiv.org/abs/2506.20938)

### 2025-06-26

* [Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU](https://arxiv.org/abs/2506.20187)
* [MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection](https://arxiv.org/abs/2506.19884)

### 2025-06-25

* [Shelby: Decentralized Storage Designed to Serve](https://arxiv.org/abs/2506.19233)
* [Vertex addition to a ball graph with application to reliability and area coverage in autonomous swarms](https://arxiv.org/abs/2506.19197)
* [Binsparse: A Specification for Cross-Platform Storage of Sparse Matrices and Tensors](https://arxiv.org/abs/2506.19175)

### 2025-06-24

* [Leveraging Cloud-Fog Automation for Autonomous Collision Detection and Classification in Intelligent Unmanned Surface Vehicles](https://arxiv.org/abs/2506.18024)
* [Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems](https://arxiv.org/abs/2506.17551)
* [Leveraging Large Language Model for Intelligent Log Processing and Autonomous Debugging in Cloud AI Platforms](https://arxiv.org/abs/2506.17900)
* [VeriLocc: End-to-End Cross-Architecture Register Allocation via LLM](https://arxiv.org/abs/2506.17506)

### 2025-06-23

* [TrainVerify: Equivalence-Based Verification for Distributed LLM Training](https://arxiv.org/abs/2506.15961)

### 2025-06-19

* [All is Not Lost: LLM Recovery without Checkpoints](https://arxiv.org/abs/2506.15461)
* [eLLM: Elastic Memory Management Framework for Efficient LLM Serving](https://arxiv.org/abs/2506.15155)
* [Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching](https://arxiv.org/abs/2506.14852)
* [Efficient Serving of LLM Applications with Probabilistic Demand Modeling](https://arxiv.org/abs/2506.14851)

### 2025-06-18

* [Keigo: Co-designing Log-Structured Merge Key-Value Stores with a Non-Volatile, Concurrency-aware Storage Hierarchy (Extended Version)](https://arxiv.org/abs/2506.14630)

### 2025-06-17

* [Serving Large Language Models on Huawei CloudMatrix384](https://arxiv.org/abs/2506.12708)
* [HarMoEny: Efficient Multi-GPU Inference of MoE Models](https://arxiv.org/abs/2506.12417)
* [NaSh: Guardrails for an LLM-Powered Natural Language Shell](https://arxiv.org/abs/2506.13028)
* [Semantic Scheduling for LLM Inference](https://arxiv.org/abs/2506.12204)

### 2025-06-16

* [A retrospective on DISPEED -- Leveraging heterogeneity in a drone swarm for IDS execution](https://arxiv.org/abs/2506.11800)
* [SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding](https://arxiv.org/abs/2506.11309)

### 2025-06-13

* [TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference](https://arxiv.org/abs/2506.10470)
* [HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration](https://arxiv.org/abs/2506.10401)

### 2025-06-12

* [Understanding the Performance and Power of LLM Inferencing on Edge Accelerators](https://arxiv.org/abs/2506.09554)
* [SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving](https://arxiv.org/abs/2506.09397)
* [ScalableHD: Scalable and High-Throughput Hyperdimensional Computing Inference on Multi-Core CPUs](https://arxiv.org/abs/2506.09282)
* [EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model](https://arxiv.org/abs/2506.09061)

### 2025-06-11

* [Recipes for Pre-training LLMs with MXFP8](https://arxiv.org/abs/2506.08027)

### 2025-06-10

* [Addressing tokens dynamic generation, propagation, storage and renewal to secure the GlideinWMS pilot based jobs and system](https://arxiv.org/abs/2506.07379)
* [Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage](https://arxiv.org/abs/2506.06472)
* [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://arxiv.org/abs/2506.06579)

### 2025-06-09

* [Beyond the Buzz: A Pragmatic Take on Inference Disaggregation](https://arxiv.org/abs/2506.05508)

### 2025-06-06

* [FlashDMoE: Fast Distributed MoE in a Single Kernel](https://arxiv.org/abs/2506.04667)
* [SkimROOT: Accelerating LHC Data Filtering with Near-Storage Processing](https://arxiv.org/abs/2506.04507)
* [Knowledge-Guided Attention-Inspired Learning for Task Offloading in Vehicle Edge Computing](https://arxiv.org/abs/2506.04456)
* [Inference economics of language models](https://arxiv.org/abs/2506.04645)

### 2025-06-05

* [Cascadia: A Cascade Serving System for Large Language Models](https://arxiv.org/abs/2506.04203)
* [Crowd-SFT: Crowdsourcing for LLM Alignment](https://arxiv.org/abs/2506.04063)
* [Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs](https://arxiv.org/abs/2506.03296)

### 2025-06-04

* [Adaptive Configuration Selection for Multi-Model Inference Pipelines in Edge Computing](https://arxiv.org/abs/2506.02814)
* [Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM](https://arxiv.org/abs/2506.02490)
* [D-Rex: Heterogeneity-Aware Reliability Framework and Adaptive Algorithms for Distributed Storage](https://arxiv.org/abs/2506.02026)
* [Evaluating the Efficacy of LLM-Based Reasoning for Multiobjective HPC Job Scheduling](https://arxiv.org/abs/2506.02025)
* [NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs](https://arxiv.org/abs/2506.02024)
* [DistMLIP: A Distributed Inference Platform for Machine Learning Interatomic Potentials](https://arxiv.org/abs/2506.02023)
* [Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing](https://arxiv.org/abs/2506.02006)

### 2025-06-03

* [Adaptive, Efficient and Fair Resource Allocation in Cloud Datacenters leveraging Weighted A3C Deep Reinforcement Learning](https://arxiv.org/abs/2506.00929)
* [Advancing AI-assisted Hardware Design with Hierarchical Decentralized Training and Personalized Inference-Time Optimization](https://arxiv.org/abs/2506.00002)

### 2025-06-02

* [Distributed Intelligence in the Computing Continuum with Active Inference](https://arxiv.org/abs/2505.24618)
* [SkyLB: A Locality-Aware Cross-Region Load Balancer for LLM Inference](https://arxiv.org/abs/2505.24095)
* [EmbAdvisor: Adaptive Cache Management for Sustainable LLM Serving](https://arxiv.org/abs/2505.23970)

### 2025-05-30

* [Sustainable Carbon-Aware and Water-Efficient LLM Scheduling in Geo-Distributed Cloud Datacenters](https://arxiv.org/abs/2505.23554)
* [MemAscend: System Memory Optimization for SSD-Offloaded LLM Fine-Tuning](https://arxiv.org/abs/2505.23254)
* [Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism](https://arxiv.org/abs/2505.23219)
* [Accelerating AllReduce with a Persistent Straggler](https://arxiv.org/abs/2505.23523)

### 2025-05-29

* [Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference](https://arxiv.org/abs/2505.21919)

### 2025-05-27

* [DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systems](https://arxiv.org/abs/2505.19847)
* [Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs](https://arxiv.org/abs/2505.19481)

### 2025-05-26

* [H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips](https://arxiv.org/abs/2505.17548)
* [Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models](https://arxiv.org/abs/2505.17826)

### 2025-05-23

* [Edge-First Language Model Inference: Models, Metrics, and Tradeoffs](https://arxiv.org/abs/2505.16508)
* [Recursive Offloading for LLM Serving in Multi-tier Networks](https://arxiv.org/abs/2505.16502)

### 2025-05-22

* [Balanced and Elastic End-to-end Training of Dynamic LLMs](https://arxiv.org/abs/2505.14864)

### 2025-05-21

* [ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs](https://arxiv.org/abs/2505.14468)

### 2025-05-20

* [HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving](https://arxiv.org/abs/2505.12658)
* [Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture](https://arxiv.org/abs/2505.11916)
* [Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference](https://arxiv.org/abs/2505.13345)

### 2025-05-19

* [TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference](https://arxiv.org/abs/2505.11329)
* [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://arxiv.org/abs/2505.11432)
* [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://arxiv.org/abs/2505.11415)

### 2025-05-16

* [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://arxiv.org/abs/2505.09999)

### 2025-05-15

* [ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor](https://arxiv.org/abs/2505.09142)

### 2025-05-14

* [Fused3S: Fast Sparse Attention on Tensor Cores](https://arxiv.org/abs/2505.08098)
* [Patchwork: A Unified Framework for RAG Serving](https://arxiv.org/abs/2505.07833)

### 2025-05-13

* [PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications](https://arxiv.org/abs/2505.07203)
* [SneakPeek: Data-Aware Model Selection and Scheduling for Inference Serving on the Edge](https://arxiv.org/abs/2505.06641)
* [Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference](https://arxiv.org/abs/2505.06461)
* [SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models](https://arxiv.org/abs/2505.07680)
* [QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration](https://arxiv.org/abs/2505.06481)
* [Towards Efficient LLM Storage Reduction via Tensor Deduplication and Delta Compression](https://arxiv.org/abs/2505.06252)

### 2025-05-12

* [Understanding Stragglers in Large Model Training Using What-if Analysis](https://arxiv.org/abs/2505.05713)

### 2025-05-09

* [Walrus: An Efficient Decentralized Storage Network](https://arxiv.org/abs/2505.05370)
* [Exploring Influence Factors on LLM Suitability for No-Code Development of End User IoT Applications](https://arxiv.org/abs/2505.04710)
* [HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights](https://arxiv.org/abs/2505.04846)

### 2025-05-08

* [Prism: Unleashing GPU Sharing for Cost-Efficient Multi-LLM Serving](https://arxiv.org/abs/2505.04021)
* [Rollbaccine : Herd Immunity against Storage Rollback Attacks in TEEs [Technical Report]](https://arxiv.org/abs/2505.04014)
* [Splitwiser: Efficient LM inference with constrained resources](https://arxiv.org/abs/2505.03763)

### 2025-05-06

* [Large Language Model Partitioning for Low-Latency Inference at the Edge](https://arxiv.org/abs/2505.02533)
* [Opt-GPTQ: An Optimized GPTQ Combining Sparse Attention and Quantization Techniques](https://arxiv.org/abs/2505.02351)
* [HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferences](https://arxiv.org/abs/2505.01968)
* [HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://arxiv.org/abs/2505.02795)

### 2025-05-05

* [CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems](https://arxiv.org/abs/2505.01164)

### 2025-04-30

* [Leveraging Neural Graph Compilers in Machine Learning Research for Edge-Cloud Systems](https://arxiv.org/abs/2504.20198)
* [GenTorrent: Scaling Large Language Model Serving with An Overley Network](https://arxiv.org/abs/2504.20101)
* [Tempo: Application-aware LLM Serving with Mixed SLO Requirements](https://arxiv.org/abs/2504.20068)
* [OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification](https://arxiv.org/abs/2504.20964)

### 2025-04-29

* [Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration](https://arxiv.org/abs/2504.19516)
* [Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation](https://arxiv.org/abs/2504.19232)
* [semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage](https://arxiv.org/abs/2504.19867)
* [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://arxiv.org/abs/2504.19720)

### 2025-04-28

* [EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration](https://arxiv.org/abs/2504.18154)

### 2025-04-24

* [Preemption Aware Task Scheduling for Priority and Deadline Constrained DNN Inference Task Offloading in Homogeneous Mobile-Edge Networks](https://arxiv.org/abs/2504.16792)
* [Real-time Bayesian inference at extreme scale: A digital twin for tsunami early warning applied to the Cascadia subduction zone](https://arxiv.org/abs/2504.16344)
* [HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing](https://arxiv.org/abs/2504.16112)

### 2025-04-23

* [SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference](https://arxiv.org/abs/2504.15720)
* [High-Throughput LLM inference on Heterogeneous Clusters](https://arxiv.org/abs/2504.15303)
* [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://arxiv.org/abs/2504.15302)
* [D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving](https://arxiv.org/abs/2504.15299)
* [Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling](https://arxiv.org/abs/2504.15296)
* [StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation](https://arxiv.org/abs/2504.15930)
* [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://arxiv.org/abs/2504.15302)

### 2025-04-22

* [SLO-Aware Scheduling for Large Language Model Inferences](https://arxiv.org/abs/2504.14966)
* [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://arxiv.org/abs/2504.14775)
* [Joint Optimization of Offloading, Batching and DVFS for Multiuser Co-Inference](https://arxiv.org/abs/2504.14611)
* [MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core](https://arxiv.org/abs/2504.14960)
* [Optimizing SLO-oriented LLM Serving with PD-Multiplexing](https://arxiv.org/abs/2504.14489)

### 2025-04-18

* [You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models](https://arxiv.org/abs/2504.12471)

### 2025-04-17

* [Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures](https://arxiv.org/abs/2504.11750)
* [Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading](https://arxiv.org/abs/2504.11816)
* [70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float](https://arxiv.org/abs/2504.11651)

### 2025-04-16

* [Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints](https://arxiv.org/abs/2504.11320)

### 2025-04-15

* [Optimal Graph Stretching for Distributed Averaging](https://arxiv.org/abs/2504.10289)
* [Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project](https://arxiv.org/abs/2504.10013)
* [MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints](https://arxiv.org/abs/2504.09345)
* [Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training](https://arxiv.org/abs/2504.09307)
* [DynaServe: Unified and Elastic Tandem-Style Execution for Dynamic Disaggregated LLM Serving](https://arxiv.org/abs/2504.09285)
* [SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting](https://arxiv.org/abs/2504.08850)
* [DARIS: An Oversubscribed Spatio-Temporal Scheduler for Real-Time DNN Inference on GPUs](https://arxiv.org/abs/2504.08795)
* [PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters](https://arxiv.org/abs/2504.08791)
* [SLOs-Serve: Optimized Serving of Multi-SLO LLMs](https://arxiv.org/abs/2504.08784)
* [MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions](https://arxiv.org/abs/2504.09474)

### 2025-04-14

* [Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devices](https://arxiv.org/abs/2504.08242)

### 2025-04-11

* [Token Level Routing Inference System for Edge Devices](https://arxiv.org/abs/2504.07878)

### 2025-04-09

* [Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training](https://arxiv.org/abs/2504.06095)
* [HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference](https://arxiv.org/abs/2504.05897)

### 2025-04-08

* [IntentContinuum: Using LLMs to Support Intent-Based Computing Across the Compute Continuum](https://arxiv.org/abs/2504.04429)
* [HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs](https://arxiv.org/abs/2504.03871)
* [FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling](https://arxiv.org/abs/2504.03775)
* [Adaptive Orchestration for Inference of Large Foundation Models at the Edge](https://arxiv.org/abs/2504.03668)
* [LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks](https://arxiv.org/abs/2504.03665)
* [PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices](https://arxiv.org/abs/2504.03664)

### 2025-04-07

* [LLMSched: Uncertainty-Aware Workload Scheduling for Compound LLM Applications](https://arxiv.org/abs/2504.03444)

### 2025-04-04

* [FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention](https://arxiv.org/abs/2504.02211)

### 2025-04-02

* [AMP4EC: Adaptive Model Partitioning Framework for Efficient Deep Learning Inference in Edge Computing Environments](https://arxiv.org/abs/2504.00407)

### 2025-04-01

* [OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training](https://arxiv.org/abs/2503.23830)
* [MVDRAM: Enabling GeMV Execution in Unmodified DRAM for Low-Bit LLM Acceleration](https://arxiv.org/abs/2503.23817)

### 2025-03-31

* [Niyama : Breaking the Silos of LLM Inference Serving](https://arxiv.org/abs/2503.22562)

### 2025-03-28

* [Robust DNN Partitioning and Resource Allocation Under Uncertain Inference Time](https://arxiv.org/abs/2503.21476)
* [Optimizing Multi-DNN Inference on Mobile Devices through Heterogeneous Processor Co-Execution](https://arxiv.org/abs/2503.21109)
* [Scalability Evaluation of HPC Multi-GPU Training for ECG-based LLMs](https://arxiv.org/abs/2503.21033)

### 2025-03-27

* [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://arxiv.org/abs/2503.20552)
* [Harmonia: A Multi-Agent Reinforcement Learning Approach to Data Placement and Migration in Hybrid Storage Systems](https://arxiv.org/abs/2503.20507)
* [L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis](https://arxiv.org/abs/2503.20263)

### 2025-03-26

* [Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization](https://arxiv.org/abs/2503.19050)

### 2025-03-25

* [Jenga: Effective Memory Management for Serving LLM with Heterogeneity](https://arxiv.org/abs/2503.18292)
* [Risk Management for Distributed Arbitrage Systems: Integrating Artificial Intelligence](https://arxiv.org/abs/2503.18265)
* [WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training](https://arxiv.org/abs/2503.17924)
* [PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling](https://arxiv.org/abs/2503.17707)
* [A Generative Caching System for Large Language Models](https://arxiv.org/abs/2503.17603)

### 2025-03-24

* [Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulation](https://arxiv.org/abs/2503.16893)
* [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://arxiv.org/abs/2503.16585)

### 2025-03-21

* [SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models](https://arxiv.org/abs/2503.15921)
* [ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism](https://arxiv.org/abs/2503.15758)

### 2025-03-20

* [Efficient allocation of image recognition and LLM tasks on multi-GPU system](https://arxiv.org/abs/2503.15252)
* [Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices](https://arxiv.org/abs/2503.14932)
* [RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving](https://arxiv.org/abs/2503.14649)

### 2025-03-19

* [Do Large Language Models Understand Performance Optimization?](https://arxiv.org/abs/2503.13772)

### 2025-03-18

* [Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environments](https://arxiv.org/abs/2503.12228)
* [FAILS: A Framework for Automated Collection and Analysis of LLM Service Incidents](https://arxiv.org/abs/2503.12185)

### 2025-03-17

* [Beyond A Single AI Cluster: A Survey of Decentralized LLM Training](https://arxiv.org/abs/2503.11023)
* [LLMPerf: GPU Performance Modeling meets Large Language Models](https://arxiv.org/abs/2503.11244)
* [Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores](https://arxiv.org/abs/2503.10725)

### 2025-03-14

* [SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading](https://arxiv.org/abs/2503.10377)
* [Collaborative Speculative Inference for Efficient LLM Inference Serving](https://arxiv.org/abs/2503.10325)
* [MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching](https://arxiv.org/abs/2503.09716)

### 2025-03-13

* [Performance Models for a Two-tiered Storage System](https://arxiv.org/abs/2503.08966)
* [Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference](https://arxiv.org/abs/2503.09304)
* [Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge](https://arxiv.org/abs/2503.09114)

### 2025-03-12

* [TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems](https://arxiv.org/abs/2503.08415)
* [Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference](https://arxiv.org/abs/2503.08311)
* [Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices](https://arxiv.org/abs/2503.08223)
* [Accelerating MoE Model Inference with Expert Sharding](https://arxiv.org/abs/2503.08467)
* [FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework](https://arxiv.org/abs/2503.08461)
* [DynTaskMAS: A Dynamic Task Graph-driven Framework for Asynchronous and Parallel LLM-based Multi-Agent Systems](https://arxiv.org/abs/2503.07675)

### 2025-03-11

* [Seesaw: High-throughput LLM Inference via Model Re-sharding](https://arxiv.org/abs/2503.06433)
* [eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference](https://arxiv.org/abs/2503.06823)
* [Distributed Graph Neural Network Inference With Just-In-Time Compilation For Industry-Scale Graphs](https://arxiv.org/abs/2503.06208)

### 2025-03-10

* [Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching](https://arxiv.org/abs/2503.05248)
* [Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts](https://arxiv.org/abs/2503.05447)

### 2025-03-07

* [Dynamic Pricing for On-Demand DNN Inference in the Edge-AI Market](https://arxiv.org/abs/2503.04521)
* [Speculative MoE: Communication Efficient Parallel MoE Inference with Speculative Token and Expert Pre-scheduling](https://arxiv.org/abs/2503.04398)
* [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://arxiv.org/abs/2503.04302)

### 2025-03-06

* [Enhancing Memory Efficiency in Large Language Model Training Through Chronos-aware Pipeline Parallelism](https://arxiv.org/abs/2503.03182)
* [Environment-Aware Dynamic Pruning for Pipelined Edge Inference](https://arxiv.org/abs/2503.03070)

### 2025-03-05

* [SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling](https://arxiv.org/abs/2503.02550)
* [CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory](https://arxiv.org/abs/2503.02354)
* [VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference](https://arxiv.org/abs/2503.02236)
### 2025-03-04

* [Improving inference time in multi-TPU systems with profiled model segmentation](https://arxiv.org/abs/2503.01025)

### 2025-03-03

* [ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs](https://arxiv.org/abs/2502.21231)
* [TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval](https://arxiv.org/abs/2502.20969)
* [Cicada: A Pipeline-Efficient Approach to Serverless Inference with Decoupled Management](https://arxiv.org/abs/2502.20959)
* [SkyStore: Cost-Optimized Object Storage Across Regions and Clouds](https://arxiv.org/abs/2502.20818)
* [LADs: Leveraging LLMs for AI-Driven DevOps](https://arxiv.org/abs/2502.20825)

### 2025-02-28

* [SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks](https://arxiv.org/abs/2502.19913)

### 2025-02-27

* [CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuning](https://arxiv.org/abs/2502.18910)

### 2025-02-25

* [FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference](https://arxiv.org/abs/2502.15804)
* [Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization](https://arxiv.org/abs/2502.15763)
* [LoXR: Performance Evaluation of Locally Executing LLMs on XR Devices](https://arxiv.org/abs/2502.15761)
* [DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices](https://arxiv.org/abs/2502.15735)

### 2025-02-24

* [Towards Swift Serverless LLM Cold Starts with ParaServe](https://arxiv.org/abs/2502.15524)
* [FlexPie: Accelerate Distributed Inference on Edge Devices with Flexible Combinatorial Optimization[Technical Report]](https://arxiv.org/abs/2502.15312)

### 2025-02-21

* [Serving Models, Fast and Slow:Optimizing Heterogeneous LLM Inferencing Workloads at Scale](https://arxiv.org/abs/2502.14617)
* [Optimizing the Longhorn Cloud-native Software Defined Storage Engine for High Performance](https://arxiv.org/abs/2502.14419)
* [CarbonEdge: Leveraging Mesoscale Spatial Carbon-Intensity Variations for Low Carbon Edge Computing](https://arxiv.org/abs/2502.14076)
* [LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention](https://arxiv.org/abs/2502.14866)
* [LLM4FaaS: No-Code Application Development using LLMs and FaaS](https://arxiv.org/abs/2502.14450)

### 2025-02-20

* [Autellix: An Efficient Serving Engine for LLM Agents as General Programs](https://arxiv.org/abs/2502.13965)

### 2025-02-19

* [SparkAttention: High-Performance Multi-Head Attention for Large Models on Volta GPU Architecture](https://arxiv.org/abs/2502.12784)
* [Distributed On-Device LLM Inference With Over-the-Air Computation](https://arxiv.org/abs/2502.12559)
* [Understanding Silent Data Corruption in LLM Training](https://arxiv.org/abs/2502.12340)
* [Semantica: Decentralized Search using a LLM-Guided Semantic Tree Overlay](https://arxiv.org/abs/2502.10151)

### 2025-02-18

* [Scalable and Cost-Efficient ML Inference: Parallel Batch Processing with Serverless Functions](https://arxiv.org/abs/2502.12017)
* [BagChain: A Dual-functional Blockchain Leveraging Bagging-based Distributed Learning](https://arxiv.org/abs/2502.11464)
* [DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization](https://arxiv.org/abs/2502.11058)
* [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880)
* [DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services](https://arxiv.org/abs/2502.11417)
* [Local-Cloud Inference Offloading for LLMs in Multi-Modal, Multi-Task, Multi-Dialogue Settings](https://arxiv.org/abs/2502.11007)

### 2025-02-17

* [λScale: Enabling Fast Scaling for Serverless Large Language Model Inference](https://arxiv.org/abs/2502.09922)

### 2025-02-14

* [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://arxiv.org/abs/2502.09334)

### 2025-02-13

* [Memory Offloading for Large Language Model Inference with Latency SLO Guarantees](https://arxiv.org/abs/2502.08182)
* [HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment](https://arxiv.org/abs/2502.07903)
* [Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers](https://arxiv.org/abs/2502.08145)

### 2025-02-11

* [MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing](https://arxiv.org/abs/2502.06643)
* [fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving](https://arxiv.org/abs/2502.05370)

### 2025-02-10

* [EcoServe: Designing Carbon-Aware AI Inference Systems](https://arxiv.org/abs/2502.05043)
* [WaferLLM: A Wafer-Scale LLM Inference System](https://arxiv.org/abs/2502.04563)

### 2025-02-07

* [HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference](https://arxiv.org/abs/2502.03589)
* [InfinitePOD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers](https://arxiv.org/abs/2502.03885)

### 2025-02-05

* [LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models](https://arxiv.org/abs/2502.02406)
* [Longer Attention Span: Increasing Transformer Context Length with Sparse Graph Processing Techniques](https://arxiv.org/abs/2502.01659)

### 2025-02-04

* [OCTOPINF: Workload-Aware Inference Serving for Edge Video Analytics](https://arxiv.org/abs/2502.01277)
* [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://arxiv.org/abs/2502.00722)
* [General Coded Computing in a Probabilistic Straggler Regime](https://arxiv.org/abs/2502.00645)
* [Leveraging InfiniBand Controller to Configure Deadlock-Free Routing Engines for Dragonflies](https://arxiv.org/abs/2502.01214)

### 2025-02-03

* [Infer-EDGE: Dynamic DNN Inference Optimization in 'Just-in-time' Edge-AI Implementations](https://arxiv.org/abs/2501.18842)

### 2025-01-30

* [Dual-Lagrange Encoding for Storage and Download in Elastic Computing for Resilience](https://arxiv.org/abs/2501.17275)

### 2025-01-29

* [On the Shape Containment Problem within the Amoebot Model with Reconfigurable Circuits](https://arxiv.org/abs/2501.16892)

### 2025-01-28

* [Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference](https://arxiv.org/abs/2501.16103)
* [Aging-aware CPU Core Management for Embodied Carbon Amortization in Cloud LLM Inference](https://arxiv.org/abs/2501.15829)
* [HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location](https://arxiv.org/abs/2501.14808)
* [HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs platform with Heterogeneous AI Accelerators](https://arxiv.org/abs/2501.14794)
* [DeServe: Towards Affordable Offline LLM Inference via Decentralization](https://arxiv.org/abs/2501.14784)
* [Dynamic Adaptation in Data Storage: Real-Time Machine Learning for Enhanced Prefetching](https://arxiv.org/abs/2501.14771)

### 2025-01-27

* [Locality-aware Fair Scheduling in LLM Serving](https://arxiv.org/abs/2501.14312)

### 2025-01-22

* [Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization](https://arxiv.org/abs/2501.12388)
* [DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference](https://arxiv.org/abs/2501.10375)
* [AdaServe: SLO-Customized LLM Serving with Fine-Grained Speculative Decoding](https://arxiv.org/abs/2501.12162)
* [Glinthawk: A Two-Tiered Architecture for High-Throughput LLM Inference](https://arxiv.org/abs/2501.11779)

### 2025-01-20

* [Over-the-Air Multi-Sensor Inference with Neural Networks Using Memristor-Based Analog Computing](https://arxiv.org/abs/2501.10245)

### 2025-01-17

* [PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks](https://arxiv.org/abs/2501.09367)

### 2025-01-15

* [PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving](https://arxiv.org/abs/2501.08192)
* [HgPCN: A Heterogeneous Architecture for E2E Embedded Point Cloud Inference](https://arxiv.org/abs/2501.07767)

### 2025-01-14

* [CoCoI: Distributed Coded Inference System for Straggler Mitigation](https://arxiv.org/abs/2501.06856)
* [Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping](https://arxiv.org/abs/2501.06589)

### 2025-01-13

* [A Practical Cross-Layer Approach for ML-Driven Storage Placement in Warehouse-Scale Computers](https://arxiv.org/abs/2501.05651)

### 2025-01-10

* [Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing](https://arxiv.org/abs/2501.05313)

### 2025-01-09

* [Collaborative Inference Acceleration with Non-Penetrative Tensor Partitioning](https://arxiv.org/abs/2501.04489)
* [Scalable Data Notarization Leveraging Hybrid DLTs](https://arxiv.org/abs/2501.04571)

### 2025-01-07

* [TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms](https://arxiv.org/abs/2501.02600)

### 2025-01-06

* [Efficient LLM Inference with Activation Checkpointing and Hybrid Caching](https://arxiv.org/abs/2501.01792)

### 2025-01-03

* [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://arxiv.org/abs/2501.01005)
* [Dynamic Optimization of Storage Systems Using Reinforcement Learning Techniques](https://arxiv.org/abs/2501.00068)

awesome-papers-LMsys

awesome-papers-LMsys Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

awesome-papers-LMsys

awesome-papers-LMsys Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README