## kvcached: A New Approach to LLM GPU Serving A new project, **kvcached**, aims to improve GPU utilization for Large Language Model (LLM) serving, particularly on shared GPU systems. The core idea is a virtualized, elastic KV cache, addressing the issue of expensive virtual memory management operations on GPUs (CUDA/HIP). Current GPU virtual memory operations can be slow due to host-device synchronization and driver overhead. kvcached tackles this by utilizing indirection tables within kernels, allowing for efficient memory remapping and offloading to cold storage without blocking the GPU pipeline. Discussion highlights that while dynamic allocation of entire servers exists, kvcached focuses on optimizing smaller models running on existing GPUs. Concerns were raised about the blog post's writing style, with some suggesting it was AI-generated due to stylistic choices like excessive bolding and em-dash usage – a claim the author denies. The project is actively working on paging and offloading features to further enhance memory efficiency.