LLM KV Cache Systems Engineer

About the role

We are seeking an LLM KV Cache Systems Engineer to design and build key‑value cache subsystems that power large‑language‑model inference. This systems‑focused position requires deep expertise in memory allocation, lifecycle management, compression, and distributed serving, with a strong emphasis on source‑level implementation.

Key responsibilities

• Design block and page allocators, KV cache lifecycle management, fragmentation control, eviction, and preemption mechanisms.
• Optimize KV layouts, handle GQA/MQA, apply quantization, compression, or sparsity while balancing accuracy and performance.
• Build KV transfer and disaggregated‑serving mechanisms, including prefill/decode separation and RDMA/NVLink/CXL transport.
• Develop KV‑aware request routing, batching, admission control, scheduling, and multi‑tenant isolation under latency and SLA constraints.
• Integrate and productionize KV technologies in frameworks such as vLLM, SGLang, TensorRT‑LLM, LMDeploy, Mooncake, or DistServe, adding observability and failure handling.

Required profile

• Master’s degree or Ph.D. in Computer Science, Computer Engineering, Distributed Systems, High‑Performance Computing, or a related field.
• Deep understanding of transformer attention, prefill/decode execution, memory behavior, and modern LLM serving architectures.
• Direct source‑level design, implementation, or research experience in an LLM KV cache subsystem.
• Demonstrated impact on at least one KV‑specific metric (capacity, hit rate, fragmentation, transfer bandwidth, latency, etc.).

Required skills

• C++
• Python
• Linux
• CUDA or equivalent accelerator development
• Distributed‑system performance engineering
• RDMA, NVLink, CXL transport technologies