TSINGHUA BREAKTHROUGH: AKV Cache System Cuts LLM Cost 27.7%...

Tsinghua researchers released AKV, an SGLang-based serving system for cache engineering. LLM agents drop and reposition KV cache instead of summarizing history, cutting average cost 27.7% across 4 age