Cleared 1m ago · sgl-project/sglang #10585 · Python · ★ 39k · Apache-2.0

[Feature] Integrate KVPress in order to try more KV cache optimization heuristics easily

What is this? In long context-serving instances, the KV cache can grow to a large size in O(n) complexity, and optimal compression can help us reach sub linear scaling of memory growth. KVPress library integrates many unique strategies to make it easier to try different…

What we checked

Similar unclaimed issues