Anyesh/EVOKE
EVict and recOver KV cache Entries. Selective KV cache eviction and recovery for long-context LLM inference.
### Overview:
EVOKE is a reversible KV-cache eviction and recovery system designed specifically for long-context language model inference. It addresses the issue of working set overflow by selectively evicting and recovering key-value pairs from a limited memory budget, ensuring that the model retains relevant context without losing important information.
### Key Features:
- **Reversible KV-Eviction**: EVOKE allows for reversible eviction, where evicted key-value pairs can be recovered without losing the original computation.
- **Selective KV-Eviction**: It selectively evicts and recovers key-value pairs based on their importance, ensuring that the most relevant context is retained.
- **Recompute-Free Recall**: The system provides recompute-free recall of evicted key-value pairs, maintaining the original model state without recomputation.
### Tech Stack:
- **Python**: The primary language used for implementing EVOKE.
- **RoPE (Relative Position Encoding)**: Used in the splice mechanism to handle the repositioning of key-value pairs after eviction.
### Why It Matters:
EVOKE is crucial for maintaining the effectiveness and efficiency of long-running language model sessions, ensuring that the model retains the most relevant context without losing important information. This system is particularly valuable in scenarios where the working set of a language model grows beyond its memory budget, such as in continuous interaction sessions or in scenarios where the model needs to handle long sequences of queries and responses.
### Repository:
The repository for EVOKE is located at [Anyesh/EVOKE](https://github.com/Anyesh/EVOKE).
### GitHub Stars:
The repository has been starred by 8 developers