actypedef/SharQ
Code for the paper “SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference”
### Overview:
SharQ is a project focused on bridging activation sparsity and FP4 quantization for large language models (LLMs) in inference. The repository aims to provide practical modes for quantization, including a dense FP4 baseline (`NVFP4`), a fused sparse-residual FP4 kernel path (`SHARQ`), and a pure PyTorch fake-quantized, fake-sparse simulation (`SHARQ_SIM`) for accuracy-only reference. The project is built around the idea of using a sparse FP4 main path to capture important activations and a dense FP4 residual path to recover the loss from sparsification and quantization.
### Key Features:
- **SHARQ Mode**: A fused sparse-residual FP4 kernel path that combines sparse and dense FP4 operations to achieve efficient inference.
- **SHARQ_SIM Mode**: A PyTorch-based simulation that provides a reference path for quantization without the need for CUDA extension.
- **NVFP4 Mode**: A dense FP4 baseline that is the default mode for quantization.
### Tech Stack:
The project utilizes Python 3.10, PyTorch, and CUDA 12.8. It includes a CUDA extension for efficient kernel paths and low-level benchmarks for correctness, performance, and ablation studies.
### Why It Matters:
SharQ offers a practical solution for quantizing large language models while maintaining performance and accuracy. By combining sparse and dense FP4 operations, it provides a balance between sparsity and precision, making it suitable for real-world applications where model size and inference speed