QwenLM/ParScale
Parallel Scaling Law for Language Model — Beyond Parameter and Inference Time Scaling
### Overview:
Parallel Scaling Law for Language Model (ParScale) introduces a new scaling paradigm that leverages parallel computation during both training and inference time. This approach aims to address the limitations of parameter and inference-time scaling by providing a more efficient and universal scaling solution.
### Key Features:
1. **Parallel Computation**: Utilizes $P$ diverse and learnable transformations to the input, executing forward passes of the model in parallel, and dynamically aggregating the $P$ outputs.
2. **Logarithmic Scaling Law**: The theoretical and empirical establishment of a logarithmic scaling law, suggesting that scaling with $P$ parallel streams is comparable to scaling the number of parameters by $O(\log P)$.
3. **Universal Applicability**: Works with any model architecture, optimization method, data, or downstream task, unlike inference-time scaling which requires specialized data and limited application.
4. **Superior Inference Efficiency**: Can use up to **22x less memory increase** and **6x less latency increase** compared to parameter scaling that achieves the same performance improvement (batch size=1).
5. **Cost-Efficient Training via Two-Stage Strategy**: Training a parallel-scaled model does not require starting from scratch, with a two-stage training strategy using only a small amount of data post-training.
### Tech Stack:
The primary language used for the project is Python.
### Why It Matters:
- **Efficient Scaling**: ParScale provides an efficient alternative to parameter and inference-time scaling, making it suitable for larger models.
- **Universal Scalability**: Works with any