Python
834 stars
Halo is an open-source framework built by White Circle for training large language and multimodal models
### Overview: Halo is an open-source framework designed by White Circle for training large language and multimodal models. It supports various training methods and parallelism techniques, making it highly efficient and scalable.
### Key Features:
- **Parallelism Techniques:** EP, CP, TP, ETP, FSDP2, BF16 training, distributed rollouts.
- **Training Methods:** Pre-training, SFT, preference optimization, fully asynchronous multi-turn RL.
- **Model Integration:** Trainers subclass Hugging Face or TRL trainers, checkpoints remain standard SafeTensors, and `from_pretrained` continues to work.
- **Performance Optimizations:** DeepEP V2, FlashAttention 4, Liger, Grouped GEMM, padding-free packing, AdamWBF16, and more.
### Tech Stack: Python
### Why It Matters:
- **Direct Training:** Trainers subclass Hugging Face or TRL trainers, checkpoints remain standard SafeTensors, and `from_pretrained` continues to work.
- **Parallelism:** EP wraps MoE blocks, CP wraps attention, and TP/ETP shard weights in place. There is no separate distributed implementation of the model.
- **Distributed Stack:** FSDP2, DTensor, and DeviceMesh provide the underlying primitives, while DeepEP handles the all-to-all communication for Expert Parallelism. EP, CP, TP, and ETP each have their own setting; EP combines with CP, TP, or ETP across GPUs and nodes, and other pairings are rejected at config time.
-