Python
14 stars
Advanced LLM fine-tuning techniques: SFT (LoRA, QLoRA, DoRA, P-/Prefix-Tuning), GRPO, DPO, ORPO, KTO & PPO; composable correctness/format rewards + LLM-as-a-Judge evals (DeepEval, Evidently AI) across math, multi-hop, medical & general QA on Llama 3, Mistral, Phi-4, Gemma & Qwen3. Built on TRL, PEFT & Unsloth.
Overview: This repository contains advanced techniques for fine-tuning large language models (LLMs) across various domains, including math reasoning, multi-hop question answering, medical question answering, and general question answering. It utilizes Supervised Fine-Tuning, Reinforcement Learning with Group Relative Policy Optimization (GRPO), and Preference Alignment, among other methods.
Key features:
- **LoRA (Low-rank Weight Updates)**: Low-rank weight updates applied to attention and feed-forward projection matrices.
- **QLoRA (4-bit NF4 Quantization)**: LoRA with 4-bit NF4 quantization of the base model via BitsAndBytes.
- **DoRA (Weight-Decomposed LoRA)**: Decomposes weights into magnitude and direction, applies LoRA to the directional component.
- **P-Tuning (Trains a Small Encoder Network)**: Trains a small encoder network to produce continuous prompt embeddings.
- **Prefix-Tuning (Prepends Trainable Prefix Vectors)**: Prepends trainable prefix vectors to the key and value tensors of every attention layer.
- **Datasets**: Includes datasets like ARC, TriviaQA, FactScore, PopQA, and Earnings Calls for various domains.
Tech stack: The repository is built on HuggingFace TRL, PEFT, and Unsloth, with reward functions powered by DeepEval and Evidently AI.
Why it matters: This repository provides comprehensive implementations of fine-tuning techniques for LLMs across multiple domains, enhancing the performance and applicability of LLMs in various tasks.