Projects

### Overview: SelfAug is a method designed to mitigate catastrophic forgetting during the fine-tuning process of Retrieval-Augmented Generation (RAG). It achieves this by aligning input-sequence logits with the original model distribution, ensuring that the fine-tuned model retains its general capability while learning downstream RAG responses without the need for replay data or expensive response-generation pipelines. ### Key Features: - **Preserves Original Model Distribution**: SelfAug ensures that the fine-tuned model remains close to the base model on the input sequence, maintaining its general capability. - **No Replay Data or Validation Loop**: It eliminates the need for additional replay data and avoids the complexity of response-generation validation loops. - **KL-Divergence Alignment Loss**: Adds a self-distribution alignment loss alongside the downstream negative log-likelihood loss, optimizing task learning and original-distribution preservation together. ### Tech Stack: - **Python**: The primary language used for implementation. ### Why It Matters: SelfAug is crucial for RAG fine-tuning, particularly in mitigating catastrophic forgetting, which is a common issue in this type of fine-tuning where the model quickly loses its ability to perform well on downstream tasks after training on a specific task. By preserving the original model distribution, SelfAug helps maintain the model's general capability, ensuring that it continues to perform well across different tasks, even after fine-tuning on a specific task. ### Repository: The repository is hosted on GitHub at <https://github.com/USTC-StarTeam/SelfAug>. It includes the official code for the paper "SelfAug: Mitigating

Overview: This project provides an end-to-end guide for beginners and developers to learn LLM fine-tuning using the Unsloth, LoRA, PEFT, and Google Colab technologies. Key features: - Learn LLM fine-tuning - Use Unsloth, LoRA, PEFT, and Google Colab - Comprehensive guide for beginners and developers Tech stack: Jupyter Notebook Why it matters: This project offers a practical, step-by-step guide for anyone looking to start or improve their understanding of LLM fine-tuning, making it accessible and useful for both beginners and developers.

### Overview: The Rare AI Archive is an open-source project aimed at closing the diagnostic gap for rare genetic diseases. It leverages decentralized post-training, clinician validation, and federated deployment to create and deploy AI diagnostic tools that improve with every interaction. ### Key Features: - **Context Creators**: Patients, families, and clinicians contribute structured clinical vignettes. - **Validators**: Rare disease centers and research partners validate and curate training data. - **Model Builders**: Open-source ML engineers fine-tune condition-specific models and publish them on HuggingFace. ### Tech Stack: Python ### Why It Matters: The Rare AI Archive aims to close the diagnostic gap for rare genetic diseases, reducing the average diagnostic odyssey from 5-7 years to just a few weeks. By leveraging the expertise of rare disease specialists, it creates a more accurate and effective diagnostic tool that can improve with every clinician interaction, ultimately leading to better patient outcomes and treatments.

### Overview: This project focuses on breast cancer classification using deep learning models, specifically enhanced ResNet-50 architecture with fine-tuning and optimization techniques. The proposed method achieves 98.46% accuracy on the breast ultrasound image dataset, making it a significant improvement over conventional approaches. ### Key Features: - **Enhanced ResNet-50 Architecture:** Modified with dilated convolutions and Squeeze-and-Excitation blocks for improved performance. - **Bayesian Optimization:** Combined with Population-Based Training for efficient hyperparameter tuning. - **Transfer Learning:** Fine-tuning of pre-trained models on medical imaging datasets. - **Data Augmentation:** Techniques to enhance model robustness and accuracy. ### Tech Stack: - Jupyter Notebook ### Why It Matters: This project demonstrates the effectiveness of deep learning models in improving medical diagnosis, particularly in the context of breast cancer classification. The use of advanced optimization techniques and specialized architectures makes the models suitable for real-world clinical applications, potentially leading to earlier detection and better treatment outcomes. ### Repository: - **Repository:** Yashmaini30/Breast-Cancer-Detection - **GitHub Stars:** 6 - **GitHub Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50, AlexNet, VGG16, and GoogLeNet with fine-tuning, feature extraction, and Bayesian optimization. ### Primary Language: - **Primary Language:** Jupyter Notebook ### GitHub Description: - **Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50

### Overview: Tuna is a Python library designed to simplify the fine-tuning of transformer-based models, making the process more efficient and user-friendly. ### Key Features: - **Multiple Fine-tuning Methods**: Supports LoRA, Prompt tuning (P-tuning), Full parameter training (SFT), and Domain-Adaptive Fine-Tuning (DAFT). - **Flexible & Chainable**: Allows chaining different fine-tuning methods to create customized training pipelines. - **Quick Start**: Requires `pip install tuna` to get started. ### Tech Stack: Python ### Why It Matters: Tuna simplifies the complex process of fine-tuning transformer-based models, making it accessible and efficient for developers and researchers alike.

Overview: This project is a legal advisory chatbot system designed to assist users by providing accurate and up-to-date legal information related to Vietnamese law. It leverages advanced NLP techniques and a large dataset of legal documents from official Vietnamese websites to ensure reliable and contextually relevant responses. Key features: - Vietnamese Law Knowledge Base - Contextual Understanding - Efficient Legal Information Retrieval - User-Friendly Interface Tech stack: Python Why it matters: This chatbot system offers a valuable resource for individuals, businesses, and legal professionals seeking quick access to legal knowledge, making it a valuable tool in various professional and personal contexts.

### Overview: This project focuses on cache-aware scheduling, memory allocation, spill selection, and pipeline compression for Directed Acyclic Graphs (DAGs) structured compute graphs on SIMD/NPU accelerators. Motivated by challenges in Large Language Model (LLM) inference runtime and compiler systems, the project aims to provide a compact codebase for studying scheduling and memory-pressure decisions on public graph instances. ### Key Features: - **Cache-pressure-aware topological scheduling**: Balances operator readiness, cache residency, and DMA traffic. - **Contiguous multi-pool memory allocation**: Decides SPILL operations when cache capacity is insufficient. - **Pipeline compression**: Estimates pipeline slack under execution-unit constraints. ### Tech Stack: Python 3.10+ ### Why it Matters: This project provides a compact and reproducible solution for studying scheduling and memory-pressure decisions on LLM and NPU scheduling problems, offering insights into the challenges encountered in modern inference workloads and compiler systems.

### Overview: This repository contains Jupyter Notebook tutorials for pre-training and fine-tuning transformer models using PyTorch and the Hugging Face Transformers library. It offers comprehensive guidance on implementing these tasks with practical examples and detailed explanations. ### Key Features: - **Pretraining Transformer in PyTorch**: Explains how to pre-train 67 transformer models on custom datasets. - **Fine-tuning Transformers in PyTorch**: Guides through the process of fine-tuning 73 transformer models for text classification tasks, including automatic model detection and no code changes necessary. - **Dataset Processing**: Provides a function `movie_reviews_to_file` for converting text data into a format suitable for training transformers. - **Compatibility**: Tested with 67 pretrained models and compatible with various transformer architectures. - **Customization**: The notebook can be easily adapted for different datasets and fine-tuning tasks. - **Prerequisites**: Basic Python knowledge and familiarity with PyTorch and Transformers library. ### Tech Stack: Jupyter Notebook ### Why it Matters: This repository offers a practical and flexible approach to fine-tuning and pre-training transformer models, making it ideal for developers looking to enhance their machine learning skills.

### Overview: This repository provides a guide for running distilled and quantized versions of the DeepSeek-R1 model on Amazon EC2, optimizing inference performance without significantly compromising accuracy. ### Key Features: - Distilled models - Quantized models - Dynamic quantization for faster token/sec without sacrificing accuracy ### Tech Stack: Python ### Why It Matters: This approach allows running the model on smaller instances, reducing costs and making it more feasible to deploy on a wide range of hardware configurations.

Overview: aihappey-ai is a stateless, open-source AI gateway that exposes a unified endpoint for accessing a vast collection of models, media, skills, agents, and more from multiple providers. Key features: - Access to 150k+ models and provider-native capabilities - Stateless design, no smart routing or admin portal - Managed agents and skills across 150k+ models - Managed by Vercel UI streaming, no hidden logic maze - Managed by managed agents and skills across 150k+ models Tech stack: C# Why it matters: aihappey-ai provides a convenient and efficient way to access a wide range of AI models and capabilities, making it easier for developers to integrate AI into their projects without the need for complex routing or management.

### Overview: This project is a production-inspired LLM inference engine built from scratch in PyTorch. It implements KV caching, continuous batching, paged memory management, and async serving, with benchmarks at each optimization layer. Key features include a 2.22x decode speedup, 118x throughput, 8.7x memory reduction, and 11.1x system throughput. ### Key Features: - **KV Cache:** Eliminates redundant attention recomputation, resulting in a 2.19x speedup. - **Batched Inference:** Saturates GPU compute, offering a 118x throughput improvement. - **Continuous Batching:** Dynamic scheduling to eliminate idle GPU slots, boosting system throughput by 11.1x. - **Paged KV Cache:** Reduces memory usage by 8.7x compared to contiguous pre-allocation. - **Async Serving:** FastAPI server with backpressure and routing for efficient inference. ### Tech Stack: - **Python** - The primary language used for development and optimization. ### Why It Matters: Understanding the design decisions behind production LLM serving systems, such as vLLM, TGI, and TensorRT-LLM, is crucial. This project isolates and benchmarks each optimization layer, providing a comprehensive understanding of what matters and by how much in LLM inference. ### Repository: - **Repository:** achi9629/llm-inference-engine - **GitHub Stars:** 6 - **GitHub Description:** A from scratch LLM inference

Overview: This project fine-tunes a small language model to extract structured JSON from casual expense messages, aiming to measure both the task gain and the retention of general ability. Key features: - Task gain on structured JSON extraction - Retention of general ability - Baseline measurement before training - Forgetfulness check with two different scoring methods Tech stack: Python Why it matters: This project provides a clear demonstration of the benefits and potential pitfalls of LoRA fine-tuning, specifically focusing on the catastrophic forgetting issue that is often overlooked in most projects.

Overview: This hands-on aiDAPTIV training course is designed to teach developers, researchers, and students how to fine-tune large language models (LLMs) locally using the aiDAPTIV technology. The course includes detailed lesson materials, datasets, code examples, and project files to provide a comprehensive learning experience. Key features: - Fine-tuning LLMs on your own hardware or a provided remote AI Training PC (AITPC) - Optimizing GPU VRAM usage with aiDAPTIV cache memory - Using both GUI and CLI workflows for training and inference - Applying fine-tuning to real-world datasets such as company handbooks or public figure transcripts Tech stack: aiDAPTIV technology Why it matters: This course offers practical skills to run on-prem AI cost-efficiently and securely, making it ideal for developers, researchers, and students looking to enhance their AI capabilities.

### Overview: Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads. ### Key features: - 🚀 AWS & IBM Cloud support - ⚡ Automated infrastructure: MachineSets, autoscaling, ClusterAutoscaler - 🧩 Operators & platform services: NVIDIA GPU Operator, Node Feature Discovery, Descheduler, KEDA, Gateway API, Kuadrant, Authorino, cert-manager - 🕵️ Observability: NetObserv, LokiStack, Grafana dashboards (WIP) - 🔬 Example manifests for KServe LLMInferenceService & KV-cache routing - 🧪 Precise-prefix cache-aware experiments - 🔄 GitOps App-of-Apps via Argo CD (WIP) ### Tech stack: Jupyter Notebook ### Why it matters: This lab provides a streamlined and reproducible environment for evaluating large language models, ensuring consistent and reliable results across different environments and configurations.

Overview: AIOpsServing 是一个开源项目，旨在为算法为中心的研发、测试、评估和服务提供轻量级流程支持。 Key features: - 支持多种计算后端，包括阿里云函数计算（FC）、大数据平台 ODPS、分布式平台 Ray 和容器 Docker。 - 负责算法迭代、归档、发布和回滚的生命周期管理。 - 提供离线和在线算法性能评估。 - 计划整合异常检测、行为画像等模型仓库。 Tech stack: Python Why it matters: AIOpsServing 为算法中心的研发、测试、评估和服务提供了一个轻量级的流程支持，简化了模型的发布、生命周期管理和性能评估，提高了开发效率和质量。

Overview: AloLab is an iterative optimization tool designed to improve the accuracy and structure of AI agent models, specifically focusing on small language models. It automates the process of refining prompts to ensure models produce reliable and structured output. Key features: - Automatically optimizes prompts for accuracy and structure. - Uses real data to detect and correct errors in the model. - Improves the prompt using an optimizer LLM. - Iterates until performance converges. Tech stack: Python Why it matters: AloLab streamlines the optimization process for AI models, ensuring they become more accurate and structured without the need for fine-tuning, thus reducing fine-tuning costs and improving the overall quality of the output. Repository: alomana-lab/alolab GitHub stars: 6 GitHub description: Iterative optimization for AI agents. Turn models into accurate, structured, production-ready systems.

Overview: This project demonstrates how to stream LLM responses using AWS API Gateway Websockets and Lambda. Key features: - AWS API Gateway Websockets - Lambda function to process and stream LLM responses - Integration with AWS SNS for message delivery Tech stack: Node.js Why it matters: This project provides a practical example for developers looking to integrate LLMs with AWS services for real-time communication.

### Overview: This project focuses on Word Sense Disambiguation (WSD) for the Word-in-Context (WiC) disambiguation task. Two BERT-based models are proposed: a feature-based approach and a fine-tuning approach. The models are trained using GlossBERT, an adaptation of the BERT model for WSD tasks. The project aims to improve disambiguation accuracy by experimenting with different approaches and leveraging pre-trained models. ### Key features: - Two BERT-based models for WiC disambiguation - Feature-based and fine-tuning approaches - GlossBERT adaptation for WSD tasks - Experimenting with different WSD models and approaches ### Tech stack: Python ### Why it matters: This project contributes to the field of natural language processing by improving the accuracy of Word Sense Disambiguation (WSD) tasks, specifically for the Word-in-Context (WiC) scenario. By experimenting with different models and approaches, it aims to enhance the current state-of-the-art in WSD for WiC tasks, potentially leading to better applications in text analysis and information retrieval.

### Overview: This project implements a Universal Fine-Tuning Pipeline for the NVIDIA NitroGen model, designed to work with any gameplay dataset generated via the NitroGen/BizHawk toolchain. It supports the model's Diffusion Transformer architecture, enabling efficient fine-tuning on consumer hardware. ### Key Features: - **Universal Architecture**: Automatically handles datasets from any console (NES, SNES, PS3, Xbox) using a dynamic gamepad serialization strategy. - **DiT Support**: Specifically engineered for NitroGen's Diffusion Transformer architecture, predicting a time-horizon of actions from a single frame. - **Continuous Action Space**: Maps inputs to a rigorous 21-dimensional continuous vector space (Analogs + Buttons). - **LoRA Integration**: Uses Low-Rank Adaptation (PEFT) for efficient fine-tuning on consumer hardware. ### Tech Stack: Python ### Why it Matters: This project is essential for developers looking to integrate artificial intelligence into retro gaming, making it easier to create AI agents that can learn from gameplay data. The universal architecture and DiT support make it adaptable to various consoles, while the continuous action space ensures a robust learning environment.

Overview: This project profile details the performance of a 26B mixture-of-experts model running on a CPU, achieving a token throughput of 124 tokens per second (tokens/sec) with no GPU. The model was built from scratch using only the CPU, achieving a speed of 40 tokens/sec for a single stream and 124 tokens/sec for serving multiple requests simultaneously. Key features: - Running Gemma-4 26B at 124 tokens/sec on a CPU with no GPU - Lossless token throughput, meaning the output tokens are exactly the same as those produced by a regular Q4_0 model - Speculative decoding unlocks significant performance gains - Running 3 out of 8 experts to improve perplexity on raw text Tech stack: Shell Why it matters: This project showcases the potential of self-hosting large language models without relying on GPUs, demonstrating the feasibility of achieving high performance on a regular desktop setup.