Projects

### Overview: AutoTree is a tree execution engine designed for large language model (LLM) inference, specifically tailored for LLMs like Qwen. It optimizes the execution of LLMs by sharing prefix key-value state across branches, forked candidate continuations with copy-on-write pages, and leveraging a Rust scheduler to prune or continue them under a hard token budget. This approach aims to improve performance and efficiency in LLM inference tasks. ### Key Features: - **Tree Execution Engine**: AutoTree executes LLM reasoning as a tree, optimizing resource usage and performance. - **Prefix KV State Sharing**: Shares prefix key-value state across branches to reduce redundant computations. - **Forking and Pruning**: Forks candidate continuations with copy-on-write pages to prevent unnecessary computations, and prunes or continues them under a hard token budget. - **OpenAI-Compliant API**: Adopting AutoTree means changing one `base_url`, ensuring compatibility with the OpenAI API. ### Tech Stack: - **Python**: The primary language used for the backend logic. - **Rust**: The Rust scheduler is used to manage and optimize the execution of LLMs. ### Why It Matters: AutoTree is designed to optimize LLM inference by leveraging the strengths of both Python and Rust. The shared prefix key-value state across branches and the efficient scheduling mechanism provided by Rust make it particularly effective for LLM inference tasks. By sharing state, AutoTree reduces redundant computations and improves performance, making it a valuable tool for developers working with large language models. ### Repository: The repository is hosted

Overview: This project showcases the implementation of Monte Carlo Tree Search (MCTS) combined with a large language model to generate and evaluate logical reasoning steps, or "chains-of-thought," for advanced problem-solving. Key features: - **Monte Carlo Tree Search (MCTS)**: Efficiently explores possible reasoning paths to find optimal solutions. - **Chain-of-Thought Generation**: Utilizes Ollama to generate concise and relevant reasoning steps. - **Feasibility and Evaluation Checks**: Ensures generated chains are logically feasible and correct. - **Modular Architecture**: Clean and maintainable codebase organized into well-defined modules. - **Extensible Design**: Easily add new features, evaluation metrics, or integrate with other models. Tech stack: Python Why it matters: This project aims to explore the synergy between MCTS algorithms and advanced language models to enhance problem-solving capabilities through iterative reasoning and evaluation, potentially improving the efficiency and accuracy of logical reasoning in complex problem-solving scenarios.

### Overview: Kirin is a self-hosted and local-first application designed for inference and Rich-Application-Graph (RAG) processing on consumer-grade hardware. It leverages 12 CPUs for inference and supports multiple neural network types, including pre-defined templates, fine-tuning, and custom neural networks. The project is built using FastAPI and Docker, providing a scalable and efficient development environment. ### Key Features: - **Inference Mode (default)**: Chat with a small language model at consumer-grade hardware using 8 CPUs. - **RAG Mode**: Chat with a small language model based on pre-processed datasets. - **CRM Features**: Multiple users can log in, chat, and save chat history in JSON format. - **Neural Network Definition**: Supports pre-defining neural networks as templates, fine-tuning, and custom neural networks through an intuitive UI. ### Tech Stack: - 🐳 Dockerized - 🐘 Asynchronous PostgreSQL - 🐍 FastAPI ### Why It Matters: Kirin is designed for efficient inference and RAG processing, leveraging the power of 12 CPUs for optimal performance. The use of FastAPI and Docker ensures a robust and scalable development environment, making it ideal for local-first applications. ### Repository: - **Repository**: [GitHub](https://github.com/SkywardAI/kirin) - **GitHub Stars**: 6 - **Primary Language**: Python ### Roadmap: - **Inference Mode (default)**: Chat with the small language model at consumer-grade hardware with 8 CPUs

Overview: This project focuses on evaluating and comparing state-of-the-art Bird's-Eye View (BEV) detection models, including BEVFusion, PointPillars, 3DSSD, and CenterPoint. Key features: - BEV visualization tools for top-down view analysis - Multi-model comparison across KITTI and nuScenes datasets - Comprehensive evaluation metrics (mAP, precision, recall, IoU) - Automated artifact export (point clouds, bounding boxes, predictions, visualizations) - BEVFusion support with custom CUDA operations compilation Tech stack: Python Why it matters: This project provides a comprehensive framework for evaluating and comparing BEV-based 3D object detection models, making it valuable for researchers and developers working in autonomous driving and related fields.

### Overview: SILKern is a deterministic sparse-index localization framework optimized for Apple silicon (MLX/Metal) and NVIDIA GPUs (Triton). It ensures stable ordering and provides a Python oracle, making it reproducible and reliable. The framework handles context-parallel attention efficiently, with a focus on performance and accuracy. ### Key features: - **Stable Ordering**: Ensures consistent and predictable results across different environments. - **Python Oracle**: Provides a Python interface for easy integration with existing projects. - **Deterministic Sparse-Index Localization**: Optimized for sparse attention queries, filtering and translating positions efficiently. - **Context-Parallel Attention**: Handles decode steps on multiple ranks, improving performance through parallel processing. ### Tech stack: Python ### Why it matters: SILKern is crucial for applications requiring high-performance sparse attention, such as natural language processing, machine learning, and other fields where context parallelism is essential. Its deterministic and stable ordering ensure consistent results across different environments, making it a valuable tool for developers and researchers.

### Overview: Medibeng-Orpheus-3b-0.1-ft Fine-Tuning is a TTS model designed for bilingual Bengali-English code-switching in healthcare, optimized for seamless patient-doctor interactions. ### Key Features: - **Bilingual Code-Switching:** Handles seamless transitions between Bengali and English in a conversational setting. - **Healthcare Focus:** Tailored for use in healthcare, simulating medical dialogues between patients and doctors. - **Accelerated Fine-Tuning:** Utilizes **HuggingFace’s TRL** and **Unsloth** for a 2x faster training process. ### Tech Stack: Python ### Why It Matters: This model is crucial for providing accurate and contextually appropriate speech in healthcare settings, enhancing patient-doctor communication and improving patient outcomes. Repository: The-Data-Dilemma/Medibeng-Orpheus-3b-0.1-ft-Fine-Tuning

### Overview: SelfAug is a method designed to mitigate catastrophic forgetting during the fine-tuning process of Retrieval-Augmented Generation (RAG). It achieves this by aligning input-sequence logits with the original model distribution, ensuring that the fine-tuned model retains its general capability while learning downstream RAG responses without the need for replay data or expensive response-generation pipelines. ### Key Features: - **Preserves Original Model Distribution**: SelfAug ensures that the fine-tuned model remains close to the base model on the input sequence, maintaining its general capability. - **No Replay Data or Validation Loop**: It eliminates the need for additional replay data and avoids the complexity of response-generation validation loops. - **KL-Divergence Alignment Loss**: Adds a self-distribution alignment loss alongside the downstream negative log-likelihood loss, optimizing task learning and original-distribution preservation together. ### Tech Stack: - **Python**: The primary language used for implementation. ### Why It Matters: SelfAug is crucial for RAG fine-tuning, particularly in mitigating catastrophic forgetting, which is a common issue in this type of fine-tuning where the model quickly loses its ability to perform well on downstream tasks after training on a specific task. By preserving the original model distribution, SelfAug helps maintain the model's general capability, ensuring that it continues to perform well across different tasks, even after fine-tuning on a specific task. ### Repository: The repository is hosted on GitHub at <https://github.com/USTC-StarTeam/SelfAug>. It includes the official code for the paper "SelfAug: Mitigating

Overview: This project provides an end-to-end guide for beginners and developers to learn LLM fine-tuning using the Unsloth, LoRA, PEFT, and Google Colab technologies. Key features: - Learn LLM fine-tuning - Use Unsloth, LoRA, PEFT, and Google Colab - Comprehensive guide for beginners and developers Tech stack: Jupyter Notebook Why it matters: This project offers a practical, step-by-step guide for anyone looking to start or improve their understanding of LLM fine-tuning, making it accessible and useful for both beginners and developers.

### Overview: The Rare AI Archive is an open-source project aimed at closing the diagnostic gap for rare genetic diseases. It leverages decentralized post-training, clinician validation, and federated deployment to create and deploy AI diagnostic tools that improve with every interaction. ### Key Features: - **Context Creators**: Patients, families, and clinicians contribute structured clinical vignettes. - **Validators**: Rare disease centers and research partners validate and curate training data. - **Model Builders**: Open-source ML engineers fine-tune condition-specific models and publish them on HuggingFace. ### Tech Stack: Python ### Why It Matters: The Rare AI Archive aims to close the diagnostic gap for rare genetic diseases, reducing the average diagnostic odyssey from 5-7 years to just a few weeks. By leveraging the expertise of rare disease specialists, it creates a more accurate and effective diagnostic tool that can improve with every clinician interaction, ultimately leading to better patient outcomes and treatments.

### Overview: This project focuses on breast cancer classification using deep learning models, specifically enhanced ResNet-50 architecture with fine-tuning and optimization techniques. The proposed method achieves 98.46% accuracy on the breast ultrasound image dataset, making it a significant improvement over conventional approaches. ### Key Features: - **Enhanced ResNet-50 Architecture:** Modified with dilated convolutions and Squeeze-and-Excitation blocks for improved performance. - **Bayesian Optimization:** Combined with Population-Based Training for efficient hyperparameter tuning. - **Transfer Learning:** Fine-tuning of pre-trained models on medical imaging datasets. - **Data Augmentation:** Techniques to enhance model robustness and accuracy. ### Tech Stack: - Jupyter Notebook ### Why It Matters: This project demonstrates the effectiveness of deep learning models in improving medical diagnosis, particularly in the context of breast cancer classification. The use of advanced optimization techniques and specialized architectures makes the models suitable for real-world clinical applications, potentially leading to earlier detection and better treatment outcomes. ### Repository: - **Repository:** Yashmaini30/Breast-Cancer-Detection - **GitHub Stars:** 6 - **GitHub Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50, AlexNet, VGG16, and GoogLeNet with fine-tuning, feature extraction, and Bayesian optimization. ### Primary Language: - **Primary Language:** Jupyter Notebook ### GitHub Description: - **Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50

### Overview: Tuna is a Python library designed to simplify the fine-tuning of transformer-based models, making the process more efficient and user-friendly. ### Key Features: - **Multiple Fine-tuning Methods**: Supports LoRA, Prompt tuning (P-tuning), Full parameter training (SFT), and Domain-Adaptive Fine-Tuning (DAFT). - **Flexible & Chainable**: Allows chaining different fine-tuning methods to create customized training pipelines. - **Quick Start**: Requires `pip install tuna` to get started. ### Tech Stack: Python ### Why It Matters: Tuna simplifies the complex process of fine-tuning transformer-based models, making it accessible and efficient for developers and researchers alike.

Overview: This project is a legal advisory chatbot system designed to assist users by providing accurate and up-to-date legal information related to Vietnamese law. It leverages advanced NLP techniques and a large dataset of legal documents from official Vietnamese websites to ensure reliable and contextually relevant responses. Key features: - Vietnamese Law Knowledge Base - Contextual Understanding - Efficient Legal Information Retrieval - User-Friendly Interface Tech stack: Python Why it matters: This chatbot system offers a valuable resource for individuals, businesses, and legal professionals seeking quick access to legal knowledge, making it a valuable tool in various professional and personal contexts.

### Overview: This project focuses on cache-aware scheduling, memory allocation, spill selection, and pipeline compression for Directed Acyclic Graphs (DAGs) structured compute graphs on SIMD/NPU accelerators. Motivated by challenges in Large Language Model (LLM) inference runtime and compiler systems, the project aims to provide a compact codebase for studying scheduling and memory-pressure decisions on public graph instances. ### Key Features: - **Cache-pressure-aware topological scheduling**: Balances operator readiness, cache residency, and DMA traffic. - **Contiguous multi-pool memory allocation**: Decides SPILL operations when cache capacity is insufficient. - **Pipeline compression**: Estimates pipeline slack under execution-unit constraints. ### Tech Stack: Python 3.10+ ### Why it Matters: This project provides a compact and reproducible solution for studying scheduling and memory-pressure decisions on LLM and NPU scheduling problems, offering insights into the challenges encountered in modern inference workloads and compiler systems.

### Overview: This repository contains Jupyter Notebook tutorials for pre-training and fine-tuning transformer models using PyTorch and the Hugging Face Transformers library. It offers comprehensive guidance on implementing these tasks with practical examples and detailed explanations. ### Key Features: - **Pretraining Transformer in PyTorch**: Explains how to pre-train 67 transformer models on custom datasets. - **Fine-tuning Transformers in PyTorch**: Guides through the process of fine-tuning 73 transformer models for text classification tasks, including automatic model detection and no code changes necessary. - **Dataset Processing**: Provides a function `movie_reviews_to_file` for converting text data into a format suitable for training transformers. - **Compatibility**: Tested with 67 pretrained models and compatible with various transformer architectures. - **Customization**: The notebook can be easily adapted for different datasets and fine-tuning tasks. - **Prerequisites**: Basic Python knowledge and familiarity with PyTorch and Transformers library. ### Tech Stack: Jupyter Notebook ### Why it Matters: This repository offers a practical and flexible approach to fine-tuning and pre-training transformer models, making it ideal for developers looking to enhance their machine learning skills.

### Overview: This repository provides a guide for running distilled and quantized versions of the DeepSeek-R1 model on Amazon EC2, optimizing inference performance without significantly compromising accuracy. ### Key Features: - Distilled models - Quantized models - Dynamic quantization for faster token/sec without sacrificing accuracy ### Tech Stack: Python ### Why It Matters: This approach allows running the model on smaller instances, reducing costs and making it more feasible to deploy on a wide range of hardware configurations.

Overview: aihappey-ai is a stateless, open-source AI gateway that exposes a unified endpoint for accessing a vast collection of models, media, skills, agents, and more from multiple providers. Key features: - Access to 150k+ models and provider-native capabilities - Stateless design, no smart routing or admin portal - Managed agents and skills across 150k+ models - Managed by Vercel UI streaming, no hidden logic maze - Managed by managed agents and skills across 150k+ models Tech stack: C# Why it matters: aihappey-ai provides a convenient and efficient way to access a wide range of AI models and capabilities, making it easier for developers to integrate AI into their projects without the need for complex routing or management.

### Overview: This project is a production-inspired LLM inference engine built from scratch in PyTorch. It implements KV caching, continuous batching, paged memory management, and async serving, with benchmarks at each optimization layer. Key features include a 2.22x decode speedup, 118x throughput, 8.7x memory reduction, and 11.1x system throughput. ### Key Features: - **KV Cache:** Eliminates redundant attention recomputation, resulting in a 2.19x speedup. - **Batched Inference:** Saturates GPU compute, offering a 118x throughput improvement. - **Continuous Batching:** Dynamic scheduling to eliminate idle GPU slots, boosting system throughput by 11.1x. - **Paged KV Cache:** Reduces memory usage by 8.7x compared to contiguous pre-allocation. - **Async Serving:** FastAPI server with backpressure and routing for efficient inference. ### Tech Stack: - **Python** - The primary language used for development and optimization. ### Why It Matters: Understanding the design decisions behind production LLM serving systems, such as vLLM, TGI, and TensorRT-LLM, is crucial. This project isolates and benchmarks each optimization layer, providing a comprehensive understanding of what matters and by how much in LLM inference. ### Repository: - **Repository:** achi9629/llm-inference-engine - **GitHub Stars:** 6 - **GitHub Description:** A from scratch LLM inference

Overview: This project fine-tunes a small language model to extract structured JSON from casual expense messages, aiming to measure both the task gain and the retention of general ability. Key features: - Task gain on structured JSON extraction - Retention of general ability - Baseline measurement before training - Forgetfulness check with two different scoring methods Tech stack: Python Why it matters: This project provides a clear demonstration of the benefits and potential pitfalls of LoRA fine-tuning, specifically focusing on the catastrophic forgetting issue that is often overlooked in most projects.

Overview: This hands-on aiDAPTIV training course is designed to teach developers, researchers, and students how to fine-tune large language models (LLMs) locally using the aiDAPTIV technology. The course includes detailed lesson materials, datasets, code examples, and project files to provide a comprehensive learning experience. Key features: - Fine-tuning LLMs on your own hardware or a provided remote AI Training PC (AITPC) - Optimizing GPU VRAM usage with aiDAPTIV cache memory - Using both GUI and CLI workflows for training and inference - Applying fine-tuning to real-world datasets such as company handbooks or public figure transcripts Tech stack: aiDAPTIV technology Why it matters: This course offers practical skills to run on-prem AI cost-efficiently and securely, making it ideal for developers, researchers, and students looking to enhance their AI capabilities.

### Overview: Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads. ### Key features: - 🚀 AWS & IBM Cloud support - ⚡ Automated infrastructure: MachineSets, autoscaling, ClusterAutoscaler - 🧩 Operators & platform services: NVIDIA GPU Operator, Node Feature Discovery, Descheduler, KEDA, Gateway API, Kuadrant, Authorino, cert-manager - 🕵️ Observability: NetObserv, LokiStack, Grafana dashboards (WIP) - 🔬 Example manifests for KServe LLMInferenceService & KV-cache routing - 🧪 Precise-prefix cache-aware experiments - 🔄 GitOps App-of-Apps via Argo CD (WIP) ### Tech stack: Jupyter Notebook ### Why it matters: This lab provides a streamlined and reproducible environment for evaluating large language models, ensuring consistent and reliable results across different environments and configurations.