Projects

### Overview: This project is a powerful Markdown chunking tool that intelligently protects atomic elements like code blocks, math formulas, tables, and lists, and scores each chunk based on quality dimensions. It is designed for RAG pipelines and LLM fine-tuning workflows. ### Key Features: - **Structure-Aware Chunking**: Parses heading hierarchy to build a document outline, then splits and merges sections intelligently. - **Atomic Element Protection**: Code fences, display/inline math, Markdown tables, HTML tables, and lists are replaced with placeholders before splitting, then restored. - **5-Dimension Quality Scoring**: Each chunk receives a 0-100 score based on length adequacy, heading presence, sentence completeness, atomic integrity, and content variety. - **Semantic Affinity Merging**: Short chunks are merged with their most semantically related neighbor using heading-path similarity. - **Context Preservation**: Sub-chunks receive a `> Context: Chapter > Section > Subsection` blockquote prefix when a section is split. - **Visual 3-Pane Workspace**: Upload, preview, edit, save, and export chunks through an intuitive web UI. - **Dual-Mode Usage**: Works both as a standalone web application and as an importable Node.js library function. - **Export Formats**: JSON and JSONL. - **i18n**: Full English and Simplified Chinese support with browser language auto-detection. - **No Database Required**: All storage is file-system based. ### Tech Stack: This project is built with Next.js 14 (App Router),

Overview: FlexiCache is a hierarchical KV-cache management system designed to address the memory bottleneck in large language model serving, particularly in long-context and long-generation workloads. By leveraging temporal stability of attention heads, FlexiCache optimizes GPU memory usage, reduces attention computation and host-device transfer overhead, while maintaining model quality. Key features: - Hierarchical KV-cache placement - Periodic reranking for stable heads - Classification of heads as stable or unstable - Retention of full context in host memory for stable heads Tech stack: Python Why it matters: FlexiCache efficiently manages the KV cache in large language models, reducing memory usage and improving performance, especially in high-throughput scenarios, without compromising model accuracy.

Overview: Ladder is a small open model fine-tuning and evaluation pipeline designed for competitive programming, specifically targeting Codeforces-style problems. It leverages QLoRA training and runs on free GPUs, ensuring everything runs on Kaggle or Colab T4s, making it verifiable and adaptable for various environments. Key features: - Fine-tuning and evaluation pipeline for competitive programming - Verifiable evaluation without relying on another model's output or BLEU scores - Runs on free GPUs, excluding full fine-tuning, bf16, and flash-attention - Data consists of reasoning traces from Codeforces problems, decontaminated against benchmarks - Training uses 4-bit QLoRA on Qwen2.5-Coder-3B-Instruct - Eval generates a program and runs it against up to 20 test cases, reporting pass@k Tech stack: Python Why it matters: Ladder is crucial for developers looking to improve their skills in competitive programming, providing a reliable and verifiable method to fine-tune models for Codeforces-style problems without the need for expensive GPU resources. Repository: NiLabs-Org/ladder Primary language: Python GitHub stars: 6 GitHub description: Ladder: small open models fine-tuned for competitive programming. QLoRA training + verifiable eval that runs on Kaggle/Colab GPUs.

Overview: StrangerAI is a conversational AI model designed to enhance user experience by providing personalized interactions. It leverages the power of HuggingFace Spaces SDK to create a seamless interface, making it ideal for developers looking to deploy AI solutions quickly. Key features: - Personalized Conversations: Tailored interactions based on user preferences and history. - Easy Deployment: Integrates seamlessly with HuggingFace Spaces SDK, ensuring quick deployment. - User-Friendly Interface: Utilizes GRadio for a smooth and intuitive experience. Tech stack: Python Why it matters: StrangerAI is a powerful tool for developers aiming to create AI-driven applications that can engage users effectively and efficiently. Repository: PRITHIVSAKTHIUR/StrangerAI Primary language: Python GitHub stars: 6 GitHub description: Turning Ideas to Product - StrangerAI - StrangerZone. Recommended to Deploy inside Huggingface Spaces SDK as GRADIO README excerpt: 

### Overview: LM Warden is a self-hosted OpenAI-compatible language model API that leverages NVIDIA GPUs. It allows developers to access and manage large language models directly on their hardware, providing a controlled environment with per-key usage tracking and real-time GPU usage monitoring. ### Key Features: - **Self-Hosted with Docker Compose**: Runs on Linux and macOS with Docker, providing a secure and isolated environment. - **Control Plane**: Manages models and keys, ensuring per-key usage tracking and usage attribution. - **Model Lifecycle Management**: Tracks model lifecycles and ensures proper key minting and usage tracking. - **GPU Monitoring**: Real-time monitoring of GPU usage, power draw, and token consumption. - **API Gateway**: Provides a secure and isolated API gateway for developers to interact with the model. ### Tech Stack: - **Python**: The primary programming language used for development and deployment. - **Docker Compose**: Used for container orchestration and deployment. - **vLLM v0.32.0**: The underlying engine for language models. - **llama.cpp**: The model engine used for the API. ### Why It Matters: LM Warden simplifies the process of accessing and managing large language models, ensuring that developers can focus on their applications without worrying about the underlying infrastructure. It provides a controlled environment, per-key usage tracking, and real-time GPU monitoring, making it an ideal solution for developers looking to integrate language models into their projects. ### GitHub Metadata: - **Repository**: Podwarden/lm-warden - **Stars**:

### Overview: A simple Natural Language Processing (NLP) model utilizing Docker, FastAPI, ONNX, and Multilingual Mini-LM. The model is designed for live applications and can be accessed through a Streamlit interface. ### Key Features: - **Model**: A Multilingual Mini-LM model optimized for NLP tasks. - **Technology Stack**: Docker, FastAPI, ONNX, and Multilingual Mini-LM. - **Environment Variables**: - `TIMEOUT`: Controls worker timeouts to prevent resource exhaustion. - `WORKERS_PER_CORE`: Determines the number of workers based on CPU cores. - `MAX_WORKERS`: Limits the number of simultaneous processes. - `LOG_LEVEL`: Configures the logging level for Gunicorn. ### Tech Stack: - Docker - FastAPI - ONNX - Multilingual Mini-LM ### Why It Matters: This model is optimized for real-time applications, making it suitable for live interfaces and ensuring efficient resource management. The use of Docker ensures portability and ease of deployment across different environments.

### Overview: AutoTree is a tree execution engine designed for large language model (LLM) inference, specifically tailored for LLMs like Qwen. It optimizes the execution of LLMs by sharing prefix key-value state across branches, forked candidate continuations with copy-on-write pages, and leveraging a Rust scheduler to prune or continue them under a hard token budget. This approach aims to improve performance and efficiency in LLM inference tasks. ### Key Features: - **Tree Execution Engine**: AutoTree executes LLM reasoning as a tree, optimizing resource usage and performance. - **Prefix KV State Sharing**: Shares prefix key-value state across branches to reduce redundant computations. - **Forking and Pruning**: Forks candidate continuations with copy-on-write pages to prevent unnecessary computations, and prunes or continues them under a hard token budget. - **OpenAI-Compliant API**: Adopting AutoTree means changing one `base_url`, ensuring compatibility with the OpenAI API. ### Tech Stack: - **Python**: The primary language used for the backend logic. - **Rust**: The Rust scheduler is used to manage and optimize the execution of LLMs. ### Why It Matters: AutoTree is designed to optimize LLM inference by leveraging the strengths of both Python and Rust. The shared prefix key-value state across branches and the efficient scheduling mechanism provided by Rust make it particularly effective for LLM inference tasks. By sharing state, AutoTree reduces redundant computations and improves performance, making it a valuable tool for developers working with large language models. ### Repository: The repository is hosted

Overview: This project showcases the implementation of Monte Carlo Tree Search (MCTS) combined with a large language model to generate and evaluate logical reasoning steps, or "chains-of-thought," for advanced problem-solving. Key features: - **Monte Carlo Tree Search (MCTS)**: Efficiently explores possible reasoning paths to find optimal solutions. - **Chain-of-Thought Generation**: Utilizes Ollama to generate concise and relevant reasoning steps. - **Feasibility and Evaluation Checks**: Ensures generated chains are logically feasible and correct. - **Modular Architecture**: Clean and maintainable codebase organized into well-defined modules. - **Extensible Design**: Easily add new features, evaluation metrics, or integrate with other models. Tech stack: Python Why it matters: This project aims to explore the synergy between MCTS algorithms and advanced language models to enhance problem-solving capabilities through iterative reasoning and evaluation, potentially improving the efficiency and accuracy of logical reasoning in complex problem-solving scenarios.

### Overview: Kirin is a self-hosted and local-first application designed for inference and Rich-Application-Graph (RAG) processing on consumer-grade hardware. It leverages 12 CPUs for inference and supports multiple neural network types, including pre-defined templates, fine-tuning, and custom neural networks. The project is built using FastAPI and Docker, providing a scalable and efficient development environment. ### Key Features: - **Inference Mode (default)**: Chat with a small language model at consumer-grade hardware using 8 CPUs. - **RAG Mode**: Chat with a small language model based on pre-processed datasets. - **CRM Features**: Multiple users can log in, chat, and save chat history in JSON format. - **Neural Network Definition**: Supports pre-defining neural networks as templates, fine-tuning, and custom neural networks through an intuitive UI. ### Tech Stack: - 🐳 Dockerized - 🐘 Asynchronous PostgreSQL - 🐍 FastAPI ### Why It Matters: Kirin is designed for efficient inference and RAG processing, leveraging the power of 12 CPUs for optimal performance. The use of FastAPI and Docker ensures a robust and scalable development environment, making it ideal for local-first applications. ### Repository: - **Repository**: [GitHub](https://github.com/SkywardAI/kirin) - **GitHub Stars**: 6 - **Primary Language**: Python ### Roadmap: - **Inference Mode (default)**: Chat with the small language model at consumer-grade hardware with 8 CPUs

Overview: This project focuses on evaluating and comparing state-of-the-art Bird's-Eye View (BEV) detection models, including BEVFusion, PointPillars, 3DSSD, and CenterPoint. Key features: - BEV visualization tools for top-down view analysis - Multi-model comparison across KITTI and nuScenes datasets - Comprehensive evaluation metrics (mAP, precision, recall, IoU) - Automated artifact export (point clouds, bounding boxes, predictions, visualizations) - BEVFusion support with custom CUDA operations compilation Tech stack: Python Why it matters: This project provides a comprehensive framework for evaluating and comparing BEV-based 3D object detection models, making it valuable for researchers and developers working in autonomous driving and related fields.

### Overview: SILKern is a deterministic sparse-index localization framework optimized for Apple silicon (MLX/Metal) and NVIDIA GPUs (Triton). It ensures stable ordering and provides a Python oracle, making it reproducible and reliable. The framework handles context-parallel attention efficiently, with a focus on performance and accuracy. ### Key features: - **Stable Ordering**: Ensures consistent and predictable results across different environments. - **Python Oracle**: Provides a Python interface for easy integration with existing projects. - **Deterministic Sparse-Index Localization**: Optimized for sparse attention queries, filtering and translating positions efficiently. - **Context-Parallel Attention**: Handles decode steps on multiple ranks, improving performance through parallel processing. ### Tech stack: Python ### Why it matters: SILKern is crucial for applications requiring high-performance sparse attention, such as natural language processing, machine learning, and other fields where context parallelism is essential. Its deterministic and stable ordering ensure consistent results across different environments, making it a valuable tool for developers and researchers.

### Overview: Medibeng-Orpheus-3b-0.1-ft Fine-Tuning is a TTS model designed for bilingual Bengali-English code-switching in healthcare, optimized for seamless patient-doctor interactions. ### Key Features: - **Bilingual Code-Switching:** Handles seamless transitions between Bengali and English in a conversational setting. - **Healthcare Focus:** Tailored for use in healthcare, simulating medical dialogues between patients and doctors. - **Accelerated Fine-Tuning:** Utilizes **HuggingFace’s TRL** and **Unsloth** for a 2x faster training process. ### Tech Stack: Python ### Why It Matters: This model is crucial for providing accurate and contextually appropriate speech in healthcare settings, enhancing patient-doctor communication and improving patient outcomes. Repository: The-Data-Dilemma/Medibeng-Orpheus-3b-0.1-ft-Fine-Tuning

### Overview: SelfAug is a method designed to mitigate catastrophic forgetting during the fine-tuning process of Retrieval-Augmented Generation (RAG). It achieves this by aligning input-sequence logits with the original model distribution, ensuring that the fine-tuned model retains its general capability while learning downstream RAG responses without the need for replay data or expensive response-generation pipelines. ### Key Features: - **Preserves Original Model Distribution**: SelfAug ensures that the fine-tuned model remains close to the base model on the input sequence, maintaining its general capability. - **No Replay Data or Validation Loop**: It eliminates the need for additional replay data and avoids the complexity of response-generation validation loops. - **KL-Divergence Alignment Loss**: Adds a self-distribution alignment loss alongside the downstream negative log-likelihood loss, optimizing task learning and original-distribution preservation together. ### Tech Stack: - **Python**: The primary language used for implementation. ### Why It Matters: SelfAug is crucial for RAG fine-tuning, particularly in mitigating catastrophic forgetting, which is a common issue in this type of fine-tuning where the model quickly loses its ability to perform well on downstream tasks after training on a specific task. By preserving the original model distribution, SelfAug helps maintain the model's general capability, ensuring that it continues to perform well across different tasks, even after fine-tuning on a specific task. ### Repository: The repository is hosted on GitHub at <https://github.com/USTC-StarTeam/SelfAug>. It includes the official code for the paper "SelfAug: Mitigating

Overview: This project provides an end-to-end guide for beginners and developers to learn LLM fine-tuning using the Unsloth, LoRA, PEFT, and Google Colab technologies. Key features: - Learn LLM fine-tuning - Use Unsloth, LoRA, PEFT, and Google Colab - Comprehensive guide for beginners and developers Tech stack: Jupyter Notebook Why it matters: This project offers a practical, step-by-step guide for anyone looking to start or improve their understanding of LLM fine-tuning, making it accessible and useful for both beginners and developers.

### Overview: The Rare AI Archive is an open-source project aimed at closing the diagnostic gap for rare genetic diseases. It leverages decentralized post-training, clinician validation, and federated deployment to create and deploy AI diagnostic tools that improve with every interaction. ### Key Features: - **Context Creators**: Patients, families, and clinicians contribute structured clinical vignettes. - **Validators**: Rare disease centers and research partners validate and curate training data. - **Model Builders**: Open-source ML engineers fine-tune condition-specific models and publish them on HuggingFace. ### Tech Stack: Python ### Why It Matters: The Rare AI Archive aims to close the diagnostic gap for rare genetic diseases, reducing the average diagnostic odyssey from 5-7 years to just a few weeks. By leveraging the expertise of rare disease specialists, it creates a more accurate and effective diagnostic tool that can improve with every clinician interaction, ultimately leading to better patient outcomes and treatments.

### Overview: This project focuses on breast cancer classification using deep learning models, specifically enhanced ResNet-50 architecture with fine-tuning and optimization techniques. The proposed method achieves 98.46% accuracy on the breast ultrasound image dataset, making it a significant improvement over conventional approaches. ### Key Features: - **Enhanced ResNet-50 Architecture:** Modified with dilated convolutions and Squeeze-and-Excitation blocks for improved performance. - **Bayesian Optimization:** Combined with Population-Based Training for efficient hyperparameter tuning. - **Transfer Learning:** Fine-tuning of pre-trained models on medical imaging datasets. - **Data Augmentation:** Techniques to enhance model robustness and accuracy. ### Tech Stack: - Jupyter Notebook ### Why It Matters: This project demonstrates the effectiveness of deep learning models in improving medical diagnosis, particularly in the context of breast cancer classification. The use of advanced optimization techniques and specialized architectures makes the models suitable for real-world clinical applications, potentially leading to earlier detection and better treatment outcomes. ### Repository: - **Repository:** Yashmaini30/Breast-Cancer-Detection - **GitHub Stars:** 6 - **GitHub Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50, AlexNet, VGG16, and GoogLeNet with fine-tuning, feature extraction, and Bayesian optimization. ### Primary Language: - **Primary Language:** Jupyter Notebook ### GitHub Description: - **Description:** Transfer learning-based breast cancer classification using ultrasound images. Includes ResNet-50

### Overview: Tuna is a Python library designed to simplify the fine-tuning of transformer-based models, making the process more efficient and user-friendly. ### Key Features: - **Multiple Fine-tuning Methods**: Supports LoRA, Prompt tuning (P-tuning), Full parameter training (SFT), and Domain-Adaptive Fine-Tuning (DAFT). - **Flexible & Chainable**: Allows chaining different fine-tuning methods to create customized training pipelines. - **Quick Start**: Requires `pip install tuna` to get started. ### Tech Stack: Python ### Why It Matters: Tuna simplifies the complex process of fine-tuning transformer-based models, making it accessible and efficient for developers and researchers alike.

Overview: This project is a legal advisory chatbot system designed to assist users by providing accurate and up-to-date legal information related to Vietnamese law. It leverages advanced NLP techniques and a large dataset of legal documents from official Vietnamese websites to ensure reliable and contextually relevant responses. Key features: - Vietnamese Law Knowledge Base - Contextual Understanding - Efficient Legal Information Retrieval - User-Friendly Interface Tech stack: Python Why it matters: This chatbot system offers a valuable resource for individuals, businesses, and legal professionals seeking quick access to legal knowledge, making it a valuable tool in various professional and personal contexts.

### Overview: This project focuses on cache-aware scheduling, memory allocation, spill selection, and pipeline compression for Directed Acyclic Graphs (DAGs) structured compute graphs on SIMD/NPU accelerators. Motivated by challenges in Large Language Model (LLM) inference runtime and compiler systems, the project aims to provide a compact codebase for studying scheduling and memory-pressure decisions on public graph instances. ### Key Features: - **Cache-pressure-aware topological scheduling**: Balances operator readiness, cache residency, and DMA traffic. - **Contiguous multi-pool memory allocation**: Decides SPILL operations when cache capacity is insufficient. - **Pipeline compression**: Estimates pipeline slack under execution-unit constraints. ### Tech Stack: Python 3.10+ ### Why it Matters: This project provides a compact and reproducible solution for studying scheduling and memory-pressure decisions on LLM and NPU scheduling problems, offering insights into the challenges encountered in modern inference workloads and compiler systems.

### Overview: This repository contains Jupyter Notebook tutorials for pre-training and fine-tuning transformer models using PyTorch and the Hugging Face Transformers library. It offers comprehensive guidance on implementing these tasks with practical examples and detailed explanations. ### Key Features: - **Pretraining Transformer in PyTorch**: Explains how to pre-train 67 transformer models on custom datasets. - **Fine-tuning Transformers in PyTorch**: Guides through the process of fine-tuning 73 transformer models for text classification tasks, including automatic model detection and no code changes necessary. - **Dataset Processing**: Provides a function `movie_reviews_to_file` for converting text data into a format suitable for training transformers. - **Compatibility**: Tested with 67 pretrained models and compatible with various transformer architectures. - **Customization**: The notebook can be easily adapted for different datasets and fine-tuning tasks. - **Prerequisites**: Basic Python knowledge and familiarity with PyTorch and Transformers library. ### Tech Stack: Jupyter Notebook ### Why it Matters: This repository offers a practical and flexible approach to fine-tuning and pre-training transformer models, making it ideal for developers looking to enhance their machine learning skills.