Projects

### Overview: This repository provides a guide for running distilled and quantized versions of the DeepSeek-R1 model on Amazon EC2, optimizing inference performance without significantly compromising accuracy. ### Key Features: - Distilled models - Quantized models - Dynamic quantization for faster token/sec without sacrificing accuracy ### Tech Stack: Python ### Why It Matters: This approach allows running the model on smaller instances, reducing costs and making it more feasible to deploy on a wide range of hardware configurations.

Overview: aihappey-ai is a stateless, open-source AI gateway that exposes a unified endpoint for accessing a vast collection of models, media, skills, agents, and more from multiple providers. Key features: - Access to 150k+ models and provider-native capabilities - Stateless design, no smart routing or admin portal - Managed agents and skills across 150k+ models - Managed by Vercel UI streaming, no hidden logic maze - Managed by managed agents and skills across 150k+ models Tech stack: C# Why it matters: aihappey-ai provides a convenient and efficient way to access a wide range of AI models and capabilities, making it easier for developers to integrate AI into their projects without the need for complex routing or management.

### Overview: This project is a production-inspired LLM inference engine built from scratch in PyTorch. It implements KV caching, continuous batching, paged memory management, and async serving, with benchmarks at each optimization layer. Key features include a 2.22x decode speedup, 118x throughput, 8.7x memory reduction, and 11.1x system throughput. ### Key Features: - **KV Cache:** Eliminates redundant attention recomputation, resulting in a 2.19x speedup. - **Batched Inference:** Saturates GPU compute, offering a 118x throughput improvement. - **Continuous Batching:** Dynamic scheduling to eliminate idle GPU slots, boosting system throughput by 11.1x. - **Paged KV Cache:** Reduces memory usage by 8.7x compared to contiguous pre-allocation. - **Async Serving:** FastAPI server with backpressure and routing for efficient inference. ### Tech Stack: - **Python** - The primary language used for development and optimization. ### Why It Matters: Understanding the design decisions behind production LLM serving systems, such as vLLM, TGI, and TensorRT-LLM, is crucial. This project isolates and benchmarks each optimization layer, providing a comprehensive understanding of what matters and by how much in LLM inference. ### Repository: - **Repository:** achi9629/llm-inference-engine - **GitHub Stars:** 6 - **GitHub Description:** A from scratch LLM inference

Overview: This project fine-tunes a small language model to extract structured JSON from casual expense messages, aiming to measure both the task gain and the retention of general ability. Key features: - Task gain on structured JSON extraction - Retention of general ability - Baseline measurement before training - Forgetfulness check with two different scoring methods Tech stack: Python Why it matters: This project provides a clear demonstration of the benefits and potential pitfalls of LoRA fine-tuning, specifically focusing on the catastrophic forgetting issue that is often overlooked in most projects.

Overview: This hands-on aiDAPTIV training course is designed to teach developers, researchers, and students how to fine-tune large language models (LLMs) locally using the aiDAPTIV technology. The course includes detailed lesson materials, datasets, code examples, and project files to provide a comprehensive learning experience. Key features: - Fine-tuning LLMs on your own hardware or a provided remote AI Training PC (AITPC) - Optimizing GPU VRAM usage with aiDAPTIV cache memory - Using both GUI and CLI workflows for training and inference - Applying fine-tuning to real-world datasets such as company handbooks or public figure transcripts Tech stack: aiDAPTIV technology Why it matters: This course offers practical skills to run on-prem AI cost-efficiently and securely, making it ideal for developers, researchers, and students looking to enhance their AI capabilities.

### Overview: Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads. ### Key features: - 🚀 AWS & IBM Cloud support - ⚡ Automated infrastructure: MachineSets, autoscaling, ClusterAutoscaler - 🧩 Operators & platform services: NVIDIA GPU Operator, Node Feature Discovery, Descheduler, KEDA, Gateway API, Kuadrant, Authorino, cert-manager - 🕵️ Observability: NetObserv, LokiStack, Grafana dashboards (WIP) - 🔬 Example manifests for KServe LLMInferenceService & KV-cache routing - 🧪 Precise-prefix cache-aware experiments - 🔄 GitOps App-of-Apps via Argo CD (WIP) ### Tech stack: Jupyter Notebook ### Why it matters: This lab provides a streamlined and reproducible environment for evaluating large language models, ensuring consistent and reliable results across different environments and configurations.

Overview: PulmoScope is a comprehensive AI system designed for pulmonary lesion detection and localization on chest CT scans. It integrates three independent models to provide a multi-stage decision pipeline, ensuring that the user is informed when the model's confidence is not reliable. Key features: - **Multi-model decision pipeline.** Combines three models with different roles to provide a multi-stage decision. - **Disagreement-aware output.** When the classifier and segmenter disagree, the system returns `uncertain` and indicates which model flagged the slice. - **Risk-adaptive sensitivity.** Adjusts the classifier threshold based on the questionnaire risk score. Tech stack: Python, TensorFlow, FastAPI, scikit-learn, OpenCV, MIT License, Educational Use Only. Why it matters: PulmoScope addresses the issue of model failure by providing a clear indication when a model's confidence is not reliable, ensuring that the user is informed and can make an informed decision.

Overview: AIOpsServing 是一个开源项目，旨在为算法为中心的研发、测试、评估和服务提供轻量级流程支持。 Key features: - 支持多种计算后端，包括阿里云函数计算（FC）、大数据平台 ODPS、分布式平台 Ray 和容器 Docker。 - 负责算法迭代、归档、发布和回滚的生命周期管理。 - 提供离线和在线算法性能评估。 - 计划整合异常检测、行为画像等模型仓库。 Tech stack: Python Why it matters: AIOpsServing 为算法中心的研发、测试、评估和服务提供了一个轻量级的流程支持，简化了模型的发布、生命周期管理和性能评估，提高了开发效率和质量。

Overview: AloLab is an iterative optimization tool designed to improve the accuracy and structure of AI agent models, specifically focusing on small language models. It automates the process of refining prompts to ensure models produce reliable and structured output. Key features: - Automatically optimizes prompts for accuracy and structure. - Uses real data to detect and correct errors in the model. - Improves the prompt using an optimizer LLM. - Iterates until performance converges. Tech stack: Python Why it matters: AloLab streamlines the optimization process for AI models, ensuring they become more accurate and structured without the need for fine-tuning, thus reducing fine-tuning costs and improving the overall quality of the output. Repository: alomana-lab/alolab GitHub stars: 6 GitHub description: Iterative optimization for AI agents. Turn models into accurate, structured, production-ready systems.

Overview: This project demonstrates how to stream LLM responses using AWS API Gateway Websockets and Lambda. Key features: - AWS API Gateway Websockets - Lambda function to process and stream LLM responses - Integration with AWS SNS for message delivery Tech stack: Node.js Why it matters: This project provides a practical example for developers looking to integrate LLMs with AWS services for real-time communication.

### Overview: This project focuses on Word Sense Disambiguation (WSD) for the Word-in-Context (WiC) disambiguation task. Two BERT-based models are proposed: a feature-based approach and a fine-tuning approach. The models are trained using GlossBERT, an adaptation of the BERT model for WSD tasks. The project aims to improve disambiguation accuracy by experimenting with different approaches and leveraging pre-trained models. ### Key features: - Two BERT-based models for WiC disambiguation - Feature-based and fine-tuning approaches - GlossBERT adaptation for WSD tasks - Experimenting with different WSD models and approaches ### Tech stack: Python ### Why it matters: This project contributes to the field of natural language processing by improving the accuracy of Word Sense Disambiguation (WSD) tasks, specifically for the Word-in-Context (WiC) scenario. By experimenting with different models and approaches, it aims to enhance the current state-of-the-art in WSD for WiC tasks, potentially leading to better applications in text analysis and information retrieval.

### Overview: This project implements a Universal Fine-Tuning Pipeline for the NVIDIA NitroGen model, designed to work with any gameplay dataset generated via the NitroGen/BizHawk toolchain. It supports the model's Diffusion Transformer architecture, enabling efficient fine-tuning on consumer hardware. ### Key Features: - **Universal Architecture**: Automatically handles datasets from any console (NES, SNES, PS3, Xbox) using a dynamic gamepad serialization strategy. - **DiT Support**: Specifically engineered for NitroGen's Diffusion Transformer architecture, predicting a time-horizon of actions from a single frame. - **Continuous Action Space**: Maps inputs to a rigorous 21-dimensional continuous vector space (Analogs + Buttons). - **LoRA Integration**: Uses Low-Rank Adaptation (PEFT) for efficient fine-tuning on consumer hardware. ### Tech Stack: Python ### Why it Matters: This project is essential for developers looking to integrate artificial intelligence into retro gaming, making it easier to create AI agents that can learn from gameplay data. The universal architecture and DiT support make it adaptable to various consoles, while the continuous action space ensures a robust learning environment.

Overview: This project profile details the performance of a 26B mixture-of-experts model running on a CPU, achieving a token throughput of 124 tokens per second (tokens/sec) with no GPU. The model was built from scratch using only the CPU, achieving a speed of 40 tokens/sec for a single stream and 124 tokens/sec for serving multiple requests simultaneously. Key features: - Running Gemma-4 26B at 124 tokens/sec on a CPU with no GPU - Lossless token throughput, meaning the output tokens are exactly the same as those produced by a regular Q4_0 model - Speculative decoding unlocks significant performance gains - Running 3 out of 8 experts to improve perplexity on raw text Tech stack: Shell Why it matters: This project showcases the potential of self-hosting large language models without relying on GPUs, demonstrating the feasibility of achieving high performance on a regular desktop setup.

**Overview:** This project provides a framework for evaluating and comparing different customization techniques for foundation models, including RAG, fine-tuning, and a hybrid approach combining both methods. **Key features:** - Evaluates performance, cost, and latency across RAG, fine-tuning, and hybrid approaches. - Utilizes a synthetic dataset for experimentation. - Supports customization of the environment settings, knowledge base configuration, data source configuration, RAG, fine-tuning, and evaluation configurations. **Tech stack:** Python **Why it matters:** This framework helps developers determine the most effective approach for specific use cases by analyzing performance, cost, and latency across different customization techniques, ultimately aiding in the optimization of foundation models for business needs.

Overview: This project is a PyTorch implementation for fine-tuning the Segment Anything Model (SAM) on medical images, specifically designed for gastrointestinal endoscopy image segmentation. It includes comprehensive data preprocessing, multi-GPU distributed training, and model evaluation tools. Key features: - Fine-tuning SAM on custom medical datasets - Single-GPU and multi-GPU distributed training - Comprehensive data preprocessing and augmentation - Model evaluation and visualization tools Tech stack: Python Why it matters: This project provides a complete pipeline for fine-tuning SAM on medical images, making it a valuable resource for researchers and developers working in medical image segmentation.

### Overview A reasoning critic evaluates the quality of reasoning behind an answer, not just its correctness. This project fine-tunes open-source LLMs, such as Qwen and Mistral, into this role using a corpus of hand-written critique examples and external instruction data. The models are evaluated against a held-out benchmark to assess their performance. ### Key Features - Fine-tuning of open-source LLMs into reasoning-process evaluators. - Evaluates the quality of reasoning behind an answer. - Uses a corpus of hand-written critique examples mixed with external instruction data. - Evaluates against a held-out benchmark. - Two models were trained on identical data and compared head-to-head. ### Tech Stack Python ### Why it Matters Fine-tuning pulled both models toward the capability level embedded in the training data, from opposite directions. Qwen's trap detection came down (0.975 → 0.900, mild forgetting) while Mistral's came up (0.800 → 0.850, a gain): they converged toward the corpus. Format proved base-dependent: Mistral followed the output contract from the system prompt alone (base verdict rate 1.000), while Qwen needed the fine-tune to learn it (0.525 → 0.950). At n=40 a 0.05 delta is two questions, so the convergence is consistent but not statistically proven — and because it rests on the trap-detection metric, it is now best read as suggestive rather than

Overview: A toolkit designed to explore and build with NVIDIA NIMs, including LLMs, 3D generation, and speech recognition. Key features: - Local exploration of NVIDIA NIMs - Support for multiple NIM services including LLMs, image generation, 3D generation, and speech recognition Tech stack: FastAPI, Celery, Nuxt.js, Vue 3, Redis, Prometheus, Grafana, Flower Why it matters: Simplifies the process of experimenting with NVIDIA NIMs and enhances the development experience with enhanced functionality and monitoring tools. Repository: briancaffey/nvidia-nim-kit GitHub stars: 6 GitHub description: A toolkit to explore and build with NVIDIA NIMs README excerpt: 

### Overview DINOv2-LoRA Segmentation is a remote sensing semantic segmentation framework that leverages pretrained DINOv2 visual representations and parameter-efficient fine-tuning through Low-Rank Adaptation (LoRA). ### Key Features - **DINOv2 Foundation Model**: Pretrained visual representations for efficient transfer learning. - **CNN-Based Decoder**: Integrates CNN-based architectures for effective semantic segmentation. - **Decoder Architecture Benchmarking**: Evaluates multiple decoder architectures for optimal performance. - **Linear Probing**: Trains the decoder only, reducing the number of parameters to be fine-tuned. - **Full Fine-Tuning**: Fine-tunes the entire network, updating all parameters for better performance. - **LoRA-Based Adaptation**: Low-Rank Adaptation (LoRA) for efficient parameter-efficient transfer learning. - **Parameter-Efficient Transfer Learning**: Reduces the number of trainable parameters while preserving knowledge from the pretrained model. - **Remote Sensing Segmentation**: Land-cover semantic segmentation on remote sensing imagery. ### Why It Matters This project explores and benchmarks various decoder architectures and adaptation strategies, including linear probing, full fine-tuning, and LoRA-based transfer learning, to improve remote sensing semantic segmentation on the LOVEDA dataset.

### Overview: Plumb is a cutting-edge project that fine-tunes a multimodal Qwen3.8-27B model to excel at AIA G702/G703 pay-application review. The project focuses on extracting and verifying structured data from invoices, lien waivers, and change orders, ensuring every math error and discrepancy is corrected. This approach aims to improve accuracy and speed, making it a valuable tool for contractors and project managers. ### Key Features: - Fine-tuning a multimodal Qwen3.8-27B model on Modal/GCP free credits. - Extraction of SOV lines and G702 summary totals. - Cross-checking with contract SOV, prior approved apps, and change orders. - Emitting PASS or line-level exceptions with corrected figures. - Verifying arithmetic and cross-referencing with entitlements and waivers. - Emitting dense, deterministic RLVR rewards with every check recomputable. ### Tech Stack: Python ### Why It Matters: - The exact recipe that beat a frontier model at 1.7B in ReconForge, now 16× bigger. - Pure document-to-structured-data + deterministic reconciliation. - Real, dollar-denominated pain: 15% of pay apps are rejected for math errors. - Competitive gap: Siteline/Rabbet/Procore digitize but don't adjudicate against contract entitlements. - Perfect-level target: Every check is recomputable arithmetic. - The verifier gives dense, deterministic RLVR rewards. ### Directory Layout:

### Overview: A minimal language model pipeline designed to run on a laptop, from raw text to a tokenizer, pretraining, inference, fine-tuning, and reinforcement learning techniques. ### Key features: - Tokenizer - Model - Pretraining - Inference - Fine-tuning - Reinforcement Learning for Human Preferences (RLHF) and Verifiable Rewards (RLVR) ### Tech stack: JavaScript ### Why it matters: The pipeline is designed to be simple and accessible, making it suitable for developers and researchers alike. It includes all stages necessary for building and fine-tuning a language model, from raw text to a trained model that can be used with other tools. ### Repository: GitHub: changkun/minllm ### GitHub stars: 6 ### GitHub description: A minimal pipeline for making a language model, small enough to run on a laptop and written to be read. It goes from raw text to a tokenizer, a pretrained model, an inference engine, a fine-tuned chat model, and a model post-trained with reinforcement learning, first from human preferences (RLHF) and then from verifiable rewards (RLVR), and ends with that model in the format that other tools run. ### Stages: - **Tokenizer**: Byte-level BPE: training, encoding, decoding, special tokens - **Model**: Decoder-only transformer with RMSNorm, RoPE, SwiGLU and grouped-query attention - **Pretraining**: Data pipeline, AdamW, learning rate schedule, checkpoints: a model that writes stories