Projects

### Overview: This project serves as a reference for implementing a complete MLOps pipeline, covering MLflow Model Registry, batch and real-time serving with FastAPI, drift monitoring using PSI and KS, and Airflow for scheduled retraining with a promotion gate. The synthetic dataset used is a seeded housing-style regression dataset. ### Key features: - MLflow Model Registry - Batch and real-time serving with FastAPI - Drift monitoring using PSI and KS - Prometheus alerts and Grafana dashboard - Airflow-scheduled retraining with a promotion gate ### Tech stack: Python ### Why it matters: This project provides a comprehensive guide for developers interested in implementing MLOps, showcasing best practices for model deployment, monitoring, and retraining.

### Overview: This project provides a sample implementation of a machine learning model serving on Kubernetes using Docker and Helm charts. The model is hosted in a Kubernetes deployment, and the necessary manifests and Helm chart are included to deploy the application. ### Key Features: - **Kubernetes Deployment**: The model is deployed using a Kubernetes deployment manifest. - **Docker Image**: The model is built and deployed using a Docker image. - **Helm Chart**: A Helm chart is used to automate the deployment and management of the application. ### Tech Stack: - **Go Template**: The project uses Go Template for the Kubernetes deployment manifest. ### Why it Matters: This project serves as a practical example for developers to understand how to deploy machine learning models on Kubernetes using Docker and Helm charts. It demonstrates the integration of machine learning with container orchestration tools, making it valuable for developers looking to deploy ML models in production environments.

### Overview: This field guide provides a comprehensive overview of Inference Engineering, covering the path from product constraints and model mechanics through hardware, runtimes, optimization techniques, modalities, and production operations. It includes 24 plain HTML pages in both English and Traditional Chinese, making it accessible to developers worldwide. ### Key Features: - **Bilingual Content**: The guide is available in both English and Traditional Chinese, catering to a global audience. - **Structured Pages**: The site is organized into 24 pages, each focusing on a specific topic, with English and Traditional Chinese versions. - **Canonical URLs**: The generated static site ensures that canonical URLs, hreflang URLs, and `sitemap.xml` are correctly set, ensuring proper indexing and SEO. - **Responsive Design**: The WCAG 2 AA harness ensures the site is accessible and usable on various devices and browsers. ### Tech Stack: The guide is built using HTML, CSS, and JavaScript, with a focus on low-contrast Neo-Brutalist design principles. ### Why It Matters: This field guide is essential for developers looking to understand and implement Inference Engineering, a critical aspect of modern machine learning and artificial intelligence. It provides a clear and accessible path from theoretical concepts to practical implementation, ensuring that readers can effectively apply the knowledge to their projects.

### Overview: llmctl is a tool designed to manage local model services on a GPU that cannot hold them all at once. It simplifies the process of switching between services safely and efficiently. ### Key features: - **Mutex Switching**: `use` stops conflicting services and starts the target. `heavy` services are globally exclusive; `light` ones may coexist. - **Safe Teardown**: stop script → port-release wait → SIGTERM → SIGKILL → VRAM-drain confirmation via `nvidia-smi`. Any failure aborts the switch instead of stacking one heavy service on top of another. - **Busy Guard**: an optional per-service probe (any shell command; exit 0 = working) prevents stopping a service mid-task, unless `--force` is given. - **Health Probing**: a service counts as started only when its health endpoint returns 200. - **Speed Benchmark**: `bench` measures decode and prefill tok/s. - **Power Governance**: optional per-service power limits applied automatically on switch (calibration SOP in `docs/SOP_POWER.md`). ### Tech stack: Python ### Why it matters: llmctl simplifies the process of managing model services on a GPU, ensuring safe and efficient switching without the risk of conflicting services.

### Overview: Deploy ML model updates to a fraction of traffic, watch Prometheus metrics for both versions, and automatically promote or roll back — no human required. FastAPI serving + Redis traffic split + Prometheus health checks + Streamlit dashboard. Part of a 3-project MLOps platform. ### Key features: - Traffic split using Redis - Continuous monitoring with Prometheus - Automatic promotion or rollback based on model performance - Streamlit dashboard for live traffic split and model comparison ### Tech stack: FastAPI + Uvicorn + Redis + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit +

### Overview: This project is a production-oriented medical video inference pipeline designed for surgical phase recognition. Built with PyTorch, MONAI, ONNX, TensorRT, and NVIDIA Triton Inference Server, it addresses the challenges of real-time medical video systems, including deterministic preprocessing, video decoding and sampling, temporal stability, model export, accelerator-specific optimization, model serving, concurrent request handling, latency and throughput measurement, and reproducible deployment. ### Key Features: - **Model Training**: PyTorch and MONAI training for surgical phase recognition. - **Model Export**: ONNX export for model portability. - **Optimization**: TensorRT optimization for faster inference. - **Deployment**: NVIDIA Triton Inference Server for concurrent request handling and high throughput. ### Tech Stack: - Python - PyTorch - MONAI - ONNX - TensorRT - NVIDIA Triton Inference Server ### Why It Matters: Real-time medical video systems require more than just model accuracy. They must also address deterministic preprocessing, video decoding and sampling, temporal stability, model export, accelerator-specific optimization, model serving, concurrent request handling, latency and throughput measurement, and reproducible deployment. This project treats surgical phase recognition as an inference systems problem, not just a classification problem, making it suitable for real-world applications in the medical field. ### Repository: The repository includes training, evaluation, model export, TensorRT deployment tooling, Triton serving configuration, streaming inference, and concurrent performance benchmarking. It is designed for Cholec80 laparoscopic cholecystectomy task

Overview: StreamInfer is a local inference-serving project designed for exploring adaptive batching, backpressure, model hot-swap, and lightweight service metrics. It provides WebSocket request/response, HTTP prediction, and adaptive batching features. Key features: - WebSocket request/response - HTTP prediction - Adaptive batching - Backpressure - Model hot-swap - Metrics - Load-test reports - Benchmark sweeps - Benchmark gates Tech stack: Python Why it matters: StreamInfer simplifies the process of inspecting serving patterns by providing a small, Python-based inference service, making it easier to understand latency and throughput tradeoffs without the complexity of larger frameworks. Repository: GoparapukethaN/streaminfer Primary language: Python GitHub stars: 1 GitHub description: Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.

### Overview: This is a production-ready ML inference service template built on BentoML, designed for easy deployment and monitoring. ### Key features: - **BentoML service** with `/health`, `/ready`, `/predict`, `/metrics`, `/version` endpoints - **Multi-stage Dockerfile** — slim runtime image, non-root user, health check - **docker-compose** with optional Prometheus and Grafana services - **GitHub Actions CI** — lint (ruff), test (pytest), Docker build, BentoML build - **Prometheus-compatible metrics** — request latency histogram, prediction count, error count - **Grafana dashboard** — pre-built JSON for latency, throughput, and error rate panels - **Structured JSON logging** for production observability - Supports **PyTorch** and **TensorFlow** model frameworks ### Tech stack: Python ### Why it matters: This template provides a comprehensive solution for deploying ML models, ensuring high-quality production services with built-in monitoring and observability features.

### Overview: Leafline is a crop-disease inference service running on Kubernetes, designed to efficiently utilize expensive GPUs without causing delays for farmers on slow connections. It includes three tiers: CPU, GPU, and an on-device browser tier, with a focus on GPU scheduling and autoscaling. ### Key features: - GPU scheduling and autoscaling - Three-tiered architecture (CPU, GPU, and browser) - M1 and M2 cluster setup - Models and calibration - Router and ladder implementation - Balloon preemption and GPU sharing - Real GPU node and cold start measurement ### Tech stack: Python ### Why it matters: Leafline aims to optimize the use of expensive GPU resources by ensuring they are utilized efficiently without causing delays for farmers on slow connections. The project's focus on GPU scheduling and autoscaling, combined with the measurement of the ladder's performance on field photographs, highlights its importance in improving the reliability and performance of crop-disease inference services.

Overview: jrjModelRegistry is a simple and secure model registry designed for storing, versioning, and serving machine learning models using S3 and MongoDB. Key features: - 📦 Store models in S3-compatible storage with ZIP encryption - 🗃️ Track metadata in MongoDB - ⚡ Dynamically serve models using built-in FastAPI routes and dashboard - ✅ Works with any Python object (models, transformers, even DataFrames) Tech stack: Python Why it matters: This library simplifies the process of managing and serving machine learning models, making it ideal for both small projects and production environments.

Overview: This project provides a PyTorch model training and inference service accessible via a FastAPI REST API. It streamlines the process of deploying and utilizing machine learning models. Key features: - Model training and inference through PyTorch - RESTful API for easy integration with other systems - Dockerized environment for consistent deployment across different environments Tech stack: Python, PyTorch, FastAPI, Docker Why it matters: This service simplifies the deployment and management of machine learning models, making it easier to integrate with existing infrastructure and applications.

Overview: A pure C# implementation of Gemini 2.5 Pro Preview for CPU inference, designed for developers. Key features: - Pure C# implementation - Gemini 2.5 Pro Preview ported to C# - Int8 inference capabilities Tech stack: C# Why it matters: This project offers a lightweight, efficient solution for developers looking to integrate AI models into their applications without the need for complex dependencies or additional scripting languages.

### Overview: TextOps is an end-to-end ML platform designed for reproducible experimentation, automated model governance, versioned artifact management, and decoupled model serving. The project uses a small text-classification problem to engineer infrastructure around an ML system, focusing on the surrounding system rather than model complexity. ### Key Features: - **Reproducible Pipeline**: Tracks and manages the entire lifecycle from versioned data to inference. - **Automated Model Governance**: Promotes models based on both aggregate performance and specific class metrics. - **Versioned Artifact Management**: Ensures that artifacts are stored and versioned correctly. - **Decoupled Model Serving**: Allows for safe introduction and reuse of new model implementations without restructuring the entire pipeline. ### Tech Stack: Python ### Why It Matters: TextOps explores the infrastructure surrounding a small but imperfect classification problem, addressing questions in a production setting that go beyond just model performance. It focuses on the surrounding system, making it useful for understanding and improving the overall ML ecosystem in a production environment. ### Repository: JohnStephens1/TextOps ### Primary Language: Python ### GitHub Stars: 1 ### GitHub Description: End-to-end ML platform for reproducible experimentation, automated model governance, versioned artifact management, and decoupled model serving.

Overview: Platform Genesis Cortex is a cutting-edge AI platform designed for developers, offering a comprehensive suite of tools for building, deploying, and managing machine learning models. Key features: - Model Serving - Model Training - Model Deployment - Model Monitoring Tech stack: Python, TensorFlow, PyTorch, Docker, Kubernetes Why it matters: Platform Genesis Cortex simplifies the process of developing and deploying AI models, making it easier for developers to integrate AI into their projects without the need for extensive technical expertise.

### Overview: InferHub is a production-grade AI inference platform featuring model registry, deployment orchestration, autoscaling, GPU scheduling, observability, and Kubernetes-ready architecture. ### Key features: - Model registry and versioning - Real deployments with health checks and autoscaling - Simulated GPU scheduling algorithms - Kubernetes-ready architecture - Structured JSON logging and rate limiting - Prometheus metrics and Grafana dashboards - Real Kubernetes manifests and Helm chart ### Tech stack: Python ### Why it matters: InferHub provides a robust and scalable solution for deploying and managing AI models, ensuring high availability and performance through real-world implementation and Kubernetes-ready architecture.

Overview: This repository contains the deployment recipes, engine patches, and measurements for serving **DeepSeek-V4.1-Flash-1017** in its uncensored form on NVIDIA DGX Spark (GB10) nodes. The stacks include two-node TP2 with TensorFold's Zig engine over an EXL3 2.9 bpw repack, and four-node TP4 with either the same Zig engine or SGLang over the native FP8/FP4 checkpoint. Key features: - TP2 (2 nodes) with TensorFold's Zig engine over an EXL3 2.9 bpw repack - TP4 (4 nodes) with either the same Zig engine or SGLang over the native FP8/FP4 checkpoint Tech stack: Shell Why it matters: The repository details the deployment of four working stacks on DGX Spark nodes, including TP2 with TensorFold's Zig engine and TP4 with either the same engine or SGLang over the native FP8/FP4 checkpoint, providing insights into the performance and efficiency of these configurations. Repository: Keenni/DeepSeek-V4.1-Flash-1017-Flash-1017-UNCENSORED-DGX-Sparks GitHub stars: 1 GitHub description: Switchless TP4/TP2 for DeepSeek-V4.1-Flash-1017 (uncensored) on DGX Spark (GB10). TensorFold Zig TP4 on a switchless 200G fibre ring (A

Overview: BentoML is a high-performance model serving framework that simplifies the deployment and management of machine learning models. It provides scripts and configurations to streamline the process and accelerate deployment. Key features: - Model deployment tools - Service creation methods - Visualization techniques - Integration with various machine learning frameworks - Advanced features for optimized deployments Tech stack: Python Why it matters: BentoML streamlines the deployment of machine learning models, making it easier to manage and scale model services in production environments.

Overview: This project combines ONNX inference with Rust and WebAssembly, providing a validated Triton gateway for efficient inference and monitoring. Key features: - ONNX inference compiled to WebAssembly for the browser - Triton v2 HTTP protocol via a stdlib-only client - Input safety checks, finite-number checks, and element budgets - Deterministic first-dimension batching before calls to Triton - Prometheus metrics and Grafana integration - 100% coverage of the gateway core Tech stack: Python, Rust, WebAssembly, Triton, Prometheus, Grafana Why it matters: This project offers a robust and efficient solution for deploying ONNX inference in production environments, ensuring reliability and performance. Repository: MrKyaw/Onnx_Rust_Web_Assembly_Triton_Promethus_Grafana GitHub stars: 1 GitHub description: ONNX inference in Rust/WebAssembly plus a validated Triton gateway: batching, readiness, Prometheus metrics, Grafana, Docker Compose. 83 Python tests, 4 Rust tests, 100% gateway coverage.

Overview: stateswap is a project that leverages RWKV-7 as a base model to train N personas with initial states (S0) and hot-swaps them at serving time. This approach allows for the manipulation of style, persona, and task modes without altering the frozen weights, achieving significant improvements in performance and efficiency. Key features: - Train each persona into a few-MB initial state (S0) - Hot-swap the initial state at serving time - Use gradient descent to bake a "virtual prefix" into the initial state - Achieve a persona hot-swap latency of 2.6–4.4 ms - Utilize a 1.5B model with 1.59 MB of parameters for persona hot-swapping Tech stack: Python Why it matters: This project demonstrates the potential of using RWKV-7 as a base model to train personas, achieving significant improvements in performance and efficiency without altering the frozen weights, making it a valuable contribution to the field of language models. Repository: Nice0o0/stateswap Primary language: Python GitHub stars: 1 GitHub description: 🐱 One RWKV-7 base, N personas: train each persona into a few-MB initial state (S0) and hot-swap it at serving time. API, WebUI, plus reproducible S0 research: phase transitions, persona anatomy, inheritance.