AI Trending

Project directory

Projects

Open-source AI projects tracked by the leaderboard, ordered by GitHub stars.

HTML 1 stars

Data-System-School/inference-engineering-distilled

A bilingual field guide to Inference Engineering: the path from product constraints and model mechanics through hardware, runtimes, optimization techniques, modalities, and production operation.

### Overview: This field guide provides a comprehensive overview of Inference Engineering, covering the path from product constraints and model mechanics through hardware, runtimes, optimization techniques, modalities, and production operations. It includes 24 plain HTML pages in both English and Traditional Chinese, making it accessible to developers worldwide. ### Key Features: - **Bilingual Content**: The guide is available in both English and Traditional Chinese, catering to a global audience. - **Structured Pages**: The site is organized into 24 pages, each focusing on a specific topic, with English and Traditional Chinese versions. - **Canonical URLs**: The generated static site ensures that canonical URLs, hreflang URLs, and `sitemap.xml` are correctly set, ensuring proper indexing and SEO. - **Responsive Design**: The WCAG 2 AA harness ensures the site is accessible and usable on various devices and browsers. ### Tech Stack: The guide is built using HTML, CSS, and JavaScript, with a focus on low-contrast Neo-Brutalist design principles. ### Why It Matters: This field guide is essential for developers looking to understand and implement Inference Engineering, a critical aspect of modern machine learning and artificial intelligence. It provides a clear and accessible path from theoretical concepts to practical implementation, ensuring that readers can effectively apply the knowledge to their projects.
Python 1 stars

Dqz00116/llmctl

llmctl manages local model services on a GPU that can't hold them all at once. One YAML registry describes your services

### Overview: llmctl is a tool designed to manage local model services on a GPU that cannot hold them all at once. It simplifies the process of switching between services safely and efficiently. ### Key features: - **Mutex Switching**: `use` stops conflicting services and starts the target. `heavy` services are globally exclusive; `light` ones may coexist. - **Safe Teardown**: stop script → port-release wait → SIGTERM → SIGKILL → VRAM-drain confirmation via `nvidia-smi`. Any failure aborts the switch instead of stacking one heavy service on top of another. - **Busy Guard**: an optional per-service probe (any shell command; exit 0 = working) prevents stopping a service mid-task, unless `--force` is given. - **Health Probing**: a service counts as started only when its health endpoint returns 200. - **Speed Benchmark**: `bench` measures decode and prefill tok/s. - **Power Governance**: optional per-service power limits applied automatically on switch (calibration SOP in `docs/SOP_POWER.md`). ### Tech stack: Python ### Why it matters: llmctl simplifies the process of managing model services on a GPU, ensuring safe and efficient switching without the risk of conflicting services.
Python 1 stars

Emart29/ml-canary-deploy

Deploy ML model updates to a fraction of traffic, watch Prometheus metrics for both versions, and automatically promote or roll back - no human required. FastAPI serving + Redis traffic split + Prometheus health checks + Streamlit dashboard. Part of a 3-project MLOps platform.

### Overview: Deploy ML model updates to a fraction of traffic, watch Prometheus metrics for both versions, and automatically promote or roll back — no human required. FastAPI serving + Redis traffic split + Prometheus health checks + Streamlit dashboard. Part of a 3-project MLOps platform. ### Key features: - Traffic split using Redis - Continuous monitoring with Prometheus - Automatic promotion or rollback based on model performance - Streamlit dashboard for live traffic split and model comparison ### Tech stack: FastAPI + Uvicorn + Redis + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit + Altair + Click + Rich + MinIO + Docker + Docker Compose + PostgreSQL + SQLAlchemy + Prometheus + Streamlit +
Python 1 stars

Gopal3746/real-time-surgical-inference

Real-time surgical phase recognition pipeline — PyTorch/MONAI training, ONNX export, TensorRT optimization, and NVIDIA Triton serving with concurrency benchmarking on Cholec80.

### Overview: This project is a production-oriented medical video inference pipeline designed for surgical phase recognition. Built with PyTorch, MONAI, ONNX, TensorRT, and NVIDIA Triton Inference Server, it addresses the challenges of real-time medical video systems, including deterministic preprocessing, video decoding and sampling, temporal stability, model export, accelerator-specific optimization, model serving, concurrent request handling, latency and throughput measurement, and reproducible deployment. ### Key Features: - **Model Training**: PyTorch and MONAI training for surgical phase recognition. - **Model Export**: ONNX export for model portability. - **Optimization**: TensorRT optimization for faster inference. - **Deployment**: NVIDIA Triton Inference Server for concurrent request handling and high throughput. ### Tech Stack: - Python - PyTorch - MONAI - ONNX - TensorRT - NVIDIA Triton Inference Server ### Why It Matters: Real-time medical video systems require more than just model accuracy. They must also address deterministic preprocessing, video decoding and sampling, temporal stability, model export, accelerator-specific optimization, model serving, concurrent request handling, latency and throughput measurement, and reproducible deployment. This project treats surgical phase recognition as an inference systems problem, not just a classification problem, making it suitable for real-world applications in the medical field. ### Repository: The repository includes training, evaluation, model export, TensorRT deployment tooling, Triton serving configuration, streaming inference, and concurrent performance benchmarking. It is designed for Cholec80 laparoscopic cholecystectomy task
Python 1 stars

GoparapukethaN/streaminfer

Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.

Overview: StreamInfer is a local inference-serving project designed for exploring adaptive batching, backpressure, model hot-swap, and lightweight service metrics. It provides WebSocket request/response, HTTP prediction, and adaptive batching features. Key features: - WebSocket request/response - HTTP prediction - Adaptive batching - Backpressure - Model hot-swap - Metrics - Load-test reports - Benchmark sweeps - Benchmark gates Tech stack: Python Why it matters: StreamInfer simplifies the process of inspecting serving patterns by providing a small, Python-based inference service, making it easier to understand latency and throughput tradeoffs without the complexity of larger frameworks. Repository: GoparapukethaN/streaminfer Primary language: Python GitHub stars: 1 GitHub description: Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.
Python 1 stars

IbLahlou/inference-pipeline-template

Reusable production ML inference template: BentoML service, Dockerized API, CI/CD, health checks, metrics, and monitoring hooks.

### Overview: This is a production-ready ML inference service template built on BentoML, designed for easy deployment and monitoring. ### Key features: - **BentoML service** with `/health`, `/ready`, `/predict`, `/metrics`, `/version` endpoints - **Multi-stage Dockerfile** — slim runtime image, non-root user, health check - **docker-compose** with optional Prometheus and Grafana services - **GitHub Actions CI** — lint (ruff), test (pytest), Docker build, BentoML build - **Prometheus-compatible metrics** — request latency histogram, prediction count, error count - **Grafana dashboard** — pre-built JSON for latency, throughput, and error rate panels - **Structured JSON logging** for production observability - Supports **PyTorch** and **TensorFlow** model frameworks ### Tech stack: Python ### Why it matters: This template provides a comprehensive solution for deploying ML models, ensuring high-quality production services with built-in monitoring and observability features.
Python 1 stars

IronNathanAlvares/leafline

Crop-disease inference on Kubernetes: a three-tier ladder, GPU scheduling, KEDA autoscaling on queue depth rather than CPU, and warm headroom for a cold start you cannot scale away. M1 and M2 built and measured; the on-device tier runs in the browser. The measurement says the ladder fails on field photos, and the README says so.

### Overview: Leafline is a crop-disease inference service running on Kubernetes, designed to efficiently utilize expensive GPUs without causing delays for farmers on slow connections. It includes three tiers: CPU, GPU, and an on-device browser tier, with a focus on GPU scheduling and autoscaling. ### Key features: - GPU scheduling and autoscaling - Three-tiered architecture (CPU, GPU, and browser) - M1 and M2 cluster setup - Models and calibration - Router and ladder implementation - Balloon preemption and GPU sharing - Real GPU node and cold start measurement ### Tech stack: Python ### Why it matters: Leafline aims to optimize the use of expensive GPU resources by ensuring they are utilized efficiently without causing delays for farmers on slow connections. The project's focus on GPU scheduling and autoscaling, combined with the measurement of the ladder's performance on field photographs, highlights its importance in improving the reliability and performance of crop-disease inference services.
Python 1 stars

JRJSolutions/jrjModelRegistry

Simple and secure model registry for storing, versioning, and serving ML models using S3 and MongoDB.

Overview: jrjModelRegistry is a simple and secure model registry designed for storing, versioning, and serving machine learning models using S3 and MongoDB. Key features: - 📦 Store models in S3-compatible storage with ZIP encryption - 🗃️ Track metadata in MongoDB - ⚡ Dynamically serve models using built-in FastAPI routes and dashboard - ✅ Works with any Python object (models, transformers, even DataFrames) Tech stack: Python Why it matters: This library simplifies the process of managing and serving machine learning models, making it ideal for both small projects and production environments.
Unknown 1 stars

JeffMLee1223/pytorch-model-service

PyTorch model training and inference service exposed through a FastAPI REST API.

Overview: This project provides a PyTorch model training and inference service accessible via a FastAPI REST API. It streamlines the process of deploying and utilizing machine learning models. Key features: - Model training and inference through PyTorch - RESTful API for easy integration with other systems - Dockerized environment for consistent deployment across different environments Tech stack: Python, PyTorch, FastAPI, Docker Why it matters: This service simplifies the deployment and management of machine learning models, making it easier to integrate with existing infrastructure and applications.
C# 1 stars

JohnClaw/gemma-2-2b-it.cs

gemma-2-2b-it int8 cpu inference in one file of pure C#

Overview: A pure C# implementation of Gemini 2.5 Pro Preview for CPU inference, designed for developers. Key features: - Pure C# implementation - Gemini 2.5 Pro Preview ported to C# - Int8 inference capabilities Tech stack: C# Why it matters: This project offers a lightweight, efficient solution for developers looking to integrate AI models into their applications without the need for complex dependencies or additional scripting languages.
Python 1 stars

JohnStephens1/TextOps

End-to-end ML platform for reproducible experimentation, automated model governance, versioned artifact management, and decoupled model serving.

### Overview: TextOps is an end-to-end ML platform designed for reproducible experimentation, automated model governance, versioned artifact management, and decoupled model serving. The project uses a small text-classification problem to engineer infrastructure around an ML system, focusing on the surrounding system rather than model complexity. ### Key Features: - **Reproducible Pipeline**: Tracks and manages the entire lifecycle from versioned data to inference. - **Automated Model Governance**: Promotes models based on both aggregate performance and specific class metrics. - **Versioned Artifact Management**: Ensures that artifacts are stored and versioned correctly. - **Decoupled Model Serving**: Allows for safe introduction and reuse of new model implementations without restructuring the entire pipeline. ### Tech Stack: Python ### Why It Matters: TextOps explores the infrastructure surrounding a small but imperfect classification problem, addressing questions in a production setting that go beyond just model performance. It focuses on the surrounding system, making it useful for understanding and improving the overall ML ecosystem in a production environment. ### Repository: JohnStephens1/TextOps ### Primary Language: Python ### GitHub Stars: 1 ### GitHub Description: End-to-end ML platform for reproducible experimentation, automated model governance, versioned artifact management, and decoupled model serving.
Unknown 1 stars

Junwu0615/PG-Cortex

Platform Genesis AI Platform • LLMOps, ML Platform & Model Serving

Overview: Platform Genesis Cortex is a cutting-edge AI platform designed for developers, offering a comprehensive suite of tools for building, deploying, and managing machine learning models. Key features: - Model Serving - Model Training - Model Deployment - Model Monitoring Tech stack: Python, TensorFlow, PyTorch, Docker, Kubernetes Why it matters: Platform Genesis Cortex simplifies the process of developing and deploying AI models, making it easier for developers to integrate AI into their projects without the need for extensive technical expertise.
Python 1 stars

Jyotiraditya3005/inferhub

Production-grade AI inference platform featuring model registry, deployment orchestration, autoscaling, GPU scheduling, observability, and Kubernetes-ready architecture.

### Overview: InferHub is a production-grade AI inference platform featuring model registry, deployment orchestration, autoscaling, GPU scheduling, observability, and Kubernetes-ready architecture. ### Key features: - Model registry and versioning - Real deployments with health checks and autoscaling - Simulated GPU scheduling algorithms - Kubernetes-ready architecture - Structured JSON logging and rate limiting - Prometheus metrics and Grafana dashboards - Real Kubernetes manifests and Helm chart ### Tech stack: Python ### Why it matters: InferHub provides a robust and scalable solution for deploying and managing AI models, ensuring high availability and performance through real-world implementation and Kubernetes-ready architecture.
Shell 1 stars

Keenni/DeepSeek-V4.1-Flash-1017-UNCENSORED-DGX-Sparks

Switchless TP4/TP2 for DeepSeek-V4.1-Flash-1017 (uncensored) on DGX Spark (GB10). TensorFold Zig TP4 on a switchless 200G fibre ring (AOT, no CUDA JIT). Plus MiaAI-Lab TensorFold TP4 native FP8 with prompt cache on disk, SGLang TP4, TP2 Zig + EXL3 2.9bpw.

Overview: This repository contains the deployment recipes, engine patches, and measurements for serving **DeepSeek-V4.1-Flash-1017** in its uncensored form on NVIDIA DGX Spark (GB10) nodes. The stacks include two-node TP2 with TensorFold's Zig engine over an EXL3 2.9 bpw repack, and four-node TP4 with either the same Zig engine or SGLang over the native FP8/FP4 checkpoint. Key features: - TP2 (2 nodes) with TensorFold's Zig engine over an EXL3 2.9 bpw repack - TP4 (4 nodes) with either the same Zig engine or SGLang over the native FP8/FP4 checkpoint Tech stack: Shell Why it matters: The repository details the deployment of four working stacks on DGX Spark nodes, including TP2 with TensorFold's Zig engine and TP4 with either the same engine or SGLang over the native FP8/FP4 checkpoint, providing insights into the performance and efficiency of these configurations. Repository: Keenni/DeepSeek-V4.1-Flash-1017-Flash-1017-UNCENSORED-DGX-Sparks GitHub stars: 1 GitHub description: Switchless TP4/TP2 for DeepSeek-V4.1-Flash-1017 (uncensored) on DGX Spark (GB10). TensorFold Zig TP4 on a switchless 200G fibre ring (A
Python 1 stars

Md-Emon-Hasan/BentoML

BentoML is a high-performance model serving framework it provides various scripts and configurations to help streamline and deployment process.

Overview: BentoML is a high-performance model serving framework that simplifies the deployment and management of machine learning models. It provides scripts and configurations to streamline the process and accelerate deployment. Key features: - Model deployment tools - Service creation methods - Visualization techniques - Integration with various machine learning frameworks - Advanced features for optimized deployments Tech stack: Python Why it matters: BentoML streamlines the deployment of machine learning models, making it easier to manage and scale model services in production environments.
Python 1 stars

MrKyaw/Onnx_Rust_Web_Assembly_Triton_Promethus_Grafana

ONNX inference in Rust/WebAssembly plus a validated Triton gateway: batching, readiness, Prometheus metrics, Grafana, Docker Compose. 83 Python tests, 4 Rust tests, 100% gateway coverage.

Overview: This project combines ONNX inference with Rust and WebAssembly, providing a validated Triton gateway for efficient inference and monitoring. Key features: - ONNX inference compiled to WebAssembly for the browser - Triton v2 HTTP protocol via a stdlib-only client - Input safety checks, finite-number checks, and element budgets - Deterministic first-dimension batching before calls to Triton - Prometheus metrics and Grafana integration - 100% coverage of the gateway core Tech stack: Python, Rust, WebAssembly, Triton, Prometheus, Grafana Why it matters: This project offers a robust and efficient solution for deploying ONNX inference in production environments, ensuring reliability and performance. Repository: MrKyaw/Onnx_Rust_Web_Assembly_Triton_Promethus_Grafana GitHub stars: 1 GitHub description: ONNX inference in Rust/WebAssembly plus a validated Triton gateway: batching, readiness, Prometheus metrics, Grafana, Docker Compose. 83 Python tests, 4 Rust tests, 100% gateway coverage.
Python 1 stars

Nice0o0/stateswap

🐱 One RWKV-7 base, N personas: train each persona into a few-MB initial state (S0) and hot-swap it at serving time.API, WebUI, plus reproducible S0 research: phase transitions, persona anatomy, inheritance.

Overview: stateswap is a project that leverages RWKV-7 as a base model to train N personas with initial states (S0) and hot-swaps them at serving time. This approach allows for the manipulation of style, persona, and task modes without altering the frozen weights, achieving significant improvements in performance and efficiency. Key features: - Train each persona into a few-MB initial state (S0) - Hot-swap the initial state at serving time - Use gradient descent to bake a "virtual prefix" into the initial state - Achieve a persona hot-swap latency of 2.6–4.4 ms - Utilize a 1.5B model with 1.59 MB of parameters for persona hot-swapping Tech stack: Python Why it matters: This project demonstrates the potential of using RWKV-7 as a base model to train personas, achieving significant improvements in performance and efficiency without altering the frozen weights, making it a valuable contribution to the field of language models. Repository: Nice0o0/stateswap Primary language: Python GitHub stars: 1 GitHub description: 🐱 One RWKV-7 base, N personas: train each persona into a few-MB initial state (S0) and hot-swap it at serving time. API, WebUI, plus reproducible S0 research: phase transitions, persona anatomy, inheritance.
Python 1 stars

Nikita3005/enterprise-customer-intelligence-platform

Production-ready Machine Learning platform for customer intelligence featuring FastAPI, MLflow, SHAP explainability, model serving, and automated testing.

Overview: This project is a production-ready Machine Learning platform designed for customer intelligence, featuring FastAPI, MLflow, SHAP explainability, model serving, and automated testing. Key features: - Customer Churn Prediction - Customer Lifetime Value Prediction - Customer Segmentation - Explainable AI using SHAP - MLflow Experiment Tracking - FastAPI Model Serving - Dockerized Deployment - Drift Detection - Automated Testing & CI Tech stack: Python Why it matters: This project demonstrates the complete lifecycle of a production Machine Learning system, combining reproducible data pipelines, experiment tracking, explainability, production-grade APIs, monitoring, testing, and maintainable software architecture.
Python 1 stars

RohitDusane/mlops-fastapi-docker-gke

Built an end-to-end MLOps pipeline for diabetes risk prediction using FastAPI, Docker, AWS ECR, and K3s Kubernetes on AWS EC2. Automated CI/CD deployment with GitHub Actions and integrated MLflow Model Registry for model versioning and dynamic production model loading.

Overview: This project showcases an end-to-end MLOps pipeline for diabetes risk prediction, utilizing FastAPI, Docker, AWS ECR, and K3s Kubernetes on AWS EC2. The pipeline includes automated CI/CD deployment with GitHub Actions and integrates MLflow Model Registry for model versioning and dynamic production model loading. Key features: - Automated CI/CD pipeline - Production-ready architecture - Full automation - Scalable Kubernetes deployment - Secure cloud-based deployment - Well-documented code Tech stack: Python Why it matters: This project demonstrates how to build and deploy a production-ready machine learning model using Docker, Kubernetes, and AWS, providing a comprehensive solution for deploying ML models in a production environment. Repository: RohitDusane/mlops-fastapi-docker-gke
Python 1 stars

RudraTyagi1135/insurance-premium-prediction

Overview: This project is a machine learning application designed to predict insurance premium risk categories based on customer demographic and lifestyle information. Key features: The system performs feature engineering, risk scoring, premium category prediction, probability estimation, and prediction logging through an interactive Streamlit dashboard. It uses Streamlit, Pandas, Pydantic, YAML configuration, and scikit-learn-compatible inference pipelines. Tech stack: Python Why it matters: This application is a live deployment of a machine learning model that estimates insurance premiums, providing valuable insights into customer risk categories and helping insurers make informed decisions. Repository: RudraTyagi1135/insurance-premium-prediction Primary language: Python GitHub stars: 1 GitHub description: No description provided.