Project directory

Projects

Open-source AI projects tracked by the leaderboard, ordered by GitHub stars.

Python 0 stars

galafis/distributed-model-inference-engine

Distributed model inference engine with REST/gRPC serving, circuit breaker, load balancing and A/B testing

Overview: A production-grade distributed inference engine that serves machine learning models through REST and gRPC APIs, designed to handle the realities of ML serving at scale. Key features: - Multi-Framework Support: Serve sklearn, PyTorch, and ONNX models through a single API. - Circuit Breaker: Prevents cascading failures before they take down the cluster. - Request Batching: Squeezes throughput out of every GPU cycle. - A/B Testing: Roll out model updates without holding breath. - Prometheus Metrics: Provides visibility into production. Tech stack: Python, FastAPI, PyTorch, scikit-learn, NumPy, Docker, gRPC, pytest. Why it matters: Ensures reliable, scalable, and efficient ML inference in production environments.

Python 0 stars

gaurav-sng2002/mlops-model-serving-platform

Dockerized ML model-serving API with health checks, metrics endpoint, Kubernetes manifests, and CI pipeline.

### Overview: This project is a Dockerized ML model-serving API that automates the process of setting up production-ready APIs for any trained model. It includes features such as universal model loading, registry integration, dynamic batching, A/B testing, observability, and auto-revert functionality. The platform is designed to save time and effort for data scientists by providing a quick and scalable solution for deploying models. ### Key Features: - **Universal Model Loader**: Automatically detects and loads models from various formats like `.pkl`, `.pt`, `.h5`, `.onnx`, and `.json`. - **Registry Integration**: Directly fetch models from MLflow, S3, or local filesystem. - **Dynamic Batching**: High-throughput async batching for efficient handling of traffic spikes. - **A/B Testing**: Built-in traffic routing based on percentages, custom headers, or sticky user IDs. - **Observability**: Prometheus metrics and Grafana dashboards for out-of-the-box observability. - **Auto-Rollback**: Automatically reverts to the previous model version if error rates spike. ### Tech Stack: - Python ### Why It Matters: This platform simplifies the process of deploying models by automating the setup of production APIs, reducing the need for custom wrappers and manual configuration. It saves time and effort for data scientists, allowing them to focus on their core responsibilities without worrying about the infrastructure setup. ### Repository: The project is hosted on GitHub at [gaurav-sng2002/mlops-model-serving-platform](https://github.com/gaurav-sng200

Python 0 stars

george-mountain/serving-diffusion-models

Production-style async serving for image-editing/generation diffusion models via two interchangeable backends — vLLM Omni and SGLang — with FastAPI gateways, Celery + Redis job queues, Docker Compose, and a React frontend.

Overview: This repository demonstrates the production-style async serving of image-editing diffusion models using two interchangeable backends: vLLM Omni and SGLang. The project includes FastAPI gateways, Celery job queues, Docker Compose orchestration, and a React frontend. Both methods share a common architecture, utilizing FastAPI for the gateway, Celery for the worker, and Docker for containerization. Key features: - **vLLM Omni Backend**: Uses `vllm/vllm-omni` for image edits, with an optional authentication feature. - **SGLang Backend**: Utilizes `lmsysorg/sglang` for diffusion, requiring a Bearer API key for authentication. - **FastAPI Gateway**: Accepts image and prompt, immediately returns a task ID. - **Celery Worker**: Pulls jobs from Redis, runs inference, and writes results to shared storage. - **Docker Compose**: Orchestrates services, and `make` commands manage the build and start/stop processes. - **React Frontend**: Handles client interactions, with test scripts available for client-side testing. Tech stack: Python, FastAPI, Celery, Docker, Redis, Docker Compose, React. Why it matters: This project showcases a robust, production-ready solution for serving image-editing diffusion models, providing a clear, concise guide for developers looking to integrate similar functionality into their projects. Repository: george-mountain/serving-diffusion-models

Unknown 0 stars

giriaryan694-a11y/ai-infra-wordlists

Curated wordlists for discovering exposed AI/ML infrastructure endpoints, including model servers, pipelines, vector databases, and experiment tracking systems.

Overview: This repository contains curated, high-signal collections of wordlists specifically designed for AI/ML infrastructure reconnaissance. It targets modern AI stacks, including model servers, pipelines, vector databases, and experiment tracking systems. Key features: - Model Serving Layer - Pipeline & Orchestration - Data & Vector Storage - Experiment & Model Lifecycle - Dev & Interactive Environments - Object Storage Tech stack: Python Why it matters: This repository fills the gap in traditional wordlists, which are focused on web apps and lack coverage of modern AI-specific infrastructure. It helps security researchers and red-hat team members identify and exploit exposed endpoints in AI/ML systems, ensuring they target the right targets and avoid false positives. Repository: giriaryan694-a11y/ai-infra-wordlists Primary language: Unknown GitHub stars: 0 GitHub description: Curated wordlists for discovering exposed AI/ML infrastructure endpoints, including model servers, pipelines, vector databases, and experiment tracking systems.

TypeScript 0 stars

girithc/inference

gpu kernels for ocr, vision and reasoning llms. hosted on modal

### Overview: Inference is a project that provides GPU-based deployments for various AI models including OCR, vision, and reasoning. It is hosted on Modal and supports open models like GLM-OCR, DeepSeek-OCR, Qwen-Image-2.1, YuE2-3B, and Qwen-3.8-27B. The project includes a Next.js TypeScript web UI for deploying models and monitoring performance metrics such as throughput and A/B testing. ### Key Features: - **GPU Deployments**: Inference supports GPU-based deployments for multiple AI models. - **Web UI**: A Next.js TypeScript web UI for deploying models, including an OCR view for real-time transcription, a reasoning chat canvas for live thinking and answers, and a speech view for voice-to-text and text-to-speech integration. - **Model Registration**: Deployments can be registered in-app with base URL, model name, and optional key, and requests are proxied through the Next server. ### Tech Stack: TypeScript ### Why it Matters: Inference simplifies the deployment and management of AI models, providing a user-friendly web interface for real-time monitoring and performance analysis.

TypeScript 0 stars

girithc/inference-gym

gpu kernels for ocr, vision and reasoning llms. hosted on modal

Overview: A comprehensive GPU kernel repository for various AI models, including OCR, vision, and reasoning, hosted on Modal. Key features: - GPU kernels for OCR, vision, and reasoning models. - Deployments for zai-org/GLM-OCR, deepseek-ai/DeepSeek-OCR, Qwen/Qwen-Image-2.1, m-a-p/YuE2-3B, Qwen/Qwen3.8-27B, and WarmBloodAban/Minimax-h3_Singularity. - A web UI for serving open models on Modal, featuring a web canvas with real-time performance metrics and interactive features. Tech stack: TypeScript Why it matters: This repository provides a versatile platform for deploying and serving AI models, offering real-time performance insights and interactive visualizations, making it useful for developers and researchers alike. Repository: girithc/inference-gym Primary language: TypeScript GitHub stars: 0 GitHub description: gpu kernels for ocr, vision and reasoning llms. hosted on modal

Python 0 stars

gitblame-hemanth/fastapi-ml-template

Production-ready FastAPI template for deploying ML models with Redis caching, Prometheus metrics, and Docker

### Overview: This is a production-ready FastAPI template designed specifically for deploying machine learning models as REST APIs. It includes built-in health checks, caching, metrics, rate limiting, and authentication, making it easy to focus on your model without worrying about infrastructure. The template supports various models like sklearn, HuggingFace, and custom wrappers, and comes with Docker support for a Docker-ready setup. ### Key features: - **Health Checks**: `/health`, `/readiness`, and `/liveness` endpoints for Kubernetes probes. - **Single & Batch Inference**: `/api/v1/predict` and `/api/v1/predict/batch` with automatic input validation. - **Redis Caching**: Deterministic request hashing with configurable TTL for efficient inference. - **Prometheus Metrics**: Request latency histograms, inference counters, and model load time at `/metrics`. - **Rate Limiting**: Per-client sliding window rate limiter via middleware. - **API Key Authentication**: Optional header-based API key gate. - **Structured Logging**: JSON-formatted logs via `structlog` with request ID correlation. - **Docker Ready**: Multi-stage Dockerfile, docker-compose with Redis and Prometheus, non-root runtime user. - **Model Registry**: Plug-and-play model registration supporting sklearn, HuggingFace, and custom wrappers. - **CORS Middleware**: Configurable allowed origins. ### Tech stack: - Python 3.11 - FastAPI 0.115 - Redis - Prometheus - Docker ### Why it matters: This template provides a standardized project structure with

Jupyter Notebook 0 stars

go-bananas-wwj/embedding-api

FastAPI service for regional geospatial embeddings, downstream inference, custom task heads and SAM3 interaction.

Overview: This repository provides a FastAPI service for regional geospatial embeddings, downstream inference, custom task heads training, and SAM3 interaction, suitable for the XuanNv workflow. Key features: - FastAPI service for regional geospatial embeddings - Downstream inference capabilities - Custom task heads training - SAM3 interaction - Precomputed Harbin embedding color preview Tech stack: Jupyter Notebook Why it matters: This repository offers a comprehensive solution for regional geospatial embeddings, making it a valuable tool for developers working in the field of geospatial data processing and analysis.

Jupyter Notebook 0 stars

gracecode42/ml-07-applied

Applied Machine Learning - Final Week - Applied

Overview: This project is designed to help developers understand and analyze a deployed machine learning model by systematically probing it with various inputs. Key features: - Systematically probing a deployed ML model with different inputs - Observing how predictions change as input features are varied - Identifying decision boundaries and edge cases - Interpreting model behavior from the outside Tech stack: Jupyter Notebook Why it matters: This project provides a practical way for developers to learn and apply machine learning concepts, focusing on the practical aspect of interrogating and understanding deployed models. Repository: gracecode42/ml-07-applied Primary language: Jupyter Notebook GitHub stars: 0 GitHub description: Applied Machine Learning - Final Week - Applied README excerpt: This project focuses on learning to interrogate a deployed ML model by probing it systematically with different inputs. We learn to: call a live prediction API from a notebook, vary input features and observe how predictions change, identify decision boundaries and edge cases, and interpret model behavior from the outside. The project uses Python 3.14 and includes links to the workflow guide, Python version, and license information. The example notebook and working files are provided, and additional packages are listed. The project uses the requests package for making calls and includes a command reference for running the project. The findings and visuals are discussed, and additional project instructions and a citation are provided.

Shell 0 stars

hBonnet875/dgx-spark-inference-stack

Serve the home! Inference stack for your Nvidia DGX Spark aka the Grace Blackwell AI supercomputer on your desk. Mostly vLLM based for now and single-spark. For the not-so-rich buddies. If you want latest/in-testing, look at the branches

Overview: This project serves as the inference stack for the Nvidia DGX Spark, a powerful AI supercomputer. It is designed for developers looking to leverage advanced AI capabilities on a desktop. Key features: - Single Spark deployment - VLLM-based inference engine - Primarily shell-based for simplicity Tech stack: Shell Why it matters: This stack provides a cost-effective way for developers to experiment with advanced AI models, making cutting-edge technology accessible to those who might not have access to large-scale AI infrastructure.

Python 0 stars

hammasbuilds/model-serving-platform

Python · A/B + canary + shadow · SLO monitor — Multi-model serving: A/B testing with sticky assignment, canary rollout, shadow traffic, per-version SLOs and auto-rollback that knows the difference between a bad canary and an upstream outage. 31 tests.

Overview: This project is a Python-based model serving platform designed for ML engineers, offering features like A/B testing with sticky assignment, canary rollout, shadow traffic, per-version SLOs, and auto-rollback. It includes a web UI for easy management and is used in the LLM Control Room. Key features: - A/B testing with sticky assignment - Canary rollout - Shadow traffic - Per-version SLOs - Auto-rollback of a breaching canary Tech stack: Python Why it matters: This platform provides a reliable and efficient way to deploy and manage multiple model versions, ensuring that canary testing and rollback are handled seamlessly, making it ideal for ML engineers looking to implement these features in their workflows.

Python 0 stars

harveybc/prediction_provider

FastAPI inference service for financial models: pluggable feeders and predictors/decision making models, multi-role auth and billing, serving entry/exit signals over HTTP

### Overview: A FastAPI-based inference service designed for financial models, offering a pluggable feeder and predictor chain, multi-role authentication, billing, and SQL persistence. It serves entry/exit signals over HTTP, ensuring deterministic and secure predictions. ### Key Features: - **Feeder and Predictor Chain**: Loads prediction and decision-model plugins (Keras/ONNX, ideal oracles, CSV replay). - **Multi-Role Authentication**: Supports admin, provider, client, and evaluator roles. - **Billing and Persistence**: Tracks and records billing and prediction persistence. - **HTTP Endpoints**: Provides prediction, health, info, and metrics endpoints. - **Deterministic Serving Mechanics**: Uses nested `mechanics/` package for deterministic serving with hash-verified artifact loading. ### Tech Stack: Python ### Why It Matters: This service is crucial for financial models and decision-making, providing a robust and secure platform for real-time predictions and analytics.

Python 0 stars

hawkz3d/laptop-qwen-deploy

AI Infra 实践：大模型推理服务部署、CPU/GPU 混合执行、显存与 KV Cache 优化、API 服务化及部署自动化。

### Overview: 本项目展示了在消费级硬件上实现大模型推理服务的实践。通过优化资源使用、显存与上下文配置、性能与兼容性排查，以及部署自动化，成功在6GB显存、32GB内存的系统上完成了35B MoE量化模型的本地部署，并提供了OpenAI兼容的API服务，支持Web客户端调用。 ### Key features: - **模型部署与运行时兼容性排查** - **CPU/GPU 混合推理与显存预算** - **推理性能分析与配置对照实验** - **模型服务化与应用接入** - **部署自动化与基础运行维护** ### Tech stack: Python ### Why it matters: 通过本项目，展示了如何在有限资源下高效运行大模型，将其封装为应用能够调用的服务，并通过实验和日志定位性能与兼容性问题，为AI基础设施实践提供了宝贵的经验和方法。

Python 0 stars

hhharrrshhh/tower-failure-model-serving-stack

### Overview: Predicting mobile telecommunications cell tower failures up to 48 hours in advance using multivariate sensor telemetry, this end-to-end MLOps serving stack includes feature engineering, low-latency API serving, container orchestration, metrics scraping, live Grafana dashboards, automated Prometheus alerting, empirical load-testing benchmarks, and an automated self-healing Linux watchdog. ### Key Features: - **Model Preparation**: Zero-skew preprocessing with median imputation and domain interaction features. - **Low-Latency Serving**: FastAPI microservice with Pydantic v2 input validation and boundary checks. - **Container Orchestration**: Multi-container Docker Compose deployment. - **Observability & Alerting**: Prometheus metrics scraping, Grafana visualizations, and two production alert rules. - **Empirical Benchmarking**: Locust load tests across 10, 100, and 500 concurrency levels. - **Linux Operations & Reliability**: Autonomous Bash watchdog. ### Tech Stack: - Python 3.11 - FastAPI - Docker Compose - Prometheus - Grafana - Locust ### Why It Matters: This stack enables telecom operators to predict failures up to 48 hours in advance, shifting maintenance from reactive emergency recovery to proactive scheduled servicing, thereby reducing downtime and improving service quality.

Python 0 stars

honeyuheony/MLCC_BE

MLCC 적층 얼라인먼트 검사 시스템

### Overview: This project is an MLCC (Multilayer Ceramic Capacitor) inspection system designed for use in the quality control process of Samsung's semiconductor manufacturing. The system aims to provide easy access to quality evaluation data models and contribute to quality analysis and model performance enhancement by professionals involved in the process. ### Key Features: - Celery-based model serving - Implementation of an API for analyzing monitoring data ### Tech Stack: Python ### Why It Matters: This project is crucial for ensuring the quality of semiconductor products, which are essential components in modern electronics. By providing a user-friendly interface for quality analysis and model performance improvement, it helps maintain high standards and reduces defects, ultimately improving the overall quality and reliability of the products. Repository: honeyuheony/MLCC_BE Primary language: Python GitHub stars: 0 GitHub description: MLCC 적층 얼라인먼트 시스템

Python 0 stars

hscspring/lightinfer

Make your model inference to http server. 一个为“同步重计算模型”量身设计的异步推理服务桥接器。

### Overview: LightInfer is a developer-friendly solution designed to bridge the gap between synchronous heavy model inference and high-performance asynchronous web frameworks. It efficiently isolates heavy computations in dedicated, managed worker threads while maintaining a fully asynchronous, high-concurrency web frontend, ensuring seamless integration with Python's FastAPI. ### Key Features: - **🛡️ Zero-Blocking Architecture:** Seamlessly merges a high-concurrency Async Web Frontend with dedicated Sync Worker Threads for heavy lifting. - **⚡ Efficient Bridge:** Utilizes a specialized `AsyncResponseBridge` for zero-thread-overhead waiting. - **🌊 Advanced Streaming Support:** Native Server-Sent Events (SSE) for LLM text generation and binary streaming for real-time audio/video generation. - **🧩 Dead Simple Integration:** Just wrap any Python class with an `infer` method. We handle the rest. - **🔒 Context Isolation:** Each worker runs in its own thread, ensuring thread-safety for libraries like PyTorch. ### Tech Stack: Python 3.8+ ### Why It Matters: Are you facing the "Blocking Loop" problem? Running heavy, synchronous model inference inside an `async` web framework like FastAPI will freeze your entire server. LightInfer instantly solves this by acting as a high-performance bridge, efficiently isolating heavy computations in dedicated, managed worker threads while maintaining a fully asynchronous, high-concurrency web frontend.

Python 0 stars

hztBUAA/deepseek-toolkit

Practical toolkit for DeepSeek models — fine-tuning, serving, evaluation, and data preparation

Overview: A comprehensive toolkit for fine-tuning, serving, evaluation, and data preparation of DeepSeek models. Key features: - Interactive Chat - Fine-Tuning - Evaluation - Serving - Benchmarking - Data Preparation Tech stack: Python Why it matters: This toolkit simplifies the process of deploying and evaluating DeepSeek models, making it easier for developers to integrate these models into their projects. Repository: hztBUAA/deepseek-toolkit Primary language: Python GitHub stars: 0 GitHub description: Practical toolkit for DeepSeek models — fine-tuning, serving, evaluation, and data preparation README excerpt: The Swiss Army Knife for DeepSeek Models. Practical tools for fine-tuning, inference optimization, evaluation, and deployment of DeepSeek models.

Python 0 stars

iPrakharV/ModelMesh

Trainable ML inference mesh with MPS training, load-aware routing, caching, failure replay, benchmarks, Docker, and a live dashboard.

### Overview: ModelMesh is a trainable ML inference gateway designed for testing ML service behavior under various conditions such as traffic, caching, and worker failures. It includes a FastAPI gateway, load-aware routing, caching, failure replay, and a live dashboard for monitoring and benchmarking. ### Key Features: - **FastAPI Gateway**: Accepts prediction requests, checks a cache, routes to a worker, retries on failure, and keeps basic latency and error metrics. - **Load-Aware Routing**: Tracks recent latency, failures, and in-flight requests to optimize routing. - **Model Artifact**: Exported JSON model artifact for model inference. - **Metrics and Dashboard**: Provides basic latency and error metrics, as well as a live dashboard for monitoring. ### Tech Stack: - Python ### Why It Matters: ModelMesh is essential for developers and researchers looking to evaluate and optimize the performance of ML inference services under various conditions, ensuring reliability and efficiency in production environments. ### Repository Layout: The project is organized into several directories: - `gateway/` for the FastAPI gateway and related components. - `worker/` for model workers and exported model artifacts. - `training/` for MPS training and evaluation scripts. - `bench/` for benchmarking and replay scripts. - `scripts/` for smoke checks and operational helpers. - `tests/` for unit tests. - `docs/` for architecture, deployment, and model card documentation. - `deploy/compose/` for local Docker Compose files. - `requirements/` for runtime, development, and training dependencies. -

Python 0 stars

iarsingh/kubernetes-ml-model-serving-platform

Python, probes and HPA flags

Overview: This project is a Kubernetes deployment gate for ML model serving, ensuring that only models with enabled probes, horizontal pod autoscaling, and a pinned container image are allowed to be served. Key features: - Built with Python, pytest, Docker, and GitHub Actions - Scope: Local lab with tests; does not call a hosted model or apply production changes Tech stack: Python, pytest, Docker, GitHub Actions Why it matters: Ensures that only models with necessary health checks, scaling capabilities, and a fixed image are deployed, preventing unvetted models from being shipped to production.

Python 0 stars

iarsingh/ml-model-serving-api

Serve a versioned local price model.

### Overview A FastAPI service that serves a versioned house price model from a local weight file. The endpoint `POST /predict` loads `models/house-price-v1.json` and scores a house based on square feet and bedrooms. The model is not retrained, and requests are rejected if they are missing required features. ### Key features - Serve a versioned local price model. - Load `models/house-price-v1.json` for predictions. - Reject requests without provided features. ### Tech stack - Built with: Python, FastAPI, pytest, GitHub Actions ### Why it matters A local lab with tests; one fixed model version and no training pipeline.