Project directory

Projects

Open-source AI projects tracked by the leaderboard, ordered by GitHub stars.

Python 1 stars

bdllhdrss3/EmbedServe

A high-performance, local embedding service for running models like Qwen, providing semantic text embeddings via FastAPI, with caching and concurrency control.

Overview: This project is a high-performance, local embedding service powered by Qwen models, designed to provide semantic text embeddings. It includes caching, concurrency control, and flexible configuration options. Key features: - 🚀 High Performance: Optimized for throughput with concurrency control - 💾 Intelligent Caching: Disk-based caching with configurable size limits - 📊 Monitoring: Cache hit/miss statistics for performance tuning - 🔧 Configurable: Environment variables for all settings - 🐳 Containerized: Ready-to-use Docker support - 🔄 Format Options: Return embeddings as JSON lists or binary numpy arrays - 🔌 Model Flexibility: Easy to swap embedding models Tech stack: Python Why it matters: This service addresses the challenges of running embedding models efficiently at scale, providing a self-hosted solution that reduces costs, ensures privacy, minimizes latency, and offers customization options. Repository: bdllhdrss3/EmbedServe Primary language: Python GitHub stars: 1 GitHub description: A high-performance, local embedding service for running models like Qwen, providing semantic text embeddings via FastAPI, with caching and concurrency control.

Python 1 stars

belarusian/practice

Production ML lab for learning end-to-end ML systems: vision/audio/text classification, embeddings, serving APIs, active learning, orchestration, and AWS deployment. Includes Marcus Aurelius philosopher training as a fine-tuning learning exercise.

### Overview: This repository is a production-oriented ML lab designed for hands-on experience with PyTorch, model training, active learning, embedding retrieval, model serving, and AWS deployment. It includes a fine-tuned Marcus Aurelius philosopher model for learning purposes. ### Key Features: - **Vision Classification**: PyTorch-based image classification. - **Audio Classification**: PyTorch-based audio classification. - **Text Classification**: Hugging Face transformer-based text classification. - **Text Embeddings**: PyTorch-based text embeddings for semantic search. - **Active Learning**: Relabel queues for model improvement. - **Model Serving**: FastAPI inference service for vision and text models. - **AWS Deployment**: Ephemeral GPU training on EC2 Spot, cheap CPU serving on EC2, S3 for artifacts, and ECR for images. ### Tech Stack: Python ### Why It Matters: This lab provides a comprehensive approach to learning and practicing the entire lifecycle of a production ML system, from data curation to model deployment, without the high costs associated with managed services.

CSS 1 stars

camilo315853/ai-launcher

Manage and launch AI coding assistants from one CLI with persistent local context across sessions, ensuring privacy and ease of use.

Overview: A CLI-based tool for managing and launching AI coding assistants, ensuring persistent local context across sessions for enhanced privacy and ease of use. Key features: - Persistent local context across sessions - Manage and launch AI coding assistants - CLI-based interface Tech stack: CSS Why it matters: This project offers a convenient and secure way to manage AI coding assistants, making it easier for developers to work efficiently without compromising privacy. Repository: camilo315853/ai-launcher Primary language: CSS GitHub stars: 1

HCL 1 stars

canonical/autoscaling-model-serving

This repository hosts the Autoscaling Model Server bundle

Overview: This repository provides the Autoscaling Model Serving solution, consisting of Istio, Knative, and Kserve Controller, enabling model server deployment in any Kubernetes cluster. Key features: - Istio - Knative - Kserve Controller Tech stack: Terraform Why it matters: This solution facilitates model server deployment across various Kubernetes clusters, making it accessible from outside the cluster, enhancing flexibility and scalability.

Python 1 stars

ciliverse/cilitrain

Lightweight open-source AI training platform on Kubernetes: submit Python scripts as batch Jobs, queue with priorities and quotas, track curves, register models, run batch and online inference. No CRDs. | 轻量开源 K8s AI 训练平台

### Overview: CiliTrain is a lightweight, open-source AI training platform on Kubernetes that transforms Python scripts into standard `batch/v1` Jobs with resource guardrails, a priority queue with per-user quotas, live logs, training & Pod resource curves, artifact harvesting, model registration, and batch and online inference capabilities. It simplifies the process of training, scheduling, and managing AI models without the need for custom CRDs or in-cluster operators. ### Key Features: - **Batch Job Submission**: Submit Python scripts via a web interface or built-in templates, with hyperparameter management and environment variable support. - **Resource Management**: Track and monitor training and Pod resource usage with charts and curves. - **Model Registry**: Register and compare finished runs' models, allowing for easy model versioning and download. - **Scheduling**: Queue runs instead of rejecting them, with priority and quota management for both namespace and user-level limits. - **Model Inference**: Deploy models as long-running services, serving predictions via a single API endpoint. ### Tech Stack: - **Python**: The primary programming language used for development and job submission. - **Kubernetes**: The underlying infrastructure for orchestrating and managing the cluster. ### Why It Matters: CiliTrain simplifies the process of training, scheduling, and managing AI models, making it accessible to developers without the need for complex customizations. It provides a streamlined workflow for training, scheduling, and inference, reducing the complexity and overhead associated with traditional AI platforms.

Go 1 stars

codeROCK-IZZY/distributed-ai-inference

Model serving that stays predictable under overload: a Go gateway with per-tenant quotas, admission control sized with Little's law, deadlines, fan-out, retries and replica ejection, in front of batching ONNX workers. Kubernetes with KEDA, tested on kind in CI.

### Overview: This project is a Go-based inference platform designed to handle unpredictable load, ensuring predictability even when demand exceeds capacity. It includes admission control, tenant quotas, deadlines, fan-out, retries, and replica ejection, making it ideal for teams serving models to multiple applications from shared infrastructure. ### Key Features: - **Admission Control Sized with Little's Law:** Admits only as much work as the replicas can finish within the latency budget, ensuring no timeouts. - **Tenant Quotas and Priorities:** Each tenant has a token bucket based on texts, not requests, and can be shed first if it exceeds capacity. - **Deadlines End to End:** Requests are dropped before running the model to ensure deadlines are met. - **Fan-Out:** Large requests are split into chunks and sent to different replicas in parallel, merging results in order. - **Failure Handling:** Replicas are routed by power of two choices on outstanding work, retries on failed calls, and ejected if they fail repeatedly. - **Dynamic Batching:** Concurrent requests share one model call with a bounded queue. - **Kubernetes-Ready:** Uses headless services for discovery, readiness tied to a warmed model, graceful drain, PodDisruptionBudgets, NetworkPolicies, and restricted Pod Security profile. ### Tech Stack: Go 1.25 (gateway, load generator) · Python 3.13 · FastAPI · ONNX Runtime (bge-small-en-v1.5 embeddings, MiniLM cross-encoder reranker) · Prometheus · Grafana · Docker · Kubernetes (K

Python 1 stars

codychampion/local-ml-workbench

Local MLOps workbench for self-hosted model serving, RAG experiments, and evaluation infrastructure.

### Overview: This self-hosted machine learning workbench provides a practical environment for running local AI experiments, including data collection, annotation, training, evaluation, inference, model tracking, and local language model serving. It combines the benefits of local-first development with the operational structure needed for serious components like GPU-backed LLMs, dataset labeling, model evaluation, experiment metadata, and a research knowledge vault. ### Key features: - Object storage: MinIO-compatible S3 storage - Local LLM serving: vLLM with OpenAI-compatible API - Experiment workbench: JupyterLab, Hydra configs, pipeline containers - Dataset inspection: FiftyOne + MongoDB - Annotation: Label Studio - Tracking: AIM experiment tracking - Chat over notes: Khoj + Postgres over the knowledge vault - API surface: FastAPI server - Registry patterns: Docker registry + model registration scripts ### Tech stack: Python ### Why it matters: This workbench sits at the "messy middle" between notebooks and production, providing a practical lab environment for testing ideas quickly without losing the operational pieces that make ML work trustworthy. It keeps the workflow local-first, reproducible, and modular while supporting serious components like GPU-backed LLMs, dataset labeling, model evaluation, experiment metadata, and a research knowledge vault.

Python 1 stars

daniele21/korgis

Local AI runtime control plane for running, scheduling and observing LLM, VLM and ASR models behind one stable API.

Overview: Korgis is a local AI runtime control plane designed to manage and observe models like LLMs, VLMs, and ASR systems, providing a stable API for applications to interact with. Key features: - Run and manage local AI models behind a stable API. - Support for multiple runtimes and models. - Resource management and budget enforcement. - Observability and diagnostics for runtime state. - Reproducibility of evaluations and comparisons. - Integration with OpenAI-compatible APIs for text generation. Tech stack: Python. Why it matters: Korgis addresses the challenge of moving AI from the cloud to local infrastructure, focusing on the runtime and control plane to ensure reliable and manageable local AI model execution. Repository: daniele21/korgis Primary language: Python GitHub stars: 1 GitHub description: Local AI runtime control plane for running, scheduling and observing LLM, VLM and ASR models behind one stable API.

Python 1 stars

datamata-io/mata-server

MATA Server is a lightweight inference runtime designed to serve computer vision models built on the MATA (Model-Agnostic Task Architecture)

Overview: MATA Server is a lightweight inference runtime designed for serving computer vision models built on the MATA ecosystem. It provides a unified API for detection, segmentation, classification, pose estimation, OCR, depth, and VLM tasks, enabling on-demand model loading, memory-aware eviction, and real-time streaming inference. Key features: - Unified inference API - REST + WebSocket - On-demand model loading - Memory-aware eviction - Multi-source model pulls - `mata.v1` response schema - GPU + CPU support - API key authentication - OpenAPI docs - Docker-ready Tech stack: Python Why it matters: MATA Server simplifies the deployment of MATA models by providing a consistent, versioned API and efficient memory management, making it easier to integrate with existing applications and workflows. Repository: datamata-io/mata-server GitHub stars: 1

Python 1 stars

debu-sinha/cross-region-model-serving-dab

Production-ready Databricks Asset Bundle for cross-region ML model serving using Delta Sharing. Deploy models and feature tables across workspaces with zero-copy data access and automated online feature store sync.

Overview: This project provides a production-ready Databricks Asset Bundle for cross-region model serving, utilizing Delta Sharing to securely share models and feature tables across Databricks workspaces. It includes automated online feature store synchronization and zero-copy data access, making it ideal for deploying models and feature tables across different regions. Key features: - Securely shares models and feature tables across Databricks workspaces using Delta Sharing - Automatic online feature table synchronization - Deploying models and feature tables with zero-copy data access - Automated online feature store sync Tech stack: Python Why it matters: This tool simplifies the process of sharing and deploying ML models across different regions, ensuring low-latency inference and easy deployment without manual intervention. Repository: debu-sinha/cross-region-model-serving-dab Primary language: Python GitHub stars: 1 GitHub description: Production-ready Databricks Asset Bundle for cross-region ML model serving using Delta Sharing. Deploy models and feature tables across workspaces with zero-copy data access and automated online feature store sync.

Python 1 stars

deepakreddy0811/mlops-pipeline

End-to-end ML pipeline — MLflow experiment tracking, containerised FastAPI serving, and PSI-based data drift monitoring

### Overview: This repository contains an end-to-end machine learning pipeline designed for churn prediction. The pipeline includes experiment tracking with MLflow, a containerized FastAPI serving API, and PSI-based data drift monitoring. ### Key features: - Experiment tracking with MLflow for detailed experiment logs. - Containerized FastAPI serving API for production deployment. - PSI-based data drift monitoring to detect drift in production data. ### Tech stack: Python ### Why it matters: Training a model is just the beginning. The real challenge lies in knowing which run produced the artifact in production, ensuring the model is serving without skew, and detecting when it starts to fail. This pipeline addresses these issues by providing a comprehensive lifecycle for machine learning models.

Java 1 stars

developertogo/velo-sentinel

Production-grade Java 25 Virtual Thread inference gateway bridging NVIDIA Triton → Dynamo with Earliest Deadline First (EDF) priority queuing, adaptive batching, and async shadow validation.

### Overview: Velo-Sentinel is a production-grade Java 25 Virtual Thread inference gateway designed for Tier-1 AI organizations. It serves as the mission-critical orchestration layer for transitioning from legacy NVIDIA Triton to the next-generation NVIDIA Dynamo disaggregated inference framework. ### Key Features: - **Concurrency**: Prefill/Decode separation and cache-aware routing. - **Resilience**: Circuit breakers, hedging, and fault injection. - **Efficiency**: Optimizing TFLOPS via intelligent request grouping. - **Governance**: PII scrubbing, authentication, and audit logging. - **Hardware**: Full-stack integration with Velo-Core. ### Tech Stack: Java 25 Virtual Threads ### Why it Matters: Velo-Sentinel provides a foundational computational platform for global-scale ML/AI applications, bridging the gap between legacy infrastructure and disaggregated, hardware-aware serving through the Four Pillars of Inference Scale: Concurrency, Resilience, Efficiency, and Governance. ### GitHub Stars: 1 ### Repository: developertogo/velo-sentinel

Python 1 stars

dipayan90/ml-model-server

Cookie cutter online ML model serving solution

### Overview: This project offers a pre-built REST server for scoring machine learning models, providing easy configuration, error handling, and cross-domain access support. ### Key features: - Score models by providing configurations. - Inbuilt error handling and standard logging. - Configurable Base Path for the API. - Supports CORS for cross-domain access. - Inbuilt Swagger documentation. - Compatible with PMML, Pickle, and simple lookup model serialization formats. ### Tech stack: Python ### Why it matters: This solution streamlines the process of deploying and scoring machine learning models, making it easier for developers to integrate and deploy models without the need for extensive setup.

Python 1 stars

dudeperf3ct/11-cortex-deploy

Overview: Cortex is a powerful tool for deploying, managing, and scaling machine learning models in production environments, particularly on AWS. It supports serverless workloads, automated cluster management, CI/CD, and observability integrations. Key features: - Serverless workloads - Automated cluster management - CI/CD and observability integrations Tech stack: Python Why it matters: Cortex simplifies the deployment process, requiring only two configuration files, making it an efficient solution for developers looking to deploy machine learning models quickly and easily. Repository: dudeperf3ct/11-cortex-deploy Primary language: Python GitHub stars: 1 GitHub description: No description provided. README excerpt:

Python 1 stars

dudeperf3ct/12-serverless-deploy

### Overview: This project demonstrates the deployment of a sentiment classifier application using Serverless Framework, a platform for developing and deploying serverless applications. The application is built using Python and is deployed to AWS Lambda for efficient execution. ### Key Features: - Sentiment classifier model deployed using Serverless Framework. - Docker-based development and testing environment. - Serverless Framework for easy deployment and monitoring. - AWS Lambda integration for serverless execution. - Swagger UI for API documentation. - CI/CD integration via Serverless Dashboard. ### Tech Stack: Python ### Why it Matters: This project showcases the capabilities of Serverless Framework in deploying serverless applications, including Docker-based development, AWS Lambda integration, and CI/CD integration. It provides a practical example of how to build, test, and deploy a sentiment classifier using serverless technologies.

Jupyter Notebook 1 stars

dudeperf3ct/6-ml-fastapi-aws-serverless

### Overview: This project demonstrates the deployment of a fastapi ML application to AWS using continuous delivery with CodePipeline and Elastic Beanstalk. ### Key features: - Deploy ML model to AWS using FastAPI - Continuous delivery using CodePipeline - Elastic Beanstalk for environment creation and application deployment ### Tech stack: Python (Jupyter Notebook) ### Why it matters: This project showcases the integration of ML models with serverless architectures, emphasizing the ease of deployment and management through AWS services.

Python 1 stars

dudeperf3ct/8-fastapi-tests-gcp-gke

Overview: This project involves deploying a sentiment analysis model using FastAPI on Google Cloud Platform (GCP) using Google Kubernetes Engine (GKE). Key features: - Containerizing different components of the project - Writing tests and testing individual modules using `pytest` - Using `trunk` for automatic code checking, formatting, and linting - Deploying the application on GKE Tech stack: Python Why it matters: This project demonstrates best practices in containerization, testing, code quality, and deployment on cloud platforms, making it valuable for developers looking to improve their skills in these areas. Repository: dudeperf3ct/8-fastapi-tests-gcp-gke Primary language: Python GitHub stars: 1 GitHub description: No description provided.

Rust 1 stars

edurdias/capyctl

Control what runs next: run vLLM and SGLang models on your own GPUs behind one OpenAI-compatible endpoint.

Overview: CapyCTL is a powerful tool designed to manage and optimize GPU usage for running models like vLLM, SGLang, and TensorFold. It ensures that models are efficiently utilized, with models being parked and woken up as needed, and models are managed behind a single OpenAI-compatible endpoint. Key features: - CapyCTL can run models on up to two GPUs per machine, ensuring efficient GPU utilization. - It supports custom engines for inference servers, allowing for easy integration with various models. - It can run models on a single machine or a server with multiple GPUs, ensuring optimal performance. - It can park models when not in use, and wake them up when needed, ensuring continuous availability. - It supports automatic GPU selection and memory accounting on identical GPUs. - It allows for the deployment of models across multiple GPUs on the same machine, with automatic switching and parking. - It has an OpenAI-compatible API for easy integration with other systems. - It provides request IDs and metrics for tracking and debugging. - It manages engine logs in memory, ensuring prompt and answer security. - It supports running inside containers, making it easy to deploy on unified-memory machines and discrete GPUs. - It supports Linux on x86-64 and ARM64, and NVIDIA GPUs. Tech stack: Rust Why it matters: CapyCTL simplifies the management of GPU resources, ensuring efficient utilization of models and models' memory. It provides a single, unified endpoint for managing multiple models, making it easier to integrate with other systems and manage models efficiently.

Python 1 stars

emmanueladutwum123/cv-resnet-pipeline

ResNet-18/CIFAR-10 → ONNX → C++17 ONNX Runtime serving: 4.03 ms p50 (batch 1), 231 img/s (batch 64), PyTorch↔ONNX parity to 8e-7, thread-sweep tuned (2.85x)

Overview: This project is an end-to-end computer vision pipeline that trains a ResNet-18 model from scratch on CIFAR-10, exports it to ONNX format, and serves it using a C++ ONNX Runtime binary. The pipeline ensures PyTorch and ONNX Runtime parity to an accuracy of 8e-7. Key features: - PyTorch training - ONNX export - C++ ONNX Runtime serving Tech stack: Python Why it matters: This pipeline demonstrates the parity and performance of PyTorch and ONNX Runtime, ensuring that the model's predictions are consistent and accurate across different environments. Repository: emmanueladutwum123/cv-resnet-pipeline Primary language: Python GitHub stars: 1 GitHub description: ResNet-18/CIFAR-10 → ONNX → C++17 ONNX Runtime serving: 4.03 ms p50 (batch 1), 231 img/s (batch 64), PyTorch↔ONNX parity to 8e-7, thread-sweep tuned (2.85x)

Go 1 stars

ethan-k-francis/ml-production-pipeline

End-to-end ML serving with automated drift detection

Overview: This project demonstrates an end-to-end machine learning (ML) in production setup, focusing on automated drift detection to ensure model accuracy remains reliable. Key features: - Train a model from example data - Serve predictions through a web API - Monitor incoming data against a baseline to detect drift Tech stack: Go Why it matters: This project helps developers understand how to train, serve, and monitor ML models in a production environment, specifically addressing the issue of data drift and ensuring model accuracy remains stable and reliable. Repository: ethan-k-francis/ml-production-pipeline Primary language: Go GitHub stars: 1 GitHub description: End-to-end ML serving with automated drift detection