Project directory

Projects

Open-source AI projects tracked by the leaderboard, ordered by GitHub stars.

Python 0 stars

iarsingh/multi-model-inference-platform

Python, several models, one champion

Overview: This project is an MLOps lab designed to serve multiple models with a single champion, using Python, pytest, and GitHub Actions. Key features: - Registering a second model with `POST /models` - Promoting a model with `POST /promote` - Reading the current champion with `GET /champion` - Serving models without rolling a cluster Tech stack: Python, pytest, GitHub Actions Why it matters: This project is a local proof-of-concept for MLOps, demonstrating how to register and promote models without affecting a cluster, and using pytest for testing.

Python 0 stars

igorb2800/SmarterRouter

SmarterRouter: An intelligent LLM gateway and VRAM-aware router for Ollama, llama.cpp, and OpenAI. Features semantic caching, model profiling, and automatic failover for local AI labs.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It includes semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing memory usage and providing a seamless gateway for various AI tools, ensuring efficient and stable operation in local AI labs.

Unknown 0 stars

ishandutta2007/Awesome-AI-Model-Build-Systems

Top AI Model Build Systems (Opensource) Star if you like it!

### Overview: This repository tracks notable SaaS platforms and open-source projects powering the AI Model Build Systems stack. These technologies deliver the distributed computing, cluster orchestration, GPU sharing, experiment management, and low-latency serving infrastructure necessary to build, fine-tune, and deploy foundation models and custom deep learning architectures at scale. ### Key features: - Distributed Training Frameworks - GPU Orchestrators - Model Serving Engines ### Tech stack: Open-source projects and SaaS platforms. ### Why it matters: This guide covers the entire landscape with concrete pricing, capacity metrics, and GitHub star analytics, making it useful for anyone managing multi-node H100/B200 clusters on Kubernetes, seeking serverless GPU inference, or designing self-hosted post-training pipelines.

Unknown 0 stars

ishandutta2007/Awesome-Cloud-AI-LLM-Service

Top Cloud AI LLM Service (Opensource) Star if you like it!

Overview: This repository is a curated directory of top Cloud AI platforms, enterprise LLM APIs, high-throughput inference engines, gateways, and AI observability tools. Key features: - Model inference & serving engines (vLLM, SGLang, Ollama, llama.cpp, TensorRT-LLM) - Cloud AI platforms & frontier model APIs (OpenAI API, Anthropic Claude, Google Cloud Vertex AI, AWS Bedrock, Azure OpenAI) - LLM proxies & multi-provider gateways (LiteLLM, Portkey Gateway, RouteLLM) - AI observability, tracing, & LLMOps (Langfuse, Arize Phoenix, OpenLIT) - Vector databases for enterprise RAG (Qdrant, Milvus, Chroma, Weaviate) Tech stack: Python, JavaScript, TypeScript, Docker, Kubernetes, AWS, Azure, Google Cloud, Anthropic, OpenAI, Qdrant, Milvus, Chroma, Weaviate, Langfuse, Arize Phoenix, OpenLIT, LiteLLM, Portkey Gateway, RouteLLM, vLLM, SGLang, Ollama, llama.cpp, TensorRT-LLM, Microsoft Azure, Alphabet Google Cloud, Amazon Web Services, OpenAI API, Anthropic Claude, Google Cloud Vertex AI, AWS Bedrock, Azure OpenAI. Why it matters: This repository provides essential metrics and insights into the current state of the Cloud AI and enterprise LLM services market, helping developers and architects make informed decisions.

Unknown 0 stars

ishandutta2007/Awesome-Cloud-ML-APIS

Top Cloud ML APIS (Opensource) Star if you like it!

Overview: This repository is a curated catalog of the best Cloud Machine Learning APIs, serverless inference endpoints, managed model serving platforms, and open-source AI infrastructure frameworks. It aims to provide a comprehensive overview of the current landscape and market dynamics in the Cloud AI and Machine Learning API sector. Key features: - Comprehensive comparison of top-tier cloud machine learning platforms, serverless inference APIs, and managed endpoint services. - Sorted by company scale (market cap / latest valuation in descending order). - Includes detailed information on pricing, free tiers, and trial limits. Tech stack: The repository is built using Markdown, GitHub, and various open-source tools and frameworks. Why it matters: This repository serves as a valuable resource for developers, researchers, and anyone interested in staying updated on the latest advancements and market trends in the Cloud AI and Machine Learning API sector. It provides a curated list of the best resources and platforms available, helping users make informed decisions and stay ahead in this rapidly evolving field.

Unknown 0 stars

ishandutta2007/Awesome-Deep-Learning-Deployment

Top Deep Learning Deployment (Opensource) Star if you like it!

### Overview: This repository provides a comprehensive directory of the best tools, frameworks, and hosted platforms for **Deep Learning Deployment**, **LLM Inference Optimization**, **Model Graph Compilation**, and **Production Model Serving** across Cloud, Kubernetes, and Edge hardware. ### Key Features: - Model Serving - Inference Engines - Model Compilers - Edge & On-Device ### Tech Stack: Open-source. ### Why it Matters: This repository offers a production-tested directory for deploying trained deep learning and LLMs into production with minimal latency, maximum token throughput, and cost-efficient hardware utilization. ### Repository: ishandutta2007/Awesome-Deep-Learning-Deployment ### GitHub Stars: 0 ### GitHub Description: Top Deep Learning Deployment (Opensource) Star if you like it!

Unknown 0 stars

ishandutta2007/Awesome-Enterprise-AI-Machine-Learning-Platform

Top Enterprise AI Machine Learning Platform (Opensource) Star if you like it!

### Overview: Enterprise AI and ML platforms empower global organizations to build, fine-tune, orchestrate, deploy, observe, and govern machine learning and generative AI workloads at mission-critical enterprise scale. ### Key Features: - **Data Preparation & Feature Stores**: High-throughput ingestion, point-in-time correctness, and unified feature access across offline training and online low-latency inference. - **AutoML & Model Training**: Neural Architecture Search (NAS), automated hyperparameter optimization, distributed multi-node GPU clusters, and reproducible training pipelines. - **Model Packaging & Registry**: Containerized artifact versioning, lineage tracking, experiment cataloging, and compliance auditability. - **Model Serving & High-Throughput Inference**: Dynamic batching, PagedAttention, KV cache management, multi-protocol serving (gRPC, REST), and Kubernetes autoscaling. - **Observability, Governance & Safety**: Data drift detection, concept drift tracking, Hallucination/Safety firewalls, and comprehensive security measures. ### Tech Stack: Open-source projects curated for enterprise AI/ML platforms, sovereign MLOps infrastructure, AutoML frameworks, feature stores, and high-performance model serving engines. ### Why It Matters: These platforms enable organizations to build, fine-tune, orchestrate, deploy, observe, and govern machine learning and generative AI workloads at enterprise scale, ensuring mission-critical performance and compliance.

Python 0 stars

ishandutta2007/Awesome-Kubernetes-ML-Inference

Top Kubernetes ML Inference (Opensource) Star if you like it!

### Overview: This repository is a curated list of notable Kubernetes ML inference tools, focusing on both SaaS platforms and open-source projects. It provides a comprehensive comparison of model-serving platforms and LLM serving engines, covering everything from classical predictive models to large language models (LLMs) and GPU optimization for production inference at scale. ### Key features: - **SaaS Hosted Platforms**: Compare pricing and free tiers for popular SaaS platforms like Google Vertex AI, AWS SageMaker, Azure Machine Learning, and others. - **Open-source GitHub Projects**: Rank open-source projects based on GitHub stars, including vLLM, SGLang, KServe, Triton, BentoML, and 18 more. - **GPU Optimization**: Focus on projects that optimize for GPU usage, ensuring efficient inference on Kubernetes clusters. - **Production Inference**: Tools that support production-grade autoscaling and intelligent request routing for Kubernetes environments. ### Tech stack: Python ### Why it matters: This repository is essential for developers looking to deploy, scale, and serve machine learning models on Kubernetes. It provides a clear, up-to-date guide to the best tools and projects available, helping teams make informed decisions for their Kubernetes ML inference needs.

Unknown 0 stars

ishandutta2007/Awesome-LLM-Application-Architectures

Top LLM Application Architectures (Opensource) Star if you like it!

Overview: This repository is a curated list of SaaS products and open-source GitHub projects focused on LLM application architectures, including model serving, inference optimization, and production RAG systems. It provides comprehensive information on pricing, free tiers, company sizes, and GitHub stars to help developers find the best tools for their needs. Key features: - Model Serving - Inference Optimization - Production RAG Systems Tech stack: Open-source emphasis Why it matters: This repository is essential for developers looking to optimize their LLM applications, ensuring they have the best tools and resources available for deployment and scalability.

Unknown 0 stars

ishandutta2007/Awesome-Machine-Learning-Platform

Top Machine Learning Platform (Opensource) Star if you like it!

### Overview: This repository is a curated directory of notable commercial machine learning platforms and open-source projects that cover the entire ML lifecycle, from data preparation and experiment tracking to model deployment, monitoring, and governance. It aims to provide a comprehensive and vendor-neutral view of the industry. ### Key Features: - 🚀 Enterprise SaaS Machine Learning Platforms - 🚀 Open-Source MLOps Infrastructure - 🚀 Feature Stores - 🚀 Experiment Tracking & Model Registry - 🚀 Distributed Computing & Training - 🚀 Feature Stores & Data Versioning - 🚀 Model Serving & Inference Infrastructure - 🚀 AutoML & Model Optimization - 🚀 Data Annotation & Labeling ### Tech Stack: Python, Docker, Kubernetes, AWS, Azure, Google Cloud, Apache Spark, TensorFlow, PyTorch, Scikit-learn, XGBoost, LightGBM, Hugging Face Transformers ### Why It Matters: This repository is essential for developers looking to understand and implement the full ML lifecycle without being tied to a specific vendor. It provides a clear, vendor-neutral overview of the industry's leading tools and platforms. ### Repository: Repository: ishandutta2007/Awesome-Machine-Learning-Platform ### GitHub Stars: GitHub stars: 0 ### GitHub Description: Top Machine Learning Platform (Opensource) Star if you like it! ### Table of Contents: - 📊 Market Insights & Sector Fragmentation - 🏢 SaaS & Hosted

Unknown 0 stars

ishandutta2007/Awesome-Managed-Enterprise-Dedicated-Platform-Plans

Top Managed Enterprise Dedicated Platform Plans (Opensource) Star if you like it!

Overview: The **Awesome Managed Enterprise & Dedicated Platform Plans** repository is a curated list of enterprise GPU cloud infrastructure, managed dedicated model endpoints, and open-source Kubernetes inference orchestrators. Key features: - Guaranteed GPU throughput - Enterprise SLAs and predictable billing - Data privacy and VPC peering - Self-hosted Kubernetes alternatives Tech stack: None specified Why it matters: This repository provides a comprehensive solution for organizations looking to move generative AI and deep learning workloads from prototype to mission-critical production, offering dedicated GPU capacity, enterprise SLAs, and secure, private network isolation.

Unknown 0 stars

ishandutta2007/Awesome-Mlops-Machine-Learning-Platform

Overview: This repository is a curated list of top SaaS platforms and open-source tools designed for modern MLOps and Machine Learning Platforms. It covers everything from data versioning and hyperparameter tuning to continuous training pipelines, feature stores, model registry, monitoring, and scalable model serving. Key features: - SaaS & Hosted Enterprise ML Platforms - Open-Source GitHub Projects & Frameworks - How to Contribute - Support & Community - Disclaimer Tech stack: Unknown Why it matters: This repository provides a comprehensive directory of essential tools and platforms for managing the complete enterprise machine learning lifecycle efficiently. Repository: ishandutta2007/Awesome-Mlops-Machine-Learning-Platform GitHub stars: 0 GitHub description: No description provided.

Unknown 0 stars

ishandutta2007/Awesome-Model-Inference-Servers

Top Model Inference Servers (Opensource) Star if you like it!

Overview: This repository is a curated directory of high-performance Large Language Model (LLM) serving engines, open-source inference runtimes, and managed SaaS AI silicon platforms. It tracks the latest developments in modern AI inference, focusing on algorithmic optimizations that address GPU memory bandwidth and KV cache utilization. Key features: - PagedAttention: Virtual memory paging for dynamic KV cache allocation to eliminate memory fragmentation. - RadixAttention & Prefix Caching: Tree-structured KV cache sharing across multi-turn agent conversations and RAG system prompts. - Continuous & In-Flight Batching: Injecting new requests at token boundaries instead of waiting for batch completion. - Speculative Decoding & Medusa: Predicting future tokens in parallel with verification by the target foundational LLM. - Custom AI Accelerators & LPUs: Delivering ultra-low first-token latency and high throughput with domain-specific silicon. Tech stack: Unknown Why it matters: This directory provides a comprehensive overview of the latest advancements in AI inference, helping developers and researchers stay informed about cutting-edge technologies and optimizations.

Unknown 0 stars

ishandutta2007/Awesome-Model-Serving-Platform

Top Model Serving Platform (Opensource) 🌟 Star if you like it! 🌟

### Overview: This repository is a curated list of SaaS products, cloud managed services, and open-source GitHub projects focused on model serving, LLM inference, GPU optimization, and Kubernetes AI deployment. It aims to provide a comprehensive overview of the market and industry trends in the model serving and inference space. ### Key Features: - **Model Serving Platforms**: Comprehensive comparison of leading commercial and managed model serving platforms, including pricing, free tiers, and managed services. - **Open-Source Projects**: Curated list of open-source GitHub projects for model serving, LLM inference, GPU optimization, and Kubernetes AI deployment. - **How to Contribute**: Guidelines on how to contribute to the project, including issues, pull requests, and documentation. - **Support & Sponsorship**: Information on how to support the project financially. - **Star History**: Track of the project's star count on GitHub. - **Disclaimer**: A statement to clarify the project's purpose and limitations. ### Tech Stack: The project uses Markdown, GitHub, and other open-source tools for its documentation and management. ### Why it Matters: The dedicated **Model Serving & AI Inference Platform Market** is estimated at **$2.9 Billion – $3.06 Billion in 2026**, growing at a CAGR of ~43.3% toward 2030. The market is moderately to highly fragmented, with tech giants dominating enterprise cloud infrastructure, specialized serverless platforms capturing high developer mindshare, and open-source engines preventing a single "winner-take-all" outcome.

Unknown 0 stars

ishandutta2007/Awesome-Multimodal-AI-Frameworks

Top Multimodal AI Frameworks (Opensource) Star if you like it!

### Overview: Multimodal AI Frameworks are the cutting-edge infrastructure for AI, orchestrating models that perceive, cross-reason, and synthesize information across text, high-resolution imagery, streaming audio, continuous video, and physical robotic actions. ### Key features: - Serving runtimes for unified omnimodal models - Vision-Language Models (VLMs) - Omnimodal Orchestration - Agent-Native Execution Frameworks ### Tech stack: Python, JavaScript, Docker, Kubernetes, and various machine learning frameworks. ### Why it matters: Empowering researchers, systems engineers, and founders to build, deploy, and scale unified omnimodal models across text, vision, audio, video, robotics, and actions. Repository: ishandutta2007/Awesome-Multimodal-AI-Frameworks GitHub stars: 0 GitHub description: Top Multimodal AI Frameworks (Opensource) Star if you like it!

Unknown 0 stars

ishandutta2007/Awesome-Serverless-GPU-AI-Compute

Top Serverless GPU AI Compute (Opensource) Star if you like it!

### Overview: The **Awesome Serverless GPU & AI Compute** project is a curated directory of the best serverless GPU and AI compute resources, platforms, and frameworks. It aims to help developers find the most suitable tools and services for their needs. ### Key Features: - **Top Open-Source GitHub Projects:** A directory of the most popular and useful open-source projects in the serverless GPU and AI compute space. - **Recommended Architecture Stack for Self-Hosting:** A recommended architecture stack for those looking to host their own GPU and AI compute infrastructure. - **Star History:** A visual representation of the project's popularity on GitHub, with a star count of 0 stars indicating the project is not yet starred by the community. - **Support & Community:** Information on how to support the project and engage with the community. - **Disclaimer & Notes:** A brief disclaimer and notes about the project's purpose and usage. ### Tech Stack: The project is built using the following technologies: - Markdown - GitHub ### Why It Matters: This directory is essential for developers looking to optimize their serverless GPU and AI compute infrastructure. It provides a comprehensive overview of the best resources and tools available in the market, helping users make informed decisions and stay up-to-date with the latest developments in the field.

JavaScript 0 stars

jdelles/ai-infra-learning

AI infrastructure course of study

### Overview: This project is an in-depth study of AI infrastructure, focusing on the foundational components of model serving, MLOps, and GPU systems. It serves as a guide for understanding these areas at a lower level than typical day-to-day work requires. ### Key Features: - ML Pipelines: Feature Stores, Training Orchestration, Model Registry - Experiment Tracking: MLflow, W&B - CI/CD for ML: Testing Models, Data Validation, Deployment Gates - Monitoring: Drift Detection, Logging Predictions, Alerting - Serving Patterns: Batch vs. Real-Time, Synchronous vs. Async - ONNX and Model Optimization - Triton Inference Server Deep Dive - vLLM Architecture, Continuous Batch, PagedAttention - Benchmarking: Latency, Throughput, Cost - Kubernetes & Cloud-Native ML: KServe, Seldon, Kubeflow Pipelines - GPU Node Management: Taints, Tolerations, Resource Quotas - Distributed Inference with Ray Serve and Distributed Training: DDP, FSDP, Tensor Parallelism ### Tech Stack: JavaScript ### Why It Matters: Understanding AI infrastructure is crucial for developing robust and scalable machine learning systems. This project provides a comprehensive overview of key components and practices, essential for anyone looking to build and manage AI models effectively.

Python 0 stars

jkaneko80/SmarterRouter

SmarterRouter: An intelligent LLM gateway and VRAM-aware router for Ollama, llama.cpp, and OpenAI. Features semantic caching, model profiling, and automatic failover for local AI labs.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for local AI labs, featuring semantic caching, model profiling, and automatic failover. Key features: - Semantic caching for faster response times - Model profiling for optimal performance - Automatic failover for seamless switching Tech stack: Python Why it matters: SmarterRouter optimizes local AI labs by enhancing performance and reliability through advanced features like semantic caching and automatic failover, making it a valuable tool for developers and researchers.

Python 0 stars

jmweb-org/servectl

Serve a model file over HTTP with health and Prometheus metrics.

Overview: `servectl` is a command-line utility designed to serve a model file over HTTP, providing health and Prometheus metrics. It simplifies the process of deploying models by eliminating the need for writing a FastAPI app each time. Key features: - Serve a model file over HTTP - Include health and Prometheus metrics - Load any joblib/pickle artifact that exposes a scikit-learn-style `predict` method Tech stack: Python Why it matters: `servectl` streamlines the deployment of models by providing a fast and easy-to-use solution, eliminating the need for manual setup and simplifying the process of serving models over HTTP with built-in health checks and Prometheus metrics.

Python 0 stars

jonswain/ml-model-api

A FastAPI for hosting ML models

Overview: This project is a FastAPI-based machine learning model API designed to predict LogS. Key features: - Real-time prediction capabilities - Easy integration with machine learning models - Dockerized for easy local development and deployment Tech stack: Python Why it matters: This project simplifies the process of deploying machine learning models, making it accessible for developers and researchers alike. Repository: jonswain/ml-model-api Primary language: Python GitHub stars: 0 GitHub description: A FastAPI for hosting ML models