Projects

### Overview This production-oriented machine learning platform ranks financial experiences such as credit card offers, loan promotions, and personalized financial insights based on historical engagement signals and contextual features. ### Key Features - Historical Engagement Features - Contextual Features - TensorFlow Modeling - Automated Validation - Real-Time Serving - Load Testing ### Tech Stack Python ### Why It Matters The system addresses the challenge of determining which financial experience is most relevant to a customer, enabling banks to show the most engaging offers in real-time and improve customer engagement and retention.

Overview: Fadenstack is an open-source, self-hosted AI platform designed for teams, providing a unified AI system that includes chat, GPU cluster management, and integration with apps and agents through an OpenAI-compatible LLM gateway. Key features: - Chat for team collaboration - GPU cluster management - Integration with apps and agents via an OpenAI-compatible LLM gateway - Support for multiple languages including English, French, German, Spanish, and Chinese - CLI for installation, upgrades, and backups Tech stack: Python Why it matters: Fadenstack simplifies the management of AI resources and integrates seamlessly with existing teams, making it easier to deploy and manage AI models within an organization. Repository: fadenstack/fadenstack GitHub stars: 2 GitHub description: Open-source, self-hosted AI platform for teams. Run private AI chat, manage Ray/vLLM GPU clusters, and connect apps and agents through an OpenAI-compatible LLM gateway and MCP.

Overview: This is an asynchronous FastAPI service designed for real-time machine learning predictions and text embeddings, with a target latency of sub-200 ms. It loads the model once, runs inference in a thread pool, and includes features like strict validation, built-in observability, and support for Docker deployment. Key features: - Async FastAPI endpoints for `/embed`, `/predict`, `/similarity`, `/health`, `/stats` - Real-time predictions with zero-shot text classification or a trained scikit-learn classifier - Sentence embeddings as 384-dimensional L2-normalised vectors (`all-MiniLM-L6-v2`) - Sub-200 ms target for model loading and inference - Built-in observability with `X-Process-Time-Ms` and `X-Request-ID` headers, and p50/p95/p99 latency tracking - Strict validation with Pydantic v2 schemas and batch-size/text-length limits - Safe by design with inference timeout (HTTP 504) and consistent JSON error responses - Tested with a pytest suite that runs without downloading the model - Load-test tooling with Locust file and benchmark script - Container-ready with Dockerfile and health check Tech stack: Python, FastAPI, PyTorch, Docker, Pydantic v2, SentenceTransformer Why it matters: This service is optimized for real-time predictions and text embeddings, ensuring low latency and high performance, making it suitable for applications requiring fast and reliable machine learning services.

Overview: This project is a production-grade machine learning pipeline designed for predicting the likelihood of a user completing a target action, such as submitting an auto-booking request, during a session on a website. Key features: - Use of CatBoost for binary classification due to the high imbalance between the classes (97% negative class / 3% positive class). - Integration of FastAPI for serving the model, Docker for infrastructure, and MLflow for tracking and tuning. - Production-grade pipeline with ROC-AUC of 0.879 on imbalanced data. - Includes PostgreSQL for data storage, Prometheus for monitoring, and pytest for testing. Tech stack: Python, CatBoost, FastAPI, Docker, Airflow, MLflow, Optuna, Hydra, PostgreSQL, Prometheus, pytest. Why it matters: This pipeline is designed to predict the probability of a user completing a target action during a session, leveraging the high imbalance in the classes and the need to rank sessions based on the probability of conversion rather than just classifying. Repository: fenderfeniks/sber_autopodpiska Primary language: HTML GitHub stars: 2 GitHub description: Production-grade ML-пайплайн предсказания целевого действия в сессии: CatBoost + FastAPI + Airflow + MLflow, ROC-AUC 0.879 на несбалансированных данных

### Overview: This project is an OpenAI-compatible LLM serving gateway built with FastAPI, vLLM, Redis quotas, Prometheus/Grafana, GPU metrics, benchmarks, Docker, Kubernetes, Helm, GitOps, Terraform, and CI/CD. It provides a stable API contract, quotas, routing, metrics, deployment automation, and release discipline, making it a valuable tool for teams needing an LLM serving gateway. ### Key Features: - **API Surface**: `/v1/models`, `/v1/chat/completions`, OpenAI-compatible request/response schemas. - **Streaming**: Server-Sent Events proxying with time-to-first-token measurement. - **Backends**: No-GPU mode, vLLM mode, mock backend. - **Quotas and Limits**: Redis-backed RPM/TPM/concurrency limits. - **Metrics**: Prometheus, Grafana, GPU telemetry. - **Deployment**: Docker, Kubernetes, Helm, GitOps, Terraform, Argo CD, CI/CD. ### Tech Stack: - Python - FastAPI - vLLM - Redis - Prometheus - Grafana - Docker - Kubernetes - Helm - GitOps - Terraform - CI/CD ### Why It Matters: This project demonstrates how to build, operate, benchmark, secure, and release an LLM serving gateway without pretending to be a full enterprise GPU scheduler. It shows how to implement API surface, streaming, backends, quotas, metrics, deployment, and release discipline, making it a valuable tool for AI platform engineering,

### Overview: Welcome to the **Awesome Agent Deployment Platform** repository! This curated list features production-grade SaaS platforms, serverless GPU providers, and open-source model serving frameworks designed for hosting, scaling, and deploying Large Language Models (LLMs), AI agents, and Machine Learning workflows. ### Key Features: - **SaaS / Hosted Platforms**: Compare top managed AI deployment platforms based on company valuation and market size. - **Open-Source GitHub Projects**: Discover and contribute to open-source projects that support AI agent hosting and deployment. - **Contribute**: Share your expertise by contributing to the repository. - **Star History**: See how others have embraced this platform. - **Support & Sponsorship**: Show your support and consider sponsoring the project. ### Tech Stack: - Python, JavaScript, Docker, Kubernetes, AWS, GCP, Azure, and more. ### Why It Matters: - **Market Insights**: The global AI Infrastructure & Agent Deployment Platform market is growing rapidly, with significant fragmentation among leading players. - **Developer Tools**: Provides a comprehensive ecosystem for hosting, scaling, and deploying AI models and agents efficiently. --- Repository: ishandutta2007/Awesome-Agent-Deployment-Platform

Overview: This repository is a curated list of top Foundation Model Platforms and open-source projects, focusing on SaaS and open-source solutions for hosting, fine-tuning, and serving large language models and multimodal foundation models. Key features: - Commercial SaaS platforms: Together AI, Fireworks AI, Mosaic AI, Groq, Baseten - Open-source GitHub projects: vLLM, Ollama, vLLM, Mosaic AI, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI, Groq, Baseten, vLLM, Ollama, Together AI, Fireworks AI,

Overview: MLServe is a production ML inference layer for Elixir, designed to handle concurrency, isolation, and dynamic upgrades without downtime. Key features: - Supervision of models - Worker pools for efficient handling of concurrent requests - Dynamic batching to optimize resource usage - Caching of model results for faster service - Telemetry for monitoring performance - Zero-downtime model upgrades for seamless versioning Tech stack: Elixir, Telemetry Why it matters: MLServe simplifies the process of deploying and managing machine learning models in a production environment, ensuring high availability and efficient resource utilization.

Overview: This project is an end-to-end ML lifecycle platform designed for reproducible training, policy-based promotion, progressive delivery, and rollback. It leverages MLflow as the control plane and integrates with both local and hosted GCP staging environments. Key features: - Reproducible training - Policy-based promotion - Progressive delivery - Rollback dry-run Tech stack: Python Why it matters: This platform enables developers and operators to manage the entire lifecycle of machine learning models efficiently, ensuring consistency and reliability across different environments. Repository: jellewillekes/ml-lifecycle-platform Primary language: Python GitHub stars: 2 GitHub description: End-to-end ML lifecycle platform with reproducible training, policy-based promotion, progressive delivery, and rollback. README excerpt: # ML Lifecycle Platform [](https://github.com/jellewillekes/ml-lifecycle-platform/actions/workflows/ci.yml) [](https://github.com/jellewillekes/ml-lifecycle-platform/actions/workflows/e2e.yml) [](https://github.com/jellewillekes/ml-lifecycle-platform/actions/runs

Overview: Sluice is a GPU inference platform designed for Kubernetes, enabling model scaling from zero and burst-proof intake. It uses a queue-driven approach to manage GPU resources, ensuring that workers self-terminate when the queue drains, and traffic bursts deepen the queue instead of failing. Key features: - **Queue-driven intake**: Traffic bursts deepen the queue, avoiding 503 errors. - **Self-terminating workers**: Workers own their lifecycle, and the control plane only scales up and reaps exited pods. - **Scheduler-placed GPU pods**: Workers run on normal pod specs with GPU selectors and tolerations. - **State-aware scaling**: Holds GPU stockouts and surfaces reasons for unschedulable pods. - **Ordered, multi-cluster placement**: Workers can place across many clusters and clouds. - **GPU packing**: Model replicas are packed on one GPU, and the `worker.instances` config option allows more replicas on a single GPU. - **Interface-first & config-driven**: Queue, ObjectStore, AppRegistry, Cache, InferenceObjects are Protocol drivers selected by config. Tech stack: Python Why it matters: Sluice simplifies GPU inference management, ensuring efficient resource allocation and robust scaling, making it a valuable tool for developers working with Kubernetes environments.

### Overview: The AI Runtime Platform is an OpenAI-compatible execution plane on Kubernetes, designed for efficient and cost-effective AI inference. It leverages MCP, vLLM, and policy to ensure optimal performance and cost management. ### Key Features: - **OpenAI-compatible Gateway**: Routes requests based on control plane decisions, ensuring health, latency, and cost-aware routing. - **Control Plane Decisions**: Enforces execution-plane concerns directly on the request path, enhancing security and reliability. - **vLLM Serving**: Provides GPU-serving examples for vLLM, demonstrating advanced inference capabilities. - **KServe Inference Service**: Facilitates scalable inference with KEDA, ensuring efficient resource utilization. - **OpenTelemetry**: Integrates trace and metric collection for improved observability and performance monitoring. - **GitOps Integration**: Enables seamless deployment and management with Argo CD. - **CI/CD Pipeline**: Automates the release process, ensuring consistent updates and testing. ### Tech Stack: - Python 3.12+ - FastAPI - vLLM - KServe - KEDA - OpenTelemetry - GitOps ### Why It Matters: - A gateway should decide *how* to execute, not just proxy to one backend, ensuring better performance and security. - Execution-plane concerns are closely tied to the AI Infrastructure Control Plane, providing a cohesive and secure execution environment. ### Repository: - Repository: `justrunme/ai-runtime-platform` - GitHub Stars: `2` - GitHub Description: "OpenAI-compatible AI execution

**Overview:** This project focuses on design and architecture in large-scale data mining, including topics such as data engineering, data integration, data storage, data processing, and data exploration. **Key features:** - Data Engineering - Data Processing - Data Integration - Data Storage - Data Modeling - Data Exploration - Data Mining - Large Scale Machine Learning - Data Mining Platforming **Tech stack:** Python, SQL, Apache Spark, TensorFlow, and Apache Hadoop. **Why it matters:** This project is crucial for developers looking to enhance their skills in large-scale data mining and design, providing a comprehensive approach to handling and analyzing large datasets effectively.

Overview: A local FastAPI serving platform for studying request scheduling and explicit token-level inference on a Hugging Face causal model. Key features: - Real prefill plus one-token decode using Hugging Face `DynamicCache` - Continuous waiting/running sequence sets with iteration-level admission and compatible real-model prefill/decode batches - Fixed-block KV capacity reservation, ownership tables, reference counts, watermarks, and exhaustion handling - Experimental physical PyTorch K/V pages with block-table gather and reference attention - Bounded radix KV snapshot lookup with LRU eviction, exact bypass, and suffix-only prefill - Prefill-first, decode-first, balanced, and optional real chunked-prefill execution - Model-aware context-window validation and optional startup warm-up - No-batching and dynamic micro-batching schedulers - Queue, concurrency, prompt, output, and deadline limits - Cancellation-aware request futures and graceful queue shutdown - Cancellation-aware interleaved SSE streaming with token IDs, indices, finish reason, usage, JSON/Prometheus metrics, W3C trace correlation, and structured request logs - Repeatable benchmark clients, Docker, and one-pod kind manifests - Typed CPU/CUDA worker abstractions, deterministic simulated GPU execution, and an in-memory worker registry with rich inventory and staged suspect/unhealthy health - Round-robin and least-queue global routing with structured decision explanations - Supervised worker heartbeats and token-bounded local worker execution - Residency-aware, memory-safe, and estimated-completion placement policies - Atomic local

Overview: Frontier Signal is a cutting-edge AI engineering radar designed specifically for AI engineers. It automatically tracks the latest developments across leading research labs, engineering blogs, arXiv, and Hacker News, filtering, summarizing, and delivering the most relevant updates in a concise and organized manner. Key features: - Filters and ranks content based on relevance - Automatically removes duplicates and generates concise summaries - Supports multiple delivery channels including Lark, Slack, Discord, Telegram, and terminal - Focuses on AI engineering, including agent systems, model serving, evaluations, context engineering, memory, MCP, tool use, AI coding, RAG, RSI, distributed training, and inference, while excluding funding, policy, and generic tech news Tech stack: Python Why it matters: Frontier Signal provides a comprehensive and efficient way for AI engineers to stay informed about the latest advancements in AI and related fields, ensuring they receive only the most relevant and impactful updates. Repository: leungll/FrontierSignal Primary language: Python GitHub stars: 2 GitHub description: An AI engineering radar for frontier research, tools, and systems - filtering, ranking, summarizing, and delivering the updates that matter every day. [简体中文](./README.zh-CN.md) · [System design](./docs/design.md) · [Contributing](./CONTRIBUTING.md)  ![Python](https://img.shields.io/badge/Python-3.11+-3776

Overview: This project demonstrates how to use Ray Serve for serving machine learning models, specifically focusing on binary classification using Wine Quality dataset. Key features: - Using Ray Serve for model serving - Binary classification with Wine Quality dataset - Implementation using Python Tech stack: Python Why it matters: This project showcases the practical application of Ray Serve, a tool for deploying machine learning models, in a real-world scenario, making it useful for developers looking to deploy their models efficiently. Repository: mpolinowski/ray-serve-model Primary language: Python GitHub stars: 2

### Overview: This project is a deep learning-based AI inference application designed for lithium-ion battery remaining useful life (RUL) prediction. It integrates CEEMDAN-based signal decomposition, Transformer/DNN representation learning, BMAML-SVGD-style few-shot adaptation, and FastAPI inference APIs with a React dashboard. The system allows for cycle-level RUL prediction, SoH and degradation trend visualization, uncertainty band, comparison of representative test batteries, precomputed prediction loading, live reinference, and feature importance/explainability views. ### Key Features: - **CEEMDAN-Based Signal Decomposition**: Separates battery degradation signals into local fluctuation, long-term aging trend, and residual trend. - **Transformer/DNN Representation Learning**: Learns complex patterns from sequence and summary features. - **BMAML-SVGD Few-Shot Adaptation**: Enables long-range predictions from limited early-cycle observations. - **FastAPI Inference APIs**: Provides real-time predictions and live re-inference capabilities. - **React Dashboard**: Displays RUL, SoH, trend, uncertainty, and live re-inference. - **Precomputed Prediction Loading**: Facilitates quick access to prediction results. - **Feature Importance/Explainability Views**: Provides insights into model predictions. ### Tech Stack: - Python - FastAPI - Transformers - DNN - CEEMDAN - BMAML-SVGD - React - SHAP ### Why It Matters: This project addresses the challenge of making long-range RUL predictions from limited early-cycle observations, demonstrating the potential of deep

### Overview: Opod is a self-hosted AI gateway that allows you to serve multiple GPUs from a single endpoint, making it ideal for teams using GPUs. It supports various GPU vendors and models, ensuring compatibility with different hardware configurations. ### Key Features: - **Engine-agnostic**: Supports models from llama.cpp, vLLM, SGLang, MLX-LM, and Ollama. - **Load-aware routing**: Automatically routes traffic based on load to optimize performance. - **Model sharding**: Shards models across multiple GPUs to distribute the workload. - **API keys and Prometheus/OpenTelemetry integration**: Provides secure access and monitoring tools. - **Go-based technology**: Built on Go, ensuring high performance and reliability. ### Tech Stack: - **Go**: The primary programming language used for development. - **Apache-2.0 license**: Ensures the project remains open-source and compliant with copyright laws. ### Why It Matters: Opod simplifies the process of deploying multiple GPUs for AI inference, making it easier and more efficient for teams to leverage their existing hardware without the need for additional infrastructure. The engine-agnostic nature and load-aware routing ensure that the system can adapt to changing workloads, optimizing performance and resource utilization.

### Overview: ModelSpec is an open, declarative specification designed to capture and validate the operational requirements of AI models, particularly LLMs, ensuring smooth deployment and production readiness. ### Key features: - **Model Identity**: Specifies the model's family, task, framework, and precision. - **Runtime Configuration**: Details GPU type, count, batch limits, sequence length, and memory. - **Operations**: Defines serving protocols, ports, concurrency, health probes, and timeouts. - **Scaling**: Sets minimum and maximum replicas, latency targets, and RPS targets. - **Observability**: Specifies metrics, logging, and tracing expectations. - **Pipeline Dependencies**: Includes guardrails, embeddings, and RAG components for dependencies. ### Tech stack: Python ### Why it matters: ModelSpec makes critical operational knowledge explicit, machine-readable, and version-controlled, ensuring smooth deployment and production readiness of AI models, particularly LLMs. ### Repository: paralleliq/modelspec ### Primary language: Python ### GitHub stars: 2 ### GitHub description: ModelSpec is an open, declarative specification for describing how AI models especially LLMs are deployed, served, and operated in production. It captures execution, serving, and orchestration intent to enable validation, reasoning, and automation across modern AI infrastructure.

### Overview: This project introduces 4-bit GPU kernels designed specifically for training and inference with Mixture-of-Experts (MoE) models. These kernels efficiently compute matrix multiplication directly on packed weights, avoiding full-size BF16 weight copies and leveraging 4-bit precision for faster performance. ### Key Features: - **4-bit Precision**: Computes directly on packed 4-bit weights, reducing memory footprint and computational overhead. - **Efficient Launching**: Uses a single GPU launch for active experts, eliminating the need for full-size BF16 weight copies. - **Triton Support**: Includes Triton kernels for grouped NF4 and native MXFP4 matrix multiplication, training gradients, INT4 decode, FP8 attention, and streaming expert weights. - **Reference Implementations**: Ships with packers, reference implementations, and byte-level checks. - **Training with Loggetta**: Intended for use with Loggetta, an experts4bit-qlora runtime, to integrate the kernel into training models. ### Tech Stack: The project is written in Python and uses the following libraries: - **Triton**: For efficient matrix multiplication and other operations. - **Torch**: For training and inference. - **bitsandbytes**: For handling quantization and precision. ### Why It Matters: - **Performance**: Reduces memory usage and computational overhead, making it suitable for models with limited GPU resources. - **Training Efficiency**: Allows for faster training times without compromising model accuracy. - **Scalability**: Works well with models that can benefit from 4-bit precision, such

Overview: Modalkit is a Python framework designed for deploying machine learning models on Modal, a serverless infrastructure, with a standardized ML architecture, configuration-driven deployments, and production features like authentication, batch processing, and error handling. Key features: - Modal integration for serverless deployment - Authentication via Modal proxy auth - Cloud storage: S3, GCS, and R2 bucket mounting - Queue integration: Optional TaskIQ, SQS, or custom queue backends - Batch inference with configurable batch sizes - Type safety with Pydantic models - Pre-configured tooling: ruff, pre-commit - Error handling and structured logging Tech stack: Python Why it matters: Modalkit simplifies the deployment of machine learning models on Modal by providing a standardized architecture, declarative configuration, and production-ready features, making it easier for developers to integrate ML models into their serverless applications. Repository: prassanna-ravishankar/modalkit GitHub stars: 2 GitHub description: A Python framework for deploying ML models on Modal from YAML config, with auth, queues, batching and storage mounts