Projects

Overview: ChronoAPI is a small, opinionated service layer built on FastAPI, OpenAPI, and Prometheus, designed for real-time forecasting using IBM's TTM model. It provides HTTP and optional Kafka endpoints for inference, fine-tuning, streaming, and operational metrics. Key features: - `POST /forecast` for configurable inference - `POST /finetune` for fine-tuning - Optional Kafka consumer for streaming - Prometheus metrics and structured logs - Optional API key and request ID headers Tech stack: Python 3.11+, FastAPI, OpenAPI, Prometheus, Docker Why it matters: TTM models are CPU-friendly, FastAPI provides type safety, and Prometheus and optional JSON logging fit standard observability stacks. Repository: fraware/chronoapi Primary language: Python GitHub stars: 1 GitHub description: A RESTful API for real-time forecasting using IBM Research’s TTM model </pre> **Forecast multivariate time series in production** using IBM Granite **Tiny Time Mixer (TTM)** models. FastAPI · OpenAPI · Optional Kafka · Prometheus · Docker-ready [](https://www.python.org/downloads/) [![FastAPI](https://img.shields.io/badge/FastAPI-API-009688?style=flat-square&logo=

Overview: Llamactl is a local-first AI infrastructure control plane designed for deploying, managing, and benchmarking models across machines and cloud providers through a CLI, web dashboard, and MCP. Key features: - Deploy, manage, and benchmark models across machines and cloud providers through a CLI, web dashboard, and MCP. - Single-machine workflow unchanged, `llamactl bench`, `llamactl server start`, `llamactl catalog list` all still work against the local node without any configuration. - Remote agents, any machine on the LAN can run `llamactl agent serve`, and the control plane drives it via HTTPS + bearer auth + pinned self-signed TLS. - Cloud + gateway nodes as first-class peers, OpenAI, Anthropic, Together, Groq, Mistral, sirius-gateway, embersynth — each shows up in kubeconfig with the same `--node` addressing as a self-hosted agent. - Declarative workloads, `ModelRun` manifests (a model bound to a node with args) and `NodeRun` manifests (an agent's desired infra stack — llama.cpp build pinned to a commit, optional sirius / embersynth sidecars) are `apply`-able, `get`-able, `delete`-able. - Infra deploy, the agent ships with a registry of known packages, side-by-side versioned layout under `~/.llamactl/infra/<pkg>/<version>/`, atomic symlink flip to activate, service-unit templates for launchd + systemd.

Overview: This project implements a fully featured Feature Store built entirely in Python, demonstrating fundamental concepts of feature management in ML pipelines. It includes feature registration with automatic versioning, online storage for real-time services, offline storage for historical point-in-time queries, encumbered pipelines, and a searchable metadata catalog with lineage tracking. Key features: - Feature Store Online (Key-Value storage for recent feature values) - Feature Store Offline (Historical storage with point-in-time query capabilities) - Encumbered Pipelines (Encapsulated transformations for feature derivation) - Metadata Catalog (Searchable catalog with metadata, tags, and lineage tracking) Tech stack: Python 3.11+, Pandas, NumPy, Matplotlib, Seaborn, pytest, Docker, Jupyter Why it matters: This project serves as an excellent educational resource to understand how Feature Stores function internally, providing a comprehensive overview of the key components and their interactions. Repository: galafis/feature-store-engineering GitHub stars: 1 GitHub description: Feature Store Engineering - Professional Python project Documentation: README.md GitHub license: MIT Overview: This project implements a fully featured Feature Store built entirely in Python, demonstrating fundamental concepts of feature management in ML pipelines. It includes feature registration with automatic versioning, online storage for real-time services, offline storage for historical point-in-time queries, encumbered pipelines, and a searchable metadata catalog with lineage tracking. Key features: - Feature Store Online (Key-Value storage for recent feature values) - Feature Store Offline (Historical storage with point-in-time

Overview: This project showcases an automated engineering workflow for a machine learning inference service, connecting model training, artifact generation, API integration, automated testing, Docker packaging, and CI execution into a single reproducible pipeline. Key features: - Automated model training - Artifact generation - API inference service - Automated testing - Docker packaging - GitHub Actions CI validation Tech stack: Python, scikit-learn, NumPy, joblib, FastAPI, Pydantic, Uvicorn, pytest, httpx, Docker, GitHub Actions. Why it matters: This project provides a comprehensive demonstration of an automated engineering workflow for machine learning inference services, ensuring that the system is robust, maintainable, and easily reproducible. Repository: gbadedata/ml-training-pipeline-ci Primary language: Python GitHub stars: 1 GitHub description: Automated ML engineering workflow connecting model training, artifact generation, API integration, testing, Docker packaging, and CI into one reproducible pipeline.

Overview: This project showcases the complete engineering lifecycle of a machine learning service, from training a model offline to deploying it as a scalable API with monitoring, containerization, CI/CD, Kubernetes, and cloud infrastructure. Key features: - Model artifact serialization - FastAPI inference API - Pydantic validation - Prometheus monitoring - Kubernetes deployment - GitHub Actions CI/CD - Docker containerization - AWS ECS Fargate deployment - Render public deployment Tech stack: Python, scikit-learn, NumPy, joblib, FastAPI, Pydantic, Uvicorn, pytest, Docker, Docker Compose, Prometheus, Grafana, Locust, GitHub Actions, GitHub Container Registry (GHCR), Amazon Elastic Container Registry (ECR), Kubernetes, Ingress NGINX, AWS ECS Fargate, Application Load Balancer, Render, AWS ECS Fargate, AWS Application Load Balancer, AWS ECS Fargate, AWS Application Load Balancer. Why it matters: This project demonstrates best practices in machine learning engineering and deployment, covering everything from model training to production-ready API deployment, ensuring a smooth and efficient transition from experimentation to production. Repository: gbadedata/ml_inference_api GitHub stars: 1 GitHub description: Production-style ML inference service taking a trained model from notebook to deployed API with monitoring, containerisation, CI/CD, Kubernetes, and cloud infrastructure.

### Overview: This project benchmarks, optimizes, and serves the TabFM tabular foundation model online. It focuses on exact KV cache, torch.compile, FastAPI, Triton, and KServe, aiming to achieve low latency and optimal performance on both a laptop and NVIDIA L4 in GKE. ### Key features: - **Exact KV Cache:** TabFM re-reads the entire training table on every request, but this can be cached exactly, reducing latency to 129 ms. - **Torch Compilation:** Using `torch.compile` and CUDA graphs, the forward pass time is reduced from 111 ms to 28 ms. - **Triton Optimization:** Triton was optimized to handle the high load, but it was found to be a bottleneck, especially with multiple client processes. ### Tech stack: Python ### Why it matters: This project demonstrates the effectiveness of exact KV caching and optimized model serving, showcasing how to efficiently serve a large tabular model online, even on a single GPU.

Overview: This project serves an AI model on GPU infrastructure using the same DevOps toolchain as the `devops-cicd-pipeline` project. Key features: - GPU Kubernetes cluster - vLLM model service - chat UI - CI/CD with GitHub Actions - GitOps with Argo CD - GPU and model observability - Automated canary release and rollback Tech stack: Terraform, Ansible, Kubernetes (k3s), Helm, KServe, Envoy Gateway, cert-manager, vLLM, Llama-3.2-3B-Instruct, Prometheus, Grafana, Loki, Alertmanager, DCGM Exporter, Flask (chat UI) Why it matters: This project demonstrates how the same DevOps toolchain can be applied to a different workload, specifically an AI model on a GPU, rather than just padding out a roadmap.

Overview: This repository is a closed-loop recommendation system template built with Python, focusing on training, evaluation, registration, drift detection, and rollout policies. It uses a numpy-only serving engine and includes features like a single configuration file, virtual environment setup, and a registry for managing models and interactions. Key features: - A single configuration file for dependency management and entry points - Virtual environment created from `pyproject.toml` and `uv.lock` - Numpy-only serving engine for the core component - Registry for model management and interactions - Drift detection and drift-checking mechanisms - Rollout policies for different deployment scenarios Tech stack: Python Why it matters: This template provides a structured approach to building recommendation systems, focusing on efficiency and minimal dependencies, making it suitable for both academic and production use cases. Repository: hoangtung386/recsys-template GitHub stars: 1 GitHub description: Closed-loop recommender system template with numpy-only serving engine, training, registry, drift detection, and rollout policies.

Overview: This project is an end-to-end MLOps reference that demonstrates the implementation of various tools and technologies in a Kubernetes environment. Key features: - MLflow for model tracking and versioning - FastAPI for serving the model with health, readiness, and metrics endpoints - Docker for containerization - Kubernetes for deployment and orchestration - Helm for application management - Prometheus and Grafana for monitoring - Terraform for infrastructure as code - GitHub Actions for CI/CD Tech stack: Python, MLflow, FastAPI, Docker, Kubernetes, Helm, Prometheus, Grafana, Terraform, GitHub Actions Why it matters: This project serves as a comprehensive guide for developers looking to implement MLOps practices using modern tools and technologies, providing a clear example of how to set up and manage a production-level MLOps platform on Kubernetes.

Overview: Locus is an engine-neutral inference control plane designed to handle placement across engines, ensuring policy-awareness and efficient resource management. Key features: - Policy-aware placement - Fleet policy enforcement - Model and state reuse optimization Tech stack: Rust Why it matters: Locus simplifies inference placement by ensuring policies are applied consistently across engines, optimizing resource usage, and providing a flexible and efficient control plane for various compute and model-state placement scenarios.

### Overview: NGramStore is a general-purpose backend for learned n-gram embeddings, designed for efficient compression, storage, and exact queries. It preserves original addressing and vector bytes, making it suitable for applications like FastText and Engram-style language models. ### Key features: - Lossless storage and efficient queries for learned n-gram memory. - Native C + PyTorch implementation. - Immutable CPU tables. - Registered string keys for DATrie indexing and structural mark-compact planning. - Numeric/hash IDs for direct use. - Shared table engine for lossless block encoding, original-ID forwarding, and bounded decoded caches. - Optional n-gram structure for prefix/suffix representative selection and mark-compact export. - Model applications for FastText word embeddings and Engram/TinyEngram LLM memory, preserving model semantics. ### Tech stack: Python ### Why it matters: NGramStore offers a shared backend for learned n-gram tables, providing byte-preserving storage and optional exact key indexing. It addresses the tradeoff between n-gram identity, capacity, and fidelity, offering a stable solution for applications like FastText and Engram-style language models.

Overview: Instill AI organization profile and default configuration repository. Key features: - Code and templates for various projects - Documentation and community health files - Reusable code snippets and templates Tech stack: Unknown Why it matters: Provides a centralized hub for Instill AI projects, facilitating code sharing, documentation, and community collaboration.

### Overview: The global AI inference and model serving infrastructure market is projected to reach over $85 billion by 2030, growing at a compound annual growth rate (CAGR) of over 35%. This rapid expansion is driven by the increasing demand for continuous real-time inference and production workflows. ### Key Features: - **Dynamic Batching**: Enables efficient handling of large batches of requests. - **Paged Attention**: Optimizes memory usage and performance for complex models. - **KV-Cache Sharing**: Reduces latency by sharing key-value pairs across multiple requests. - **Scale-to-Zero Elasticity**: Automatically scales resources up or down based on demand. ### Tech Stack: - **Python**: Used for development and integration with other tools. - **Kubernetes**: Utilized for scalable and efficient deployment of models. - **Open-Source Frameworks**: Such as BentoML and Triton, providing cost-effective solutions for engineering teams. ### Why It Matters: This project offers a curated directory of SaaS inference platforms, open-source model serving engines, serverless GPU clouds, and Kubernetes-native MLOps infrastructure, making it a valuable resource for developers looking to optimize their AI and ML models for production use. ### Repository: ishandutta2007/Awesome-ML-Model-Serving ### GitHub Stars: 1 ### GitHub Description: Top ML Model Serving (Opensource) Star if you like it!

### Overview: CustomerIQ is a customer intelligence and churn prediction platform that leverages FastAPI for API development, scikit-learn for model training, and Pydantic for strict validation. The project includes SHA-256 artifact verification, Pydantic validation, and an honest model card, making it a valuable tool for understanding and predicting customer churn. ### Key Features: - **Churn Scoring**: Returns the estimator's positive-class score unchanged. - **Strict Contracts**: Rejects unknown categories, extra fields, inconsistent telecom services, oversized batches, and non-finite numbers. - **Honest Metadata**: Exposes artifact hashes, limitations, uncalibrated status, and threshold provenance through `/model-info`. - **Protected Serving**: Verifies artifacts before deserialization and applies bounded request, rate, CORS, and safe-error controls. ### Tech Stack: - FastAPI - scikit-learn - Pydantic ### Why it Matters: CustomerIQ provides a robust and secure platform for churn prediction, ensuring that only valid and reliable data is used in its operations. This approach helps in making informed decisions and improving customer retention strategies.

Overview: OmniPredict Pro is a unified PyTorch Deep Learning & AutoML studio designed for both tabular and computer vision data. It automatically cleans, trains, optimizes, and predicts using zero manual coding. Key features: - Universal Tabular Engine - Computer Vision Ingestion - Pretrained Vision CNNs and Tabular ResNet - Bayesian AutoML (Optuna) - Live Training & Diagnostics - Interactive Live Predictions - Global Execution Lock - PyTorch 2.6+ Standalone Registry Tech stack: Python Why it matters: OmniPredict Pro simplifies the process of training and deploying neural networks on tabular and computer vision datasets, making it an essential tool for data scientists and developers.

Overview: This project is an embedding service using mosec, compatible with OpenAI's API, designed for developers. Key features: - Embedding service with mosec - Compatible with OpenAI embedding API - Supports various embedding models from the Hugging Face Embedding LeaderBoard Tech stack: Python Why it matters: This service simplifies the process of embedding data, making it easier for developers to integrate into their projects, especially when working with the OpenAI API.

Overview: This repository offers standalone latency benchmarking pipelines for Large Language Models (LLMs) running on AMD GPUs using the vLLM inference engine and AMD's Model Automation and Dashboarding (MAD) framework. Key features: - ROCm 6.3 + vLLM 0.8.5 pipeline for Aya Expanse 32B - ROCm 7.0 + vLLM 0.14 pipeline for GPT OSS Safeguard 20B Tech stack: Python Why it matters: This repository provides reproducible benchmarking pipelines and visualization scripts for evaluating the performance of LLMs on AMD GPUs, aiding in the optimization of inference times in production environments.

### Overview: StreamInfer is a real-time ML inference engine designed to bridge the gap between model performance in a notebook and its deployment in production. It automatically finds the optimal balance between throughput and latency, enabling high-concurrency inference with minimal latency. ### Key Features: - **WebSocket Streaming**: Clients send data, and predictions are returned in real-time. - **Adaptive Batching**: Accumulates requests and flushes on batch size or timeout, ensuring optimal performance. - **Backpressure**: Token bucket rate limiter and slow consumer detection prevent one bad client from overwhelming the server. - **Model Hot-Swap**: Swaps models with zero downtime via SIGHUP or API call, allowing seamless model updates without downtime. ### Tech Stack: StreamInfer is built using Python, leveraging the power of the language for efficient and scalable real-time inference. ### Why it Matters: StreamInfer simplifies the deployment of machine learning models in production environments, ensuring that models work seamlessly in both development and production settings. By dynamically batching requests and handling backpressure, it ensures that models run efficiently and predictably, even under high load. ### Architecture: - **WebSocket Streaming**: Clients send data, and predictions are returned in real-time. - **Adaptive Batching**: Requests are aggregated into batches, and batches are flushed when either the batch size is reached or the timeout is exceeded. - **Backpressure**: Rate limiting and slow consumer detection prevent one client from overwhelming the server. - **Model Hot-Swap**: Swaps models with zero downtime via SIGHUP or API call, allowing for seamless

Overview: Fornax is a self-hosted open-weight model serving solution that freezes weights, serves, and benchmarks models on a single Go binary. This approach ensures weight security and cost efficiency, allowing users to control their own GPUs and bypass third-party rate limits. Key features: - Weight freeze: Models are pinned to specific revisions and checksums, ensuring they remain secure and up-to-date. - Cost and control: Users can run their own GPUs at a lower cost compared to cloud-based solutions, and they can control their own weight storage. - Multi-GPU fleet: Supports Kubernetes for a fleet of GPUs, and an installed binary for a single-GPU host. - End-to-end inference: Provides OpenAI Chat, Anthropic Messages, and OpenAI Responses endpoints, ensuring compatibility with various client SDKs. - Real-time performance monitoring: Measures and reports latency and throughput for accurate performance metrics. Tech stack: Go Why it matters: Fornax offers a secure, cost-effective, and flexible solution for running open-weight models, allowing users to control their own GPUs and bypass third-party rate limits, making it ideal for those looking to run models on their own infrastructure. Repository: latere-ai/fornax GitHub stars: 1 GitHub description: Fornax is a self-hosted open-weight model serving solution that freezes weights, serves, and benchmarks models on a single Go binary. This approach ensures weight security and cost efficiency, allowing users to control their own GPUs and bypass third-party rate limits. Key features: - Weight freeze: Models are pinned to specific revisions and

Overview: `quantpreflight` is a tool designed to determine if a quantized AI model will run on your GPUs before downloading it. Key features: - Checks formats, kernels, sharding, tokens, files, and memory from headers only. - Answers in seconds, reads headers, not weights. - Needs no GPU, runs on a laptop or in CI. - Checks the whole mix, 16 rules cover model, format, engine, GPU, and GPU count. - Tells you what works, gives GPU count and context length for `vllm serve` command. - Tested on real hardware, predicted 26 of 27 real vLLM loads on Tesla T4. - Finds known vulnerabilities, checks vLLM or SGLang version against published advisories. - Fits your workflow, supports CLI, GitHub Action, pre-commit hook, SARIF output, and browser scanner. Tech stack: Python 3.10 or newer. Why it matters: `quantpreflight` helps developers quickly identify compatibility issues with quantized AI models, ensuring they can run efficiently on their GPUs without wasting time or resources.