Projects

Overview: This project is a FastAPI-based model inference service designed for an AI internship portfolio. It includes request validation, service layer, CLI smoke tests, and unit tests. Key features: - Request validation - Pure-Python service layer - CLI smoke tests - Unit tests Tech stack: Python Why it matters: This project demonstrates separating model logic from API routes, validating request payloads, returning structured prediction responses, exposing health checks, writing tests for the inference service, and documenting example requests and outputs.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI, featuring semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing resource usage and ensuring smooth operation across different environments.

### Overview: This project focuses on deploying computer vision models with NVIDIA Triton Inference Server, specifically using YOLOv8 and EfficientNet pipelines. The goal is to build practical experience with on-premise GPU inference serving, model optimisation, ensemble pipelines, C++ backends, and benchmark-driven bottleneck analysis. The project follows a deliberate experimental arc, addressing each bottleneck and revealing the next one. ### Key features: - **Model Optimisation:** Deploying YOLOv8 and EfficientNet pipelines using ONNX and TensorRT. - **Ensemble Pipelines:** Implementing YOLOv8 and EfficientNet in an ensemble setup. - **C++ Preprocessing Backend:** Custom C++ backend for preprocessing input images. - **Benchmarking:** Real benchmark numbers for performance analysis. - **GPU Constrained Edge Setup:** Emulating industrial CV deployments with GPU-constrained inference servers. ### Tech stack: Python ### Why it matters: This project provides practical experience in deploying computer vision models on GPUs, optimising models for inference, and benchmarking performance. It simulates the kind of stack used in industrial CV deployments, making it valuable for developers looking to build robust and efficient inference systems.

### Overview: This project is a detailed exploration of the internals of a language model serving system, focusing on key components such as a KV cache, static and continuous batching, and a plain `transformers` forward pass. The system is benchmarked against vLLM, aiming to understand the performance and efficiency of LLM serving mechanisms. ### Key features: - **KV Cache**: A custom implementation for storing and retrieving model outputs. - **Static Batching**: A method for processing multiple requests in parallel. - **Continuous Batching**: Real-time processing of incoming requests. - **Benchmarking**: Stage-by-stage benchmarking against vLLM to evaluate performance. ### Tech stack: Python ### Why it matters: This project provides insights into the performance and efficiency of LLM serving mechanisms, specifically focusing on KV caching, batching, and benchmarking against vLLM. The results can help developers optimize their own LLM serving systems for better performance and resource utilization. ### Repository: Repository: simon-bouchard/llm-serving-internals Primary language: Python GitHub stars: 0 GitHub description: Hand-rolled LLM serving internals (KV cache, static and continuous batching) on a plain `transformers` forward pass, benchmarked stage by stage against vLLM. Work in progress. ### Stages: - **Stage 1 (Naive `generate()`)**: Benchmarked with Qwen2.5-0.5B-Instruct, fp16, batch size 1, 128 input / 128

### Overview: This repository is a production-ready MLOps pipeline for financial forecasting using LSTM models. It includes a selected Keras model, its training-fitted scalers, and a FastAPI service for model inference. The service is deployed with Docker, and features model versioning, monitoring with MLflow, Prometheus, and Grafana dashboards. ### Key features: - **Model Versioning**: The repository includes an MLflow model registry with explicit promotion and rollback capabilities. - **FastAPI Service**: A FastAPI endpoint for model predictions, featuring input validation, health checks, and durable SQLite audit records. - **Docker Deployment**: A Docker Compose stack for the API, MLflow, Prometheus, and Grafana, ensuring the serving container runs without root and does not require a GPU. - **Historical Error Metrics**: Provisioned dashboards with historical error metrics and feature PSI, and tested alert rules. - **Backup and Restore**: Automated numerical parity checks, a paced HTTP load test, and backup, restore, and rollback rehearsals. ### Tech stack: - **Python**: The primary programming language used in the project. - **Docker**: For containerization and deployment. - **MLflow**: For model versioning and tracking. - **Prometheus**: For monitoring and alerting. - **Grafana**: For dashboards and monitoring. - **SQLite**: For audit records. ### Why it matters: This repository provides a production-ready solution for financial forecasting, ensuring model reproducibility and monitoring through MLflow, Prometheus, and Grafana dashboards. The inclusion of

### Overview: VisionFlow Inference 是基于 NVIDIA Triton Inference Server 构建的视觉模型推理服务，为 VisionFlow 提供人脸检测与特征提取、人物特征提取、图文特征、姿态估计和图像分割等能力。通过 HTTP/gRPC 提供统一推理接口，支持按需加载模型及其依赖，提供模型来源记录、ONNX 导出和 TensorRT 推理引擎构建工具，并结合动态批处理、健康检查和 Prometheus 指标支持部署与运维；同时提供基于 OpenCV 的独立 CPU 人脸检测演示，便于快速展示项目效果。 ### Key features: - 多模型服务：支持目标检测、姿态、分割、人脸特征和图文特征等接口。 - 可复现的模型维护：记录来源、SHA256、转换配置和制品身份，支持从源权重导出 ONNX 并在目标设备上构建 TensorRT engine。 - 推理调度：按模型配置使用动态 batch；Jetson Orin 提供可选 GPU 前后处理路径。 - 可观测性：Triton 暴露健康状态和 Prometheus 指标；提供模型就绪检查和运维说明。 ### Tech stack: Python ### Why it matters: 通过 Triton Inference Server 构建的视觉模型推理服务，为 VisionFlow 提供了高效、灵活且可扩展的模型推理能力，支持多种任务如人脸检测、特征提取、姿态估计和图像分割等，满足了不同应用场景的需求。

### Overview: Switchyard is a developer-focused platform for serving ONNX models with bounded batching, canary releases, automatic rollback, and Prometheus monitoring. It features real-time inference, CPU benchmarking, and a reproducible environment. ### Key Features: - **Bounded Batching**: Ensures consistent performance with predictable queue times. - **Canary Releases**: Automatically roll back faulty versions to restore stability. - **Automatic Rollback**: Restores the champion version after observed errors or latency breaches. - **CPU Benchmarking**: Measures and records the performance of CPU-based inference. - **Sticky Hash Routing**: Ensures traffic is directed to the correct version of the model. - **Quality Gate**: Requires candidates to pass a holdout quality gate before being promoted to the production environment. - **Optimistic Revisions**: Allows for manual adjustments to the rollback policy without downtime. - **Minimal Startup**: Trains and exports models, verifies artifacts, and warms both sessions before starting. ### Tech Stack: Python ### Why It Matters: Switchyard provides a reliable and reproducible environment for model serving, ensuring consistent performance and safety through automatic rollback and canary release mechanisms. This platform is essential for developers looking to deploy and test machine learning models in a controlled and stable manner.

Overview: This repository is a reference implementation of an end-to-end MLOps pipeline for online causal inference. It demonstrates how causal machine learning models can be trained and tracked on Databricks, packaged as a web service, deployed to Kubernetes via CI/CD, and monitored in production on Microsoft Azure. Key features: - Train, deploy, and monitor causal machine learning models for online inference using Double Machine Learning on Databricks. - Serve causal models on Azure Kubernetes Service via CI/CD. - Use DoWhy and EconML Double Machine Learning (DML) for causal inference. - Implement causal forests, meta-learners, and doubly-robust learners for accurate causal estimates. Tech stack: Python, Databricks, MLflow, FastAPI, Kubernetes, Azure, Bicep, Docker, GitHub Actions, Azure Container Registry. Why it matters: This project showcases the intersection of causal inference and production MLOps, emphasizing the importance of handling causal inference tasks in a production environment, where models are not just trained but also deployed and monitored for real-time inference.

Overview: This project is a binary classifier designed to predict chess blunder probability at move 15, using XGBoost for the model. The ROC-AUC score is 0.68, and the blunder rate is approximately 14.5% per side. Key features: - `player_elo`: Player's Elo rating - `elo_diff`: Player Elo minus opponent Elo - `time_control`: Base time in seconds (600 / 900 / 1800) - `eval_at_snapshot`: Stockfish eval at move 15 from player's perspective - `eval_volatility`: Std dev of player's evals over moves 1–15 - `eval_trend`: Slope of player's last 5 evals (positive = improving) - `material_balance`: Player material minus opponent material (centipawns) - `material_imbalance`: Absolute difference in material - `king_attackers`: Number of opponent pieces attacking the player's king at move 15 - `has_castled`: Whether the player has castled by move 15 Tech stack: The project is built with Jupyter Notebook, XGBoost, FastAPI, Docker, MLflow, and Stockfish evals from Lichess open data. Why it matters: This model helps chess players and coaches identify potential blunders early, potentially improving their game strategy and performance.

### Overview: This project is a personal experiment in multi-model serving using FastAPI and Docker, focusing on hosting two domain-specific fine-tuned GPT-2 models behind a single GPU-backed endpoint. The deployment includes a GPU benchmarking suite to evaluate performance across different worker/thread configurations and client concurrency levels on NVIDIA L4 instances. ### Key Features: - **Multi-model Endpoint**: Serves two fine-tuned GPT-2 models (`domain_a` and `domain_b`) via a single `/generate/` endpoint, with the model chosen per request. - **Model Selection**: Each request specifies which model to route to, ensuring a single deployment serves both domains without cold-starting. - **Worker/Thread/Concurrency Benchmarking**: Includes a GPU benchmarking suite that load-tests the deployment across various configurations (workers, threads, concurrency) on NVIDIA L4 instances, providing insights into throughput and error rates. ### Tech Stack: - **FastAPI**: The backend API server. - **Docker**: Containerized the application for reproducibility and ease of deployment across different environments. - **Gunicorn**: Process-level parallelism for CPU-bound preprocessing, and async I/O for handling requests. ### Why It Matters: - **Single Endpoint, Multi-Model**: Simplifies operational complexity by serving multiple models through a single endpoint, reducing the need for separate services. - **Gunicorn + uvicorn workers**: Utilizes process-level parallelism for CPU-bound tasks and async I/O for I/O-bound tasks, optimizing performance. - **Two Separate Models**: Keeps the inference code simple and allows

### Overview: FastAPI ML Serving is a production-ready framework for serving multiple models with versioning, dynamic batching, A/B testing, and health checks. ### Key features: - **Multi-model support** — Serve multiple models with versioning. - **Dynamic batching** — Automatically batch requests for GPU efficiency. - **A/B testing** — Split traffic between model versions. - **Health checks** — Implement liveness and readiness probes. - **Metrics** — Provide Prometheus-compatible metrics endpoint. - **Graceful shutdown** — Cleanly drain requests during shutdown. ### Tech stack: Python ### Why it matters: FastAPI ML Serving simplifies the deployment and management of machine learning models, ensuring high performance and reliability in production environments.

Overview: This project is a real-time ML inference server built with FastAPI and WebSocket, designed for batched predictions, model hot-reloading, health monitoring, and feature caching. It supports multiple model backends and enables A/B testing. Key features: - Batched predictions - Model hot-reloading - Health monitoring - In-memory feature cache - A/B testing capabilities Tech stack: Python Why it matters: This server addresses the challenges of serving predictions at low latency, model hot-reloading, and managing model rollouts in a production-grade environment. Repository: sonimaharshi1999/realtime-inference-server Primary language: Python GitHub stars: 0 GitHub description: FastAPI + WebSocket-based real-time ML inference server: batched predictions, model hot-reload, health monitoring, and an in-memory feature cache. Supports multiple model backends (scikit-learn, ONNX) with A/B testing capabilities.

Overview: This project is a production-grade ML inference service that counts total and occupied industrial ducts in images using a multitask EfficientNet-B0 model served via FastAPI with dynamic batching and Prometheus metrics. Key features: - Production-grade ML inference service - Counts total and occupied industrial ducts in images - EfficientNet-B0 model served via FastAPI - Dynamic request batching for improved throughput - Prometheus metrics for performance monitoring - 38 tests and Locust load testing Tech stack: FastAPI, PyTorch 2.x, Docker (multi-stage), Prometheus, structlog, Locust Why it matters: This service is designed to be highly scalable and performant, making it suitable for real-world industrial applications where accuracy and speed are critical.

Overview: This project is a Flask-based Iris Classifier model serving application that uses FastAPI for the backend and Streamlit for the frontend. It automates the CI/CD pipeline using GitHub Actions, Docker, and Google Cloud Platform (GCP) VMs for deployment. Key features: - FastAPI + Streamlit for backend and frontend - 5-fold cross-validation for model selection - Automatic deployment to GCP Compute Engine instances - CI/CD pipeline with GitHub Actions - Docker image for deployment - Health checks and external validation Tech stack: Python, FastAPI, scikit-learn, Docker, GitHub Actions, GCP, Streamlit Why it matters: This project automates the entire pipeline from training, validation, and model selection to deployment, ensuring high-quality and reliable model serving.

Overview: This project provides a mock implementation for deploying models in the SPIRA Inference System, using Python as the tech stack. Key features: - A template for model classes that need to be followed for proper inference system compatibility. - Instructions on how to deploy a new model server to an Inspirada machine. - Examples of expected input and output for model predictions. Tech stack: Python Why it matters: This project serves as a mock implementation for developers to understand and practice the process of deploying models in the SPIRA Inference System, ensuring compatibility with the system's requirements and providing a clear example of the steps involved.

Overview: The Tac repository serves as a comprehensive technical compendium for Agentic AI, detailing the entire stack from token economics to production evaluation. Key features: - Tokens & Cost - Context Windows - Sampling - LLM Serving - Prompt Caching - Latency - KV Cache & Quantization - Rate Limits & Concurrency - MoE Architecture - Reasoning Models - Agent Frameworks - Orchestration - Tool Use & Function Calling - Context Management - Evaluations - Prompt Injection & Security Tech stack: VitePress Why it matters: Provides a structured reference for AI builders, offering production decision framing, technical depth, and real-world gotchas in production scenarios. Repository: srmdn/tac GitHub stars: 0 GitHub description: The Agent Stack — technical compendium for Agentic AI: tokens, inference, caching, serving, frameworks, and orchestration.

Overview: This is an end-to-end ML lifecycle platform that supports training, packaging, serving, validation, and rollback of models using scikit-learn and PyTorch. It includes a registry for versioned artifacts, FastAPI for multi-version serving, and parity checks for offline-to-online prediction accuracy. Key features: - Model training - Model packaging - Serving-time validation - Offline-to-online prediction parity check Tech stack: Python Why it matters: This platform ensures that model releases are debuggable, repeatable, and safe to ship by providing a complete lifecycle management system for ML models. Repository: srn91/ml-training-serving-platform Primary language: Python GitHub stars: 0 GitHub description: ML lifecycle platform with scikit-learn/PyTorch training, artifact registry metadata, multi-version FastAPI serving, parity checks, and monitoring.

Overview: This project is a local-first ranking system that generates query-item candidates, compares two ranking models, records experiment metadata and a comparison report, and serves top-k results through a FastAPI endpoint. Key features: - Generates query-item candidates - Compares two ranking models - Records experiment metadata and a comparison report - Serves top-k results through a FastAPI endpoint Tech stack: Python Why it matters: This project focuses on the end-to-end ranking workflow, including features, offline evaluation, artifact selection, and top-k serving, making it suitable for production-style engineering systems. Repository: srn91/ranking-serving-engine Primary language: Python GitHub stars: 0 GitHub description: Artifact-backed ranking service with query-item features, NDCG/MAP evaluation, model comparison, and FastAPI top-k serving.

### Overview: This TensorFlow Serving tutorial provides a comprehensive guide on how to export, load, version, and serve models using gRPC and REST APIs. The tutorial covers Python, Go, and Rust clients, along with monitoring and production deployment strategies. ### Key Features: - **Models Export and Inspection**: Demonstrates how to export models in SavedModel format and inspect their signatures. - **Versioning and Labels**: Explains how to manage versions and apply labels to models. - **Loading Models**: Walks through the process of loading models from config files and version policies. - **Serving**: Discusses the differences between gRPC and REST APIs, and how to configure server-side batching. - **Calling the Server**: Shows how to call the server from Python, Go, and Rust clients, including error handling and response decoding. - **Requests and Responses**: Details the structure of requests and responses, including tensor and example conversions. - **Monitoring**: Introduces Prometheus and Grafana for monitoring and provides examples of how to set up the dashboard. ### Tech Stack: - **Go**: Used for the Python client, Go client, and Rust client. - **Python**: The primary language for the client. - **Rust**: Built for the Rust client. - **TensorFlow Serving**: The tutorial uses TensorFlow 2.21.0. ### Why It Matters: - The tutorial provides a practical guide for developers to understand and implement TensorFlow Serving in various programming languages. - It covers the entire lifecycle of a model, from export to serving and monitoring, ensuring developers can deploy models efficiently

Overview: This technical journal focuses on practical examples, operational details, and testable engineering arguments related to building AI systems that can survive contact with reality. Key features: - **RAG Is a Search Problem Before It Is a Generation Problem** - **LoRA Adapters Are a Serving Primitive** - **Zeros Are Data: Forecasting Intermittent Demand** - **Agents Need Budgets, Not Infinite Loops** - **Model Routers Turn Benchmarks into Runtime Decisions** - **The Final Answer Is the Least Interesting Agent Trace** - **The Screenshot Is Becoming the New Bug Report** - **Your AI Assistant Should Be Allowed to Say “I Don’t Know”** - **Small Models Keep Winning the Boring Jobs** - **The Cloud Is a Great Place to Sync, Not to Live** - **Every Cache Is a Tiny Time Machine** - **Your 200,000-Token Prompt Is a Junk Drawer** - **Voice AI Has a Politeness Problem** - **Synthetic Data Is a Photocopier, Not a Camera** - **The Best Automation Leaves a Handle** Tech stack: TypeScript Why it matters: The publication favors practical examples, operational details, and testable engineering arguments over product announcements or news summaries. It links to the papers, documentation, and benchmarks behind its claims. Repository: sujikathir/field-notes Primary language: TypeScript GitHub stars: 0 GitHub description: Useful field notes on building AI systems that survive contact with reality.