Projects

Overview: This repository serves as a hands-on playground for comparing different LLM serving backends, FastAPI, vLLM, and Ollama, through an OpenAI-compatible client. Key features: - Model serving through different backends (FastAPI, vLLM, Ollama) - Client layer using raw HTTP and OpenAI SDK - Benchmarks for inference performance Tech stack: Python Why it matters: This project provides a practical way to understand and compare the performance of different LLM serving backends, making it valuable for developers looking to optimize their LLM integrations.

### Overview: gpuworker is a single-node GPU job broker designed to manage GPU resources efficiently. It ensures that multiple GPU-bound model services can run concurrently without overloading the host's VRAM. ### Key Features: - **Single-Node Management:** gpuworker manages a single GPU, ensuring that jobs can run without causing the host's VRAM to run out. - **VRAM-Budgeted Admission:** Each job type declares its peak VRAM usage, and only jobs that fit the remaining budget are admitted. - **Warm Serve Containers:** A `vram` job type can run in `serve` mode, where a container loads the model once and receives jobs over a Unix socket. - **Sandboxed Jobs:** Host-process jobs run in a locked-down sandbox, while container jobs run only if the configuration allows them. - **Reclaim Gate:** Before loading a model, the broker waits for NVML to show the previous model's memory as free, preventing VRAM admission if the memory is not actually free. - **Retries:** Jobs can be re-queued up to `max_retries` times if they fail, and a job that fails on its own is never re-run. - **GPU Faults:** After a CUDA error or a `vram` job that times out, crashes, or is killed, the broker asks the GPU monitor if the card still answers, reporting a hardware fault that pauses VRAM admission until the broker restarts. ### Tech Stack: - Python ### Why It Matters: gpuworker is a personal project for one GPU host and clients

Overview: This C# SDK provides a fully generated, up-to-date, and modern .NET 5+ SDK based on the official Baseten OpenAPI specification, supporting the latest features and breaking changes. Key features: - Fully generated C# SDK based on the official Baseten OpenAPI specification - Supports the latest features and breaking changes - Updated and automatically supported if there are no breaking changes - Modern .NET features like nullability, trimming, and NativeAOT - Compatible with .NET Framework/.NET Standard 2.0 Tech stack: C# Why it matters: This SDK is essential for developers looking to integrate Baseten's model serving and deployment platform with modern .NET applications, ensuring compatibility and staying updated with the latest features and improvements.

### Overview: This repository contains a containerized FastAPI-based inference pipeline for classification, segmentation, and detection tasks. It includes request validation, health/readiness contracts, tests, and CI, ensuring a clean and secure AI engineering architecture without publishing sensitive data or commercial models. ### Key features: - FastAPI routing - Request validation - Health/readiness contracts - Tests - CI - Docker container ### Tech stack: Python ### Why it matters: This architecture demonstrates best practices in clean AI engineering, including configuration-driven tunables, pip-installable `src/` packages, fast contract/regression tests, and reproducible releases. It ensures that the repository is secure and can be shared publicly without compromising sensitive data or commercial models. ### Repository: tushar2159/vision-model-serving-pipeline Primary language: Python GitHub stars: 0 GitHub description: Containerized FastAPI inference architecture for classification, segmentation, and detection routing with request validation, health/readiness contracts, tests, and CI.

Overview: This project provides a comprehensive guide for developing and deploying a Census income classifier using FastAPI and Python. It includes training, evaluation, and serving a model with detailed documentation and code snippets. Key features: - Train and evaluate a Census income classifier - Test model and API behavior - Serve predictions with FastAPI - Clean and preprocess data - Split data for training and evaluation - Train a classifier and report performance metrics - Evaluate categorical features and slice results - Save and load model artifacts - Implement FastAPI application for API endpoints Tech stack: Python Why it matters: This project is designed to help developers understand and implement the entire pipeline from data preparation to model deployment using FastAPI and Python, ensuring a solid foundation in machine learning and server development.

### Overview: This project is a production-shaped FastAPI service designed to serve a scikit-learn model. It includes input validation, model versioning, health checks, Prometheus metrics, structured logs, and a non-root Docker image. The model is trained and served using a versioned bundle, ensuring that only valid and up-to-date models are deployed. ### Key Features: - **Versioned Bundles**: Each `models/<version>/` holds the pipeline and metadata, ensuring that a half-written model never reaches traffic. - **Fail at Start-up, Not on the First Request**: The model loads during the lifespan hook; a broken bundle fails the deploy while `/health` reports 503. - **Validation at the Edge**: Pydantic rejects empty, non-numeric, NaN, and oversized inputs; the handler only ever sees a clean row, and the wrong number of features returns 422 with the count it expected. - **Rollback in One Call**: `POST /model/activate/2026.01.01.0001` switches back to a known-good version without redeploying. - **Observability**: Every response carries `x-request-id` and `x-response-time-ms`, logs are one JSON object per line, and `/metrics` exposes request counts, error reasons, a latency histogram, and prediction counts by label. - **Small, Non-Root Image**: Multi-stage build; the model is trained in the build stage and baked in, the runtime stage has no build tools and runs as uid

Overview: This project provides an inference stack for the Nvidia DGX Spark, a powerful AI supercomputer. It is designed for developers looking to run large language models on their desktops. Key features: - DGX Spark Inference Stack - VLLM Based - Single Spark Tech stack: Shell Why it matters: This stack is ideal for developers who want to leverage advanced AI capabilities without the need for a dedicated server, making it accessible to a wider range of users.

### Overview: This project is an end-to-end, self-hosted MLOps loop on MLflow designed for a credit-risk model. It includes reproducible training with lineage, gated champion/challenger promotion, drift monitoring, and an online scoring API that hot-swaps models when the registry alias moves. The project is small, using one Docker Compose file and a few hundred lines of Python code, focusing on the decisions that matter in production ML. ### Key features: - **Training**: Candidates (logistic regression, histogram gradient boosting) compared with stratified 5-fold CV; the best one is refit and scored on a holdout. - **Promotion**: Challenger and champion are scored on the same evaluation set, so the comparison is fair. Gates: absolute AUC floor, minimum AUC gain, calibration (Brier) may not degrade, and a fairness guardrail on the AUC gap between age groups. - **Responsible features**: `personal_status` and `foreign_worker` are excluded from the model by design. - **Drift**: Population Stability Index per feature (quantile bins for numeric, categories for categorical, unseen categories handled). `mlops drift` exits 1 on major drift, ready for cron. - **Serving**: FastAPI with strict input validation, loads `models:/credit-risk@champion` and polls the alias: promotions and rollbacks go live in seconds with an atomic swap. ### Tech stack: Python ### Why it matters: The project shows the decisions that matter in production ML, not to wrap a framework

Overview: This project builds a production-grade end-to-end fraud detection data platform using a Medallion Data Lake architecture, PySpark for batch processing, and machine learning for real-time predictions and monitoring. Key features: - PySpark-based data processing - Dockerized runtime - Strict phase separation and data contracts Tech stack: Python 3.10, PySpark 3.5.1, Docker Why it matters: This project demonstrates a production-style data pipeline design, clean phase ownership and separation, end-to-end batch ML workflow, idempotent pipeline execution, and real-world dataset handling. Repository: vinhnx0/End-to-End-Fraud-Detection-Data-ML-Pipeline Primary language: Python GitHub stars: 0 GitHub description: Production-grade end-to-end fraud detection pipeline leveraging a Medallion Data Lake architecture (Bronze/Silver/Gold), PySpark-based data processing, and machine learning for real-time prediction, API serving, and monitoring.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It includes semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing VRAM usage and providing failover mechanisms, making it essential for efficient and secure AI development in local AI labs.

Overview: This project is an end-to-end, reproducible MLOps pipeline designed for credit-default risk prediction. It generates synthetic data, trains a credit-risk model using a HistGradientBoostingClassifier, and serves the model behind a FastAPI API. Key features: - Reproducibility by construction: one global seed drives data generation, splits, and training. - No train/serve skew: preprocessing lives inside the sklearn pipeline, ensuring the exact transforms used in training run again at inference. - Lightweight model registry: every run writes a timestamped artifact and a metrics manifest, updating a `latest.json` pointer the API loads. Full audit trail without a heavyweight platform. - Schema-validated API: Pydantic v2 request/response contracts; invalid payloads get a clean 422. Auto-generated OpenAPI docs at `/docs`. - Drift monitoring built in: training captures a reference distribution into the manifest, and `/monitor/drift` scores live batches against it with PSI. Default labels arrive months late; the input distribution is observable now — see Drift monitoring. - Containerized + CI: image trains a model at build time and smoke-tests `/health`; GitHub Actions runs the test matrix and the container smoke test. Why it matters: This project emphasizes production ML engineering, not a one-off notebook, ensuring reproducibility, train/serve parity, model versioning, schema-validated serving, tests, and a Docker image that ships ready to score. Tech stack: Python GitHub stars: 0 Repository: vk86294140-cloud

Overview: shilp is a model-serving stack that allows developers to pull models from Hugging Face, reshape them, and serve them over both Ollama and OpenAI compatible APIs. It is built from scratch and runs on CPU and Apple Silicon. Key features: - Pulls models from Hugging Face - Reshapes models (convert, quantize, prune, merge LoRA) - Serves models over Ollama and OpenAI compatible APIs - Runs on CPU and Apple Silicon Tech stack: Python Why it matters: shilp provides a hands-on approach to understanding model internals, the reshape step, and the API surface, making it valuable for developers looking to build their own model-serving solutions.

### Overview: A minimal and complete MLOps loop designed for running on a laptop, featuring reproducible artifacts, immutable versioning, and observability through OpenTelemetry and Prometheus. ### Key features: - **Model Packaging & Versioning**: Every training run produces an immutable artifact directory (`model.joblib`, `metadata.json`, `baseline.json`) with a `latest.txt` pointer. - **Contract-First Serving**: Pydantic v2 request validation, HTTP 422 on bad input, and a response that carries the exact `model_version`. - **Operational Readiness**: The service is not ready until a valid artifact is loaded. - **Observability by Construction**: An inference-latency histogram, request/error counters, a predicted-class counter, and observable gauges for model readiness, model info, and PSI drift. - **Reproducibility**: Pinned dependencies, clean-checkout container build, and CI that runs the same checks locally and in GitHub Actions. ### Tech stack: Python ### Why it matters: This project demonstrates the key practices of MLOps in a small, reproducible, and easy-to-run setup, providing a clear path for developers to understand and implement MLOps on their own laptops.

### Overview: `cv-inference-server` is a real-time computer vision (CV) model serving system built as a portfolio project. It demonstrates the core building blocks of a low-latency inference service: a pluggable backend abstraction, dynamic request batching against a latency budget, fair compute scheduling with backpressure, an observability surface (Prometheus metrics), and a reproducible benchmark harness across compute paths. ### Key features: - **Dynamic Request Batching**: Coalesces bursty concurrent requests into batches without blowing a latency budget. - **Fair Compute Scheduling**: Dispatches batches fairly across compute so no model starves, applying backpressure instead of growing memory unbounded. - **Backpressure**: Correctly demuxes requests by correlation ID and disables paths (batch-of-one). - **Metrics Registry**: Counts, gauges, and histograms for observability. - **Benchmark Harness**: Reproducible CPU path benchmark with throughput, p50, p99, and mean + max metrics. ### Tech stack: C++20 ### Why it matters: `cv-inference-server` implements the pipeline for serving CV models in real time, ensuring low latency, fair scheduling, and reproducible metrics. The design degrades gracefully from a CPU host to a GPU-only laptop, making it suitable for both CPU-only and GPU-enabled systems. ### Repository: vlantonov/cv-inference-server ### GitHub stars: 0 ### GitHub description: Real-time CV model serving with batching and GPU scheduling

Overview: This project provides an inference stack for the Nvidia DGX Spark, a powerful AI supercomputer, to be used on a desktop. It is primarily based on vLLM technology and supports single-spark configuration. The focus is on accessibility for those on a budget. Key features: - DGX Spark Inference Stack - vLLM-based inference - Single-spark configuration Tech stack: Shell Why it matters: This project offers a cost-effective solution for running AI inference on a desktop, making advanced AI capabilities accessible to a wider audience.

Overview: This project is an inference solution sizing engine that calculates card configurations, explicit KV Cache memory usage, parallel strategies, token-per-watt throughput, and three-year total cost of ownership (TCO). It also assembles technical proposals, bid technical sections, whitepapers, and four-tier form specifications. Key features: - Explicit calculation of KV Cache memory usage - Parallel strategy recommendations - Token-per-watt throughput calculation - Three-year TCO output - Assembles technical proposals, bid technical sections, whitepapers, and four-tier form specifications Tech stack: Python Why it matters: This tool provides a systematic and verifiable approach to calculating card configurations, token-per-watt throughput, and TCO for inference solutions, ensuring accurate and reliable results without the need for guesswork. Repository: wangsir720/inference-solution-sizing-engine Primary language: Python GitHub stars: 0 GitHub description: Inference solution sizing engine: scenario parameters in, card sizing (with explicit KV cache memory accounting), tensor/expert parallel strategy, token-per-watt throughput and 3-year TCO out. Also assembles the technical proposal, bid technical section, whitepaper and the four-tier form spec. Missing hardware pricing is reported, not guessed. Output: The project is an inference solution sizing engine that calculates card configurations, explicit KV Cache memory usage, parallel strategies, token-per-watt throughput, and three-year TCO. It also assembles technical proposals, bid technical sections, whitepapers, and four-tier form specifications.

Overview: A simple, self-hosted vLLM (Vertex Model Language) platform designed for developers, offering a Git-friendly YAML configuration for models and profiles, with a user-friendly CLI and TUI for a seamless experience. Key features: - Git-friendly YAML configuration for models and profiles - Full lifecycle management including CLI and TUI - Self-hosted vLLM platform Tech stack: Python Why it matters: This tool simplifies the process of managing and deploying vLLM models, making it accessible for developers without requiring extensive technical knowledge.

Overview: 이 프로젝트는 우리 아이 미술관의 작업 처리 규칙을 공개 서비스로 재현하고 검증하는 프로젝트입니다. Key features: - 공개 서비스로의 재현 및 검증 - GitHub Actions를 사용한 시험 프로세스 - 계약 검사, 장애 주입 시험, E2E 검증 Tech stack: Python, GitHub Actions, Docker, Playwright, MariaDB, OpenAPI Why it matters: 이 프로젝트는 우리 아이 미술관의 작업 처리 규칙을 공개 서비스로 재현하고 검증하는 데 중요한 역할을 합니다. 이를 통해 원본 팀의 코드를 공개 서비스로 재현하고, 다양한 검증 프로세스를 통해 작업의 정확성과 안전성을 확보합니다. 또한, GitHub Actions를 통해 자동화된 검증 프로세스를 구현하여 개발자들의 생산성 향상에 기여합니다.

### Overview: InferenceGateway is a production-shaped GPU inference engine that integrates seamlessly with Ray Serve, vLLM, and NVIDIA DCGM for enhanced observability. This repository serves as the GPU data plane component of a dual-repository serving stack, enabling low-latency LLM inference through continuous batching, prefix caching, and backpressure mechanisms. ### Key Features: - Continuous Batching - PagedAttention - Prefix Caching - SSE Streaming - TTFT / TPOT / Goodput - DCGM Telemetry ### Tech Stack: - Python 3.11 / 3.12 - Ray Serve - vLLM AsyncLLMEngine - CUDA ### Why It Matters: InferenceGateway optimizes GPU utilization, improves KV-block reuse, and provides real-time GPU telemetry, making it essential for efficient and scalable LLM inference in production environments.

Overview: This repository is the production-shaped control plane for low-latency LLM inference, featuring adaptive routing, dynamic micro-batching, prefix caching, streaming, canary releases, and end-to-end observability. Key features: - Adaptive routing - Micro-batching - Prefix caching - Streaming - Canary releases - Prometheus metrics Tech stack: Python 3.11/3.12, FastAPI, Uvicorn, Pydantic v2, async httpx, Adaptive router, token bucket, warm pool, asyncio micro-batcher, LRU prefix cache, Ray Serve + vLLM `AsyncLLMEngine`, continuous batching, GPU execution, NVIDIA DCGM Exporter, Prometheus, Grafana, distributed traces, serving metrics, dashboards, alerts, Next.js 14, React 18, TypeScript, Recharts, Docker Compose, Kubernetes, Terraform, AWS ECR/EKS, GitHub Actions, pytest, pytest-asyncio, container smoke test. Why it matters: This repository provides a robust and scalable solution for deploying OpenAI-compatible LLM models, ensuring low latency and efficient resource utilization, making it suitable for real-world applications requiring high performance and reliability.