Projects

### Overview: ServeML is a production-oriented ML model serving API that utilizes a scikit-learn pipeline trained on the Iris dataset. It includes FastAPI endpoints for single and batch predictions, model metadata, and a developer console for easy access and validation. ### Key Features: - **Single and Batch Inference:** ServeML allows for both single and batch predictions using a serialized RandomForest pipeline. - **Model Metadata:** Includes model details such as class names, version, and training and evaluation metrics. - **Runtime Metrics:** Provides real-time metrics and recent activity details. - **Inference Playground:** Offers a browser-based playground for testing predictions. - **Model Versioning:** Tracks and displays the model version and status. - **Validation:** Ensures strict input validation and structured error handling. - **Metrics and Activity:** In-memory metrics and recent activity are available for process monitoring. - **Light-Theme Console:** Provides a developer console with an inference playground, model details, metrics, and API reference. - **OpenAPI Documentation:** Available at `/openapi.json` for API reference. ### Tech Stack: ServeML uses FastAPI for the backend, Python 3.13.14 for the runtime, scikit-learn for the model training, and Pydantic for validation. ### Why it Matters: ServeML simplifies the process of deploying and validating machine learning models, making it easy for developers to integrate and test their models in a production environment. The reproducible training pipeline and real-time metrics ensure that the models are reliable and up-to-date.

Overview: This project is a development MLOps platform that integrates MLflow, Kubeflow Pipelines, KServe, Evidently, Prometheus, and Grafana, providing local and remote provisioning, persistent metadata and artifact storage, access controls, and executable demonstrations of the model lifecycle. Key features: - Provisioning with local kind or OpenStack VM and Ansible configuration - Experiment tracking with MLflow backed by PostgreSQL and MinIO - ML workflows with Kubeflow pipeline and tracked training and quality gate - Model serving with KServe InferenceService, Knative autoscaling, and Istio routing - Input monitoring with Evidently workspace and reproducible drift snapshot - System monitoring with Prometheus scrape targets and provisioned Grafana dashboards - Access controls with NGINX routes, read-only observer roles, scoped kubeconfig, and local credentials - Verification with unit tests, manifest policies, smoke tests, full-platform checks, and CI Tech stack: Python Why it matters: This platform provides a reproducible Kubernetes environment for MLOps, including local and remote provisioning, artifact storage, and system monitoring, making it valuable for development and validation of machine learning models. Repository: VillaforTech/kubernetes-mlops-platform Primary language: Python GitHub stars: 0 GitHub description: Development MLOps platform: MLflow, Kubeflow Pipelines, KServe on Kubernetes, with local validation, model lineage, and observability.

Overview: This project benchmarks vision and NLP models via server inference APIs, focusing on MobileNet, ResNet20, and DistilBERT. Key features: - Server-based inference pipeline - Vision models: MobileNetV2 (CIFAR-10) and ResNet-20 (CIFAR-10) - NLP model: DistilBERT fine-tuned on AG News - Model caching and ONNX conversion for efficient reuse - Flexible device support (CPU, CUDA, MPS) - Dataset utilities for CIFAR-10 Tech stack: Python Why it matters: This project is dedicated to fulfilling a Master’s thesis in AI & ML, providing a practical application of server-based inference and benchmarking techniques.

Overview: This project is a developer-friendly inference stack designed for Nvidia DGX Spark, a powerful AI supercomputer. It serves as a primary tool for developers, providing a simple and efficient way to leverage vLLM models. Key features: - Single-Node Spark Inference Stack - vLLM Model Support - Shell-Based Interface Tech stack: Shell Why it matters: This stack is ideal for developers who need a straightforward and cost-effective solution for AI inference, making advanced AI capabilities accessible to a broader audience without the need for expensive hardware.

Overview: This project is an end-to-end ML platform that demonstrates production MLOps practices, including training pipelines, model serving, monitoring, and CI/CD using Kubernetes. Key features: - Training Service: Sklearn pipeline with MLflow experiment tracking and model registry - Serving Service: FastAPI with Prometheus metrics, health checks, and batch predictions - Monitoring Service: PSI/KL drift detection, Evidently reports, and automated alerting - Infrastructure: Docker Compose for local dev, Kubernetes manifests, Prometheus + Grafana Tech stack: Python, Docker, Kubernetes, FastAPI, MLflow, Prometheus, Grafana, MLflow Registry, Terraform, pytest, ruff, mypy Why it matters: This project showcases best practices in ML operations, ensuring that models are deployed and monitored effectively, and CI/CD pipelines are in place for continuous improvement. Repository: YZXBiz/ml-platform-demo Primary language: Python GitHub stars: 0 GitHub description: End-to-end ML platform: training pipelines, model serving, monitoring, and CI/CD with Kubernetes

Overview: This project is a production-style serving stack for a HuggingFace sentiment model, featuring a versioned model registry, shadow deployments, prediction cache, async evaluation, drift detection, observability, and a CI/CD pipeline. Key features: - Model registry with staging, shadow, production, and archived lifecycles. - Shadow deployments with live traffic mirrored to a candidate version. - Prediction cache in Redis, keyed by model version and input hash. - Async evaluation worker for scoring predictions. - Drift detection and alerting. - Prometheus + Grafana observability. - Load-tested and CI/CD pipeline. Tech stack: Python, FastAPI, Redis, PostgreSQL, Prometheus, Grafana. Why it matters: This platform provides a robust and scalable solution for deploying and managing machine learning models, ensuring high availability and reliability.

Overview: Nazra is a production Arabic sentiment analysis platform powered by AraBERT, utilizing MLflow for tracking, DVC for data versioning, and CI/CD for quality gates. It automates the entire MLOps lifecycle from training to deployment. Key features: - AraBERT-based sentiment analysis model - Automatic model retraining on drift detection - Zero-downtime model registry hot-swapping - CI/CD pipeline for continuous integration and deployment - Production-ready architecture with Docker, Prometheus, Grafana, and Alertmanager - Fast inference using ONNX Runtime and Triton - Scalable throughput and low latency Tech stack: Python, MLflow, DVC, Docker, Prometheus, Grafana, Alertmanager, MLflow, BentoML, Triton, Airflow, MinIO, Docker, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow, MLflow,

### Overview: Nazra is an end-to-end MLOps platform powered by AraBERT, designed for Arabic sentiment analysis and production-level deployment. Nazra streamlines the entire lifecycle from training to retraining, ensuring high-quality, real-time inference. ### Key Features: - **AraBERT Model**: Fine-tuned for Arabic sentiment analysis. - **Production-Sized Architecture**: Containerized services for a resilient, production-ready deployment. - **Micro-Batching**: Dynamic micro-batching for efficient inference. - **BentoML Serving**: Fast and scalable inference service. - **Monitoring & Observability**: Grafana for observability, Prometheus for telemetry, Alertmanager for routing alerts. - **Retraining & Lifecycle Management**: Automated retraining and end-to-end lifecycle management. ### Tech Stack: - **Python**: 3.12+ - **BentoML**: Containerized model serving. - **MLflow**: Tracking and deployment of models. - **Docker**: Containerization for reproducibility. - **Airflow**: Pipeline orchestration. - **Triton**: NVIDIA Triton for inference acceleration. - **Postgres**: Database for storing model weights and metrics. - **MinIO**: Object storage for model weights. - **Prometheus**: Telemetry monitoring. - **Grafana**: Observability dashboards. - **Alertmanager**: Alerting and routing. ### Why It Matters: Nazra simplifies the process of deploying and retraining sentiment analysis models, ensuring high-quality inference at scale. Its modular architecture and automated lifecycle management make it a powerful tool for developers

Overview: A developer-friendly inference stack for Nvidia DGX Spark, designed for home use and the less affluent AI enthusiasts. Primarily based on vLLM and single-spark technology. Key features: - Single Spark Inference Stack - vLLM-Based Model Serving - Easy-to-Use Interface - Primarily for Home Use and Budget-Conscious Developers Tech stack: Shell Why it matters: This stack provides a cost-effective solution for running inference tasks on a DGX Spark, making AI accessible to a broader audience without the need for expensive hardware.

### Overview: This project aims to deploy and design an AI inference platform using Kubernetes and KServe, focusing on the KServe deployment and AI inference capabilities. The platform is designed to handle model training, inference, and autoscaling, with a particular emphasis on the Iris classifier model. ### Key Features: - **KServe Deployment**: KServe is used to deploy the Iris classifier model as an InferenceService. - **Kubernetes Cluster**: A single-node Kubernetes cluster is set up using K3s, ensuring lightweight and efficient operation. - **Model Training**: Trained a Scikit-learn Iris classifier model using Random Forest. - **Persistent Storage**: Models are stored in Kubernetes PersistentVolumes for durability. - **Auto-scaling**: KServe supports autoscaling to handle varying loads. - **Scale-to-zero**: Demonstrated the ability to scale down to zero resources when idle. ### Tech Stack: - **Kubernetes (K3s)**: Lightweight, single-binary Kubernetes for easy deployment. - **Knative Serving**: Serverless framework for scalable and autoscaling deployments. - **Kubernetes Networking**: Kourier for efficient traffic routing. - **KServe**: Control plane for serving ML models. ### Why It Matters: This project showcases the practical application of KServe in deploying AI models on Kubernetes, highlighting its capabilities in managing model inference, autoscaling, and efficient resource utilization. The Iris classifier model serves as a real-world example, demonstrating how KServe can be used to deploy and manage AI models effectively. ### Repository Structure: - **README.md**: Contains project details and instructions.

Overview: This repository contains production-grade AI inference systems, a low-latency LLM serving client architecture, and streaming telemetry benchmarking. Key features: 1. **Low-Latency Streaming Client (`track1-client`)**: Asynchronous streaming token ingestion powered by `httpx` and `asyncio`. 2. **Benchmarking & Validation Artifacts**: Structured experimental runs and trace files in `artifacts/` tracking inference efficiency. 3. **Inference Engineering Systems Curriculum**: Comprehensive architecture breakdowns, token economics, and low-level attention mechanism implementations (`day17_attention.py`). Tech stack: Python 3.12+ / 3.14, Async Runtime: Python `asyncio`, `httpx`, Data Modeling & Validation: Python `dataclasses`, `pydantic`, Containerization & Compute: Docker, Docker Compose, NVIDIA CUDA GPU acceleration, Testing: `pytest`, `pytest-asyncio`. Why it matters: This repository provides a production-grade solution for AI inference systems, low-latency LLM serving, and benchmarking, ensuring high efficiency and reliability. Repository: abdulhafeezaidev/AI-Inference-Engineering Primary language: HTML GitHub stars: 0 GitHub description: Production-grade AI inference systems, low-latency LLM serving client architecture, and streaming telemetry benchmarking.

### Overview: This project is a FastAPI + ONNX Runtime microservice designed for serving image inference with advanced features such as A/B routing, optional shadow traffic, token-bucket rate limiting, payload size guards, and end-to-end observability. It includes Blue/Green promotion, Docker/Kubernetes manifests, k6 load tests, and comprehensive monitoring and tracing capabilities. ### Key features: - **A/B Routing**: Sticky cookies and optional shadow traffic. - **FastAPI + ONNX Runtime**: Batched processing. - **Rate Limiting**: Request rate limiting with token-bucket. - **Payload Size Guards**: Rejects large payloads. - **Observability**: Prometheus metrics, Grafana dashboards, structured logs, and request tracing. - **Admin API**: Live tweak A/B weights, warmup, and promote from Blue to Green. ### Tech stack: - Python ### Why it matters: This microservice provides a robust and production-grade solution for image inference, ensuring high performance, security, and easy observability. It supports A/B testing, traffic management, and comprehensive monitoring, making it ideal for environments requiring high availability and performance. ### Repository: Repository: abdulvahapmutlu/edge-serve-ab Primary language: Python GitHub stars: 0 GitHub description: This project is a FastAPI + ONNX Runtime microservice that serves image inference with A/B routing, optional shadow traffic, token-bucket rate limiting, payload size guards, and end-to-end observability (Prometheus metrics, Grafana dashboards, structured logs, request tracing

### Overview: A local inference gateway designed for bounded admission, tenant quotas, and streaming functionality. It runs the original Google FLAN-T5-small checkpoint on CPU, records request outcomes in SQLite, and uses Redis for quotas and optional tenant-scoped response caching. ### Key features: - **Bounded Admission**: Limits the number of active inference and queued work. - **Streaming Cancellation**: Stops generation cooperatively and joins the worker before returning capacity. - **Token Reservation**: Reserves output tokens in Redis, refunds unused tokens, and rejects admission when Redis is unavailable. - **Quota Management**: Routes through revision-compatible backends with circuit breakers and draining. Failover is allowed only before any response text or final usage has been emitted. - **Request Monitoring**: Retains request identity, measured usage, failures, and operator actions without persisting prompts or generated text in SQLite. Optional cached responses remain in Redis until their TTL expires. - **Operator Console**: Inspect capacity, usage, requests, drain or disable a backend, cancel work, and run a fixed streaming diagnostic with a separate tenant key. ### Tech stack: Python 3.10.11, PyTorch 1.13.1, Transformers 4.30.2 ### Why it matters: - **Efficient Resource Management**: Limits the number of active requests to prevent resource exhaustion. - **Scalable Quotas**: Tracks and manages tenant quotas to ensure fair usage. - **Real-time Monitoring**: Provides insights into request capacity, usage, and operator actions. - **Scalable

### Overview: The Ollama Runtime is a GPU-accelerated containerized Ollama model server designed for local, multi-app serving. It leverages a shared Docker bridge network to allow multiple applications to access the same local LLM. ### Key Features: - **GPU Acceleration**: Uses NVIDIA GPU passthrough via `nvidia-container-toolkit` for faster inference. A CPU fallback is available if no GPU is present. - **Shared Docker Network**: Exposes `ollama-runtime-network` (bridge) to allow any local service to reach Ollama at `http://ollama:11434`. - **Centralized Model Storage**: Models are pulled into a named Docker volume (`ollama_data`) and reused across all connected apps. - **Lifecycle Independence**: The runtime starts and stops independently of any consumer service. - **Multi-Model Support**: Ships with pull scripts for `llama3.2:3b`, `qwen2.5:3b`, and `phi3.5:3.8b` (all Q4_K_M quantized, 1.9-2.2 GB each). ### Tech Stack: - **LLM Runtime**: Ollama (latest) - **Containerization**: Docker + Docker Compose v2 - **Networking**: Docker bridge (`ollama-runtime-network`) - **Hardware Acceleration**: NVIDIA GPU via `nvidia-container-toolkit` - **Test Suite**: pytest + httpx + pyyaml ### Architecture: ```mermaid graph TD A["docker-compose

### Overview: vLLM server API probe and benchmark tool. ### Key Features: - **Endpoint Catalog**: Probes all 22 known vLLM HTTP endpoints across 5 categories. - **Model Parameter Sweep**: Tests 9 parameter dimensions (temperature, top-k, top-p, min-p, repetition penalty, structured output, streaming, stop sequences, seed) with 32 test cases per model. - **TTFT/Throughput Benchmark**: Measures time to first token (p50/p95/p99), end-to-end latency, generation tok/s, and prefill tok/s across short/medium/long prompts. - **Prometheus Metrics Reference**: Maps the `/metrics` output for use in Grafana dashboards (KV cache, queue depth, prefix cache, preemption counts). - **No vLLM SDK**: Pure `httpx` against any running vLLM server; works across vLLM versions without SDK compatibility concerns. ### Tech Stack: - **Language**: Python 3.12+ - **HTTP Client**: httpx (async) - **OpenAI-compatible Client**: openai - **CLI Output**: rich - **Config**: python-dotenv ### Why It Matters: The official documentation for vLLM is outdated, making it difficult to understand which endpoints exist in a specific version and how they behave. vllm-explorer provides a comprehensive solution by probing the API surface, testing model parameters, and benchmarking performance metrics across various models and hardware configurations. This tool helps developers and researchers understand the live state

### Overview: This project demonstrates a complete end-to-end MLOps pipeline, from training and evaluating a model to registering and serving it. It uses MLflow for tracking experiment lineage, Docker for containerization, and FastAPI for the API. The project includes CI/CD pipelines, automated tests, and health checks, ensuring reproducibility and operational readiness. ### Key Features: - **Experiment Lineage:** Parameters, dataset hashes, runtime versions, evaluation metrics, model signatures, input examples, and artifacts are recorded in MLflow. - **Governed Delivery:** Registry aliases identify the candidate and champion; promotion succeeds only when the candidate meets a configurable minimum accuracy. - **Typed Serving:** Batch predictions use named Iris features and return a class, confidence, and the configured alias URI. - **Operational Readiness:** Separate liveness and readiness checks, model metadata, and valid Prometheus exposition are available from the API. - **Delivery Discipline:** A Python package, Docker Compose stack, linting, static typing, tests, smoke checks, and GitHub Actions keep the workflow repeatable. ### Tech Stack: - Python 3.11 or 3.12 - MLflow for experiment tracking - Docker for containerization - FastAPI for the API - Prometheus for monitoring - CI/CD pipelines ### Why It Matters: This project demonstrates the entire MLOps lifecycle, from training to serving, ensuring reproducibility, governance, and operational readiness. It includes CI/CD pipelines, automated tests, and health checks, making it a comprehensive reference for developers looking

### Overview: This repository provides a local example of converting a trained machine learning model into a REST API using FastAPI. It demonstrates the process of loading a saved model, validating JSON requests, preparing model inputs, running inference, and returning predictions as JSON. ### Key Features: - Load the model once at startup. - Define the API contract with Pydantic schemas. - Prepare inputs for model inference. - Run inference using the loaded model. - Return predictions as JSON responses. ### Tech Stack: Python ### Why it Matters: This example serves as a practical guide for developers looking to integrate machine learning models into their applications using FastAPI, a popular Python framework for building APIs. The detailed steps provided help in understanding the integration process and can be adapted to various models and scenarios.

Overview: This project demonstrates the end-to-end lifecycle of a machine learning model, including training, serving predictions, monitoring sample data drift, and visualizing results with Streamlit. It uses scikit-learn for training, FastAPI for serving, and Streamlit for dashboarding. Key features: - Trains a scikit-learn model on the Wisconsin Diagnostic Breast Cancer dataset. - Uses MLflow to track training settings and evaluation scores. - Accepts 30 numeric measurements to predict class labels. - Compares sample production data with reference training data using Population Stability Index (PSI). - Displays API health, prediction counts, feature drift, and evaluation metrics in a Streamlit dashboard. - Includes Docker, tests, and CI for reproducibility. Tech stack: Python. Why it matters: This project serves as a practical guide for developers transitioning into machine learning lifecycle management, focusing on reproducibility, model monitoring, and real-time data visualization. Repository: ajayjoffice/mlops-lifecycle-monitor Primary language: Python GitHub stars: 0 GitHub description: End-to-end MLOps demo: train and track a scikit-learn model, serve predictions with FastAPI, detect sample data drift, and monitor results in Streamlit. Includes Docker, tests, and CI.

### Overview: MLServe is a minimal example of serving a scikit-learn model with FastAPI, specifically an Iris species classifier. The model is trained on scikit-learn's bundled Iris dataset, and the API provides a JSON prediction endpoint for validated input. ### Key Features: - **Model Trained on Iris Dataset:** The model is trained on the Iris dataset, and the held-out accuracy is reported at `/model`. - **Validated Input Endpoint:** The `/predict` endpoint accepts four flower measurements as named fields and returns the predicted species along with per-class probabilities. - **Input Validation:** The input is validated using Pydantic, ensuring all fields are required, numeric, and within the specified ranges. Invalid input results in a `422` response with details. ### Tech Stack: - Python 3.11+ - FastAPI - Pydantic 2 - scikit-learn - NumPy ### Why It Matters: MLServe demonstrates how to serve a scikit-learn model with FastAPI, ensuring reproducibility and accuracy through fixed seeds and stratified splits. The validated input endpoint and strict input validation make it suitable for production use cases. ### Repository: - **GitHub Repository:** akincenk/MLServe - **Primary Language:** Python 3.11+ - **GitHub Stars:** 0 - **GitHub Description:** MLServe is a minimal example of serving a scikit-learn model with FastAPI: an Iris species classifier with validated input and a JSON prediction endpoint. ### Stack: - Python 3

Overview: This repository provides a production-ready infrastructure for LLM serving experiments, including automated GPU VMs, pre-configured vLLM, full observability with Grafana, Jupyter Notebook environment, and comprehensive monitoring. Key features: - Automated Infrastructure - Pre-configured vLLM - Full Observability - Interactive Experimentation - Scalable Foundation - Research-Ready Environment Tech stack: Python Why it matters: This repository offers a structured environment for AI infrastructure engineering experiments, ensuring reliability and ease of use. Repository: akrisanov/llm-serving-lab Primary language: Python GitHub stars: 0 GitHub description: vLLM serving environment with GPU provisioning, workload telemetry, ClickHouse analytics, and Grafana observability.