Projects

Overview: This repository is a local AI infrastructure lab designed to explore the systems layer beneath model capability, focusing on model serving, observability, and deployment experiments. Key features: - Model serving experiments - Local/offline deployment constraints - GPU memory and quantization tradeoffs - Prometheus-compatible telemetry patterns - Service replacement and recovery workflows - Latency, throughput, and reliability judgment below the API layer Tech stack: Python, Docker, Prometheus, Kubernetes, Docker Compose Why it matters: This lab provides a public-safe writeup of a local compute environment, highlighting the importance of understanding the systems layer beneath model capability. By exposing details that managed APIs abstract away, it demonstrates how real compute operates, emphasizing the reliability of AI systems beyond just the model itself. Repository: EthanHNguyen/local-ai-lab Primary language: English GitHub stars: 0 GitHub description: Local AI infrastructure lab for model serving, observability, and deployment experiments.

### Overview: This project demonstrates the deployment of a Qwen model on a cloud T4 with vLLM, integrating it with a Python client. It includes local inference, readiness checks, authentication tests, and measured costs and network usage. ### Key Features: - Deploying Qwen on a cloud T4 with vLLM. - Connecting to an authenticated, OpenAI-compatible API via Python. - Local Unsloth inference and readiness checks. - Authentication tests and token usage measurement. - Cloud inference with measured resource usage and costs. - A video tutorial walk-through. ### Tech Stack: - Python 3.12 - vLLM on Modal - Unsloth API for local inference ### Why It Matters: This project showcases the seamless integration of a large language model with cloud infrastructure, ensuring secure and efficient inference. It provides a practical guide for deploying models on cloud services and integrating them with Python clients, making it valuable for developers looking to deploy models on cloud platforms.

### Overview: This repository presents a production-style industrial energy forecasting and MLOps system designed and implemented after a professional Measurement & Verification engagement. It uses curated daily analytical data and applies different models and validation rules to address a different objective. ### Key features: - Daily industrial active-energy forecasting - Separate forecasting and MLOps project - Governed by 70/30 Extra Trees–AdaBoost ensemble - FastAPI service for API - Docker deployment for scalability - Safeguards and testing - Observability for monitoring ### Tech stack: Python ### Why it matters: This project showcases the ability to adapt existing data and models to new objectives, demonstrating the flexibility and scalability of the system.

Overview: BentoML is a Python library designed for building online serving systems optimized for AI apps and model inference. It simplifies the process of creating REST APIs, Docker containers, and more. Key features: - Easily build APIs for any AI/ML model - Docker containers made simple, ensuring reproducibility and ease of deployment - Maximizes CPU/GPU utilization by leveraging serving optimization features - Fully customizable, allowing for custom APIs, task queues, and model inference - Ready for production, supporting local development, deployment to Docker, and Kubernetes Tech stack: Python Why it matters: BentoML streamlines the process of creating and deploying AI applications, making it easier to build, run, and scale models efficiently.

Overview: LitServe is a minimal Python framework designed for building custom AI inference servers. It allows developers to define and control the logic of inference, batching, and routing without the need for MLOps glue or configuration files. Key features: - Custom inference logic - 2x faster than FastAPI - Agents, RAG, pipelines, and more - Custom logic + control - Multi-GPU autoscaling - Batching + streaming - Self-host or managed - BYO model or vLLM Tech stack: Python Why it matters: LitServe provides a flexible and powerful solution for building custom inference servers, enabling developers to write their own inference engines in Python. It allows for complete control over the logic, batching, and scaling of inference, making it ideal for scenarios requiring custom logic, multiple models, agents, or non-standard pipelines.

Overview: This project classifies airline-complaint tweets into five categories using six TF-IDF models benchmarked with 10-fold stratified cross-validation. The champion model is served as a Dockerized FastAPI API. Key features: - Six TF-IDF classifiers benchmarked with 10-fold stratified cross-validation - A compact end-to-end ML project: clean data → benchmark models → select champion → serve with FastAPI → ship in Docker → deploy to the cloud - The project includes a live demo and API documentation Tech stack: Python, scikit-learn, FastAPI, Docker, AWS Lambda, ECR, Terraform Why it matters: This project provides a practical solution for classifying airline-complaint tweets into five categories, demonstrating the effectiveness of various machine learning models and showcasing the integration of different technologies for efficient deployment.

Overview: BentoML is a Python library designed to simplify the creation of online serving systems for AI apps and model inference. It allows developers to build APIs for any model, manage environments, and deploy to various environments with ease. Key features: - Easily build APIs for any AI/ML model - Docker containers made simple - Maximize CPU/GPU utilization - Fully customizable - Ready for production Tech stack: Python Why it matters: BentoML streamlines the process of building and deploying AI models, making it accessible to developers without requiring extensive knowledge of machine learning or Docker. It simplifies the deployment of models to production environments, ensuring high performance and reproducibility.

Overview: LitServe is a minimal Python framework designed for building custom AI inference servers. It allows developers to define how inference works for models, agents, RAG, or pipelines without relying on MLOps glue or configuration files. Key features: - Custom inference logic - 2x faster than FastAPI - Agents, RAG, pipelines, and more - Custom logic + control - Multi-GPU autoscaling - Batching + streaming - BYO model or vLLM - Easy setup in Python - Serverless support Tech stack: Python Why it matters: LitServe provides a flexible and powerful solution for building custom inference servers, allowing developers to write their own logic and control over the inference process, without the constraints of traditional serving tools. This makes it ideal for scenarios where custom logic, multiple models, or non-standard pipelines are required.

Overview: This project is a scalable real-time machine learning inference platform built with FastAPI, Docker, Redis, Kubernetes, Triton/vLLM, and Prometheus. Key features: - Scalable real-time ML inference platform - Production-grade observability - Support for concurrent requests and dynamic batching - Containerized system with Docker - Deployed on Kubernetes - Triton Inference Server or vLLM model serving - Prometheus metrics for system health and performance monitoring Tech stack: Python Why it matters: This platform is designed to serve models with low latency, high throughput, and production-grade observability, making it suitable for real-time applications requiring high performance and reliability. Repository: FredChang823913319/real-time-ml-inference-platform Primary language: Python GitHub stars: 0 GitHub description: A scalable real-time machine learning inference platform built with FastAPI, Docker, Redis, Kubernetes, Triton/vLLM, and Prometheus.

Overview: Pinance ML Inference is a model-serving service designed to ensure that LightGBM models used in production match the features they were trained on, preventing any discrepancies that could lead to incorrect predictions. Key features: - Ensures feature parity between the serving code and the models loaded from MinIO. - Checks the schema fingerprint against the feature lists and hashes of four models exported by the training pipeline. - Provides replayable predictions, ensuring that forecasts missed by the backend can be requested again. - Rejects models that do not match the serving schema, preventing any misconfiguration. - Runs offline with a demo script that trains toy models and replaces MinIO with a local directory. Tech stack: Python Why it matters: Ensuring that models used in production are exactly the same as those they were trained on is crucial for maintaining the integrity and accuracy of predictions, especially in real-time systems like Pinance where the slightest discrepancy could lead to incorrect outcomes. Repository: GKatzer/Pinance_ml_inference

Overview: A self-hosted vLLM solution that simplifies the process of managing and deploying large language models. Key features: - YAML-based configuration for models and profiles - Git-friendly repository for version control - CLI and TUI for a seamless user experience Tech stack: Python Why it matters: This project provides a straightforward way to manage and deploy large language models, making it accessible and easy to use for developers.

Overview: This project is a production-style MLOps implementation that trains, serves, and monitors a machine learning model behind a REST API. It uses FastAPI, scikit-learn, and Docker for its tech stack. Key features: - Model training and persistence - Real-time monitoring of latency, errors, and prediction drift - RESTful API for prediction and management Tech stack: Python, FastAPI, scikit-learn, Docker Why it matters: This project showcases the skills required to ship machine learning models to production, making it highly valued in the engineering field. Repository: Gextro/ml-serving-api Primary language: Python GitHub stars: 0 GitHub description: Production-style MLOps: train, serve a model behind a REST API, and monitor latency, errors and prediction drift live. FastAPI + scikit-learn.

Overview: This project serves as a home for the inference stack for the Nvidia DGX Spark, also known as the Grace Blackwell AI supercomputer. Primarily based on vLLM for now, it is designed for the less affluent community. The latest and testing branches are available for those seeking the most up-to-date features. Key features: - vLLM-based inference stack - Single-Spark implementation Tech stack: Shell Why it matters: This project provides a cost-effective solution for AI inference on a desktop, leveraging the power of the Nvidia DGX Spark.

Overview: This repository contains a dry-run-first operations kit designed to review and document the GLM 5.2 serving process on DGX Spark before it is deployed. Key features: - Dry-run-first operations manual and runbook set - Explicit model/runtime intent - Inventory, environment, systemd, and RouterOS templates separated from live values - Preflight, API smoke, benchmark, model-directory, download, and link-validation scripts - Test matrix covering documented serving shapes - Evidence templates distinguishing planned, observed, and unknown results Tech stack: Python Why it matters: Serving can fail at artifact provenance, runtime compatibility, memory planning, process lifecycle, or request validation. This kit ensures a launch command is not a usable-service receipt.

Overview: Measure local LLM inference performance by tracking Time to First Token (TTFT), Time Per Output Token (TPOT), and throughput with a research-grade dashboard for Ollama. Key features: - Tracks TTFT and TPOT - Displays throughput - Saves test results locally Tech stack: Python Why it matters: Provides a comprehensive view of LLM performance on personal hardware, aiding in optimizing model performance and troubleshooting issues.

Overview: HBServe is a high-performance serving engine designed for time-series foundation models, built on the time predicate framework. It optimizes model state reuse and execution strategies to enhance answer quality and freshness while handling continuous data streams. Key features: - Time predicate: Defines acceptable answer quality and freshness, a violation budget, and execution strategies for each stream. - Offline planning: Models are calibrated on representative streams to build cost-aware serving plans. - Online serving: Executes the selected plan through a standalone HTTP server, with options for GPU batching and multi-GPU routing. - Models and datasets: Supports PatchTST, Chronos-2, Toto-2, and TimesFM-3 for classification and forecasting, with data preparation pipelines and extensible model adapters. - Evaluation: Compares policies offline and benchmarks throughput, latency, and quality against Ray Serve, Clockwork, vLLM, and PrefillOnly. Tech stack: Python Why it matters: HBServe optimizes model state reuse and execution strategies, enhancing answer quality and freshness while handling continuous data streams efficiently. Repository: HBridge-R-D/HBServe Primary language: Python GitHub stars: 0 GitHub description: A serving engine for time-series foundation models, built on the time predicate.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It features semantic caching, model profiling, and automatic failover for local AI labs, enhancing performance and reliability. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter optimizes AI performance and reliability by managing local AI labs efficiently, ensuring smooth operation and minimal latency. Repository: HannahArslan1432/SmarterRouter Primary language: Python GitHub stars: 0 GitHub description: SmarterRouter: An intelligent LLM gateway and VRAM-aware router for Ollama, llama.cpp, and OpenAI. Features semantic caching, model profiling, and automatic failover for local AI labs.

### Overview: RealTime ML Demonstrates the full lifecycle of a machine learning project, from raw data to a live prediction in the browser. Two independent prediction services, one for heart disease detection and another for fuel consumption prediction, are built on stacking ensembles selected after benchmarking multiple algorithms. ### Key Features: - **Ensemble Modeling**: Voting and Stacking ensembles benchmarked against tuned individual models (Logistic Regression, SVM, k-NN, Decision Trees, Random Forest, etc.). - **Production-style Inference**: Preprocessing (Min–Max scaling + one-hot encoding) is reproduced exactly at request time to match training, ensuring live predictions match training. - **Systematic Tuning**: Every candidate model optimized with `GridSearchCV` and compared on consistent metrics (precision/accuracy vs. R²). - **Decoupled Training & Serving**: Models and scalers serialized with `pickle` and loaded by the web app, so serving needs no retraining. ### Tech Stack: - **Web**: Python 3.11.5 · Django 5.0.2 · SQLite - **Machine Learning & Data**: scikit-learn 1.2.1 · pandas 2.2.0 · NumPy 1.26.4 · Pillow 10.2.0 · pickle (serialization) - **Notebooks & Visualization**: Jupyter Notebook · Matplotlib · Seaborn - **Datasets**: `heart.csv` (clinical heart-disease — classification) · `auto-mpg.csv` (

Overview: This repository demonstrates the end-to-end process of operationalizing a machine learning model, from training to serving it as a production API. It includes a pipeline that integrates model training, evaluation, serialization, and deployment using FastAPI, Docker, and automated testing. Key features: - Model training and evaluation using a pipeline - Model serialization with metrics and feature contract - Deployment in a Docker container - Automated testing with pytest and GitHub Actions CI Tech stack: Python, FastAPI, Docker, pytest, GitHub Actions, scikit-learn, scikit-learn, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, FastAPI, Docker, CI, Fast

### Overview: InferScale is an end-to-end LLM inference benchmarking and optimization platform that offers reproducible experiments, quality gate enforcement, and a React dashboard for easy configuration and monitoring. ### Key features: - **Reproducible Experiments:** Register engines, define workloads, and run experiments to measure performance and quality. - **Quality Gate Enforcement:** Automatically enforce regression policies and compare results to ensure consistency. - **Configuration Optimization:** Search for better serving configurations based on measured outcomes. - **Durable Execution:** Coordinate independent workers with resource leases, idempotency, cancellation, and fenced writes. - **Inspectable Evidence:** Export configuration, provenance, request measurements, quality scores, and policy outcomes. - **Runs Without GPU:** Launch synthetic environment locally, with seeded experiments, and a working dashboard. ### Tech stack: FastAPI, TypeScript, React, OpenEval integration, and SQL workers. ### Why it matters: InferScale provides a comprehensive solution for optimizing LLM inference configurations, ensuring consistent and reliable performance across different environments and workloads.