Projects

### Overview: This project automates the creation of a Docker image from a ZIP file containing a model, inference script, and optional requirements file. It leverages the Docker Engine API to build the image without a Dockerfile, ensuring the ZIP is validated and the model runs as expected. ### Key Features: - **ZIP to Docker Image Conversion**: Converts a ZIP file containing a model, inference script, and optional requirements into a runnable Docker image. - **No Dockerfile Required**: The project uses the Docker Engine API to build the image, eliminating the need for a Dockerfile. - **Validation**: Validates the ZIP file on the host before proceeding, checking for path traversal, zip bombs, symlinks, and other unsafe entries. - **Build and Commit**: The build process copies files into the working directory as a TAR stream, installs dependencies, and commits the image with the model's SHA-256. - **Dry Run**: Allows you to see what would be built without actually building the image. ### Tech Stack: - Python 3.10+ - Docker Engine ### Why It Matters: This project streamlines the process of creating Docker images, making it easier to deploy models without the need for a Dockerfile. It ensures that models run as expected and that the image is validated before being committed, providing a secure and reliable deployment method.

### Overview: This project is an adaptive inference routing system designed for dynamic traffic management across a cascade of model variants, leveraging multi-objective reinforcement learning to optimize performance metrics. ### Key Features: - **Dynamic Routing**: Routes incoming requests across different model variants (quantized, pruned, distilled, full-precision) based on predicted query difficulty, SLA constraints, and real-time cluster load. - **Model Variants**: Supports four model variants with varying performance characteristics. - **Performance Metrics**: Reduces latency by 35%, maintains accuracy at 14.5%, improves throughput by 2.5x, and reduces SLA violation rate to 58.3%. - **Policy Convergence**: Achieves strong latency reduction but at the cost of significant accuracy degradation and high SLA violation rates. ### Tech Stack: Python ### Why It Matters: This system demonstrates the potential of reinforcement learning in optimizing complex multi-objective tasks, showcasing how it can dynamically adjust traffic routing to meet diverse performance and reliability requirements.

Overview: This project implements an adaptive model serving optimizer that uses UCB multi-armed bandit algorithms to dynamically select the best serving strategy for ML models. It automatically chooses between PyTorch optimized, standard PyTorch, and ONNX Runtime backends based on real-time performance metrics, achieving significant latency reduction and cost savings with zero accuracy degradation. Key features: - UCB-based multi-armed bandit algorithm for selecting optimal serving strategies - Automatic switching between PyTorch optimized, standard PyTorch, and ONNX Runtime backends - Real-time performance monitoring and optimization - Production simulation results showing 86.7% P99 latency reduction with zero accuracy degradation Tech stack: Python Why it matters: This project provides a robust solution for optimizing model serving configurations, ensuring minimal latency and cost while maintaining model accuracy, making it highly valuable for applications requiring high performance and low latency. Repository: A-SHOJAEI/adaptive-model-serving-optimizer Primary language: Python GitHub stars: 0 GitHub description: UCB bandit-based model serving optimizer with automatic latency/accuracy/cost tradeoff. 86.7% P99 latency reduction with zero accuracy degradation in production simulation.

### Overview: This project serves a calibrated wafer-defect model and monitors for drift, using FastAPI and Prometheus for monitoring. The model is frozen from a sibling checkout and drift monitors are triggered based on honest numbers, including detection latency, false-alarm rate, calibration deltas, and shifts each monitor is blind to. The architecture includes a predictor, monitors, and a retrain trigger, with metrics exposed via Prometheus and Grafana. ### Key features: - **Calibrated Model**: A frozen model (`wafer-mixed/outputs/{best.pt,thresholds.json,calibration.json}`) from a sibling checkout. - **Drift Monitoring**: Triggers based on honest numbers, including detection latency, false-alarm rate, calibration deltas, and shifts each monitor is blind to. - **Unsupervised Drift Monitoring**: Monitors compare against a committed reference snapshot frozen from the wafer-mixed test split. - **Self-contained Monitoring**: Monitors are bounded rolling windows, ensuring online path is co-tenant-safe on the GB10. - **No Training**: The model is a fixed, calibrated artifact, reused through a guarded bridge. ### Tech stack: Python ### Why it matters: This project closes the lifecycle of the wafer-defect model, serving and monitoring it, ensuring continuous improvement through drift monitoring.

### Overview: This project is an asynchronous image-classification server built using FastAPI, Redis, and PyTorch. It handles HTTP requests, batches images for inference, and runs the ResNet-18 model in parallel across multiple workers. The server can handle up to 1,200 concurrent requests and has been load-tested to 1,200 concurrent requests. ### Key features: - **Asynchronous Processing**: Uses asyncio for efficient handling of multiple requests. - **Batched Inference**: Groups images into batches and runs them through ResNet-18 in parallel. - **Redis Queue**: Utilizes Redis for managing the queue of images and responses. - **Load Testing**: Capable of handling up to 1,200 concurrent requests, with a load test that reaches 1,200 concurrent requests. ### Tech stack: - **Python**: FastAPI for the frontend, FastAPI for the backend, and asyncio for asynchronous processing. - **Redis**: For managing the queue of images and responses. - **PyTorch**: For running the ResNet-18 model in parallel across multiple workers. ### Why it matters: This project demonstrates the capabilities of asynchronous processing, batched inference, and load testing in handling high concurrency. It shows how to efficiently manage resources and handle a large number of concurrent requests using Python, FastAPI, and Redis. ### Repository: The repository is named `ALIUD1/Inference-Server` and is currently at 0 GitHub stars. It is written in Python and uses FastAPI, Redis

Overview: This project focuses on deploying and serving a trained machine learning model, including saving, loading, and serving the model through a FastAPI service. It covers topics such as validating inputs, handling errors, and managing drift. Key features: - Deploying a model - Loading and saving model artifacts - Wrapping a model in a FastAPI service - Handling inputs, validation, and prediction requests Tech stack: Jupyter Notebook Why it matters: This project demonstrates how to deploy and serve a machine learning model, making it accessible to others without requiring retraining.

Overview: This project details the architecture and functionality of a production FastAPI gateway designed for training, versioning, and serving models on Kubeflow + KServe. Key features: - Train: Starts a pipeline by name with parameters checked before any request. - Watch: Live progress tracking for components and parallel runs. - Version: Lists model versions with metrics, lineage, and artifacts. - Deploy: Converts a registered version to a KServe InferenceService in staging or production. - Undeploy / Promote: Converts a registered version to a KServe InferenceService in a different environment. Tech stack: FastAPI, Kubeflow Pipelines v2 SDK, Kubeflow Model Registry, KServe SDK, Triton, vLLM, GPU-queue scheduling with fractional GPU memory, MinIO, Docker, Kubernetes. Why it matters: The production code is proprietary, and this repository explains how it works and why it's built this way. It serves as a single entry point for every model in the software suite, ensuring stability and ease of maintenance.

### Overview: This project is a multimodal neuroscience research agent combining domain post-training, EEG analysis tools, and GPU inference optimization with vLLM, W8A8, CUTLASS, and Triton. It aims to enhance scientific workflows by integrating deterministic computation, visual interpretation, and language synthesis. ### Key Features: - **Deterministic Computation**: Exact numerical/scientific tools for tasks like band power, RMS, PSD peak, and thresholding. - **Visual Interpretation**: Vision-language model reasoning over waveforms, spectrograms, and topomaps. - **Model Synthesis**: Language-model orchestration, grounding, and answer drafting. ### Tech Stack: Python ### Why It Matters: Scientific AI needs more than just fluent language. Neuroscience workflows require precise numerical tools, deterministic statistics, and visual fidelity. This system separates the roles of deterministic computation, visual interpretation, and model synthesis, making it a comprehensive research assistant. ### Repository: AchuthReddy-16/Multimodal-Neuroscience-research-agent ### Primary Language: Python ### GitHub Stars: 0 ### Hardware for Systems Measurements: NVIDIA GeForce RTX 4090 (24 GB) ### Data: - PhysioNet EEG Motor Movement/Imagery Dataset (420 subjects, 14 runs each) ### Headline Results: - **Text Held-out Quality (base → corrected SFT v2)**: 60.0% → 86.4% - **Production W8A

Overview: A self-hosted vLLM (Vector Language Model) platform designed for developers, offering a Git-friendly YAML-based model and profile management system with a user-friendly CLI and TUI. Key features: - YAML-based model and profile management - Full lifecycle from CLI and TUI - Git-friendly for version control - Intuitive and user-friendly interface Tech stack: Python Why it matters: This project simplifies the management of vector language models, making it accessible and efficient for developers without requiring extensive technical knowledge.

Overview: A self-hosted vLLM solution designed for developers, offering a simple and Git-friendly YAML configuration for models and profiles, complete with an intuitive CLI and TUI. Key features: - YAML-based model and profile configuration - Full lifecycle management with CLI and TUI - Git-friendly versioning and branching Tech stack: Python Why it matters: Simplifies the management and deployment of large language models, making it accessible and easy to use for developers.

Overview: Self-hosted vLLM made simple, with a Git-friendly YAML for models and profiles, and a CLI and TUI for a full lifecycle management. Key features: - YAML-based model and profile management - CLI and TUI for intuitive and Git-friendly workflow - Full lifecycle management from setup to deployment Tech stack: Python Why it matters: Simplifies the self-hosting of large language models, making it accessible and easy to manage.

Overview: A Python Flask server designed to host and serve multiple machine learning models locally or remotely via ngrok tunneling. Key features: - Local model hosting - Remote model access via ngrok - Support for file uploads and downloads - Integration with local Wi-Fi and ngrok for global access Tech stack: Python Flask Why it matters: This tool simplifies the process of deploying machine learning models, making it accessible both locally and over the internet, enhancing flexibility and convenience for developers and users alike.

Overview: This project predicts the efficiency category of industrial smart machines—High, Medium, or Low—using real-time operational data. It is implemented as a Flask web application, containerized with Docker, deployable to Kubernetes, and automated via a Jenkins CI/CD pipeline. Key features: - Flask web application for real-time predictions - Docker containerized for portability - Kubernetes deployment for scalable production - Jenkins CI/CD pipeline for automated build and deployment Tech stack: Flask, Docker, Kubernetes, Jenkins Why it matters: This project showcases the complete lifecycle from data preprocessing and model training to scalable deployment, demonstrating best practices in MLOps and CI/CD.

Overview: This project is a Python-based ML model serving API designed to expose machine learning models through RESTful endpoints, ensuring reliable access to predictions from external applications. Key features: - Model loading - Prediction endpoints - Request validation - Inference pipeline - Response serialization - Error handling - Health monitoring - API documentation - Testing Tech stack: Python Why it matters: This system simplifies the deployment and access of machine learning models, making them available for integration with other services without requiring direct interaction with the underlying ML implementation. Repository: Anamikaa18/AI-Model-Serving-API Primary language: Unknown GitHub stars: 0 GitHub description: A Python-based model serving API for deploying trained machine learning models through REST endpoints with structured inference, validation, and error handling.

Overview: This project provides an inference stack for Nvidia DGX Spark, a powerful AI supercomputer, tailored for developers. It is designed to be accessible and useful for those without extensive resources. Key features: - Single-Node Spark Inference Stack - VLLM-Based Model Serving - Primarily Shell-Based Tech stack: Shell Why it matters: This project offers a practical solution for developers looking to leverage advanced AI capabilities without the need for a dedicated supercomputer, making it ideal for those on a budget or with limited resources.

Overview: This project provides a comprehensive framework for training, evaluating, exporting, serving, and observing a decoder-only language model built from scratch using PyTorch. The repository includes reproducible configurations, parity checks for ONNX and TorchScript, a production-shaped FastAPI server, and end-to-end CI. Key features: - Train, evaluate, export, serve, and observe a decoder-only language model - Implement transformer, tokenizer, and training loop from scratch - Use KV-cache parity tests, ONNX/TorchScript export verification, and production-shaped inference server - Include Prometheus instrumentation, container and Kubernetes manifests, and CI running the whole pipeline end to end on every push Tech stack: Python Why it matters: This repository provides a detailed and reproducible approach to building and deploying a transformer model from scratch, focusing on key engineering tests and practical considerations.

Overview: AsukaFrancois58/dgx-spark-inference-stack is a project designed to serve as a home for inference tasks on Nvidia DGX Spark, a powerful AI supercomputer. The primary language used is Shell, making it suitable for developers and enthusiasts looking to experiment with GPU-based inference. Key features: - Inference stack for Nvidia DGX Spark - Primarily uses vLLM for tasks - Single Spark setup - Ideal for the not-so-rich AI enthusiasts Tech stack: Shell Why it matters: This project offers a practical solution for developers to quickly test and experiment with GPU-based inference tasks, making it accessible even for those without extensive resources.

Overview: This project is a production ML inference microservice built with FastAPI and Scikit-Learn, designed for real-time and batch predictions. It includes features like strict data validation, real-time batch predictions, and model observability. Key features: - Real-time single and high-throughput batch predictions - Strict data validation - Model observability via `/metrics` endpoint - Zero-downtime hot reloading for model swapping - Multi-stage Dockerfile for lightweight production container Tech stack: Python, FastAPI, Scikit-Learn Why it matters: This microservice is essential for deploying machine learning models in a production environment, ensuring high performance, reliability, and observability.

Overview: ModelGate is a containerized machine learning inference API and Python SDK that dynamically serves tabular ML models from URLs, ensuring strict JSON schema validation and error shielding. Key features: - Dynamic artifact loading - Strict, dynamic input validation - Out-of-the-box ready - Error shielding - Docker-Native & SDK-Ready Tech stack: Python, FastAPI, Docker, Ruff, MIT License Why it matters: ModelGate simplifies the process of deploying machine learning models by providing a robust, dynamic, and safe microservice, making it ideal for production use cases where model deployment and integration with existing systems are crucial.

Overview: ServeScale is a horizontally scalable, latency-optimized machine learning model serving system built on Kubernetes, featuring ONNX dynamic quantization, FastAPI, and zero-downtime rollouts. Key features: - Kubernetes Orchestration - Inference Optimization (ONNX Runtime, Dynamic Quantization) - Zero-Downtime Rollouts - Load Verification with Locust Tech stack: Python Why it matters: ServeScale demonstrates production-serving competence by providing orchestrated scale, explicit inference optimization, and zero-downtime updates, making it ideal for real-world applications requiring high availability and performance.