Projects

Overview: This project is a lightweight backend API designed for deploying and serving machine learning models using FastAPI. It is built with Docker for containerization and Pydantic for clean inference with validation. Key features: - FastAPI for RESTful API - Pydantic for request/response validation - Docker for containerized deployment - uv for dependency management - Model serialization with pickle Tech stack: Python 3.13.1, FastAPI, Pydantic, Docker, uv, pickle Why it matters: This project simplifies the process of deploying machine learning models by providing a lightweight backend API, making it easy to integrate with existing infrastructure and ensuring clean inference with validation.

Overview: Model Registry is a service designed to manage and expose machine learning models through an API. Key features: - Save models - Fetch models - Delete models Tech stack: Python Why it matters: Model Registry simplifies the process of deploying and managing machine learning models, making it easier for developers and data scientists to share and reuse models across different projects.

### Overview: This project focuses on optimizing the deployment and performance of the vLLM model on Kubernetes, specifically using NVIDIA RTX 5060 Ti GPUs. The repository includes a Helm chart for multi-tenant GPU scheduling, custom Helm charts, and benchmarks on throughput and cost-per-token performance. ### Key features: - Multi-tenant GPU scheduling - Custom Helm charts for Kubernetes deployment - Throughput and cost-per-token benchmarks ### Tech stack: Python ### Why it matters: This project addresses the need for efficient GPU usage and cost-effectiveness in Kubernetes environments, particularly for high-throughput applications like vLLM.

### Overview: DYNAMO-SERVE is a production-grade distributed LLM inference infrastructure designed to benchmark and optimize real-world bottlenecks in LLM serving. Built with a KV cache-aware scheduler, multi-node GPU placement, and SLA-based admission control, it simulates and benchmarks the performance of LLM serving systems. ### Key features: - KV cache-aware scheduling - Multi-node GPU placement - SLA-based admission control ### Tech stack: Python ### Why it matters: DYNAMO-SERVE simulates and benchmarks the real bottlenecks in LLM serving, focusing on GPU memory fragmentation, KV cache pressure, multi-tenant interference, and tail latency under bursty workloads. It shows how a KV-cache-aware scheduler can outperform naive baselines on every metric that matters. ### Repository: aryanputta/DYNAMO-SERVE - Primary language: Python - GitHub stars: 0 - GitHub description: Distributed LLM inference infrastructure with KV cache-aware scheduling, multi-node GPU placement, and SLA-based admission control.

Overview: This project is a production-style sentiment analysis pipeline that integrates Shiny for user-friendly exploration, Plumber for backend model serving, and tidymodels for consistent scoring. Key features: - Shiny dashboard for sentiment analysis - Plumber backend for model serving - Tidymodels model bundle for consistent scoring Tech stack: R Why it matters: This project aims to provide a clean, user-friendly interface for sentiment analysis, a robust backend for model serving, and clear contract definitions for model inputs and outputs, ensuring consistent and reproducible evaluation.

Overview: This repository demonstrates how to build and deploy an XGBoost model using a Jupyter notebook, with the final algorithm hosted on Algorithmia for serving predictions. Key features: - Creating an algorithm on Algorithmia - Developing locally from a Jupyter notebook - Training and deploying an XGBoost model - Preprocessing data and setting up the model - Iterative hyperparameter tuning - Accurately predicting sentiment analysis Tech stack: Jupyter Notebook Why it matters: This repository provides a comprehensive guide on how to build and deploy machine learning models using XGBoost, making it accessible for developers to quickly create and deploy their own algorithms on Algorithmia.

Overview: This project is an AI text moderation backend that serves a Korean text classification model using FastAPI, PostgreSQL for review workflows, Docker Compose for containerized services, and integrates CI/CD with GitHub Actions for continuous testing and performance monitoring. Key features include: - FastAPI-based model lifecycle management - Pydantic-based input validation and two-step length policy - Inference isolation using `run_in_threadpool` - `allow`, `review`, and `block` confidence policies - PostgreSQL review queue and Alembic migration - X-API-Key-based Admin API - Docker Compose-based API and database environment - GitHub Actions CI and runtime health smoke tests - Request ID-based structured JSON logging - Locust scenario/stress testing and service pivot - 60 sample dataset for 56 threshold combinations comparison Why it matters: This project addresses the challenges of running the model locally versus deploying it as an API service, ensuring robust validation, efficient model lifecycle management, and proper handling of sensitive content. The confidence policy helps in managing different categories with varying confidence levels, and the threshold calibration provides a way to fine-tune the system's performance and security. Tech stack: Python, FastAPI, Pydantic, PostgreSQL, Docker Compose, GitHub Actions, Alembic, X-API-Key, Locust, Swagger UI GitHub stars: 0 Repository: bae-kh/text-moderation-api

Overview: A comprehensive inference stack designed for Nvidia DGX Spark, optimized for home use and accessible to the less affluent. Primarily based on vLLM, this stack is tailored for a single Spark instance, offering a cost-effective solution for AI development and deployment. Key features: - vLLM-based inference - Single Spark instance support - Primarily Shell-based technology Tech stack: Shell Why it matters: This stack provides a practical and affordable way to leverage Nvidia DGX Spark for AI tasks, making advanced AI capabilities accessible to those with limited resources.

### Overview: This project is an end-to-end MLOps pipeline that tracks experiments with MLflow, compares runs to select a champion, and serves the best-performing model through a FastAPI prediction endpoint. The pipeline uses the NSL-KDD intrusion-detection dataset and includes honest evaluation metrics to ensure the model's performance is reliable and trackable. ### Key Features: - **Trains** a Random Forest model on the NSL-KDD dataset with three different hyperparameter sets. - **Tracks** every run in MLflow, logging parameters, metrics, and model artifacts. - **Compares** the runs in the MLflow UI and selects the champion by Run ID. - **Serves** the champion model through a FastAPI `/predict` endpoint that classifies live traffic as Normal or Attack. ### Tech Stack: - **MLflow** for experiment tracking and model artifacts. - **scikit-learn** for the Random Forest classifier. - **pandas / numpy** for data preparation. - **FastAPI + Uvicorn** for model serving. ### Why It Matters: The project demonstrates the operational side of machine learning, showing how to track, compare, and deploy models effectively. It highlights the importance of honest evaluation metrics in security applications, where missed attacks can have severe consequences.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It includes semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing memory usage and providing a seamless gateway for local AI research.

Overview: A self-hosted vLLM solution that simplifies the process of managing large language models using YAML configurations and a CLI/TUI interface. Key features: - YAML-based model and profile management - Git-friendly workflow - CLI and TUI for easy configuration and management Tech stack: Python Why it matters: This project offers a straightforward and efficient way to manage large language models, making it ideal for developers and teams looking to integrate vLLM into their projects without the complexity of traditional deployment methods.

Overview: A TypeScript inference SDK designed for self-hosted Large Language Models (LLMs), Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and embeddings. It manages GPU lifecycle coordination, model handoffs, and provides clients for vLLM, Whisper, Chatterbox, Qwen3, and Kokoro. Key features: - GPU lifecycle coordination - Model handoffs - Clients for vLLM, Whisper, Chatterbox, Qwen3, and Kokoro Tech stack: TypeScript Why it matters: This SDK simplifies the process of deploying and managing large language models and related AI tools, making it easier for developers to integrate advanced AI capabilities into their applications without the complexity of complex infrastructure.

Overview: A self-hosted Qwen3 Large Language Model (LLM) serving solution utilizing vLLM, FP8 quantization, and an OpenAI-compatible API. This project includes a Docker Compose recipe with pinned images, persistent weights, and explicit GPU memory budgets. Key features: - Self-hosted Qwen3 LLM serving - vLLM integration - FP8 quantization - OpenAI-compatible API compatibility Tech stack: Docker Compose Why it matters: This project provides a robust, scalable, and secure LLM serving solution that can be easily deployed and managed, making it ideal for developers and teams looking to integrate advanced AI capabilities into their applications.

Overview: This project provides an inference stack for Nvidia DGX Spark, a powerful AI supercomputer, tailored for developers on a budget. It is primarily based on VLLM and supports single Spark instance. Key features: - DGX Spark inference stack - VLLM-based inference - Single Spark instance support Tech stack: Shell Why it matters: This project offers a cost-effective solution for developers looking to deploy AI inference on a DGX Spark, making advanced AI capabilities accessible to those on a budget.

### Overview: A real-time fraud-scoring inference service backed by a hand-written Raft consensus core, designed to ensure order and durability of model-version promotions. ### Key features: - Order: Ensures that only one authoritative answer is returned for the current model version. - Durability: Decisions persist even after the primary node fails. - Failover: Refuses new promotions when a majority of the registry is unreachable, ensuring data consistency. ### Tech stack: Go inference, model versioning, and Raft consensus core. ### Why it matters: This project writes the Raft consensus core, making it a valuable addition to the distributed systems toolkit, especially for applications requiring order and durability of model-version promotions. Repository: brunobrsr1/fraud-scoring-platform Primary language: Go GitHub stars: 0 GitHub description: Distributed fraud-scoring service with Go inference, model versioning, and a hand-written Raft consensus core.

### Overview: This project is a comprehensive deployment and governance framework for insurance pricing, designed to meet the stringent requirements of the ICOBS 6B.2 record-keeping regulations. ### Key features: - **Routing Determinism**: SHA-256 hash routing ensures expected allocation and policy_id consistency across 10,000 policies. - **ENBP Breach Detection**: With 2% intentional breach rate injected, the audit report flags ~2% of renewals, confirming correct logger records and breaches. - **LR Test Calibration**: Bootstrap LR test returns INSUFFICIENT_EVIDENCE at typical volumes, reflecting true statistical difficulty rather than a library bug. - **Power Analysis**: At 20% challenger allocation and 5,000 total quotes, the library reports 18-24 months to LR significance, matching theoretical calculation. ### Tech stack: Python ### Why it matters: This framework ensures operational correctness and meets the stringent requirements of the ICOBS 6B.2 record-keeping regulations, providing a reliable and efficient solution for insurance pricing.

### Overview: Ray Serve serves inference, not the entire lifecycle, in an enterprise Model Development Life Cycle (MDLC). ### Key features: - **Ray Serve as an Online Inference Layer**: Fits best for compositional inference (multi-model pipelines, RAG, agentic flows) and LLM serving on self-owned accelerators. - **Over-engineered for Single-Model Endpoints**: Not suitable for fleets of single-model, tensor-in/tensor-out endpoints. - **Autoscaling Parameters**: Verified defaults and what drives the bill. - **Cloud Deployment via Managed Kubernetes**: Architecture, autoscaling parameters, release strategies, and failure modes. - **Ray 2.57.0 and KubeRay v1.6.2**: Latest stable versions. ### Tech Stack: Python ### Why it Matters: - **Ray Serve as an Online Inference Layer**: Fits best for compositional inference (multi-model pipelines, RAG, agentic flows) and LLM serving on self-owned accelerators. - **Over-engineered for Single-Model Endpoints**: Not suitable for fleets of single-model, tensor-in/tensor-out endpoints. - **Autoscaling Parameters**: Verified defaults and what drives the bill. - **Cloud Deployment via Managed Kubernetes**: Architecture, autoscaling parameters, release strategies, and failure modes. - **Ray 2.57.0 and KubeRay v1.6.2**: Latest stable versions. ### Repository: byte-ish/ray-serve-inference-mdlc ### Primary Language: Python ### GitHub Stars: 0

Overview: This project serves object detection models on various hardware platforms, including Intel FPGA, Intel CPU, and NVIDIA GPU, with support for models like SSD, YoLov3, and Faster R-CNN. Key features: - Implement object detection models with OpenVINO and TensorRT - Support REST and gRPC for server communication - Inference engine for CPU, FPGA, and GPU - Compatible with DOTA dataset for training models - Pre-trained models available for download Tech stack: C++ and OpenVINO, TensorRT, Intel OpenCL, and NVIDIA TensorRT Why it matters: This project provides a comprehensive solution for deploying object detection models on different hardware, making it suitable for applications requiring real-time inference on various platforms.

Overview: MLauncher simplifies the deployment of machine learning models with a single command, making it an enterprise-grade platform for efficient model serving. Key features: - Simplifies model deployment with one command. - Seamlessly integrates with AWS, Azure, and Google Cloud Platform. - Uses Docker for easy environment setup. - Benefits from FastAPI for fast, easy-to-use APIs. - Offers customizable templates for specific needs. Tech stack: Python, FastAPI, Docker. Why it matters: MLauncher streamlines the deployment process, making it easier and more efficient for both professional and beginner developers to manage and serve their machine learning models.

Overview: This repository contains the definition and deployment for the Food Review Service API, a Python FastAPI microservice designed to provide information about product reviews from the Amazon Fine Food Reviews dataset. Key features: - FastAPI microservice for handling product reviews - Dataset integration and processing - Swagger API documentation for easy access Tech stack: Python, FastAPI, Docker, Docker Compose Why it matters: This service aims to simplify the process of accessing and analyzing product reviews, making it easier for developers and businesses to gather valuable insights from customer feedback.