Projects

Overview: This repository serves as a home for the inference stack for Nvidia DGX Spark, commonly known as the Grace Blackwell AI supercomputer. It is primarily designed for developers and aims to provide a simple, efficient solution for inference tasks. Key features: - Single Spark Inference Stack - VLLM-Based Inference - Shell-Based Tech Stack - Primarily for Developers Tech stack: Shell Why it matters: This stack is designed for developers who may not have access to a full-scale AI supercomputer, offering a practical and accessible solution for inference tasks.

Overview: This project designs and implements scalable infrastructure for deploying and managing machine learning models in production. Key features: - Automated model deployment - Real-time model monitoring - Load balancing for scalability Tech stack: Go Why it matters: Ensures efficient and reliable deployment of machine learning models, essential for modern data-driven applications.

### Overview: This project is a model serving platform with observability, built on Python. It includes a scikit-learn classifier trained by a reproducible script, served over FastAPI with Prometheus metrics and structured logging. The platform ensures the lifecycle of the model, from training to serving, and exposes observability endpoints for monitoring. ### Key features: - Trains a real model using scikit-learn. - Serves predictions via FastAPI with validated request/response schemas. - Exposes observability endpoints: `/metrics` for Prometheus metrics and structured JSON logs. - Reports the model's state honestly, including missing artifacts. ### Tech stack: Python ### Why it matters: This project demonstrates a practical approach to model lifecycle management, ensuring models are always up-to-date and easily deployable. The observability features provide real-time monitoring and traceability, enhancing the reliability and maintainability of the model serving platform.

### Overview: Prism-Serve is a Kubernetes control plane designed for disaggregated LLM (Large Language Model) serving. It includes prefix-aware routing, prefill/decode scheduling, and KV-cache orchestration for prism-infer workers, enhancing performance and efficiency. ### Key Features: - **FastAPI Gateway**: Provides liveness/readiness probes and a Prism token-id streaming route. - **Fixed 2P2D Deployment**: Uses Helm templates for a single active Gateway and an atomic four-worker world. - **PD Scheduler**: Selects shortest-queue prefill and most-free-slots decode based on runtime instance-count recommendations. - **KV Transfer Flow Control**: Manages dynamic high/low watermark, per-dst byte cap, and FIFO deferred queue with automatic flush. - **Recompute Fallback**: Detects timeout, resets instances, and aborts after max attempts. - **Request State Machine**: Tracks per-request `SeqState` lifecycle, guards illegal transitions, and detects stuck requests with TTFT timestamps. - **10 ms Schedule Loop**: Consists of Phase 1-6 reconciles, assigns P/D, submits KV, checks for stuck requests, flushes deferred queues, collects finished requests, and metrics. - **NATS Queue**: Wraps publish/subscribe functionality with bounded inbox, queue-group load balancing, and wildcard `kv_usage.*` subscription. - **Metrics**: Monitors and records TTFT, KV transfer, congestion, deferred depth, and slot utilization for performance tracking. ### Tech Stack: Python ### Why It Matters: Prism-Serve optim

### Overview: InferenceOps is a reproducible benchmark lab designed to study how inference systems behave under real operational conditions. It focuses on deterministic request replay, router policy comparison, controller evaluation, GPU serving, and Kubernetes deployment. ### Key features: - Deterministic request replay - Router policy comparison - Controller evaluation - GPU serving - Kubernetes deployment ### Tech stack: Python ### Why it matters: InferenceOps provides a comprehensive framework for evaluating and benchmarking inference systems, ensuring that they perform reliably under real-world operational conditions.

Overview: A self-hosted vLLM solution that simplifies the process of managing and deploying large language models using YAML configurations and a command-line interface (CLI) and terminal user interface (TUI). Key features: - YAML-based configuration for models and profiles - CLI and TUI for a seamless workflow - Git-friendly repository management Tech stack: Python Why it matters: This tool provides a straightforward and efficient way to manage and deploy large language models, making it accessible for developers without requiring extensive knowledge of complex systems.

### Overview: This project showcases a containerized gRPC service designed for machine learning model inference, focusing on practical ML engineering concepts such as API contracts, service health checks, model versioning, and inference endpoints. ### Key Features: - gRPC service for ML model inference - Protocol Buffers API contract - Health check endpoint - Prediction endpoint with confidence score - Python client for local testing - Serialized ML model artifact - Dockerized service - Local testing with `grpcurl` ### Tech Stack: Python, gRPC, Protocol Buffers, scikit-learn, Docker, grpcurl, ML model inference, API contract design ### Why it Matters: This project demonstrates several practical ML engineering concepts, including serving ML models through a structured API, defining a clear service contract with Protocol Buffers, separating server and client logic, exposing health and prediction endpoints, returning prediction confidence, managing model path and version through environment variables, and containerizing an inference service with Docker. The approach can be extended to larger ML systems, making it relevant for production-oriented healthcare analytics, Medical AI services, and other similar applications.

### Overview: This project is a real-time fraud detection system built on leakage-aware machine learning, validated across datasets (PaySim, ULB) with SHAP explainability, a cost-calibrated decision policy, and a tested FastAPI serving layer. The system scores financial transactions for fraud, explains every flagged decision with SHAP reason codes, and serves predictions through a live API. ### Key Features: - **Real-time Fraud Detection**: Scores financial transactions for fraud. - **SHAP Explainability**: Real fintech systems are required to justify every declined or flagged transaction. - **Cost-Calibrated Decision Policy**: Every high-scoring feature is stress-tested against the full dataset before being trusted. - **Two Separate Leaks Found and Rejected**: Every high-scoring feature is stress-tested against the full dataset before being trusted. - **Model Validation**: Features derived from post-transaction state look predictive but don't exist yet at decision time. - **Severe Class Imbalance**: Fraud is 0.13% of all transactions, a model that predicts "never fraud" scores 99.87% accuracy while being useless. ### Tech Stack: - **Python**: 3.13 - **XGBoost**: Model tuning with Optuna - **Optuna**: Tuning XGBoost - **FastAPI**: Serving layer - **SHAP**: Explainability - **Status**: Phase 4 complete - **License**: MIT ### Why It Matters: - **Accuracy is the Wrong Metric**: Teams that lead with accuracy are shipping broken fraud systems

### Overview: This project is a lightweight Machine Learning Model Serving API built with Python, scikit-learn, and FastAPI. The API serves house price predictions, validates inputs, supports batch inference, and tracks prediction history. ### Key features: - Train a Linear Regression model using scikit-learn - Split data into training and testing sets - Evaluate the model using MAE and R² Score - Save the trained model using joblib - Validate API inputs using Pydantic - Single house price prediction - Batch predictions - Model information endpoint - Prediction history logging ### Tech stack: - Python - Pandas - NumPy - Scikit-learn - FastAPI - Pydantic - Joblib - Postman - Git & GitHub ### Why it matters: This project demonstrates the basic flow of taking an ML model from training to serving, including data processing, model training, evaluation, saving, and API serving. It provides a simple example of how to integrate ML models into a production environment through an API, making it useful for understanding the ML → Model → API workflow.

Overview: This project provides an end-to-end MLOps pipeline for deploying and managing machine learning models on AWS cloud infrastructure. It uses tools such as Boto3, S3, Lambda, and SageMaker to automate the provisioning, training, and deployment of models. Key features: - Automated AWS infrastructure setup - Flask API for easy integration - CI/CD with GitHub Actions for continuous testing and deployment - AWS deployment using Boto3 and SageMaker Tech stack: Python, Flask, Boto3, S3, Lambda, SageMaker, GitHub Actions Why it matters: This project showcases the practical application of MLOps principles on AWS, demonstrating how to efficiently deploy and manage machine learning models using automated tools and best practices. Repository: Tania526-sudo/ML-Ops-on-AWS Primary language: Jupyter Notebook GitHub stars: 0 GitHub description: End-to-end MLOps pipeline for deploying and managing machine learning models on AWS cloud infrastructure. Supports automated provisioning, training, and deployment using tools like Boto3, S3, Lambda, and SageMaker.

Overview: This project provides a model-agnostic FastAPI-based template for deploying machine learning models for tasks such as banknote authentication. It includes features like schema-driven inference, anti-overfitting training configurations, and a serialized model artifact for easy deployment. Key features: - Model-agnostic inference API - Anti-overfitting training configuration - External model config file - FastAPI prediction endpoint - Postman-ready payload handling - Tech stack: Python 3.12+, scikit-learn, pandas / numpy, FastAPI + Uvicorn Why it matters: This template simplifies the deployment of machine learning models, making it easier to develop and deploy models without the need for extensive setup. Repository: Tharun007-TK/ml-api-template Primary language: Jupyter Notebook GitHub stars: 0 GitHub description: Model-agnostic FastAPI ML inference template with config-driven feature schema, environment-based model loading, and discoverable predict and config endpoints. Tech stack: Python 3.12+, scikit-learn, pandas / numpy, FastAPI + Uvicorn Why it matters: This project simplifies the deployment of machine learning models, making it easier to develop and deploy models without the need for extensive setup.

Overview: Astra Studio 是一款面向私有化环境的云原生 AI 开发与交付平台，覆盖数据集、在线开发、模型训练、模型登记、推理部署、资源配额与离线交付等环节。 Key features: - 支持 Notebook 和 VSCode Web 在线开发，兼容任意镜像注入 Jupyter/VSCode 工具包。 - 数据集版本、训练任务、模型登记和推理服务闭环。 - 提供目标检测、分类、分割、OCR、跟踪与 LLM 微调算法资料包。 - PostgreSQL 主库、Redis、MLflow、NFS/PVC 与 S3/MinIO ArtifactStore。 - 基于 Volcano 的组织级 CPU、内存和 GPU 配额与调度。 - NVIDIA GPU 与 MetaX GPU 资源发现、配额和工作负载注入。 - 联网部署、完整离线包、增量更新和部署后健康检查。 Tech stack: Python Why it matters: Astra Studio 是腾讯音乐cube-studio的云原生版本，旨在提供私有化部署和离线交付的解决方案，适用于企业级AI应用。 Repository: TheBigThreeCN/astra-studio-public Primary language: Python GitHub stars: 0 GitHub description: Cloud-native AI development, training and inference platform for private and offline deployments

### Overview: PneumoAI is a production-style engineering portfolio project designed for classifying chest X-rays as either Pneumonia or Normal. It includes a serving layer, async processing, monitoring, deployment infrastructure, and a gated model promotion pipeline. The system is not clinically validated or authorized for diagnostic use. ### Key features: - **Production Deployment**: Runs at [https://api.thiyaga.dev](https://api.thiyaga.dev) with the full VM stack described below. - **Model Versioning and Threshold Calibration**: All predictions are persisted to a database for audit, evaluation, and drift analysis. - **Async Inference and Monitoring**: Uses Prometheus and Grafana for monitoring and observability. - **Gated Model Promotion**: Supports gated model promotion for model lifecycle management. - **Drift Detection**: Detects model drift using drift detection signals. ### Tech stack: Python ### Why it matters: The project demonstrates how a reliable, production-ready ML system for medical image classification can be built, served, and operated. The emphasis is on model versioning, threshold calibration, async inference, monitoring, and gated model promotion. The system is not clinically validated or authorized for diagnostic use.

Overview: This project showcases a scalable machine learning microservice built with Java and Spring Boot, designed to serve AI models efficiently and integrate them into larger applications. Key features: - Model Serving: Exposes trained machine learning models via RESTful APIs. - Spring Boot: Leverages Spring Boot for rapid development and easy deployment. - Containerization: Dockerfile included for containerizing the microservice. - Scalability: Designed to be horizontally scalable to handle high request loads. - Health Checks & Monitoring: Basic endpoints for service health and integration with monitoring tools. Tech stack: Java, Spring Boot, Maven, Docker, ONNX Runtime/TensorFlow Serving (via REST). Why it matters: This microservice provides a robust and efficient way to integrate machine learning predictions into larger applications, focusing on best practices for microservice architecture, including containerization, API design, and deployment considerations for high availability and performance.

Overview: SmarterRouter is an advanced gateway and router designed for Ollama, llama.cpp, and OpenAI. It features semantic caching, model profiling, and automatic failover for local AI labs, enhancing performance and reliability. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter optimizes AI performance and reliability by intelligently managing models and resources, ensuring seamless operation across different AI tools and environments.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It features semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter optimizes performance and reliability for AI models, making it essential for efficient and secure AI development in local labs.

Overview: llamacore is a desktop client designed to manage local GGUF models, run llama-server processes, and streamline AI workflows using the llama.cpp framework. Key features: - Local GGUF model management - Running llama-server processes - Building AI workflows Tech stack: llama.cpp Why it matters: llamacore simplifies the process of managing and utilizing AI models, making it easier for developers to integrate llama.cpp into their projects.

Overview: This project is an end-to-end machine learning service that predicts taxi trip durations in New York City. It uses LightGBM, MLflow, and Docker for deployment, and includes features like champion/challenger registry, drift monitoring, and validated FastAPI with Prometheus metrics. Key features: - Predicts taxi trip durations with 62% lower error than a baseline model. - Uses MLflow for experiment tracking and model validation. - Provides a validated REST API with FastAPI and Prometheus metrics. - Includes drift monitoring and champion/challenger registry. - Uses Docker for containerization and CI/CD pipeline. Tech stack: Python, MLflow, LightGBM, Docker, MLflow, Prometheus, FastAPI. Why it matters: This service offers a more accurate and reliable prediction of taxi trip durations, improving operational efficiency and customer satisfaction.

Overview: Nexoryx_MLOps_Platform is a production-ready Kubernetes-based MLOps platform that integrates Apache Airflow, MLflow, MinIO, PostgreSQL, Redis, Spark, and other tools for efficient and scalable machine learning workflows. Key features: - Kubernetes-native architecture - MLflow tracking server - Airflow workflow orchestration - Spark job execution - Object storage using MinIO - Monitoring with Prometheus and Grafana - Persistent storage - Horizontal scaling - Ingress-based routing - Secrets management - Production-ready manifests Tech stack: Kubernetes, Docker, Apache Airflow, MLflow, Apache Spark, PostgreSQL, Redis, MinIO, JupyterHub, Grafana, Prometheus, NGINX Ingress. Why it matters: This platform provides a robust and scalable solution for deploying and managing machine learning models using Kubernetes, ensuring high availability and efficient resource utilization. Repository: ViBhuTechnologies/Nexoryx_MLOps_Platform Primary language: Unknown GitHub stars: 0 GitHub description: Production-ready Kubernetes MLOps platform with Apache Airflow, MLflow, MinIO, PostgreSQL, Redis, Spark, JupyterHub, Prometheus, Grafana, NGINX Ingress, and more. README excerpt: Production-ready MLOps platform for Kubernetes with: - Apache Airflow - MLflow - MinIO - PostgreSQL - Redis - Spark - JupyterHub - Prometheus - Grafana - NGINX Ingress ## Features - Kubernetes-native architecture - MLflow

Overview: A self-hosted vLLM (Versatile Language Learning Model) platform designed for developers, offering a Git-friendly YAML-based model and profile management system, along with a CLI and TUI for a seamless and intuitive user experience. Key features: - YAML-based model and profile management - CLI and TUI for a simple and intuitive interface - Git-friendly repository for model and profile versions - Full lifecycle management from creation to deployment Tech stack: Python Why it matters: This project simplifies the management and deployment of vLLM models, making it accessible and easy to use for developers, and provides a Git-friendly way to manage models and profiles, ensuring version control and easy collaboration.