Projects

### Overview: This repository provides a local environment for running and integrating a large language model (LLM) using `llama.cpp`-compatible serving. It includes profiles for various model sizes and configurations, allowing for easy setup and deployment of the model through an OpenAI-compatible API. ### Key Features: - **Model Profiles**: Available profiles include "cpu-safe" for conservative CPU usage, "qwen-27b" for medium/large dense models, "qwen-9b" for high-context models, and "qwen-a3b-moe" for MoE-style models with CPU offloading. - **LLM Server**: A `llama.cpp`-compatible server is used to serve the model, ensuring compatibility with the model's requirements. - **OpenAI API Integration**: The model can be accessed via an OpenAI-compatible API, allowing for seamless integration with other tools and services. - **Configuration Files**: The repository includes example `.env` files and `profiles` for easy configuration and tuning of the model. - **Optional OpenWebUI**: An optional OpenWebUI Docker Compose file is provided for a more user-friendly interface, though it is not required. ### Tech Stack: - **Shell**: The primary language used for scripting and configuration. - **Dependencies**: `bash`, `curl`, `llama.cpp` binary, `llama-server` binary, `jq` for JSON formatting, `make` for convenience checks, `ripgrep` for safety scanning, and `docker` for OpenWebUI integration. ### Why It Matters:

Overview: This project is a deployment agent for Model Serving runtimes used in RHOAI, designed to streamline the configuration and validation of model-car configurations through a LangChain supervisor pattern. Key features: - CLI-first workflow for querying model-car files - LangChain supervisor for specialized configuration, accelerator, and decision management - Configuration specialist for parsing YAML and extracting serving arguments, GPU counts, and parameter/quantization hints - Accelerator specialist for checking authentication, enumerating GPUs, and providing detailed metadata for the runtime container - Decision specialist for enforcing hardware/quantization compatibility and issuing GO/NO-GO verdicts - Quantization-aware decisioning to flag unsupported kernels and recommend safer variants - Serving-argument optimizer to apply optimized flags to the configuration - Container metadata enrichment for capturing image size and supported CPU architecture - QA specialist for staging kubeconfig, OCI pull secret, and launching a QA container - Checklist-style responses for specialist steps - Config bootstrap for reuse of cached requirements Tech stack: Python 3.12+ Why it matters: This tool simplifies the deployment and validation process for model-serving runtimes, ensuring compatibility and optimal performance across different environments and configurations. Repository: Raghul-M/Runtimes_Deployment-Agent Primary language: Python GitHub stars: 0 GitHub description: A Deployment agent for Model Serving runtimes Component in RHOAI README excerpt: # Runtimes Deployment Agent [](

### Overview: This repository is an end-to-end MLOps pipeline designed for synthetic healthcare data. It includes reproducible training, model comparison, MLflow tracking, FastAPI serving, batch scoring, and container checks. The synthetic evaluation is conducted on an 800 train / 200 test split with seed 42, yielding a best accuracy of 0.925, F1 score of 0.944, and ROC-AUC of 0.967. The exported random-forest baseline achieved an accuracy of 0.885, F1 score of 0.914, and ROC-AUC of 0.955. ### Key Features: - **Reproducible Training**: Synthetic data generation and feature transformation. - **Model Comparison**: Six different model configurations compared in a single experiment. - **MLflow Tracking**: Comprehensive tracking of model performance. - **FastAPI Serving**: Serving with Prometheus metrics for monitoring. - **Batch Scoring**: Automatic scoring of new data. - **Container Checks**: Checks and container builds on every push. ### Tech Stack: - **Python 3.11** or **3.12** - **MLflow** for tracking model performance. - **FastAPI** for serving the model. - **Prometheus** for metrics monitoring. - **NumPy** for data manipulation. - **Faker** for generating synthetic data. ### Why It Matters: This pipeline is designed for an independent portfolio simulation, not for clinical use or production deployment. The synthetic

Overview: This workshop provides a hands-on experience of building, deploying, and operating production AI agents on Amazon EKS. Participants learn to use the Strands Agents SDK and the A2A protocol, swapping backend infrastructures between an open-source stack and AWS managed services. Key features: - Deploying vLLM on AWS Inferentia - Building agents with the Strands Agents SDK - Using the A2A protocol for communication between agents - Understanding the trade-offs between open-source and AWS managed services Tech stack: Python, AWS CLI, Strands Agents SDK, AWS Bedrock, Milvus, Langfuse, AWS X-Ray, OpenAI, Sentence Transformers, FastAPI, Uvicorn, HTTPX Why it matters: This workshop equips developers with the skills to build and deploy AI agents on Amazon EKS, understanding the benefits and trade-offs of using open-source and managed services.

Overview: SmarterRouter is an advanced gateway and router designed for Ollama, llama.cpp, and OpenAI, featuring semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing resource usage and ensuring consistent performance across different environments.

Overview: SmarterRouter is an intelligent gateway and router designed for Ollama, llama.cpp, and OpenAI, featuring semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic Caching - Model Profiling - Automatic Failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by intelligently managing resources and ensuring seamless operation across different environments.

### Overview: This project is an application built with Flask that evaluates loan requests using a machine learning model. The model was trained using a dataset provided by the instructor and is serialized as `modello_mutuo.joblib`. The application includes a form for inputting the necessary parameters and handles the prediction through a simple form submission and redirection to the result page. ### Key Features: - **Form Submission**: Users can input the required parameters in a form. - **Data Transformation**: The input data is transformed into a pandas DataFrame for the model. - **Model Prediction**: The model predicts the loan eligibility based on the input data. - **Result Page**: The prediction outcome is displayed on a result page. ### Tech Stack: - **Frontend**: HTML - **Backend**: Flask ### Why It Matters: This project serves as a practical demonstration of how machine learning models can be integrated into web applications. It highlights the importance of understanding the limitations of models when dealing with real-world data, especially concerning protected attributes like gender and marital status. The project also emphasizes the need for proper data handling and model retraining when deploying models in production. ### Repository: - **Repository**: Richard-WebDev/loan-predictor-webapp-ml - **Stars**: 0 - **Description**: Web application in Flask that evaluates loan requests using a machine learning model. ### Primary Language: - **Primary Language**: HTML ### GitHub Description: The application is a web application built with Flask that evaluates loan requests using a machine learning model. It includes a form for inputting the necessary parameters and handles

Overview: This repository provides GPU-backed inference functions for Nuclio, enabling independent deployment with shared model-serving contracts. It includes functions for various computer vision tasks such as object detection and segmentation. Key features: - YOLOv5 COCO - YOLOv5 BDD - YOLOv5 wheel - YOLOR signs - Mask R-CNN COCO - MViTv2 COCO - Two-stage cuboid workflow - Lane detection - OpenPCDet KITTI Tech stack: Python Why it matters: This repository simplifies the annotation workflow by abstracting away the complexities of model loading, preprocessing, and inference, allowing the annotation platform to focus on tasks, media, labels, and annotations. Repository: RocketWill/serverless-inference-foundry Primary language: Python GitHub stars: 0 GitHub description: Deployable GPU inference functions for Nuclio with shared model-serving contracts.

Overview: This end-to-end MLOps system uses FastAPI, MLflow, Celery, Redis, Prometheus, Grafana, Docker, and GitHub Actions to train, version, and deploy an XGBoost model for predicting work productivity scores. Key features: - Streamlit frontend for user interface - XGBoost model for predicting work productivity scores - Model registry with MLflow for versioning and rollback - SHAP feature contributions for insights - LLM-generated natural language insights - Prometheus and Grafana for observability - HTTPS via Let's Encrypt - CI/CD pipeline for continuous integration and deployment Tech stack: Python, FastAPI, XGBoost, Docker, MLflow, Celery, Redis, Prometheus, Grafana, GitHub Actions Why it matters: This system provides a production-ready solution for predicting work productivity scores, offering insights into why the score is high or low, and automates the deployment process, ensuring high reliability and scalability. Repository: Ryuzaki1415/mlops-productivity-xgboost Primary language: Python GitHub stars: 0 GitHub description: End-to-end MLOps system for training, versioning, and serving an XGBoost productivity prediction model using FastAPI, MLflow Model Registry, Celery, Redis, Prometheus, Grafana, and Docker with CI/CD automation.

Overview: This project is a PyTorch to ONNX deployment pipeline that includes parity validation, batched benchmarking across runtimes, and self-describing artifact manifests. It helps ensure model consistency and performance across different environments. Key features: - Parity validation to check for numeric drift between training and serving. - Batched benchmarking across different batch sizes to ensure consistent performance. - Self-describing artifact manifests to easily identify the model, parity results, and exact bytes running. Tech stack: Python Why it matters: This project is crucial for ensuring the reliability and performance of deployed machine learning models, especially in production environments where small discrepancies can lead to significant errors. Repository: SAY-5/onnx-deploy Primary language: Python GitHub stars: 0 GitHub description: PyTorch to ONNX deployment pipeline with parity validation, batched bench harness, and self-describing artifact manifests.

Overview: This project provides versioned model artifacts, prediction contracts, drift signals, rollback metadata, and audit-ready release records for healthcare machine learning research. It aims to detect model or data degradation before aggregate metrics become misleading. Key features: - Versioned model artifacts - Prediction contracts - Drift signals - Rollback metadata - Audit-ready release records Tech stack: Python Why it matters: This platform is essential for researchers in the healthcare domain who need to ensure the reliability and accuracy of their machine learning models without clinical certification, focusing on detecting operational signals that indicate model or data degradation before aggregate metrics become misleading.

Overview: This project provides production-oriented building blocks for serving and monitoring machine learning models, including versioned artifacts, prediction contracts, latency metrics, and drift checks. It is designed to be dependency-light, allowing for local testing and easy deployment behind a FastAPI service and CI/CD pipeline. Key features: - Versioned model serving - Predictions with contracts - Latency metrics tracking - Drift detection Tech stack: Python Why it matters: This platform offers a robust solution for deploying and monitoring machine learning models, ensuring high performance and reliability.

Overview: Adaptive Inference Gateway is a production-style ImageNet inference server designed to balance batch processing efficiency with queueing latency, focusing on admission control, asynchronous scheduling, and observability. Key features: - Dynamic batching to optimize throughput and queueing latency - Bounded queue and per-request deadline to prevent infinite memory usage or delayed results - Production-style ML inference server with PyTorch and ONNX backends - Observability and reproducible benchmarks Tech stack: Python Why it matters: This educational implementation showcases the trade-offs between batch size and queueing latency in a production environment, focusing on practical engineering challenges and performance metrics. Repository: Sahil-Arifi/adaptive-inference-gateway Primary language: Python GitHub stars: 0 GitHub description: Production-style ML inference server with dynamic batching, queue backpressure, PyTorch and ONNX backends, observability, and reproducible throughput/latency benchmarks.

Overview: This project is a high-throughput, low-latency LLM inference platform optimized for LLaMA-3 and Mistral models. It leverages dynamic batching, KV-cache optimization, mixed precision inference, tensor parallelism, and PyTorch profiling for improved performance. Key features: - Dynamic batching to maximize GPU utilization - Explicit KV-cache decode loop for efficient memory management - Mixed precision inference for faster performance - Tensor parallelism for optimal model distribution across GPUs - Benchmarking and profiling for accurate performance metrics - Observability with Prometheus and Grafana dashboards - Production deployment with Docker and Kubernetes (HPA autoscaling) Tech stack: Python Why it matters: This platform provides a robust solution for serving high-throughput, low-latency inference for large language models like LLaMA-3 and Mistral, ensuring optimal performance and efficient resource utilization. Repository: SapnilPatel/llm-inference-optimizer Primary language: Python GitHub stars: 0 GitHub description: High-throughput, low-latency LLM inference platform for LLaMA-3 & Mistral — dynamic batching, KV-cache optimization, FP16/BF16 mixed precision, tensor parallelism, with PyTorch profiling, Prometheus/Grafana observability, Docker & Kubernetes (HPA) deployment.

Overview: This repository demonstrates the end-to-end machine learning workflow using MLflow, FastAPI, Docker, and GitHub Actions CI. It trains an Iris classification model, tracks and registers it with MLflow, serves the registered model through a FastAPI REST API, containers the application with Docker, and validates the Docker build using GitHub Actions CI. Key features: - MLflow Experiment Tracking - MLflow Model Registry - Docker Containerization - FastAPI Model Serving - GitHub Actions CI Validation Tech stack: Python 3.12, scikit-learn, MLflow, FastAPI, Pydantic, NumPy, Uvicorn, Docker, Docker Compose, GitHub Actions, SQLite. Why it matters: This project showcases best practices in machine learning model deployment, including model tracking, model registry, Docker containerization, and CI/CD integration using GitHub Actions. Repository: SarathKumarKomathukattil/mlflow-docker-cicd-azure Primary language: Python GitHub stars: 0 GitHub description: MLOps project using MLflow, FastAPI, Docker Compose, and GitHub Actions CI for model tracking, registry, and API serving.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It features semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances AI performance by optimizing resource usage, ensuring reliable operation, and accelerating model execution, making it essential for AI labs and developers working with large language models.

Overview: A comprehensive inference stack designed for Nvidia DGX Spark, providing a robust solution for AI development and deployment on a desktop. Key features: - VLLM-based inference engine - Single Spark cluster management - Flexible configuration for various use cases - Easy integration with existing development environments Tech stack: Shell-based scripting for automation and orchestration Why it matters: This stack is essential for developers looking to leverage the power of Nvidia DGX Spark for AI research and development, ensuring efficient and scalable inference capabilities on a personal workstation.

### Overview: A comprehensive operational layer for model inference, designed for high performance and reliability. ### Key features: - Versioning and rollback - A/B routing - Circuit breaking for backend health checks - Per-feature drift detection - Redis-backed response cache - Prometheus metrics collection - Atomic rate limiting - Model version management - Triton inference backend - Image processing pipeline ### Tech stack: Python ### Why it matters: This platform ensures efficient, reliable, and scalable model inference, optimizing resource usage and reducing downtime. ### Repository: Shakhtar-Sankur/ML-Model-Serving-Platform ### Primary language: Python ### GitHub stars: 0 ### GitHub description: The operational layer around model inference rather than the inference itself: versioned rollout, A/B routing, circuit breaking, per-feature drift detection, caching and metrics. Design target 1,200 QPS. ### Overview: This repository provides a complete operational framework for model inference, designed to handle high traffic volumes and ensure smooth operation. It includes all the necessary components for effective service management, from versioning and routing to caching and performance monitoring. The platform is optimized for high throughput and low latency, making it ideal for real-world applications where reliability and performance are critical.

Overview: This project is a production-grade self-healing microservice mesh designed for real-time fraud detection, featuring versioned models, live traffic routing, automated retraining, observability, and fault tolerance. Key features: - Versioned models using MLflow registry - Canary traffic routing with configurable traffic splits - Circuit breakers with heuristic fallback - KS/PSI drift detection and heuristic fallback - Prefect-driven auto-retraining on drift detection - Prometheus metrics and Grafana dashboards for observability Tech stack: Python, FastAPI, MLflow, Prometheus, Grafana, Docker Compose, Kubernetes. Why it matters: This system demonstrates advanced ML engineering practices, including model versioning, safe deployments, failure recovery, data drift monitoring, auto-retraining, and observability, making it a valuable resource for real-world ML applications. Repository: ShreyasVR2545/Self_healing_ML_mesh

Overview: This repository serves as a home for the inference stack for Nvidia DGX Spark, also known as the Grace Blackwell AI supercomputer, on a desk. It is primarily based on vLLM and single-spark technology, with a focus on accessibility for those who cannot afford the latest and most advanced setups. Key features: - Single-spark inference stack - vLLM-based technology - Primarily designed for accessibility Tech stack: Shell Why it matters: This repository provides a cost-effective solution for AI inference on Nvidia DGX Spark, making advanced AI capabilities accessible to a broader audience.