Projects

### Overview: The llama-service project is a Systemd-managed llama.cpp (Vulkan backend) server designed to run behind llama-slot-proxy. It features a locally-vendored, self-updating engine, preset-based model switching, and prompt-cache tuning, all managed through a single entrypoint, `llama-tool.py`. ### Key features: - Systemd-managed llama.cpp server - Locally-vendored, self-updating engine - Preset-based model switching - Prompt-cache tuning - Runs behind llama-slot-proxy ### Tech stack: Python ### Why it matters: This project is ideal for developers looking to run a llama.cpp server behind a proxy, with easy management of models and prompt-cache tuning. The system is designed to be lightweight and efficient, making it suitable for various applications requiring llama.cpp functionality.

Overview: LLM Router is a lightweight .NET-based routing layer designed to manage and load-balance requests across multiple local LLM and image-generation inference servers. Key features: - Multi-protocol API gateway for OpenAI, Claude, and Ollama-compatible endpoints. - Tool / function calling through llama.cpp backends. - OpenAI Responses API with stateful support. - Smart routing with priority-ordered rules, preset/model affinity, and round-robin fallbacks. - Model aliases & fallback presets for clients with hardcoded model ids. - Memory groups & idle unload for GPU sharing and server management. - Image generation with stable-diffusion.cpp presets. - Embeddings with presets launched with embeddings enabled. - Health monitoring & auto-restart for backend supervision. - Model library with Hugging Face integration. - Model presets for reusable configurations. - Remote agents for running servers on other machines. - Resilient backend supervision with separate wrapper processes. - Request logging & stats dashboard for monitoring and comparison. Tech stack: C# Why it matters: LLM Router simplifies the management and load-balancing of multiple LLM and image-generation inference servers, making it easier to deploy and scale your infrastructure while ensuring high availability and performance.

Overview: SPIRA Serving API v1 is a multi-container project designed to serve inference requests to ML models and register predictions for the research project SPIRA. Key features: - Multi-container setup for efficient resource utilization - Integration with the SPIRA research project - Docker-based containers for easy deployment and scalability Tech stack: Python Why it matters: This project is essential for the research project SPIRA, providing a reliable and scalable solution for model serving and prediction registration. Repository: spirabr/SPIRA-serving-api-v1 Primary language: Python GitHub stars: 2 GitHub description: SPIRA Serving API v1 by @daitamae and @vitorguidi

Overview: This project serves as the model server for SPIRA, a project focused on collecting and analyzing data related to the respiratory system. It is designed to be used in a multi-container environment, ensuring efficient and scalable model deployment. Key features: - Model serving capabilities - Multi-container architecture - Integration with SPIRA project Tech stack: Python Why it matters: This project is crucial for the efficient and scalable deployment of models in the SPIRA project, facilitating the analysis and processing of respiratory system data. Repository: spirabr/SPIRA-serving-predictor-v1 Primary language: Python GitHub stars: 2 GitHub description: SPIRA Serving Predictor v1 by @daitamae and @vitorguidi

Overview: Local Model Gateway is a lightweight local AI workstation management system designed to coordinate model access across local agents and LLM apps, ensuring efficient and seamless interactions without the need for manual intervention. Key features: - Lightweight and efficient management of local AI workstations. - Single endpoint for coordinating model access across local agents and LLM apps. - Scheduler and dashboard for managing GPU-bound work. - OpenAI-compatible HTTP routes and a Streamable HTTP MCP endpoint. - SQLite-backed scheduler to handle GPU-bound work. - Shared scheduler and dashboard for multiple local agents and LLM apps. - Supports OpenAI-compatible requests and MCP clients. Tech stack: TypeScript Why it matters: Local Model Gateway simplifies the management of multiple local agents and LLM apps by providing a unified interface and shared scheduling, ensuring efficient resource utilization and seamless coordination without the need for manual intervention.

### Overview: This project is a Flask REST service that auto-discovers and serves 12 ML models across scikit-learn, PyTorch, and TensorFlow. It includes a React dashboard for live inference monitoring and daily JSONL event logs. The models are versioned using DVC and tracked with MLflow, ensuring reproducibility and metric gating. ### Key features: - Auto-discovery of ML models - Serving 12 ML models - Real-time inference monitoring - Daily JSONL event logs - Model registry with metric gating - React dashboard for live inference monitoring - CI pipeline for model quality assurance ### Tech stack: Python ### Why it matters: This repository answers the need for a controlled, reproducible ML workflow with a trained classifier that scores network traffic features and flags likely attacks. The machinery around the model includes DVC-versioned data, MLflow-tracked runs, a model registry with a metric-gated promotion step, drift monitoring on live traffic, and CI that fails when model quality regresses.

### Overview: EvoNano-VLLM is a lightweight inference engine built on the Nano-vLLM framework, designed to be efficient and easy to understand. It supports various optimizations such as INT8 KV Cache Quantization, AWQ Weight Quantization, and a fast offline inference capability, making it suitable for both industrial and research use cases. ### Key Features: - **INT8 KV Cache Quantization**: Enhances cacheable context length and reduces memory footprint. - **AWQ Weight Quantization (W4A16)**: Loads and runs existing AutoAWQ 4-bit quantized models without a calibration set, improving inference speed. - **Fast Offline Inference**: Comparable inference speed to vLLM. - **Readable Codebase**: Clean implementation in approximately 1,200 lines of Python. - **Optimization Suite**: Includes prefix caching, Tensor Parallelism, Torch compilation, and CUDA graph optimizations. ### Tech Stack: Python ### Why It Matters: EvoNano-VLLM provides a concise and efficient way to build and refine lightweight AI inference engines, making it valuable for developers looking to integrate AI capabilities into their projects quickly and effectively.

Overview: LLM Serving Platform is an open-source project designed to unify and manage heterogeneous GPU and model services for enterprise private AI. It offers a standardized API gateway and a unified GPU management system, enabling seamless integration and governance of AI services across different environments. Key features: - Unified GPU and model management - OpenAI-compatible API gateway for model services - Adapter for various inference engines like vLLM, SGLang, and MindSpore Serving - GPU and inference backend adapters for NVIDIA, Huawei Ascend, and other vendors - React Console for easy management of AI infrastructure, including GPU status, model deployment, and API key management Tech stack: TypeScript Why it matters: LLM Serving Platform simplifies the management and deployment of AI models and GPU resources, making it easier to integrate AI services into existing enterprise infrastructure, especially for those looking to build private AI platforms or deploy models in multiple environments. Repository: 837515265/llm-serving-platform Primary language: TypeScript GitHub stars: 1 GitHub description: Open-source model serving and heterogeneous GPU management platform for enterprise private AI. NVIDIA / Ascend / Cambricon / Hygon, unified through OpenAI-compatible APIs.

Overview: 이 프로젝트는 파인튜닝된 NLP 모델을 실제 프로덕션 환경에 배포하기 위한 실무적인 엔지니어링 과정을 다룹니다. NVIDIA Triton Inference Server를 활용하여 딥러닝 모델의 서빙 구조를 설계하고, 지연 시간과 처리량을 최소화하는 실무적인 엔지니어링 역량을 확보합니다. Key features: - PyTorch 모델을 ONNX 포맷으로 변환 - Triton 아키텍처 규칙에 맞는 디렉토리 및 설정 파일 작성 - 서버 최적화: 실시간 트래픽 효율을 위한 동적 배치 구성 - 클라이언트 통신: 백엔드 애플리케이션에서 모델로 추론 요청 및 결과 수신 Tech stack: Python, PyTorch, Triton, ONNX Why it matters: 이 프로젝트는 실무에서 사용하는 실제 환경에서 NLP 모델을 배포하는 데 필요한 실무적인 엔지니어링 기법을 다룹니다. 이를 통해 NLP 모델을 효율적으로 배포하고 사용자에게 신속하고 정확한 응답을 제공하는 데 도움이 됩니다.

### Overview: Qwen 3.8 with 27B parameters is served on a free Kaggle TPU v5e-8 via zrok and an OpenAI-compatible API. The model runs at a throughput of approximately 130 tok/s single-stream decode and 540 tok/s aggregate across 8 streams, with a maximum context length of 262,144 tokens. This setup is verified on Kaggle's free TPU v5e-8, using the Qwen 3.8-27B model in full bf16 format without quantization. ### Key Features: - **Model**: Qwen 3.8 with 27B parameters. - **Quantization**: Full bf16 format (55.6 GB). - **TPU Setup**: Kaggle TPU v5e-8 with free tier access. - **API Compatibility**: Compatible with OpenAI's API. - **Throughput**: ~130 tok/s single-stream decode, ~540 tok/s aggregate across 8 streams, ~10,000+ tok/s prefill, context up to 262,144 tokens. - **Tunnel**: Public zrok tunnel for secure and fast communication. - **Source Code**: Based on the work of **rahim3** on GitHub, using the Qwen 3-8-27b-bf16-on-kaggle-tpu-130-tok-s-api notebook. ### Tech Stack: - **

Overview: CropAI is a comprehensive AI system designed for smart agriculture, utilizing advanced machine learning and deep learning techniques to predict crop diseases, recommend optimal crops based on soil and climate conditions, and suggest the right fertilizers. The system includes a mobile app, a backend API, a dashboard, and cloud deployment on Render. Key features: - Crop Disease Detection: Trained on the PlantVillage dataset, this feature predicts crop diseases from leaf images with high accuracy. - Crop Recommendation: Recommends the best crop based on soil and climate parameters. - Fertilizer Recommendation: Suggests the right fertilizers based on crop, soil type, and nutrient requirements. Tech stack: Python, TensorFlow, FastAPI, Streamlit, Render, Pandas, NumPy, EfficientNet, Machine Learning, Deep Learning, Computer Vision. Why it matters: CropAI offers farmers a powerful tool to enhance crop health, optimize crop yields, and reduce the risk of crop diseases, thereby improving overall agricultural productivity and sustainability. Repository: AKHIL-SAURABH/CropAI-Project Primary language: Jupyter Notebook GitHub stars: 1 GitHub description: CropAI is a fully integrated, end-to-end AI system for farmers that can detect crop diseases, recommend the best crop based on soil and climate parameters, and suggest the right fertilizers. It includes a mobile app, a backend API, a dashboard, and cloud deployment on Render.

### Overview: This project is an end-to-end MLOps pipeline for emotion detection, utilizing modern DevOps and MLOps practices. It includes data versioning with DVC + AWS S3, model training and evaluation with MLflow, CI/CD via GitHub Actions, FastAPI serving, Docker containerization, AWS EC2 deployment, and experiment tracking on DagsHub. ### Key features: - Data versioning with DVC + AWS S3 - Experiment tracking with MLflow + DagsHub - Model registry for automatic promotion - Hyperparameter tuning with MLflow - Dockerized application - CI/CD pipeline with GitHub Actions - Cloud deployment on AWS EC2 - Consistent deployment across environments ### Tech stack: - Machine Learning: Scikit-learn, NLTK - API Framework: FastAPI (Cookiecutter template) - Experiment Tracking & Model Registry: MLflow with DagsHub integration - Data Versioning: DVC (backed by AWS S3) - CI/CD: GitHub Actions - Deployment: Docker + AWS EC2 - Container Registry: DockerHub - Secret Management: GitHub Secrets ### Why it matters: This project ensures reproducibility, scalability, and automation in the development and deployment of an emotion detection system. It leverages modern DevOps practices to make the application production-ready, ensuring consistent performance and easy deployment across different environments.

Overview: This project focuses on predicting breast cancer diagnosis using machine learning classification models, specifically aiming to distinguish between benign and malignant cases based on features derived from a digitized image of a fine needle aspirate (FNA) of the breast mass. Key features: - Data preprocessing - Exploratory Data Analysis (EDA) - Machine learning model training - Model evaluation metrics Tech stack: Python, NumPy, Pandas, Matplotlib, Seaborn, Scikit-learn Why it matters: This project showcases the application of machine learning in healthcare and early cancer detection, supporting medical decision-making. Repository: Abhijais4896/Breast-Cancer-prediction-using-machine-learning Primary language: Jupyter Notebook GitHub stars: 1 GitHub description: A machine learning project for predicting breast cancer diagnosis (benign or malignant) using classification models and medical datasets Tech stack: Python 🐍 Libraries & Tools: NumPy, Pandas, Matplotlib, Seaborn, Scikit-learn Why it matters: This project demonstrates how machine learning can be applied in healthcare and early cancer detection, supporting medical decision-making.

Overview: This project demonstrates the complete engineering lifecycle of fine-tuning a pretrained YOLO11 model for specialized visual damage detection. Key features: - Fine-tuning a pretrained YOLO11 model - Dataset engineering - Image curation - Human annotation - YOLO dataset preparation - Leakage-safe dataset split - Transfer learning - Fine-tuning the pretrained model - Model evaluation - Inference testing - ONNX export - ONNX runtime - FastAPI model serving - Docker - Kubernetes GPU deployment - MLOps Tech stack: PyTorch, OpenCV, ONNX, FastAPI, Docker, Kubernetes, CUDA, MLOps Why it matters: This project showcases practical experience in transfer learning, fine-tuning pretrained models, computer vision, YOLO object detection, deep learning, dataset engineering, model evaluation, model optimization, ONNX, ONNX runtime, FastAPI, Docker, Kubernetes, GPU inference, and MLOps. Repository: AlwinXavier19/yolo11-fine-tuning-visual-damage-detection GitHub stars: 1 GitHub description: Fine-tuning a pretrained YOLO11 model for visual damage detection with ONNX inference, FastAPI serving, Docker, Kubernetes GPU deployment, and MLOps.

Overview: DL Playground allows developers to run deep-learning models in the browser, supporting three input types: images, text, and tabular data. The app features four real PyTorch models trained on diverse datasets, including Russian toxicity BERT, MobileNetV3 vegetables, MNIST CNN, and a tabular ensemble model. Key features: - Realistic PyTorch models - Three input types: images, text, and tabular data - FastAPI + React frontend - Diverse datasets: Russian toxicity BERT, MobileNetV3 vegetables, MNIST CNN, tabular ensemble Tech stack: Jupyter Notebook Why it matters: DL Playground simplifies the deployment of deep-learning models, making it accessible and easy to use for developers. The app supports real models trained on various datasets, ensuring accurate predictions across different inputs.

### Overview: This project is an Async FastAPI service designed to serve a scikit-learn model, ensuring non-blocking request handling, idempotent requests, durable audit trails, and a deployable container. The model is a small multinomial logistic regression classifier trained on `load_digits` with 96.7% accuracy, and the service is optimized for production use. ### Key Features: - **Non-blocking request handling**: CPU-bound inference runs in the background, allowing the event loop to handle other tasks. - **Idempotent requests**: Repeated `request_id` queries return the stored result, avoiding redundant model runs or database conflicts. - **Durable audit trail**: Every prediction is logged to PostgreSQL, including rejected ones, ensuring data integrity. - **Model version pinning**: Clients requesting an unavailable version receive an error, not a silent model switch. - **Deployable container**: Uses multi-stage builds, non-root user access, health checks, and migrations before the port is opened. ### Tech Stack: - Python 3.12 - FastAPI 0.115 - PostgreSQL 16 ### Why It Matters: This service is crucial for applications requiring high availability and reliability, ensuring that predictions are auditable and that models can be updated without downtime. The use of PostgreSQL for audit trails and FastAPI for efficient request handling makes it highly suitable for production environments. ### Repository: - **Repository**: Aniruddha-Shukla/async-ml-serving-api - **GitHub Stars**: 1 - **GitHub Description**: Async Fast

### Overview: ClearText is an async inference platform designed to serve a slow ML model to many concurrent users without falling over. Built with FastAPI, Celery, Redis, PostgreSQL, and HuggingFace Transformers, it addresses the issue of a BERT model taking 150-300ms to classify one comment, which is problematic for 500 concurrent requests. The solution involves enqueuing the job with Celery and returning a task ID, while a separate worker picks up the job, runs inference, and the caller polls for the result. ### Key features: 1. **Asynchronous Inference:** FastAPI returns a task ID while Celery workers handle the actual inference. 2. **Model-Versioned Cache:** Redis cache key is a hash of `(model_version, text)`, ensuring cache hits are served while old entries are automatically removed. 3. **Circuit Breaker:** Before enqueuing a job, the API checks the queue length and returns a 503 error with a `Retry-After` header if the backlog is over the limit. 4. **Dead-Letter Queue:** Jobs that fail after 3 retries are not dropped; the original text, error, and traceback are pushed to a Redis list, and `/dlq/{id}/replay` re-runs one. ### Tech stack: - **API:** FastAPI, Python 3.11 - **Queue:** Celery + Redis - **Database:** PostgreSQL, SQLAlchemy, Alembic migrations - **Model:** `unitary/toxic-b

### Overview: AshML is a Kubernetes-native ML platform designed for efficient GPU training and deployment. It schedules training jobs onto real Pods, tracks experiments, verifies artifacts, and serves models with real traffic splitting. ### Key Features: - **Realworkload**: ResNet-18 on CIFAR-10, one full epoch, 390 steps over 50,000 images, 691 seconds. - **Result**: 65.59% top-1 on the complete 10,000-image test set, independently re-evaluated. - **Reproducible**: Same seed, same image digest → `0.6559` / `0.9687` twice, step for step. - **Serving**: 1,000 test images at 66.0% top-1, 8.7 ms per image on CPU. - **Recovery**: Training pod SIGKILLed mid-epoch → resumes weights, optimizer, schedule *and* data order. - **Scope**: Phases 0–5 complete, 11 ADRs, integration tests on real Postgres + MinIO in CI. ### Tech Stack: JavaScript ### Why It Matters: AshML provides a practical solution for deploying and managing machine learning models in a Kubernetes environment, ensuring efficient training, reproducibility, and real-world deployment.

### Overview: This project serves as a reference for implementing a complete MLOps pipeline, covering MLflow Model Registry, batch and real-time serving with FastAPI, drift monitoring using PSI and KS, and Airflow for scheduled retraining with a promotion gate. The synthetic dataset used is a seeded housing-style regression dataset. ### Key features: - MLflow Model Registry - Batch and real-time serving with FastAPI - Drift monitoring using PSI and KS - Prometheus alerts and Grafana dashboard - Airflow-scheduled retraining with a promotion gate ### Tech stack: Python ### Why it matters: This project provides a comprehensive guide for developers interested in implementing MLOps, showcasing best practices for model deployment, monitoring, and retraining.

### Overview: This project provides a sample implementation of a machine learning model serving on Kubernetes using Docker and Helm charts. The model is hosted in a Kubernetes deployment, and the necessary manifests and Helm chart are included to deploy the application. ### Key Features: - **Kubernetes Deployment**: The model is deployed using a Kubernetes deployment manifest. - **Docker Image**: The model is built and deployed using a Docker image. - **Helm Chart**: A Helm chart is used to automate the deployment and management of the application. ### Tech Stack: - **Go Template**: The project uses Go Template for the Kubernetes deployment manifest. ### Why it Matters: This project serves as a practical example for developers to understand how to deploy machine learning models on Kubernetes using Docker and Helm charts. It demonstrates the integration of machine learning with container orchestration tools, making it valuable for developers looking to deploy ML models in production environments.