Projects

Overview: This project provides an inference stack for Nvidia DGX Spark, also known as the Grace Blackwell AI supercomputer, designed for developers on a budget. Primarily based on vLLM, it offers a simple and efficient solution for inference tasks. Key features: - vLLM-based inference stack - Single Spark deployment - Primarily for developers on a budget Tech stack: Shell Why it matters: This project simplifies the process of deploying AI inference on Nvidia DGX Spark, making it accessible to developers without the need for a high-end supercomputer.

### Overview: This project is a production-style MLOps pipeline designed specifically for recommendation serving. It includes FastAPI for model serving, SVD and NMF recommenders, MLflow for model registry, Redis for caching, Kafka for interaction streaming, and full observability via Prometheus and Grafana. ### Key Features: - **FastAPI Model Serving**: Handles API requests. - **SVD and NMF Recommenders**: Implementing both SVD and NMF for recommendation. - **MLflow Model Registry**: Manages and tracks models. - **Redis Caching**: Efficiently stores and retrieves data. - **Kafka Interaction Streaming**: Real-time interaction events. - **Prometheus + Grafana**: Full observability dashboard. ### Tech Stack: - Python ### Why It Matters: This pipeline is optimized for production environments, ensuring high performance and reliability in recommendation serving applications.

Overview: Backplane is a governed extension runtime for open-weight language models, turning frozen models into programmable systems. Key features: - Tenant isolation with API-key hashing, SQLite namespaces, and card-stack routing. - Compatibility-aware microbatch scheduler. - Hugging Face backend using normal KV-cached generation, 4-bit loading, and bounded activation hooks. - VLLM/OpenAI-compatible backend for prompt/output processing and per-request LoRA routing. - Reproducible `.bpcard` archives signed with Ed25519. - Declarative card boundary rejecting executable archive members. - Public interactive control-plane sandbox with deterministic backend and clear labeling. Tech stack: Python. Why it matters: Backplane enables teams to deploy and manage open-weight language models securely and efficiently, ensuring model integrity and performance.

Overview: CarloDogan/vllmops is a self-hosted vLLM (Vector Language Model) made simple, providing a Git-friendly YAML for models and profiles, along with an intuitive CLI and TUI for a seamless development lifecycle. Key features: - YAML-based model and profile management - CLI and TUI for easy configuration and monitoring - Git integration for version control Tech stack: Python Why it matters: CarloDogan/vllmops simplifies the process of managing and deploying vector language models, making it accessible and easy to use for developers.

**Overview:** A production-style ML serving platform for versioned model inference, runtime model activation and rollback, observability, automated testing, containerized deployment, and CI security checks. **Key features:** - FastAPI-based ML inference service - Two versioned LightGBM model artifacts: V1 and V2 - Runtime model activation without restarting the API - Rollback to the previously active model - Model metadata and feature-schema validation - Health and readiness endpoints - Strict request validation using Pydantic - Prometheus metrics for HTTP request counts, duration, prediction counts by model version, and model inference latency by model version - Grafana dashboard for model-serving observability - Automatic Grafana datasource and dashboard provisioning - Dockerized FastAPI application - Full FastAPI + Prometheus + Grafana stack using Docker Compose - 9 automated tests covering serving, validation, lifecycle management, and metrics - GitHub Actions CI - Automated Docker image build validation - SBOM generation - Container vulnerability scanning **Tech stack:** Python **Why it matters:** This platform demonstrates how multiple ML model versions can be registered, switched at runtime, monitored, rolled back, tested, containerized, and validated through CI, providing a robust and scalable solution for deploying and managing machine learning models in production environments.

Overview: This open-source project provides an inference infrastructure for browser, edge, and GPU, utilizing WebGPU, FastAPI, PostgreSQL, and Docker workers. It includes a PostgreSQL-backed authentication system and is currently undergoing automated CI verification. Key features: - WebGPU inference console - FastAPI gateway - PostgreSQL-backed authentication - Docker workers for runtime integrations Tech stack: Python Why it matters: This project offers a robust and automated infrastructure for inference, ensuring secure and efficient processing through various runtime adapters and PostgreSQL-backed authentication. The CI verification process ensures the project's reliability and performance, making it suitable for both development and production environments.

### Overview: This repository offers comprehensive guides and practical resources for developers looking to enhance their skills in creating and managing AI gateways. ### Key Features: - **Getting Started**: Detailed tutorials and examples for beginners. - **Core Concepts**: Comprehensive guides on fundamental AI/ML principles, key components, common use cases, architectural patterns, and technology stack. - **Implementation Guide**: Step-by-step implementation of core functionalities, data pipelines, service layers, and production deployment. - **Best Practices**: Best practices for development, performance optimization, security, operational excellence, and cost management. - **Advanced Topics**: Coverage of advanced architectures, machine learning integration, data management, system scalability, and security & compliance. ### Tech Stack: Python, Git, pip, and various AI/ML libraries and frameworks. ### Why It Matters: This repository is essential for developers looking to enhance their skills in creating and managing AI gateways, providing practical resources and industry best practices to help them succeed in their projects.

### Overview: This project integrates MLflow, Unity Catalog, and Model Registry to evaluate whether a trained model outperforms a hand-picked heuristic used by the Surge Exposure Agent. The model is trained on real FEMA flood insurance claims data from the same regions as the original validation study, and the results are tracked and reported in a real-world scenario. ### Key Features: - **Model Training**: A real model is trained on the same data used to validate the hand-picked heuristic. - **Data Integration**: Buildings and claims are grouped onto a grid, ensuring a cell-based cross-validation split. - **Real Data**: The project fetches and processes real NFIP claims data from FEMA's OpenFEMA API, covering 8 coastal counties. - **Model Tracking**: MLflow tracks every run, Unity Catalog registers the results, and the project reports the comparison. ### Tech Stack: - MLflow for model tracking and deployment. - Unity Catalog for model registry and storage. - Model Serving for real-time predictions and querying. ### Why It Matters: This project aims to determine whether a trained model can outperform the heuristic used by the Surge Exposure Agent, providing a more accurate and reliable risk assessment compared to the current method.

Overview: A straightforward, self-hosted version of the vLLM model, designed for developers to easily manage and access models using YAML configurations and a command-line interface. Key features: - YAML-based model and profile configuration - CLI and TUI for easy management - Git-friendly workflow for version control Tech stack: Python Why it matters: Simplifies the process of managing and using large language models, making it accessible to developers without requiring specialized knowledge of the underlying technology.

Overview: Autorma is an end-to-end machine learning system designed for automated classification of returned items in an e-commerce warehouse. Built with production-grade practices, it includes model versioning, containerized services, batch inference, and real-time monitoring. Key features: - Model versioning - Containerized services - Batch inference - Real-time monitoring Tech stack: Python, Docker, MLflow, FastAPI, Prometheus, Pushgateway, Grafana, Streamlit Why it matters: Autorma ensures efficient and reliable classification of returned items, improving the accuracy and speed of the classification process, thereby enhancing the overall efficiency and customer satisfaction in an e-commerce warehouse. Repository: DanielPopoola/autorma Primary language: Python GitHub stars: 0 GitHub description: An end-to-end machine learning system for automated classification of returned items in an e-commerce warehouse. Built with production-grade practices including model versioning, batch inference pipelines, monitoring, and a user-friendly interface.

### Overview: This project is a customer churn prediction pipeline implemented using a local MLOps loop. It features feature engineering, scikit-learn training, MLflow-style tracking, and FastAPI model serving. The pipeline is designed to predict customer churn by scoring a synthetic dataset with logistic regression and gradient boosting models, and then scoring new customers with a FastAPI service. The project includes a Population Stability Index (PSI) check to measure marginal drift between batches. ### Key features: - Logistic regression and gradient boosting models trained on a synthetic dataset. - Stratified cross-validation for model evaluation. - Model registry for versioning and champion selection. - FastAPI service for scoring new customers. - PSI check to compare drift between batches. ### Tech stack: Python ### Why it matters: This project demonstrates the practical application of customer churn prediction using ML techniques, feature engineering, and MLOps practices. It showcases how to train models, track experiments, and deploy them using FastAPI, providing a clear example of how to build and test a customer churn prediction pipeline in a local MLOps environment.

Overview: Ce projet comprend un pipeline de détection de fraude bancaire basé sur des données de transactions de carte bancaire. Il utilise des techniques modernes comme XGBoost pour le scoring, FastAPI pour l'API, et Apache Airflow pour l'orchestration. Le pipeline est local et sans besoin de compte cloud, avec des métriques honnêtes et reproductibles. Key features: - Pipeline de détection de fraude sur transactions de carte bancaire - Scoring XGBoost - API FastAPI - Orchestration Airflow - Métriques honnêtes (PR-AUC, ROC-AUC) Tech stack: Jupyter Notebook Why it matters: Le pipeline permet d'identifier efficacement les fraudes bancaires, minimisant les coûts de fausse alerte et de fraude ratée. Les métriques honnêtes et les résultats sur un jeu de données simulé soulignent la pertinence de la performance sur des données réelles.

### Overview: Concrete Strength MLOps is a project that estimates concrete compressive strength from mixture ingredients and age, providing a validated API with conformal intervals and input-range monitoring. The project uses a regression model trained on real-world data, including split-conformal intervals, artifact identity, and input validation, to ensure reproducibility and reliability. ### Key Features: - **Model Training**: The project uses a boosted regression model (hist_gradient_boosting) to predict concrete strength with a mean absolute error (MAE) of 3.379 MPa and a coefficient of determination (R²) of 0.907 on 200 held-out feature rows. - **Conformal Prediction**: The model provides split-conformal intervals to cover 91% of the test targets, ensuring high coverage without being too wide. - **Artifact Monitoring**: The project includes input validation and ready status checks to ensure the input data is correct and ready for predictions. - **API Documentation**: The trained model is exposed through a documented batch API, making it accessible for real-world applications. ### Tech Stack: - **Python**: The project is built and tested using Python 3.11 and 3.12. - **License**: The project is released under the MIT License. - **Level**: This project is part of the Machine Learning Portfolio, level 6 of 6. ### Why It Matters: Concrete Strength MLOps demonstrates advanced system capabilities such as FastAPI, conformal prediction, model artifacts, input validation, and service tests. It

Overview: A simple, self-hosted vLLM (Virtual Language Learning Model) platform designed for developers. It provides a Git-friendly YAML configuration for models and profiles, offering a seamless and intuitive CLI and TUI for managing the lifecycle of vLLM projects. Key features: - YAML-based configuration for models and profiles - Git-friendly repository management - CLI and TUI for easy project management Tech stack: Python Why it matters: This project simplifies the process of managing and deploying vLLM models, making it accessible and efficient for developers working with language models.

### Overview: This project is a FastAPI-based API serving a trained model, packaged in a multi-stage Docker image, with a drift monitor, a served model card, and tests that assert training and serving compute features identically. ### Key features: - FastAPI service - Multi-stage Docker image - Drift monitor - Served model card - Tests asserting identical training and serving compute features ### Tech stack: Python ### Why it matters: The fix is structural, not procedural. The `src/features.py` file exists once and is imported by both `src/train.py` and `src/service.py`, ensuring divergence cannot be introduced by editing one of them. The `tests/test_parity.py` asserts that the training and serving paths match, ensuring identical training-serving skew.

Overview: This project is a research code focused on improving the eviction strategy for the vLLM V1 KV cache. It includes a detailed ablation study and an offline validation on a single workload. Key features: - Offline ablation study - Validation on a single vLLM V1 workload - Detailed documentation on migration from vLLM V0 to V1 Tech stack: Python Why it matters: The project explores the performance of different eviction strategies, particularly focusing on the impact of recency/frequency partitioning and the ghost ticket mechanism. The findings suggest that the recency/frequency partitioning is more effective than the adaptive `p` mechanism, and the ghost ticket mechanism can add significant performance benefits when decoupled from the queue-size budget. Repository: EileenClara/vllm-arc-evictor

### Overview: This project is a from-scratch implementation and benchmarking study of core optimization techniques for modern LLM inference servers, focusing on GPT-2 and a single consumer GPU (NVIDIA RTX 3050, 4 GB). The primary goal is to optimize the efficiency of LLM inference servers without training a new model. ### Key Features: - **KV-cache**: A language model generates text one token at a time. Naively, every new token re-processes the entire sequence from scratch (O(n²) work). KV-caching stores the Key/Value tensors of past tokens so each step only processes the newest token (O(1) per token). - **Static Batching**: A GPU can process many requests in parallel. Running one request at a time wastes most of its capacity. Batching N requests into a single forward pass yields close to N× the throughput at roughly the same latency. - **Dynamic Batching Server**: A real FastAPI server. Incoming requests land in a queue; a background worker collects everything that arrives within a 10ms window (up to batch size 16), processes them together, then routes each response back to its caller via an `asyncio.Future`. Under 32 concurrent requests, this delivered 8× the throughput of the naive one-at-a-time server, and cut average per-request latency from 16.7s to 1.5s. - **Continuous Batching**: Continuous batching (the core trick in vLLM) evicts finished requests from the batch mid-flight

Overview: This repository provides a curated reading path for AI infrastructure, focusing on foundational concepts and practices for engineers, architects, and technology leaders building production AI-native systems. Key features: - What AI Infrastructure Includes in an AI-Native Stack - The AI Inference Stack Explained - Inference Infrastructure is Where AI Features Survive Production - The Architecture of a Production LLM Inference Platform - Horizontal vs Vertical Scaling for GPU-Backed AI Workloads - Why AI Infrastructure Needs Token-Level Observability Tech stack: Articles on inference platforms, model gateways, caching, scaling, GPU infrastructure, and API management. Why it matters: This repository offers a comprehensive guide to understanding and implementing AI infrastructure, ensuring that the operational foundation of production systems is robust and reliable.

Overview: MOSS-VL is a simple tool designed for visual image analysis using artificial intelligence, allowing users to understand scenes, identify objects, and read text within images without needing to understand code. Key features: - Processes visual data - Describes scenes - Identifies objects - Reads text in images Tech stack: Python and OpenCV Why it matters: MOSS-VL simplifies image analysis by automating tasks that would otherwise require coding, making it accessible for non-technical users. Repository: Erikalaylafajri15/MOSS-VL Primary language: Unknown GitHub stars: 0 GitHub description: Process visual and textual data with this vision-language model for multimodal reasoning and image understanding tasks.