Projects

### Overview: The ROSClaw platform pack for the Iluvatar TY1200 is a complete solution designed to enable controlled, auditable, and agent-operable operations on the Iluvatar TY1200 embodied-AI compute box. This platform includes Python packages, skills, providers, and a digest-pinned Docker Compose deployment, making it compatible with OpenAI-compatible models. ### Key Features: - **Platform Detection and Telemetry**: Detects the hardware and provides telemetry data. - **Model Serving Daemon**: A controlled-ops daemon for model serving. - **Provider Assets**: 4 provider assets including local and remote LLM services. - **Deployment**: Docker Compose, systemd units, and an installer. - **Skills**: A platform operations skill with executable runner. ### Tech Stack: Python ### Why It Matters: This platform is designed to simplify and secure the operations of the Iluvatar TY1200, making it more accessible and auditable. The controlled operations and digest-pinned deployment ensure that the platform is reliable and secure, while the skills and providers ensure that the AI operations are both efficient and compliant with the requirements of the system. ### Repository: - **Repository**: ros-claw/ty1200-platform - **Primary Language**: Python - **GitHub Stars**: 0 - **GitHub Description**: ROSClaw platform pack for the Iluvatar TY1200 embodied-AI box: executable ops skill, controlled modeld, OpenAI-compatible providers (Cosmos-Reason2

Overview: This project is an end-to-end MLOps platform designed for credit-card fraud scoring. It leverages Kaggle GPU training, MLflow for model versioning, DVC for data versioning and pipeline management, drift monitoring, and a multi-agent FastAPI serving layer. Key features: - Imbalance-aware modeling using PR-AUC as the primary metric - Two-model benchmark: XGBoost vs PyTorch MLP - Remote GPU training via Kaggle Kernels - Single feature contract for consistent training and serving - Experiment tracking and MLflow model registry - Reproducible data and training pipeline Tech stack: Python Why it matters: This project demonstrates a complete, reproducible MLOps platform that addresses the complexities of credit-card fraud detection in a regulated environment. It shows how to handle imbalanced data, train models remotely, and ensure explainable, rule-aware decision-making in a production-ready system.

### Overview: InferenceOS is an adaptive AI inference runtime built on llama.cpp, designed for local Large Language Model (LLM) inference. It optimizes execution in real-time, profiles system hardware capabilities, and dynamically sizes microbatches to eliminate OOM errors and improve resource efficiency. ### Key Features: - **Dynamic Execution**: Adapts execution in real-time to optimize performance and prevent OOM errors. - **Layer Placement Optimization**: Uses integer linear programming to place layers efficiently, reducing VRAM pressure. - **Prefill Microbatch Management**: Dynamically sizes microbatches to avoid OOMs and improve throughput. - **Telemetry Recording**: Records telemetry in a persistent SQLite database for runtime learning and optimization. - **Hardware Profiling**: Profiles system hardware capabilities (NVIDIA, AMD, Apple Silicon, Intel Arc, and integrated GPUs) to optimize performance. ### Tech Stack: InferenceOS is built on C++20, Python, CUDA, ROCm, Vulkan, and Metal, utilizing high-performance tensor execution engines. ### Why It Matters: InferenceOS addresses critical hardware boundaries and resource management challenges faced by local LLM deployment, providing an adaptive, self-tuning runtime layer that optimizes performance and resource efficiency.

Overview: A simple, self-hosted vLLM solution with a Git-friendly YAML configuration for models and profiles, offering a user-friendly CLI and TUI for full lifecycle management. Key features: - Self-hosted vLLM solution - Git-friendly YAML configuration - Intuitive CLI and TUI for easy management Tech stack: Python Why it matters: This project simplifies the deployment and management of large language models, making it accessible and easy to use for developers and researchers alike.

Overview: Modelforge is a model-serving platform written in Go that verifies XGBoost predictions, supports dynamic request batching, canary and shadow routing, and includes PSI drift detection. Key features: - Verifies XGBoost predictions - Supports dynamic request batching - Canary and shadow routing - PSI drift detection Tech stack: Go Why it matters: Modelforge ensures consistent model predictions across different versions, supports efficient handling of concurrent requests, and includes advanced monitoring features to detect model drift.

### Overview: California Property Valuator is a machine learning app powered by FastAPI, serving a Random Forest model trained on California census data. It provides single price estimates with a margin of error and bulk CSV valuations, offering an average error of approximately $37,778. ### Key features: - **Single Estimate:** Enter a block's income, housing stats, and coordinates to get an instant price estimate with a margin of error. - **Bulk Appraisal:** Upload a CSV of multiple properties and download predictions for all of them at once. - **REST API:** `/predict` and `/predict-file` endpoints for programmatic access. ### Tech stack: - **Backend:** FastAPI, scikit-learn (Random Forest Regressor), pandas, joblib - **Frontend:** Vanilla HTML/CSS/JS, served directly by FastAPI ### Why it matters: The California Property Valuator is a valuable tool for real estate professionals and investors, offering a quick and cost-effective way to estimate property values. Its model's average error of about $37,778 makes it a reliable resource for preliminary assessments. Repository: saimera/california-property-valuator Primary language: HTML GitHub stars: 0 GitHub description: California Property Valuator is a machine learning app powered by FastAPI backend serving a Random Forest model trained on California census data.It gives single price estimates with a margin of error, plus bulk CSV valuations, with an average error of about $37.8K.

Overview: TrackAndServe is an end-to-end MLOps project that connects model experimentation with an actual serving contract. It trains a reproducible classifier, records the run with MLflow, packages the selected model, and serves predictions through a Dockerized FastAPI application. Key features: - MLflow metrics tracking - Versioned model bundle - Typed API request and response contracts - Health endpoint for orchestration checks - Container builds the model from source instead of relying on an opaque committed pickle - Unit tests cover training quality and inference shape Tech stack: Python Why it matters: TrackAndServe provides a practical solution for connecting model experimentation with a real-world serving contract, ensuring reproducibility and traceability of the model development process.

### Overview A multimodal AI service that combines vision and language understanding to answer questions about images. The model reads both the image and the text together, making it suitable for applications where visual and textual information need to be integrated. ### Key features - **Visual Question Answering:** Answers questions like "How many people are there?" or "What color is the car?" by considering both the image and the question. - **Top Answers with Confidence:** Provides five answers by default, each with a score, allowing clients to decide which answer to trust. - **REST API and Web UI:** Supports `POST /ask` for sending images and questions, `GET /health` for readiness checks, and interactive OpenAPI documentation at `/docs`. - **Input Validation:** Ensures that the image and question are properly formatted and validated before processing. ### Tech stack - **Model:** ViLT (Vision and Language Transformer) - **Framework:** FastAPI - **Deployment:** Docker ### Why it matters This service is particularly useful for developers who need a small, production-ready example of serving a vision-language model. It can be easily integrated into existing software systems, making it accessible to a wide range of applications.

### Overview: MirbadLLM is a self-hosted service that allows you to run small and medium AI models on your spare machines, managed from a single OpenAI-compatible endpoint. This eliminates the need for separate projects, runtime configurations, and ports, making it a cost-effective and efficient solution for running AI models on your existing hardware. ### Key Features: - **Self-Hosted and OpenAI-Compatible**: Runs on your existing machines, managed from a single endpoint. - **Model Registry and Key Management**: Centralized model registry and key management. - **Browser UI**: Simple and intuitive UI for managing models and configurations. - **Automatic Model Selection**: Models are automatically selected based on available resources. - **Memory and Throughput Monitoring**: Continuous monitoring of memory and throughput to ensure optimal performance. - **API Keys Management**: Automatic generation and management of API keys for secure access. ### Tech Stack: C# ### Why It Matters: MirbadLLM simplifies the management and deployment of AI models, making it a practical solution for small and medium-sized projects. It eliminates the need for separate projects and configurations, reducing complexity and maintenance overhead. The ability to run models on your existing hardware is particularly useful for those with limited resources.

Overview: This project focuses on deploying and serving a machine learning model, specifically a penguin species classifier, using Python, FastAPI, and joblib for model serialization. Key features: - Deploying and serving a trained machine learning model - Wrapping a model in a simple API for remote predictions - Saving and loading a trained model - Validating inputs and handling errors gracefully Tech stack: Python, FastAPI, joblib Why it matters: This project demonstrates the process of training a machine learning model, saving it, and then serving it over the internet, allowing others to use the model without needing to retrain it every time. Repository: sandra8918/ml-06-serving-sandra Primary language: Python GitHub stars: 0 GitHub description: Machine learning model serving project using Python, FastAPI, and a penguin species classifier. README excerpt: # ml-06-serving-sandra [](https://denisecase.github.io/pro-analytics-02/workflow-b-apply-example-project/) [](./pyproject.toml) [](./LICENSE) > Professional Python project: deploying and serving machine learning models. --- ##

### Overview: This project is an end-to-end cross-sell propensity pipeline utilizing scikit-learn, FastAPI, MLflow, and Docker. The model achieves an ROC AUC of 0.798, a 2.5x lift in the top decile, and 47 tests with a CI that is green. ### Key Features: - **Training**: Scikit-learn pipeline for imputation, scaling, one-hot encoding, and SMOTE. - **Experiment Tracking**: MLflow for optional experiment tracking. - **Serving**: FastAPI with a typed request/response contract. - **Packaging**: Docker + docker-compose for deployment. - **Quality**: Unit tests with pytest and API tests, CI (tests + image build). ### Tech Stack: TypeScript ### Why it Matters: The model's performance, as demonstrated by the ROC AUC score of 0.798, indicates a high level of predictive accuracy. The 2.5x lift in the top decile suggests significant improvement in the model's ability to identify potential customers who are likely to buy a second insurance product. The 47 tests and green CI status indicate robust and reliable model performance, which is crucial for a successful implementation in a real-world scenario. Repository: `sanju-zxt/Insurance-crosssell-mlops` Primary Language: TypeScript GitHub Stars: 0 GitHub Description: End-to-end cross-sell propensity pipeline: scikit-learn + FastAPI + MLflow registry + Docker. roc_auc 0.798, 2

### Overview: Blazar is a multi-engine and multimodal local inference platform. It routes models from various engines to the appropriate engine lane and serves them through OpenAI- and ollama-compatible APIs, with durable jobs, federation, speech, vision, and image lanes behind the same CLI. ### Key Features: - Unified OpenAI, Ollama, and Anthropic gateway. - Multi-engine serving with llama.cpp, mistral.rs, SGLang, MLX, sd.cpp, whisper.cpp, and piper lanes. - VRAM/KV fit planning, slots, admission, co-residency, lifecycle management. - Durable jobs, request cancel/interrupt, and restart-safe response chains. - Engines that update without breaking. ### Tech Stack: Rust ### Why It Matters: Blazar simplifies the process of running AI models locally, making it easier to integrate with various engines and models without the need for a hosted service. It ensures that models can be updated without breaking the system and manages resources efficiently, making it a valuable tool for developers and researchers working with local AI inference.

Overview: This project provides a FastAPI backend for the FundFirst Logistic Regression pipeline, serving live prediction and probability endpoints. Key features: - Real-time prediction and probability endpoints - Deployed on Render for easy integration - Supports Lovable's API for prediction requests Tech stack: Python Why it matters: This backend streamlines the process of making predictions using a saved scikit-learn model, ensuring quick and accurate responses without the need for complex backend logic.

Overview: This repository serves as the entry point to a multi-repo system designed for the production-grade ML platform portfolio, catering to real-time decisioning systems. Key features: - Real-time data ingestion - Point-in-time feature engineering - Feature storage and retrieval - Model training and evaluation - Real-time inference with feature reconstruction - Operational monitoring and feedback - Deterministic decisioning Tech stack: Python, Apache Flink, DuckDB, PostgreSQL, Kafka, Debezium CDC, Flink, Parquet, DuckDB, Apache Spark, Apache Hadoop, Docker, Kubernetes Why it matters: This platform is crucial for real-time decisioning systems, ensuring efficient and accurate ML operations across various layers of production.

Overview: Serkanandre/vllmops is a self-hosted vLLM solution that simplifies the process of managing and deploying large language models. It provides a Git-friendly YAML configuration for models and profiles, along with an intuitive CLI and TUI for a seamless lifecycle management. Key features: - YAML-based configuration for models and profiles - Self-hosted vLLM solution - Git-friendly configuration - Intuitive CLI and TUI for lifecycle management Tech stack: Python Why it matters: Serkanandre/vllmops offers a straightforward and efficient way to manage and deploy large language models, making it ideal for developers and teams looking to streamline their vLLM operations.

Overview: Sentinel is a model-serving gateway designed to provide observability and cost accounting for model requests, with features such as per-request cost accounting, W3C tracing, circuit breakers, and load testing capabilities. It is compatible with OpenAI and OpenAI-compatible models. Key features: - Per-request cost accounting - W3C tracing - Circuit breakers - Load testing - OpenAI-compatible Tech stack: Python Why it matters: Sentinel addresses the challenges of cost visibility, latency, and failure management in model-serving operations, providing a solution that is both efficient and cost-effective. Repository: shahriar-ahmed-seam/sentinel Primary language: Python GitHub stars: 0 GitHub description: Model-serving gateway: cheapest-capable routing, per-request cost accounting, W3C tracing, circuit breakers and load testing. OpenAI-compatible.

### Overview: This project is a CPU-only sentiment model serving solution using FastAPI, ONNX Runtime, and INT8 quantization. It includes reproducible latency, memory, and prediction-agreement benchmarks. ### Key Features: - **CPU-only Model Serving**: The project focuses on CPU-only model serving, ensuring low resource usage. - **FastAPI Integration**: Utilizes FastAPI for efficient API endpoints. - **ONNX Runtime**: Optimized implementation of ONNX Runtime for inference. - **INT8 Quantization**: Uses dynamic INT8 quantization for reduced memory and latency. ### Tech Stack: Python ### Why It Matters: The optimized implementation achieves lower p99 latency and reduced peak container memory compared to the baseline, demonstrating the effectiveness of CPU-only model serving and dynamic quantization in improving performance under controlled CPU and memory limits.

Overview: `local-inference-pool` is a multi-server GPU slot management tool designed for OpenAI-compatible inference backends, specifically for LM Studio and other compatible systems. Key features: - **Slot Management**: Tracks per-server state to prevent model swap collisions and load imbalance. - **ServerPool**: Synchronous and atomic for finding and acquiring models. - **ConcurrentDispatcher**: Queue-based dispatch to manage concurrent requests efficiently. - **ServerConfig**: Tracks per-server configuration and state. Tech stack: Python Why it matters: This tool ensures smooth and efficient inference across multiple servers, preventing model swap collisions and load imbalances, making it crucial for applications requiring high throughput and stability. Repository: shanevcantwell/local-inference-pool Primary language: Python GitHub stars: 0 GitHub description: Multi-server GPU slot management for OpenAI-compatible inference backends.

Overview: The Automated Sentiment Intelligence Engine (ASIE) is a production-grade MLOps platform designed to address the "Static Model" problem in ML, offering a modular, reproducible, and testable pipeline for the end-to-end lifecycle management of NLP sentiment models. Key features: - Modular Pipeline Design - Configuration-Driven Execution - Comprehensive Experiment Tracking - Immutable Model Promotion - Advanced Data Versioning - Secure Cloud Deployment - Kubernetes-Native Orchestration - Structured Inference Logging - Multi-Signal Drift Detection - Event-Driven Alerting Pipeline - GitOps Delivery - Progressive Delivery - Evidence-Based Promotion - Automated Rollback Tech stack: Python Why it matters: ASIE provides a robust, production-grade solution for managing and deploying NLP sentiment models, ensuring data integrity, secure deployment, and reliable monitoring, making it essential for organizations looking to improve the maturity of their MLOps practices. Repository: sharjeel6392/asie Primary language: Python GitHub stars: 0 GitHub description: The ASIE is a production-grade MLOps platform designed to solve the "Static Model" problem in ML. A Level 2 MLOps Maturity architecture—a self-healing, closed-loop system that monitors for statistical data drift, triggers automated retraining on GPU-accelerated cloud infrastructure, and redeploys optimized models without manual intervention.

### Overview: ModelForge is an end-to-end ML platform designed for model registry, experiment tracking, deployment, serving, and monitoring. Built with FastAPI, React, PostgreSQL, Redis, MinIO, and Celery, it offers features like model registry, experiment tracking, model serving, deployment management, A/B testing, monitoring, and a dashboard. ### Key Features: - **Model Registry**: Register, version, and manage ML models with semantic versioning and stage transitions. - **Experiment Tracking**: Log runs, parameters, and metrics; compare experiments side-by-side with metric history. - **Model Serving**: Real-time and batch inference with automatic model caching, input validation, and output processing. - **Deployment Management**: Deploy model versions with configurable resource limits, auto-scaling, and health checks. - **A/B Testing**: Run controlled experiments across model variants with traffic splitting. - **Monitoring**: Detect data drift, track prediction performance over time, and receive alerts on anomalies. - **Dashboard**: At-a-glance stats, recent models, deployment status, and drift summaries. ### Tech Stack: - Frontend: React 18, Tailwind CSS, Zustand, Recharts, Axios - Backend: FastAPI, SQLAlchemy 2.0 (async), Pydantic v2, structlog - Database: PostgreSQL 16 - Cache / Queue: Redis 7, Celery - Object Storage: MinIO (S3-compatible) - Infrastructure: Docker, Docker Compose, Nginx ### Why it Matters: ModelForge provides a comprehensive solution