Projects

Overview: Fornax is a self-hosted open-weight model serving solution that freezes weights, serves, and benchmarks models on a single Go binary. This approach ensures weight security and cost efficiency, allowing users to control their own GPUs and bypass third-party rate limits. Key features: - Weight freeze: Models are pinned to specific revisions and checksums, ensuring they remain secure and up-to-date. - Cost and control: Users can run their own GPUs at a lower cost compared to cloud-based solutions, and they can control their own weight storage. - Multi-GPU fleet: Supports Kubernetes for a fleet of GPUs, and an installed binary for a single-GPU host. - End-to-end inference: Provides OpenAI Chat, Anthropic Messages, and OpenAI Responses endpoints, ensuring compatibility with various client SDKs. - Real-time performance monitoring: Measures and reports latency and throughput for accurate performance metrics. Tech stack: Go Why it matters: Fornax offers a secure, cost-effective, and flexible solution for running open-weight models, allowing users to control their own GPUs and bypass third-party rate limits, making it ideal for those looking to run models on their own infrastructure. Repository: latere-ai/fornax GitHub stars: 1 GitHub description: Fornax is a self-hosted open-weight model serving solution that freezes weights, serves, and benchmarks models on a single Go binary. This approach ensures weight security and cost efficiency, allowing users to control their own GPUs and bypass third-party rate limits. Key features: - Weight freeze: Models are pinned to specific revisions and

Overview: `quantpreflight` is a tool designed to determine if a quantized AI model will run on your GPUs before downloading it. Key features: - Checks formats, kernels, sharding, tokens, files, and memory from headers only. - Answers in seconds, reads headers, not weights. - Needs no GPU, runs on a laptop or in CI. - Checks the whole mix, 16 rules cover model, format, engine, GPU, and GPU count. - Tells you what works, gives GPU count and context length for `vllm serve` command. - Tested on real hardware, predicted 26 of 27 real vLLM loads on Tesla T4. - Finds known vulnerabilities, checks vLLM or SGLang version against published advisories. - Fits your workflow, supports CLI, GitHub Action, pre-commit hook, SARIF output, and browser scanner. Tech stack: Python 3.10 or newer. Why it matters: `quantpreflight` helps developers quickly identify compatibility issues with quantized AI models, ensuring they can run efficiently on their GPUs without wasting time or resources.

Overview: This project is an AI-powered medical imaging pipeline that processes DICOM images using NVIDIA MONAI, Databricks Model Serving, and a premium interactive web viewer built with React. It includes a pre-trained spleen CT segmentation model and real-time GPU inference capabilities. Key features: - 🖼️ Premium interactive viewer with zoom, pan, and AI overlay rendering - 🤖 MONAI inference for pre-trained spleen CT segmentation model - 🚀 Databricks Model Serving for deployment - 📡 REST API for real-time inference - 📊 Bounding box overlays for visual AI findings Tech stack: React, FastAPI, MONAI, PyTorch, MLflow, Databricks Model Serving, NVIDIA MONAI, Databricks, CSS Why it matters: This pipeline provides a comprehensive solution for medical imaging analysis, enabling real-time inference and interactive visualization of medical images, making it valuable for healthcare professionals and researchers. Repository: lohitganugula/Medical-Imaging-Pipeline GitHub stars: 1 GitHub description: 🧠 AI-powered medical imaging analysis with NVIDIA MONAI, Databricks Model Serving, and an interactive web viewer built with React. Tech stack: React, FastAPI, MONAI, PyTorch, MLflow, Databricks Model Serving, NVIDIA MONAI, Databricks, CSS Why it matters: This project is an end-to-end pipeline for medical imaging analysis, combining deep learning, GPU inference, and an interactive web viewer for real-time analysis and visualization.

Overview: Pulsar Inference Stack is a model serving, file management, and recovery tool designed for NVIDIA DGX Spark clusters, enabling precise and inspectable VLLM configurations on these systems. Key features: - Manages model files and prepares them on selected machines. - Checks and starts or stops the service. - Provides a public repository for browsing, restoring, preparing, and serving published specifications. - Supports a private workbench for model onboarding and experiments. Tech stack: Python Why it matters: Pulsar simplifies the process of managing and serving machine learning models on DGX Spark clusters, ensuring efficient and reliable operations. Repository: luisefigueroa/pulsar-inference-stack Primary language: Python GitHub stars: 1 GitHub description: Model serving, model-file management, and recovery workflows for NVIDIA DGX Spark.

Overview: ModelRack is a control plane designed for managing generative AI models, offering a single source of truth, layered configuration, and isolated environments. It simplifies the setup and management of models without requiring heavy dependencies. Key features: - Zero ML dependencies in the hub. - Every model is fully isolated, with its own `.venv` and pinned `requirements.txt`. - Config is layered, with a base config, app overrides, and runtime parameters. - Config schema drives UIs, making parameters editable dynamically. - Open, extensible model types. - Supports both local and API models, with a unified interface. Tech stack: Python Why it matters: - **Zero ML dependencies in the hub.** `modelrack` depends only on `pyyaml`, `pydantic`, `typer`, `fastapi`, `uvicorn`, `watchdog`, `httpx`, `python-dotenv`, and `rich`. Your apps install `modelrack` and nothing heavy. - **Every model is fully isolated.** Each model has its own `.venv`, its own pinned `requirements.txt`, and its own isolated server. Version conflicts between models are structurally impossible — or, when several models share a compatible dependency stack, they can opt into one shared venv (`environment.shared_venv`) to build heavy dependencies like `torch` once. - **Config is layered.** `base config.yaml` → app overrides → runtime params. Later wins, deep-merged. One place to change a default; apps and UIs layer on top. -

### Overview: A Kubernetes operator designed to safely deploy decision models, ensuring that new versions are evaluated on a golden dataset before being promoted to production, preventing regressions and rollbacks. ### Key features: - Evaluates a candidate model on the golden dataset before deployment. - Blocks or rolls back models that regress. - Runs the candidate model against the golden dataset and production model. - Compares the results against predefined thresholds. - Switches the Service to the candidate model only if all gates pass, otherwise keeps production on the old model. ### Tech stack: Go ### Why it matters: Ensures that new versions of decision models are evaluated and validated on a known, stable dataset before being deployed, preventing regressions and ensuring a smooth transition to production.

### Overview: InferenceFlow is a prepaid asynchronous LLM inference platform built on FastAPI, PostgreSQL-backed queues, deadline-aware scheduling, and modular GPU workers. It includes vLLM integration, token metering, ledger billing, Stripe test payments, a Next.js dashboard, and a Python SDK. The platform supports real GPU validation and includes a mock mode and recorded walkthrough. The code has passed 254 tests and 31 scheduler regressions, and real GPU evidence is in a separate experiment report. ### Key features: - Real GPU validation - Modular GPU workers - Deadline-aware scheduling - Token metering - Ledger billing - Stripe test payments - Python SDK - Mock mode and recorded walkthrough - Real GPU cost displayed immediately after runs ### Tech stack: Python ### Why it matters: InferenceFlow provides a reliable and auditable solution for asynchronous LLM inference, ensuring real GPU validation and real-time cost display. The platform supports real GPU validation and includes a mock mode and recorded walkthrough, making it a valuable tool for developers and businesses looking to optimize their LLM inference processes.

### Overview: BentoML is an unofficial async Rust client for BentoML services, providing features such as typed calls, async task queues, file and streaming I/O, and resilient transport. This crate simplifies interaction with BentoML services by exposing their methods as HTTP endpoints. ### Key Features: - **Typed Calls**: `endpoint(route).invoke(&req)` over your own `serde` request and response types, with no codegen or per-service bindings. - **Async Task Queues**: `client.task(route)` submits `@bentoml.task` jobs, then poll status, fetch results, retry, or cancel through a `TaskHandle`. - **File and Streaming I/O**: `multipart/form-data` file inputs, raw-binary root inputs, binary responses, and chunked streaming endpoints. - **Resilient Transport**: exponential-backoff retries via `reqwest-middleware`, scoped to idempotent methods. ### Tech Stack: - Rust ### Why it Matters: This crate simplifies interaction with BentoML services, making it easier to integrate with their APIs without the need for complex codegen or per-service bindings. It supports various endpoints and provides robust error handling, making it a valuable tool for developers working with BentoML services.

### Overview: Multihull is a Python control plane designed to deploy and manage GPU inference containers on multiple providers, ensuring reliability and cost-effectiveness. It simplifies the process by translating specifications into native resources, providing a single URL for serving and failover between providers. ### Key Features: - **Deployment and Failover**: Automatically scales between providers based on request, ensuring high availability. - **Single URL Endpoint**: Provides a stable endpoint for serving inference containers, eliminating the need for multiple URLs. - **Reliability Over Cost**: Defaults to maintaining two providers warm, avoiding unnecessary scaling of fallbacks to zero, and prioritizing on-demand over spot instances. ### Tech Stack: - **Python**: The primary language for the control plane. - **Rust**: Used for the stateless router to score endpoints, retry requests, and maintain session stickiness. ### Why It Matters: Multihull simplifies the deployment and management of GPU inference containers across multiple providers, ensuring reliability and cost-effectiveness. It provides a single, stable endpoint and automatically scales between providers based on request, making it ideal for applications requiring high availability and cost optimization. ### Repository: - **Primary Language**: Python - **GitHub Stars**: 1 - **Repository**: `mishraprafful/multihull` - **GitHub Description**: Deploy always-warm GPU inference containers to many providers from one `multihull.yaml`, serve them behind one URL, and fail over between providers per request. Built for reliability over cost. ### Quickstart: ```sh uv tool install 'multihull[

Overview: OrcaBonsai-27B-Uncensored is a tool designed to improve the performance and stability of compressed AI models, specifically targeting the Ternary Bonsai 2 27B model. It removes problematic behaviors without modifying the AI's core files, ensuring a smoother and more predictable conversation experience. Key features: - Behavioral ablation to fix inconsistent AI behavior - No weight modification to preserve the original AI model's quality - No re-quantization to maintain the best version of the compressed model - Optimized for Ternary Bonsai 2 27B - Windows-friendly Tech stack: Python Why it matters: OrcaBonsai-27B-Uncensored enhances AI interactions by stabilizing and improving the performance of compressed models, making them more reliable and user-friendly. Repository: mounge4/OrcaBonsai-27B-Uncensored Primary language: Python GitHub stars: 1 GitHub description: Abliterate Ternary Bonsai 2 27B at runtime, no weight modifications, preserving original quantization.

Overview: This project is a production-style machine learning inference API for detecting fraudulent credit card transactions using FastAPI and XGBoost. It serves a trained XGBoost fraud detection model via FastAPI, containerized with Docker, and automatically deployed via a GitHub Actions CI/CD pipeline. Key features: - FastAPI-based inference service - XGBoost fraud detection model - Strict request validation using Pydantic schemas - Feature name and order alignment with training schema - Partial preprocessing during inference (Time & Amount scaling) - Clear error handling with meaningful HTTP responses - Model metadata endpoint for observability - JSON-safe prediction outputs - Dockerized for consistent deployment anywhere - CI/CD pipeline via GitHub Actions → auto-builds and pushes to Docker Hub on every push Why it matters: This project focuses on the serving and inference layer of a fraud detection system, ensuring that models are validated, preprocessed properly, and safely deployed behind an API, making them ready for real-world production use. Repository: msaad-dot/fraud-detection-api Primary language: Python GitHub stars: 1 GitHub description: Production-style ML inference API for credit card fraud detection using FastAPI and XGBoost

### Overview: This project is a YOLOv8 object-detection service built on FastAPI, utilizing PyTorch, ONNX Runtime, OpenVINO, and TensorRT backends. It supports static INT8 quantization and reproducible benchmarks, making it highly versatile and efficient. ### Key features: - **Supports Multiple Backends**: Runs on PyTorch, ONNX Runtime (CPU/CUDA, FP32/INT8), OpenVINO (FP32/INT8), and TensorRT (FP16). - **Static INT8 Quantization**: Ensures consistent performance across different environments. - **Benchmarking**: Measures accuracy, latency, and memory usage on the same machine with identical evaluation code. - **Exported Models**: Models are exported to ONNX FP32/INT8, OpenVINO IR, and TensorRT engine, ensuring compatibility and accuracy. ### Tech stack: - Python 3.10+ (FastAPI) - PyTorch - ONNX Runtime - OpenVINO - TensorRT ### Why it matters: This service is crucial for developers and researchers looking to deploy efficient and accurate object detection models in production environments. Its ability to handle multiple backends and provide reproducible benchmarks makes it highly versatile and beneficial for various applications.

Overview: This project is a policy-driven ModelOps control plane designed for canary deployments, automated model promotion, rollback, and performance benchmarking. It ensures that new model versions are introduced gradually and continuously monitors their performance and quality. Key features: - Gradual ramping of new model versions into live traffic - Ingestion of delayed ground truth through a real API surface - Comprehensive evaluation of reliability and model quality - Autonomous promotion or rollback based on policy decisions - Ability to take specific deployments out of automation without stopping the loop - Synchronization of database desired traffic split with router's actual traffic Tech stack: Python Why it matters: This control plane is crucial for ensuring the reliability and quality of model deployments, especially in high-traffic environments where latency and model accuracy are critical. Repository: negativexq/modelops-control-plane GitHub stars: 1 GitHub description: Policy-driven ModelOps control plane for canary deployments, automated model promotion, rollback, and performance benchmarking.

### Overview: TokenVector.Inference is an ultra-low-latency deep learning inference runtime and in-process micro-serving engine designed for the TokenVector AI ecosystem. It is built with strict Zero-GC memory allocation in pure C# and TokenVector (`.tkv`) language, making it ideal for high-performance CPU inference. ### Key Features: - **Ultra-Low Latency**: Designed for ultra-low latency inference, TokenVector.Inference is optimized for real-time applications. - **Zero-GC Allocation**: The engine is built with strict Zero-GC memory allocation, ensuring efficient and low-overhead operations. - **In-Process Micro-Serving**: TokenVector.Inference operates in-process, providing fast and efficient micro-serving capabilities. - **Cross-Platform Compatibility**: The engine supports CPU, TokenVector SIMT, CUDA, OpenCL, and OpenVINO, ensuring compatibility across different hardware platforms. ### Tech Stack: Pure C# with strict Zero-GC memory allocation. ### Why it Matters: TokenVector.Inference is a crucial component of the TokenVector AI ecosystem, designed to optimize deep learning inference for high-performance CPU environments. Its strict Zero-GC memory allocation and in-process micro-serving capabilities make it an essential tool for developers working with the TokenVector ecosystem. Repository: nguyenhungtran18/TokenVector.Inference Primary language: PowerShell GitHub stars: 1

### Overview: LlamaLift 是一款面向 Windows 平台的 llama.cpp 桌面控制中心，提供从运行时安装、硬件识别到性能监测和多协议 API 接入的一站式服务。它简化了对本地模型的管理，包括参数调整、服务生命周期和性能监控等功能。 ### Key features: - **运行时安装与配置**：从官方 Release 获取 Windows x64 构建，支持多种后端如 CUDA、Vulkan、CPU、SYCL、HIP。 - **硬件与模型识别**：检测并识别硬件配置，如 CPU、内存、GPU、显存等，以及模型架构、上下文、层数、GQA 和量化信息。 - **自适应参数调整**：提供三档自适应参数方案，快速、均衡、极限目标，帮助用户根据需求调整参数。 - **参数工作台**：编辑完整 llama-server 命令，解析后反向同步表单，保存并保留未知参数。 - **服务管理与启动**：管理启动、停止、重启、端口释放和异常退出，减少残留进程和端口冲突。 - **性能监测与实时面板**：实时监测系统资源和 llama.cpp `/metrics`、`/slots` 数据，展示最近 90 个采样点。 - **API Key 管理**：创建、生成、导入、脱敏查看和选择本地 Key 文件，确保密钥安全。 - **小白友好反馈**：软件内填写问题、粘

Overview: This repository contains a comprehensive, hands-on curriculum for mastering MLflow, covering everything from tracking runs to production-ready serving, evaluation, and hyperparameter tuning. Key features: - Tracking - Experiments - Autologging - Model Registry - Serving - Projects - Pipeline - Evaluation - Custom Models - Hyperparameter Tuning Tech stack: Python Why it matters: This curriculum provides a structured sequence of modules that teach you how to use MLflow from the ground up, from tracking runs to production-ready serving, evaluation, and hyperparameter tuning. The scripts run top-to-bottom and log real data to MLflow, allowing you to explore the UI as you learn. GitHub stars: 1 Repository: rohanmistry231/MLflow-Mastery

Overview: This project is an end-to-end MLFlow pipeline built with MLflow and hosted on AWS. Key features: - MLFlow integration for tracking and managing the ML workflow - AWS deployment for the ML pipeline - Detailed documentation and blog posts Tech stack: Python Why it matters: This project showcases the integration of MLflow with AWS, providing a comprehensive solution for ML workflow management and deployment.

### Overview: Este manual técnico aberto oferece um guia completo sobre a engenharia de sistemas de IA em ambientes corporativos, incluindo ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps, e deployment. ### Key features: - **ML tradicional**: Desenvolvimento e implementação de modelos de aprendizado de máquina. - **LLMs (Large Language Models)**: Exploração e aplicação de modelos de grandes linguagens. - **RAG (Relevance-Augmented Generation)**: Adaptação de modelos de linguagem para melhorar a relevância dos resultados. - **Agentes**: Implementação de sistemas de IA como agentes de negócios. - **Segurança**: Proteção e monitoramento de sistemas de IA. - **Observabilidade**: Monitoramento e análise de performance dos sistemas de IA. - **FinOps (Financial Operations)**: Gestão financeira eficiente de sistemas de IA. - **Deployment**: Implementação e manutenção de sistemas de IA em ambientes corporativos. ### Tech stack: Markdown ### Why it matters: Este guia técnico aberto é uma ferramenta valiosa para engenheiros, cientistas de dados, e profissionais de TI que desejam entender e gerenciar sistemas de IA em ambientes corporativos. ### Repository: ruanpato/production-ai-engineering ### Primary language: Markdown ### GitHub stars: 1 ### GitHub description: Handbook técnico aberto sobre eng

Overview: Triton Inference Server control plane is a comprehensive solution for managing model repositories, monitoring VRAM usage, and performing inference tasks via gRPC with FastMCP 3.2. Key features: - Model repository manipulation - VRAM monitoring - Server metrics - Config tuning - Direct gRPC tensor inference Tech stack: Python Why it matters: This tool provides a dynamic GPU infrastructure management solution for AI assistants, allowing for efficient GPU utilization and model management without the need for HTTP overhead. Repository: sandraschi/triton-mcp Primary language: Python GitHub stars: 1 GitHub description: NVIDIA Triton Inference Server control plane — model repository manipulation, VRAM monitoring, and inference over gRPC via FastMCP 3.2

### Overview: InferEdge is a production ML inference system optimized for high throughput and low latency, designed to handle 1000 requests per second with <10ms p99 latency and 99.9% uptime. It leverages Triton Inference Server, ONNX, and INT8 quantization for efficient model deployment and dynamic batching to achieve significant performance improvements. ### Key features: 1. **Model Optimization**: Supports FP32 → FP16 → INT8 with calibration. 2. **Serving**: Deployed with Triton Inference Server for dynamic batching and throughput. 3. **Observability**: Utilizes Prometheus for metrics and latency histograms. 4. **Deployment**: Runs on Docker with Kubernetes-ready health checks. ### Tech stack: Python, ONNX, Triton, Docker, Kubernetes. ### Why it matters: InferEdge simplifies the process of deploying and optimizing machine learning models for high-performance inference, making it an essential tool for developers looking to scale their ML infrastructure without compromising on performance or reliability.