Projects

### Overview: LoRA Slot Lineage is a machine-learning inference forensics challenge that reconstructs the historical identity of LoRA adapters as they move through reusable runtime slots. ### Key Features: - Reconstructs the lineage of LoRA adapters across various runtime events. - Supports slot reuse, reloads, evictions, reassignments, and runtime restarts. - Tracks logical adapter identities and specific loaded instances. - Incorporates restart, generation, epoch, and checkpoint boundaries for accurate lineage reconstruction. ### Tech Stack: Python ### Why it Matters: LoRA Slot Lineage is essential for understanding and debugging complex production inference systems that use LoRA adapters. It helps identify the correct adapter identity and logical state at any given time, ensuring accurate model behavior and performance optimization.

### Overview: dan is a system designed to serve System One models, providing fast and easy inference capabilities. It converts open-weight causal language models into System One models, returning answers in a standardized format. ### Key Features: - **System One Model Conversion:** Converts open-weight causal LM into a System One model. - **Calibrated Answers:** Returns answers in a standardized format (`noul`, `choice`, `score`), matching TypeSafe's Jev. - **Prefill Per Request:** Each request has its own prefill, ensuring consistency. - **No Generated Tokens:** No generated tokens, maintaining consistency across requests. ### Tech Stack: - Python ### Why It Matters: dan simplifies the process of converting and serving System One models, making it easier and more efficient for developers to integrate these models into their projects.

Overview: This project serves as an inference stack for Nvidia DGX Spark, providing a home-like environment for AI development. Primarily focused on VLLM models and single Spark instance, it aims to make advanced AI tools accessible to non-wealthy developers. Key features: - VLLM-based inference stack - Single Spark instance for efficient resource utilization Tech stack: Shell Why it matters: This project simplifies the process of AI development, making complex tools more accessible to individuals without extensive computing resources.

Overview: Self-hosted vLLM made simple, featuring a Git-friendly YAML configuration for models and profiles, a user-friendly CLI and TUI for managing the vLLM, and a straightforward setup process. Key features: - Git-friendly YAML configuration for models and profiles - User-friendly CLI and TUI for managing the vLLM - Full lifecycle management from installation to deployment Tech stack: Python Why it matters: Simplifies the process of setting up and managing vLLM models, making it accessible and easy to use for developers and researchers alike.

Overview: DGX Spark AI Studio is a sophisticated platform that transforms NVIDIA DGX Spark systems into intelligent multi-model neural conductors, orchestrating multiple AI models for complex tasks and creating cognitive workflow orchestration. Key features: - Intelligent model routing - Collaborative inference pipelines - Resource-aware execution Tech stack: Python, Flask, PyTorch, NVIDIA DGX Spark, NVIDIA drivers 550+ Why it matters: DGX Spark AI Studio extends AI infrastructure beyond serving models, enabling orchestration and collaboration among specialized models for complex tasks, enhancing performance and stability.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It features semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing resource usage and ensuring smooth operation in local AI labs.

Overview: This project is dedicated to creating a sophisticated computer vision application designed to detect and identify black marks on nuts, which are indicative of impurities, using cutting-edge technology. Key features: - Defect detection on nuts - Identification of black marks as impurities - Utilization of advanced computer vision algorithms Tech stack: Ruby Why it matters: This project is crucial for improving quality control in the manufacturing process, ensuring that only defect-free nuts are used, thereby enhancing product safety and customer satisfaction. Repository: redhat-ai-edge-pins-demo/redhat-ai-edge-pins-demo.github.io Primary language: Ruby GitHub stars: 0 GitHub description: This project focuses on developing a cutting-edge computer vision program to identify defects on nuts, specifically black marks representing impurities.

Overview: This repository demonstrates how to customize the Nvidia Triton server container for use with OpenShift Source to Image (s2i), enabling efficient deployment and management of machine learning models within the Kubernetes environment. Key features: - Customizable Triton server container - Support for model serving via PVC storage - Option to copy models from local storage - Use of Init Containers to set up models at runtime - Load models from an S3 bucket at runtime - Examples provided for further details Tech stack: Shell Why it matters: This project showcases best practices for integrating machine learning models with Kubernetes, making it easier to deploy and manage complex ML workflows within the cloud-native environment. Repository: redhat-na-ssa/demo-triton-yolo Primary language: Shell GitHub stars: 0 GitHub description: Customize Nvidia Triton to use OpenShift Source to Image building

Overview: A simple, self-hosted version of vLLM, offering a Git-friendly YAML configuration for models and profiles, complete with an intuitive CLI and TUI. Key features: - Self-hosted vLLM - YAML configuration for models and profiles - Git-friendly repository - CLI and TUI for easy management Tech stack: Python Why it matters: This project simplifies the process of managing and deploying large language models, making it accessible and easy to use for developers and enthusiasts alike.

### Overview: Spark-LLM is a model lifecycle management system designed for NVIDIA DGX Spark, providing stable and secure access to various LLMs and image-generation services. It ensures memory efficiency and live health checks, making it ideal for developers looking to manage and deploy models efficiently. ### Key Features: - **Memory-Aware Orchestration**: Ensures that models run efficiently on a unified memory system. - **Stable Consumer Aliases**: Ensures that external applications can reliably connect to the models without overcommitting system resources. - **Live Health Checks**: Provides live health checks to ensure the models are running correctly, rather than relying on stale process state. - **Test Mode**: Allows developers to test model operations without executing them, providing a safe environment for experimentation. - **vLLM, LLaMA.cpp, and Diffusers Support**: Supports multiple LLMs and image-generation services, including vLLM, LLaMA.cpp, and Diffusers. - **OpenAI-Compatible Endpoints**: Provides stable and secure access to OpenAI-compatible text, tool-calling, and image-generation services. ### Tech Stack: - Python 3.11 or newer - NVIDIA Container Toolkit - `python3-rich` for the terminal UI - Model weights stored locally at `/srv/models` by default - Optional: a CUDA-enabled static `llama-server` build for GGUF models ### Why It Matters: - **Efficient Memory Management**: Ensures that models run efficiently on a unified memory system, preventing overcommitment and potential crashes. - **St

### Overview: This project leverages Tabular foundational models to predict housing affordability in the United States, achieving a public score of 3.859. The TabPFN model, optimized for performance and interpretability, uses property-type features and monthly-change forecasts for ZIP-level predictions. The project includes reproducible evaluation, optimized inference, and model interpretation capabilities. ### Key Features: - TabPFN model for housing affordability prediction - Property-type features and monthly-change forecasts - Optimized inference for faster predictions - Model interpretability through grouped permutation importance and local group sensitivity ### Tech Stack: - Python 3.10+ - PyTorch with CUDA for GPU support ### Why It Matters: This project demonstrates the effectiveness of TabPFN in predicting housing affordability, achieving a public score of 3.859. The model's performance is optimized for both speed and accuracy, making it suitable for real-world applications.

### Overview: BIFROST is a correctness-first key-value state transport and storage infrastructure designed for Large Language Model (LLM) serving. It leverages immutable content-addressed objects and cross-language validation to ensure data integrity and reliability. ### Key features: - **Immutable Content-Accessed Objects**: Ensures that data is only visible after a durable store commit. - **Chunked Transfer**: Handles data in chunks for efficient transmission. - **Optional Multipath Retry**: Supports fallbacks in case of network failures. - **SQLite-backed Object Catalog**: Manages object metadata and prefixes. - **Manifests**: Provides a manifest for object identification. - **Eviction and `fsck`**: Handles object eviction and file system check. - **Replayable Fault Scenarios**: Simulates and records fault scenarios for testing. ### Tech stack: Python ### Why it matters: BIFROST is specifically designed to ensure the correctness of LLM serving by focusing on immutable and cross-language validation, making it an essential tool for maintaining the integrity of LLM models and their interactions.

### Overview: Epochless is a correctness-oriented lifecycle infrastructure designed for online-RL serving, focusing on failure-atomic publication, causal attestation, and compact proofs. It aims to maintain the identity of bytes and the causal history of emitted tokens without loss. ### Key features: - **Immutable Table Generations**: Ensures the integrity of behavior-policy revisions. - **Admission Leases and Revision-Aware KV Metadata**: Tracks and manages the lifecycle of revisions. - **Trusted Completion Fences**: Validates the completion of revisions. - **Equality-Validated Batch Decisions**: Ensures the correctness of batch decisions. - **Compact Proof Formats**: Provides a compact and checkable proof of causal history. - **Seven Finite-State TLA+ Models**: Covers lifecycle, rank publication, crash recovery, and more. ### Tech stack: Python ### Why it matters: Epochless offers a robust and efficient infrastructure for online-RL serving, ensuring the correctness and integrity of the data and its lifecycle management. ### Repository: riskywindow/Epochless ### Primary language: Python ### GitHub stars: 0 ### GitHub description: Correctness-oriented lifecycle infrastructure for continuously updated online-RL serving.

Overview: FissionSpec is an outcome-decoupled continuous batching system designed for speculative LLM serving, focusing on per-request scheduling, transactional KV state management, and reproducible CPU experiments. Key features: - Outcome-decoupled continuous batching - Per-request scheduling - Transactional KV state management - Reproducible CPU experiments Tech stack: Python Why it matters: FissionSpec addresses the issue of delayed correct hits due to missed requests in speculative LLM serving, ensuring that correct predictions do not cause incorrect outcomes to wait or be padded, thereby improving the efficiency and reliability of the system.

Overview: Switchyard is a hybrid inference control plane designed for local and remote model backends, featuring policy-based routing, typed route decisions, reproducible benchmarking, and evidence-driven optimization. Key features: - Policy-based routing: `latency_first`, `balanced`, `quality_first`, `local_only` - Admission control with per-tenant concurrency caps and queue bounds - Circuit breakers, session affinity, canary routing, shadow traffic - Typed route decisions with per-candidate scoring and fallback reasoning - MLX-LM on Apple Silicon (local), vLLM-Metal on Apple Silicon (local), vLLM-CUDA on NVIDIA GPUs (remote) - Deterministic mock backend (no GPU required) - All backends behind the same adapter interface and HTTP worker protocol Why it matters: Switchyard simplifies the management and optimization of complex inference workflows, ensuring consistent performance and reliability across different environments and configurations.

### Overview: SLO-driven compiler and adaptive runtime for LLM inference deployments. ### Key features: - **Typed Model Compilation:** Compiles a typed model, workload, hardware, cost, and SLO contract into a versioned `DeploymentPlan`. - **Feasibility Analysis:** Feasibility analysis and profiling passes. - **Calibrated Service Curves:** Calibrated service models and constrained multi-objective optimization. - **Deployment Plan:** Deployable artifacts and a hash-bound `EvidenceBundle`. - **Replay and Fault Injection:** Replay and fault injection for validation. - **Validation:** Evidence-backed validation using a deterministic CPU/mock vertical slice. ### Tech stack: Python ### Why it matters: SLOForge provides a robust and adaptable solution for LLM inference deployments, ensuring high performance and reliability through SLO-driven optimization and validation. Repository: riskywindow/slo-forge

Overview: This project, `rogelioRuiz/dust-serve-capacitor`, is a Capacitor plugin designed to orchestrate the download and serving of models for Dust, a device unified serving toolkit. It includes features such as model registry, resumable downloads with SHA-256 verification, session caching, and accelerator probing. Key features: - Model registry - Resumable downloads - SHA-256 verification - Network policy - Session caching - Inference serialization - Accelerator probing - DustCore integration Tech stack: TypeScript Why it matters: This plugin simplifies the process of managing and serving models on-device, ensuring secure and efficient model downloads and lifecycle management, making it essential for developers working with Dust and Capacitor applications. Repository: rogelioRuiz/dust-serve-capacitor Primary language: TypeScript GitHub stars: 0 GitHub description: Model download and serving orchestration for Dust — Capacitor bridge README excerpt: This project is part of the Dust ecosystem, offering features like model registry, resumable downloads, and session caching for efficient model management on devices. It integrates with DustCore for seamless discovery across plugins.

Overview: `dust-serve-kotlin` is an Android library designed for downloading, verifying, caching, and running inference on ML models. It includes features such as download management, session caching, accelerator probing, and device benchmarking. Key features: - Download management with SHA256 verification and resume support. - Session caching with LRU eviction and priority awareness. - Accelerator probing for NNAPI, GPU, and CPU with fallback. - Device benchmarking to classify hardware performance. - WorkManager integration for reliable background downloads. - Thread-safe state management with lock separation. Tech stack: Kotlin Why it matters: This library simplifies the implementation of Android ML model servers by providing a comprehensive solution for downloading, caching, and running inference on models efficiently.

Overview: dust-serve-swift is a standalone model-server business logic implementation for iOS and macOS, providing features such as background downloads, session caching, LRU eviction under memory pressure, and WiFi-only network policy enforcement. Key features: - Background downloads with SHA-256 verification - Session caching with reference counting - LRU eviction under memory pressure - WiFi-only network policy enforcement - Disk space validation Tech stack: Swift Why it matters: dust-serve-swift simplifies the integration of model-server logic with iOS and macOS apps, making it easier to manage and optimize model downloads and sessions.

Overview: batchserve is an inference server designed to handle concurrent requests efficiently, leveraging bounded queues and dynamic batching to achieve up to 17x throughput gains. It includes features such as graceful drain, Prometheus metrics, and readiness probes, making it suitable for models that benefit from batching. Key features: - Dynamic Batcher: A single asyncio queue that collects requests until a maximum batch size or wait time is reached. - Backpressure: A hard cap on the queue size to prevent latency from growing unboundedly. - Lifecycle: `/health` checks for liveness, `/ready` is false during startup and shutdown. - Observability: Metrics for request latency, batch size, and queue depth. - Two models: `TextClassifier` and `SlowModel`. - CLI commands for serving, predicting, and health checks. Tech stack: Python. Why it matters: This server is optimized for models that benefit from dynamic batching, achieving significant throughput improvements over traditional unbatched serving methods. Repository: rohanreddy79/batchserve Primary language: Python GitHub stars: 0 GitHub description: Inference server with dynamic batching and backpressure: bounded queues, graceful drain, Prometheus metrics, 17x gain README excerpt: # batchserve **An inference server that coalesces concurrent requests into size-or-time bounded batches, with bounded queues (503 + Retry-After), graceful drain, readiness probes and Prometheus metrics; measured here at 17x the throughput of unbatched serving on an accelerator-like model.** [![ci](https