Projects

### Overview: KVBridge is a production-oriented implementation and systems extension of NVIDIA's August 2026 paper, designed for memory-bounded cross-model KV-cache transfer in LLM families. It maps a source model's attention cache into a compatible target model, enabling the target to continue without replaying the entire prompt. ### Key features: - **Paper Mapper**: Per-target-layer, per-target-head ridge maps for K and V. - **Layer Selection**: Head-averaged single-source R², then top-k cross-layer features. - **Position Handling**: Exact inverse/forward RoPE using factors emitted by each model. - **Modest-Host Fitting**: Target-layer blocks, deterministic token stride, CPU/CUDA statistics. - **Distributed Path**: Mergeable sufficient statistics plus `torch.distributed` all-reduce. - **Data Plane**: SafeTensors calibration shards; no pickle deserialization. - **Artifact Plane**: Atomic writes, SHA-256, model fingerprints, measured FP32/BF16 storage. - **Runtime Safety**: Shape/architecture/numerical/latency/logit-KL gates, explicit fallback. - **Integration**: Qwen/Llama-style Hugging Face `DynamicCache` handoff adapter. - **Operations**: JSON resource planner, structured handoff events, CI, package build. - **Research**: Attention-output metrics, pinned T2 jobs, local results, multi-lab protocol. ### Tech stack: Python 3.10+ ### Why it matters: KVBridge is a critical tool for maintaining the performance

Overview: This project demonstrates how a pre-trained machine learning model can be converted into a usable application and deployed in a containerized environment. Key features: - Loads a pre-trained machine learning model using Pickle. - Provides a simple browser-based prediction interface. - Offers a REST API for programmatic predictions. - Built using Python and Flask. - Containerized using Docker. - Supports both form-based and JSON-based prediction requests. - Runs on port 5000. Tech stack: Python, Flask, NumPy, Pickle, Docker, Scikit-learn, Git/GitHub. Why it matters: This project showcases how to deploy machine learning models efficiently and securely, making them accessible through web interfaces and APIs, while maintaining the simplicity and ease of use for the end-users.

### Overview: This repository is a production-oriented MLOps reference implementation designed for real-time inference, feature validation, drift monitoring, champion/challenger governance, model promotion, and observability. ### Key Features: - Real-time inference - Feature validation - Drift monitoring - Champion/challenger governance - Model promotion - Observability ### Tech Stack: Python ### Why it Matters: This repository provides a comprehensive solution for managing and monitoring machine learning models in a production environment, ensuring continuous improvement and validation of models without the need for frequent model replacements.

### Overview: Maia LLM is a Windows desktop app designed for developers who want to use open-weight models efficiently. It allows users to rent a GPU on demand, see the cost before anything is created, and chat directly from their editor or AI agents using an Ollama-compatible API. ### Key Features: - **GPU Rental**: Rent a GPU on demand and run models on it. - **Real-Time Chat**: Chat directly from your editor or AI agents using an Ollama-compatible API. - **Catalogue**: Browse a catalogue of open-weight models with license, size, context window, and feature filters. - **Automatic GPU Rental**: Maia rents a GPU from vast.ai, chosen based on the model and user preferences. - **No Surprise Bills**: Every run starts with an estimate, and nothing is provisioned until confirmed. - **Automatic GPU Retention**: Retries startup on up to five replacement machines by default. ### Tech Stack: TypeScript ### Why it Matters: Maia LLM simplifies the process of using open-weight models by allowing users to rent a GPU on demand, see the cost before anything is created, and chat directly from their editor or AI agents using an Ollama-compatible API. This makes it easier for developers to experiment with models without the hassle of managing clusters or terminals.

### Overview: This project is an experimental vLLM runtime for Qwen3.6 deployed on Intel XPU, focusing on reproducibility and validation. The repository includes Docker images that are based on the Intel Lunar Lake XPU, maintaining a specific configuration of Qwen3.6 with parameters such as 35B-A3B-AWQ-CT-MTP-W4G32 and MTP2. The project aims to bridge the gap between Intel XPU and vLLM, ensuring that the model can be validated and verified across different environments. ### Key Features: - **Intel XPU Runtime**: The project provides a runtime environment for Intel XPU, ensuring compatibility and performance. - **Reproducible Model**: The model is validated and verified using Docker images, ensuring that the model can be easily reproduced and tested. - **Performance Optimization**: The project includes various optimizations, such as MoE (Multi-Headed Embedding) and MTP (Model-Throughput), to improve the model's performance. - **Model Validation**: The project includes a comprehensive validation process, including model preparation, execution on XPU, and performance evaluation, to ensure the model's accuracy and reliability. ### Tech Stack: - Python: The primary programming language used for the project. - Docker: The containerization technology used to package the runtime environment. - vLLM/XPU: The Intel XPU-based vLLM runtime environment. ### Why It Matters: - **Reproducibility**: The project ensures that the model can be reproduced and validated across different environments, maintaining

Overview: This project provides a KServe integration for TrustyAI, offering LIME and SHAP explanation methods to interpret model predictions. Key features: - Integrated support for LIME and SHAP explanation methods - Deployable via YAML configurations for both LIME and SHAP explainer - Supports KServe `InferenceServices` for model predictions Tech stack: Java Why it matters: This tool enhances the interpretability of AI/ML models, making them more transparent and trustworthy, which is crucial in various industries where model explainability is a critical factor.

Overview: This project is an end-to-end distributed ML recommendation system that demonstrates real-time streaming, multi-node distributed training, and fault-tolerant, scalable serving capabilities. It uses Python, Apache Kafka, Spark Structured Streaming, PyTorch DDP, FAISS, Ray Serve, Redis, PostgreSQL, Prometheus/Grafana, and Docker technologies. Key features: - Real-time streaming of events - Exactly-once processing - Distributed training across multiple nodes - Fault-tolerant, scalable serving Tech stack: Python, Apache Kafka, Spark Structured Streaming, PyTorch DDP, FAISS, Ray Serve, Redis, PostgreSQL, Prometheus/Grafana, Docker Why it matters: This project showcases how to build a robust recommendation system that can handle continuous data ingestion, distributed model training, and scalable serving under various failure scenarios, demonstrating the importance of correct system behavior in real-world applications.

Overview: InferenceServer is an ML inference server designed for serving Image classification over HTTP, utilizing JavaScript for the backend. Key features: - Image classification capabilities - Scalable and efficient inference processing - Built-in Docker support for easy deployment Tech stack: JavaScript Why it matters: This server is crucial for real-time image classification applications, providing a fast and reliable service for developers and researchers alike. Repository: vishal-keshav/InferenceServer Primary language: JavaScript GitHub stars: 1 GitHub description: An ML inference server to serve Image classification over HTTP

### Overview: This repository is a curated list of frameworks, tools, and resources for Large Language Models (LLMs) training and inference. It covers the entire pipeline from data processing to deployment, including training frameworks, pretraining implementations, end-to-end training and inference, distributed training, hardware acceleration, model compression, data engineering, privacy and security, domain-specific LLMs, RAG & vector search, evaluation and benchmarking, research tools, MLOps & platform, deployment and serving, monitoring and observability, and learning resources. ### Key Features: - Training Frameworks: Includes popular deep learning frameworks like PyTorch, transformers, JAX, Flax, and others. - Pretraining Implementations: Covers from-scratch implementations like nanoGPT and LLaMA. - End-to-End Training & Inference: Provides comprehensive resources for full training pipelines. - Inference Frameworks: Offers tools for inference and model serving. - Distributed Training: Supports distributed training with libraries like Accelerate and DeepSpeed. - Hardware Acceleration: Includes resources for hardware acceleration with DeepSpeed and Fairscale. - Fine-Tuning & PEFT: Provides fine-tuning and performance enhancement techniques with libraries like Megatron-LM and ColossalAI. - Model Compression & Quantization: Offers resources for model compression and quantization with tools like Megatron-LM and Fairscale. - Data Engineering: Includes resources for data processing and engineering. - Privacy & Security: Provides resources for privacy and security in LLMs. - Domain-Specific LLMs: Covers specific applications and

### Overview: This repository contains a set of curated examples showcasing the capabilities of the Flama framework. Each section builds upon the previous one, providing a structured and progressive learning experience from simple "hello world" applications to production-grade applications with Domain-Driven Design, AI model serving, and more. ### Key Features: - **Introduction**: Minimal apps, routing basics, and project structure. - **Getting Started**: Setting up environment and building a complete application. - **Fundamentals**: Applications, routes, components, modules, middleware, endpoints, resources, and responses. - **CLI Commands**: CLI usage: run, serve, start, model, get, upgrade. - **Advanced Topics**: Config, debug, client, pagination, background tasks, lifespan, authentication, and testing. - **Predictive AI**: Model packaging, adding models, model resources, and model components. - **Generative AI**: Overview, serving LLMs, chatbot, and MCP. - **Domain-Driven Design**: Data models, repositories, workers, resources, and full application. - **Contributing**: Contributing to Flama. ### Tech Stack: - **Language**: Python 3.13+ - **Framework**: Flama - **ORM**: SQLAlchemy - **Migrations**: Alembic - **Schemas**: Pydantic, Marshmallow, Typesystem - **Testing**: pytest, pytest-asyncio - **Package Manager**: uv ### Why It Matters: Learning a framework is best done by example. This repository provides a structured, progressive set of examples

Overview: This repository contains examples of deploying Large Language Models (LLMs) on Modal, providing a comparison of various serving frameworks. Key features: - Deploying LLMs on Modal - Benchmarking setup for LLM serving frameworks Tech stack: Python Why it matters: This repository serves as a practical guide for developers looking to deploy and benchmark large language models on the Modal platform, offering insights into different serving frameworks.

### Overview: This project focuses on legal contract analysis using advanced AI models and cloud infrastructure. It leverages the Hermes-4-14B model and QLoRA fine-tuning techniques, integrating AWS SageMaker for training and serving, and MLRun for orchestrating workflows and model management. ### Key Features: - **Model Fine-Tuning**: Uses QLoRA/PEFT and 4-bit NF4 base weights for efficient fine-tuning. - **Training Configuration**: Utilizes one SageMaker `ml.g6e.12xlarge` instance with four 48 GB L40S GPUs. - **Serving**: Deployed using SageMaker LMI/vLLM with FP8 configuration and LoRA inference components. - **Model Evaluation**: Records evaluation metrics and raw benchmark responses for performance analysis. - **GPU Serving Benchmarks**: Measures serving behavior under concurrent requests to optimize performance. ### Tech Stack: The project is built using Jupyter Notebook for experimentation and documentation. ### Why It Matters: This project provides a comprehensive solution for analyzing non-disclosure agreements, enhancing legal contract review processes by automating the classification of contract clauses and supporting evidence. The use of advanced AI models and cloud infrastructure ensures scalability and reliability, making it suitable for real-world applications in legal and contract management.

Overview: This project is an enterprise-level CVAT/Nuclio integration platform that orchestrates annotation models from remote GPU hosts, enabling fast and efficient model management and inference. Key features: - Direct remote connection: Models run on a dedicated remote host (Inference Server) and connect to CVAT via automatically generated Nuclio functions. - Multi-model type support: Handles Detectors, Trackers, and Interactors through a unified interface. - Multi-instance support: Runs the same model implementation with different weights or configurations simultaneously. - Generic configuration: Passes any custom parameters to models via the `config` block in `models.yaml`. - Auto-detection & generation: New models are detected automatically, and implementation files or `classes.txt` are generated using intelligent templates. - Auto-configuration: Model types and ports are automatically managed in `config/models.yaml`. - Nuclio generator: Dynamic deployment files for CVAT are automatically generated with type-specific annotations and specs, pre-configured to point to your remote host. - Lazy initialization: Models load only when requested, with no overhead during server startup. - Smart resource management: Automatic memory unloading for idle models. - Enterprise architecture: Fully typed, PEP8 compliant, and designed for production extensibility. Tech stack: Python Why it matters: This project is crucial for organizations looking to leverage the power of remote GPU hosts for AI model inference, while maintaining the flexibility and ease of use of CVAT. The platform's auto-detection, auto-generation, and zero-config features make it highly efficient and scalable, ensuring seamless integration

### Overview: This project is a distributed ML training and serving platform designed for minimal systems path in C++. It focuses on parameter server, inference server, and dynamic batching, aiming to provide low-latency and high-throughput service. ### Key features: - **C++20 Systems Path:** Parameter server, inference server, dynamic batcher, autoscaler, socket protocol, and CMake build. - **Distributed Training:** Deterministic data sharding, worker replicas, gradient push/pull, weighted aggregation, sync vs. async SGD. - **Serving Infrastructure:** Trained weights loaded into a C++ inference service, concurrent requests batched by size or delay window. - **Operational Signals:** Queue depth, processed batches, p99 latency, throughput, desired replica count. - **End-to-End Verification:** Baseline training, 2-worker distributed training, serving load test, C++ and Python tests. ### Tech stack: C++ ### Why it matters: This project is designed to showcase the minimal systems path for distributed ML training and serving, focusing on efficient and stable performance under varying conditions.

### Overview: This project provides a dual NVIDIA DGX Spark cluster operator that supports model switching, weight synchronization, and other advanced features for efficient and reliable model deployment. ### Key features: - **Model Switching in One Command**: Easily switch between different models using a single command. - **TP=2 Across Two Nodes**: Dual nodes with 100G RoCE links for high-speed communication. - **Weight Synchronization**: Dual-node weight sync over the CX7 fabric at a rate of 1.8 GB/s. - **Watchdog for Crash Resilience**: Automatically rebuilds nodes in pairs to prevent crashes. - **Unified OpenAI Gateway**: Single endpoint for model access, independent of the active engine. - **Ops Web Panel**: Zero-dependency web panel for monitoring and task management. ### Tech Stack: Shell ### Why it Matters: This project bridges the gap between home lab and production environments, making it easier and more efficient to deploy large language models on dual Spark clusters.

### Overview: This project is a model-swap control plane designed for a DGX Spark (GB10) running a single vLLM model at a time. The tool ensures a safe, one-click, drain-aware model swap without risking the entire system. ### Key Features: - **One-click model swap**: Stops the running container, drains the GPU, starts the target, waits for health, ensuring no risk of wedging. - **Live telemetry**: Monitors unified memory usage, GPU utilization, temperature, uptime, decode throughput, and in-flight/served request counts. - **Model registry**: Displays every recipe as a card with size, quant, context, measured speed, and "best for" tags; shows committed (official) vs. draft (unpromoted) status. - **Import from Hugging Face**: Pastes a repo ID, gets an auto-proposed recipe, edits the recipe, and optionally kicks off the weights download in one step. - **Hand-edit recipes**: Directly edit a model's raw `.env` for tuning knobs the Import form doesn't expose. - **HF lookup**: Re-fetches a model's `config.json` from Hugging Face while editing, ensuring you're never trusting a stale or misremembered spec. - **GB10 stability battery**: Runs a 9-test suite (`swap-ui/stability.py`) to validate a model's reliability on this hardware before relying on it. - **Promote workflow**: Commits and pushes the model's recipe directly to the model-configs repo from the dashboard. - **

Overview: gb10-model-configs provides model serve recipes for the DGX Spark Model Swapper, including official and draft configurations. Key features: - Official `.env` recipes for models - Draft `.env` files for UI-imported models - Experimental flag to clear after stability tests Tech stack: Bash and Python Why it matters: This repository simplifies the process of managing and promoting model configurations for the DGX Spark Model Swapper, ensuring consistency and ease of use for both developers and end-users.

Overview: This project is a real-time MLOps monitoring platform that streamlines the entire machine learning lifecycle, from data ingestion to model evaluation and deployment. Key features: - Real-time streaming inference - Live model training - Model benchmarking - Interactive observability dashboard Tech stack: TypeScript Why it matters: This platform simplifies the complex process of machine learning by providing a unified, real-time monitoring solution, making it easier for developers to manage and optimize their models. Repository: 05vukasin/ml-research-project Primary language: TypeScript GitHub stars: 0 GitHub description: Real-time MLOps monitoring platform: streaming inference, live Training Lab, model benchmarking, and an interactive observability dashboard. One command to run.

Overview: This project is an async FastAPI app designed for ML model serving using MLflow. It showcases async inference with asyncio.to_thread, local model deployment, and retraining triggers, providing a clean and modular structure for both beginners and production-ready workflows. Key features: - Async FastAPI app for ML model serving - MLflow integration - Async inference with asyncio.to_thread - Local model deployment - Retraining triggers Tech stack: Python Why it matters: This project demonstrates the use of asyncio to handle asynchronous tasks, making it a valuable resource for developers looking to improve the performance and scalability of their applications.

### Overview: This project is a biomedical signal classifier that combines Electrocardiogram (ECG) with PPG (Photoplethysmography) or accelerometer data to predict human activities. It leverages PyTorch for the model, FastAPI for the API, and Claude for generating a clinical-style report on detected anomalies. The project is designed to be used as a real product slice rather than a single-notebook prototype, featuring a Next.js dashboard for visualization and a Claude API layer for detailed explanations. ### Key features: - **Multimodal Signal Classification**: Combines ECG with PPG or accelerometer data to predict human activities. - **Model Training**: Uses PyTorch to train a model that processes time-aligned ECG, PPG, and accelerometer signals. - **API Service**: Provides a FastAPI service for predictions and a Claude API for generating clinical-style reports. - **Dashboard**: Visualizes the raw signals, predictions, and reports using Next.js. ### Tech stack: - **Model**: PyTorch - **API**: FastAPI - **Report Generation**: Claude API - **Visualization**: Next.js ### Why it matters: This project is a practical example of combining multiple signals for classification, demonstrating the potential of multimodal data in biomedical applications. The inclusion of a Claude API layer and a Next.js dashboard provides a tangible way to visualize and explain the results, making it accessible to both developers and healthcare professionals. The project's educational nature is underscored by the disclaimer that it is not a medical diagnostic tool, ensuring it remains focused on learning and