Projects

Overview: This project simulates and benchmarks the interaction between LoRA adapter swapping and KV cache fragmentation in multi-adapter language model serving systems. Key features: - Adapter swap increases KV fragmentation - Throughput penalty depends on adapter size - Hotset reduces fragmentation but costs throughput - No policy dominates for large models under heavy load Tech stack: Python Why it matters: This benchmark helps understand the trade-offs between reducing swap churn and maintaining KV cache efficiency in large multi-adapter language models, highlighting the importance of careful policy design to optimize performance and resource utilization.

### Overview: This project benchmarks the impact of prefill and decode interference on a single GPU, focusing on the TTFT inflation experienced by new requests arriving while decode is already running. ### Key Features: - **TTFT Inflation Measurement**: Measures the increase in the time to first token (TTFT) when new requests arrive on a busy server while decode is already running. - **Benchmarking**: Uses real hardware measurements to close the loop on the throughput gain from disaggregation. - **Models Tested**: Qwen2-0.5B, Qwen2-1.5B, and PyTorch models. - **Corpus Examples**: 3 examples for each (prompt length, batch size). ### Tech Stack: - Python - PyTorch - HuggingFace Transformers - CUDA ### Why It Matters: - **Realistic Measurement**: Provides accurate measurements of GPU interference, unlike theoretical models. - **Thorough Benchmarks**: Measures the impact on both prefill and decode, and their combined effect. - **Practical Insights**: Helps developers understand the trade-offs and potential bottlenecks in their own systems. ### Repository: - **Repository**: `JohnScheuer/prefill-decode-interference-bench` - **Stars**: 0 - **Description**: A benchmark for measuring the impact of prefill and decode interference on a single GPU, focusing on TTFT inflation when new requests arrive while decode is already running. ### Tech Stack: - Python - PyTorch - HuggingFace Transformers - CUDA ### Why This Exists

Overview: This benchmark evaluates the effectiveness of prompt compression strategies for large language models like GPT-2 and GPT-2-medium, focusing on the speedup of prefill and the quality of continuation. Key features: - Attention-based selection - Lexical heuristics - Stride selection - Random selection Tech stack: Python Why it matters: This benchmark tests whether removing tokens from the prompt before prefilling can improve prefill speedup and maintain continuation quality, providing insights into the efficiency and quality of prompt compression strategies.

Overview: This project synthesizes empirical data from 30+ benchmarks to create a single, auditable cost model for 2026 LLM inference stacks, providing answers to the cost of serving 1M tokens and identifying the most impactful optimization moves. Key features: - Incorporates empirical data from 30+ benchmarks - Answers: cost per 1M output tokens and which optimization moves have the most marginal impact - Includes two KPIs: $/1M output and $/1M request Tech stack: Python Why it matters: This project synthesizes data from multiple projects to provide a comprehensive and auditable cost model, addressing the need for a single, accurate cost model for LLM inference stacks. Repository: JohnScheuer/serving-cost-model-v2

Overview: This repository serves as a companion workspace for a three-part article series on enterprise LLM architecture decisions, focusing on system reality, MoE, serving quality, and ROI. Key features: - Benchmark inputs and selected raw results - Final figure assets from the article series - Toy experiments related to system design, MoE, serving, and ROI - Python scripts used to generate the materials Tech stack: Python Why it matters: This repository provides the supporting materials for the detailed article-to-material map, allowing readers to understand the practical decision chain behind enterprise LLM architecture choices. Repository: Johnny-Kao/enterprise-llm-architecture-lab Primary language: Python GitHub stars: 0 GitHub description: Supporting materials for a three-part article series on enterprise LLM architecture, MoE, serving, and ROI.

Overview: Self-hosted vLLM made simple. Git-friendly YAML for models and profiles, full lifecycle from an intuitive CLI and TUI. Key features: - Git-friendly YAML for models and profiles - Full lifecycle management from an intuitive CLI and TUI - Simple and easy to use Tech stack: Python Why it matters: This project simplifies the process of managing and deploying large language models, making it accessible and easy to use for developers.

### Overview: This project is a production-grade ML inference server written in Go, designed for real-time applications. It supports dynamic request batching, streaming responses, a hot-swappable model registry, and Prometheus metrics. ### Key features: - **Dynamic Request Batching**: Handles multiple requests efficiently. - **SSE Token Streaming**: Sends responses in real-time using SSE. - **Hot-Reloadable Model Registry**: Allows easy updates and hot-swapping of models. - **Prometheus Metrics**: Monitors and exposes server performance metrics. ### Tech stack: Built entirely on the standard Go library, without the use of any serving frameworks. ### Why it matters: This server is optimized for real-time applications, ensuring high performance and reliability. It simplifies the process of adding, updating, and removing models, making it easy for operators to manage and scale the system.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for local AI labs, featuring semantic caching, model profiling, and automatic failover. Key features: - Semantic Caching - Model Profiling - Automatic Failover Tech stack: Python Why it matters: SmarterRouter optimizes performance and ensures reliable AI development by leveraging semantic caching and automatic failover mechanisms.

Overview: SmarterRouter is an advanced gateway and router designed for Ollama, llama.cpp, and OpenAI, utilizing semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic Caching - Model Profiling - Automatic Failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing resource utilization and ensuring seamless operation across different AI environments.

### Overview: This project aims to make a trained checkpoint deployable and accurate, focusing on ONNX export, INT8 quantization, and latency benchmarks. The goal is to prove that nothing is lost during the transfer and to showcase the model's performance in a browser environment. ### Key features: - ONNX export with parity proof - INT8 quantization with accuracy re-measured - Full latency benchmarks on a 2014 CPU - Yoruba diacritics demo running entirely in the browser ### Tech stack: Python ### Why it matters: This project is a critical step in making a research model deployable and accurate, ensuring that the model's performance is maintained and that it can be used in real-world applications.

### Overview: A model serving system built around the dynamic batcher, which manages requests, prioritizes interactive traffic, and prevents hardware overload, ensuring optimal throughput and minimal shedding. ### Key features: - **Dynamic Batcher**: Manages requests, prioritizes interactive traffic, and prevents hardware overload. - **Priorities**: Prioritizes interactive traffic to ensure faster response times. - **Load Shedding**: Automatically sheds requests to prevent hardware overload, maintaining optimal throughput. ### Tech stack: Python ### Why it matters: The dynamic batcher and load shedding mechanisms ensure optimal performance and minimal shedding, making the system highly efficient and reliable.

### Overview Customer churn prediction is a critical business problem where companies aim to identify at-risk customers to prevent them from leaving. This project demonstrates the entire lifecycle of building, evaluating, and deploying a machine learning model for predicting customer churn using a Random Forest classifier. The project utilizes a synthetic dataset and is designed to be easily deployed on Kubernetes using KServe. ### Key Features - **Model Training**: A Random Forest classifier is trained on synthetic customer data. - **Model Artifact Persistence**: The trained model is saved to a local directory. - **Inference Options**: The model can be served locally with FastAPI or uploaded to S3 and loaded by KServe on Kubernetes. ### Tech Stack - **Python**: The primary programming language used for the project. - **FastAPI**: A lightweight web framework for building APIs. - **scikit-learn**: A popular machine learning library for building and evaluating models. - **DVC**: A data management tool for versioning and reproducibility. - **Kubernetes**: A container orchestration platform for deploying and scaling applications. - **AWS S3**: A cloud storage service for storing model artifacts. - **KServe**: A Kubernetes service for serving machine learning models. ### Why It Matters This project provides a practical example of how to build, evaluate, and deploy a machine learning model for customer churn prediction. It showcases the entire MLOps workflow, from data generation to model deployment, making it a valuable resource for developers looking to improve their skills in this area. ### Repository The project is hosted on GitHub at [Kevind

Overview: Runlaya is a plug-and-play inference API designed for Laya ONNX models, allowing developers to register multiple models, run predictions with per-request parameters, and interactively document the API. Key features: - Plug-and-play registration of models - Multi-model support for side-by-side inference - Per-request parameters for flexible question handling - Raw output without thresholds - Batch prediction for dataset evaluation - Interactive API documentation Tech stack: TypeScript Why it matters: Runlaya simplifies the process of deploying and using Laya ONNX models by providing a straightforward API for inference, making it easy for developers to integrate model-based functionalities into their applications. Repository: Khip01/runlaya Primary language: TypeScript GitHub stars: 0 GitHub description: Plug-and-play inference API for Laya ONNX models: register multiple models, run choice/score/noul predictions with per-request parameters.

### Overview: InferenceHub is a high-performance Microservices Pattern designed for serving AI models. It uses gRPC for binary transport, enabling 10x faster serialization and independent scaling of the application and compute layers. ### Key Features: - **⚡ binary Transport**: Uses Protocol Buffers via gRPC for 60% smaller payloads compared to JSON. - **🛡️ Polyglot Architecture**: Node.js for high-concurrency I/O and Python for optimized AI compute. - **🧠 Strict API Contracts**: Zero "Schema Drift" thanks to shared `.proto` source-of-truth. - **🐳 Docker Native**: Orchestrates complex networking between C++ based gRPC runtimes. ### Tech Stack: - **Gateway**: Node.js (Express) - **Worker**: Python 3.9 - **Protocol**: gRPC (Protobuf) - **Infrastructure**: Docker Compose ### Why it Matters: InferenceHub protects the "Brain" (AI models) from the "Noise" (Concurrent API traffic) by decoupling the application layer from the compute layer, ensuring scalable and high-performance operations. ### Tech Stack: - **Gateway**: Node.js (Express) - **Worker**: Python 3.9 - **Protocol**: gRPC (Protobuf) - **Infrastructure**: Docker Compose

Overview: This project predicts the efficiency category of industrial smart machines—High, Medium, or Low—using real-time operational data. Key features: - Machine Learning model trained to classify machine efficiency. - Web-based UI for entering operational parameters and getting instant predictions. - Containerized using Docker for portability. - Kubernetes deployment manifests for scalable production. - CI/CD-enabled via Jenkins pipeline for automated build and deployment. Tech stack: Flask, Docker, Kubernetes, and Jenkins. Why it matters: This project showcases the complete lifecycle from data preprocessing and model training to scalable deployment, demonstrating how to implement an end-to-end MLOps project using Flask, Docker, Kubernetes, and Jenkins. Repository: Kiran8053/Smart-Machines-Efficiency-Prediction Primary language: Jupyter Notebook GitHub stars: 0 GitHub description: An end-to-end MLOps project that predicts the efficiency of industrial smart machines—High, Medium, or Low—using real-time operational data. Built with Flask, Docker, and Kubernetes, and automated via a Jenkins CI/CD pipeline, it showcases the complete lifecycle from data preprocessing and model training to scalable deployment.

Overview: This project demonstrates the process of training, evaluating, and scoring a binary classifier using Azure Machine Learning (Azure ML). It includes a synthetic dataset, a holdout validation, and an artifact that is ready for Azure ML online scoring. Key features: - Synthetic ML training - Holdout release gates - Azure ML online scoring with a hashed model artifact Tech stack: Python Why it matters: This project showcases the end-to-end workflow for deploying a model on Azure ML, ensuring that the model is both explainable and ready for production use.

Overview: This project is an ordered recommendation policy for tuning Inference Cache SLOs for model-serving endpoints, focusing on high inference latency issues. Key features: - Evaluates cache hit rate, p95 latency, and request volume - Suggests TTL, warm keys, profiling, or no change based on the ordered policy - Implements CLI and decision rules using Python Tech stack: Python Why it matters: This tool helps operators quickly identify and address latency issues by prioritizing cache hit rate, latency, and traffic volume, providing a structured approach to tuning Inference Cache SLOs. Repository: Klokesh493/inference-cache-hit-rate-advisor Primary language: Python GitHub stars: 0 GitHub description: Inference cache SLO tuning advisor for model-serving endpoints.

### Overview: This project, developed by Klokesh493, calculates the consumed inference error budget from request and failure counts, helping teams compare failures against the allowed budget. ### Key Features: - Calculates consumed inference error budget from request and failure counts. - Divides observed failures by the failure allowance derived from request count and the configured budget percentage. - Reports watch at 75% and exhausted at 100% consumption. ### Tech Stack: Python ### Why It Matters: Teams need to compare failures against the allowed budget to ensure efficient resource management and prevent overconsumption.

### Overview: This project provides a tool to summarize the impact of ML incidents, including the count of unique models, endpoints, affected users, and the severity based on user count and endpoint coverage. ### Key features: - Summarizes the blast radius of ML incidents. - Counts distinct models and endpoints. - Totals reported affected users. - Assigns severity by user count and endpoint coverage. ### Tech stack: Python ### Why it matters: This tool helps responders quickly estimate the impact of ML incidents, enabling more effective communication and decision-making during incidents.

Overview: This project is a replica sizing calculator designed to estimate the number of replicas needed for model serving based on forecasted QPS, latency budget, and per-replica throughput. Key features: - Estimates replicas required for model serving from QPS, latency budget, and per-replica throughput. - Applies headroom to target QPS, divides by measured per-replica QPS, and rounds up to a minimum of one replica. Tech stack: Python Why it matters: This tool helps serving teams translate forecast requests into replica counts, ensuring they reserve headroom for demand variation, thereby optimizing resource allocation and improving model performance.