Projects

### Overview: This repository provides runnable implementations and benchmarks for robot-policy inference systems, focusing on latency measurement, dynamic batching, asynchronous action queues, model serving, and multimodal serving experiments. It answers "how does the mechanism work, and what does it cost?" by hosting interviews and bounded experiment reports. ### Key features: - Latency measurement - Dynamic batching - Asynchronous action queues - PyTorch model execution - Local model serving - SGLang serving ### Tech stack: Python ### Why it matters: This lab offers reusable, tested inference mechanisms that support the Software-to-Robotics-Inference guide, providing insights into how these mechanisms work and their associated costs. ### Repository Layout: - `docs/` contains review material and experiment guidance - `quests/` contains tested implementations of local mechanisms - `experiments/` contains benchmarks, machine-readable results, and reports - `src/robotics_inference_lab/` contains reusable inference and serving utilities ### Install and Test: Requires Python 3.11+. ```bash git clone https://github.com/yanizhang-yz/robotics-inference-lab.git cd robotics-inference-lab python -m venv .venv source .venv/bin/activate python -m pip install -e ".[dev,http]" pytest -q ``` Heavier experiment dependencies are opt-in: ```bash python -m pip install -e ".[torch]" python -m pip install -e ".[vision]" ``

Overview: NumPyForge is a pure-NumPy machine learning framework that includes classical models, deep learning, evaluation tools, and a production-oriented training and serving setup. It uses Docker for packaging, versioned artifacts, and CI-enforced tests/type checks. Key features: - Classical models: linear regression, logistic regression, softmax regression - Deep learning: dense layers, activations, dropout, backpropagation, optimizers, and MLPs - Evaluation tools: train/test splits, cross-validation, metrics, baselines, ROC/PR curves, JSON reports - Production setup: deterministic ingestion, training, MLflow-compatible tracking, versioned model artifacts, FastAPI serving, Docker packaging, structured logs, and quality system checks Tech stack: Python, NumPy, FastAPI, Docker, CI/CD, unit tests, coverage checks, type checks Why it matters: NumPyForge provides a comprehensive, production-oriented solution for ML projects, bridging the gap between algorithm implementation and operational deployment, making it easier for interviewers to evaluate the math and run the pipeline.

Overview: This project automates the deployment of machine learning models using FastAPI, Docker, and GitHub Actions for continuous model delivery and monitoring. Key features: - Automated model deployment - Continuous integration and continuous deployment (CI/CD) - Model monitoring and logging Tech stack: FastAPI, Docker, GitHub Actions Why it matters: This framework simplifies the process of deploying and monitoring machine learning models, making it easier for developers to integrate AI into their applications and ensure model reliability.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for local AI labs, offering semantic caching, model profiling, and automatic failover for seamless performance and reliability. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of local AI labs by optimizing resource usage and ensuring consistent access to AI models.

### Overview: 高光谱图像分类系统旨在通过混合3D/2D卷积、通道重标定、冻结PCA等技术，实现高光谱遥感数据的光谱-空间联合分类。系统采用Python编写的链路验证了模型的训练和推理一致性，并保留了Vue和Spring Boot的文件上传和结果下载功能入口。 ### Key Features: 1. **模型核心**：HybridSN混合3D/2D卷积与通道重标定。 2. **通道重标定**：PCA仅在训练中心像素上拟合，均值、投影矩阵与whitening参数随权重导出。 3. **模型工件**：版本化manifest、SHA-256校验、模型配置、随机种子、划分索引和逐轮训练记录。 4. **推理路径**：惰性空间patch、可配置mini-batch、`eval()`/`inference_mode()`；预测覆盖全部像素，不依赖地面真值文件。 5. **HTTP边界**：上传体积、立方体形状、波段、有限值和像素上限检查；进程内单请求推理，忙碌时返回503。 6. **回归验证**：合成数据训练、PCA数值对照、batch一致性、工件完整性、Dropout模式恢复及真实loopbackHTTPsmoke。 ### Tech Stack: - Python 3.12 - PyTorch 2.5.1 - License: MIT ### Why it Matters: 该系统为

### Overview: vLLM V1 inference systems lab aims to validate heterogeneous backends, analyze workload orchestration, characterize scheduling, and evaluate SLO-constrained load with reproducible experiments. The project covers Metal and CUDA execution environments, workload scheduling, runtime instrumentation, and SLO-constrained load evaluation. ### Key Features: - **Heterogeneous Backend Validation**: Validates Apple Silicon / vLLM-Metal and NVIDIA CUDA execution environments. - **Workload Orchestration**: Analyzes Barrier-synchronized burst, closed-loop concurrency, and deterministic open-loop arrivals. - **Runtime Instrumentation**: Measures Gauge time series, Counter increments, and Histogram deltas. - **Scheduler Characterization**: Examines Sequence admission, iteration-level token budget, and Chunked Prefill overlap. - **KV & Prefix Cache Analysis**: Estimates KV footprint, block-budget comparison, and Preemption/Recompute analysis. - **SLO-constrained Load Evaluation**: Separates offered load, achieved throughput, and Goodput, and evaluates TTFT and E2E joint constraints. ### Tech Stack: - Python - vLLM - CUDA / Metal ### Why It Matters: This project provides a comprehensive framework for validating and characterizing the performance of different execution environments, workload orchestration, scheduling, and load evaluation under SLO constraints. The reproducible experiments ensure that the results are reliable and can be easily replicated, making it valuable for developers and researchers alike.

Overview: Regulon is a comprehensive project that fine-tunes a large language model (LLM) for pharmaceutical compliance, covering the full lifecycle from synthetic data generation to deployment. The project includes QLoRA, GPTQ, and AWQ quantization techniques, and uses vLLM and SGLang for serving. Key features: - Fine-tuning of QLoRA and Unsloth models - Quantization to 4-bit FP8, AWQ, and FP8 - Evaluation using metrics like Exact Match and Adversarial Refusal Rate - Deployment using vLLM and SGLang Tech stack: Python Why it matters: This project demonstrates the practical application of fine-tuning and quantization techniques for LLMs in a real-world pharmaceutical compliance context, highlighting the challenges and benefits of deploying such models in a production environment.

### Overview: drift-aware-serving is a FastAPI model server that continuously monitors its input distribution using Prometheus metrics, latency benchmarks, and sliding-window PSI drift detection computed at scrape time. This approach ensures real-time monitoring without the need for additional services or batch jobs. ### Key Features: - **PSI Drift Detection**: Computes PSI drift detection at Prometheus scrape time, ensuring real-time monitoring. - **FastAPI Integration**: Built on FastAPI, providing a robust and efficient server. - **Prometheus Metrics**: Monitors input distribution and latency metrics. - **Latency SLO Histograms**: Provides service level objectives (SLO) for latency. - **Held-out AUC**: Evaluates model performance with a held-out AUC of 0.858. - **MIT License**: Open-source project with a clear licensing agreement. ### Tech Stack: - **Python**: Version 3.11+. - **FastAPI**: 0.14.1. - **Prometheus**: Scrape-time PSI drift detection. - **Held-out AUC**: 0.858. - **MIT License**. ### Why It Matters: This project addresses the common issue of delayed drift detection by treating observability as an integral part of the model, rather than an afterthought. By continuously monitoring the input distribution and computing PSI drift detection at scrape time, it ensures real-time monitoring and alerts on drift, providing a more timely and relevant signal for business metrics and latency alerts.

### Overview: 本项目旨在通过使用UCI SECOM数据集中的半导体制造数据，实现不良缺陷的预测。项目通过全工程闭合的评估、检定、模型改进、推理API（FastAPI + Docker）以及Docker容器化、以及对模型和参数的监控，展示了如何在不遗漏数据的情况下，实现高精度的不良缺陷预测。 ### Key features: - **模型改进与验证**: 通过多次模型改进和验证，确保模型的准确性和稳定性。 - **推理API**: 实现了基于FastAPI的推理服务，方便用户在本地或远程环境中使用。 - **Docker化**: 将推理服务部署到Docker容器中，实现高可用性和可扩展性。 - **参数监控**: 实施了对模型参数的实时监控，确保模型的性能在不断变化的环境中保持稳定。 ### Tech stack: 该项目使用了Python作为主要编程语言，结合了FastAPI和Docker进行开发。 ### Why it matters: 通过该项目，用户可以轻松地实现对半导体制造数据的不良缺陷预测，同时确保预测结果的准确性和稳定性。此外，通过使用Docker和FastAPI，项目展示了如何将推理服务部署到容器中，实现高可用性和可扩展性，这对于实时监控和性能优化都非常有利。 ### Repository: 该项目位于GitHub仓库`yuya0408/secom-defect-prediction`，目前没有公开的GitHub星星数。

Overview: This repository provides an end-to-end MLOps pipeline for online causal inference, utilizing Double Machine Learning on Databricks and serving the trained models on Azure Kubernetes Service (AKS) through a CI/CD pipeline. Key features: - Train causal machine learning models on Databricks - Deploy the models as a web service on AKS - Monitor the models in production using Azure Monitor and other tools Tech stack: Python, Databricks, MLflow, FastAPI, Kubernetes, Azure Why it matters: This project showcases the integration of causal inference techniques with production MLOps, demonstrating how causal models can be effectively trained, deployed, and monitored in a real-world scenario. Repository: Zachary-S-Cacciatore/causal-dml-measurement

### Overview: A production-shaped serving stack for a small open-source coding LLM, designed to run on a single 24GB GPU (RTX 4090) for approximately $0.50/hr. The stack includes vLLM 0.7 serving Qwen2.5-Coder-3B with FP8 weights, prefix caching, and ngram speculative decoding for FIM autocomplete. The gateway is OpenAI-compatible, and Prometheus/Grafana observability is integrated. A weekly LoRA pipeline with eval-gated hot-swap allows for zero restarts and improved performance. ### Key features: - **vLLM 0.7** serving Qwen2.5-Coder-3B with FP8 weights, prefix caching, and ngram speculative decoding for FIM autocomplete. - **LiteLLM gateway** in front, OpenAI-compatible endpoints, friendly model routes. - **Prometheus + Grafana** for server-side performance metrics, including p50/p95/p99 TTFT, tokens/sec, queue depth, KV-cache usage, GPU utilization. - **Weekly LoRA pipeline**: IDE accept/reject events → PEFT LoRA → eval gate (blocks adapters that don't beat base) → hot-swap into the running server in ~1–2s, zero restart. - **FIM load-test harness** with client-side percentile reporting. ### Tech stack: Python ### Why it matters: This stack is optimized for low cost and high performance, making it suitable for small-scale applications in the coding domain.

### Overview: ZKA Model is the Model Layer of the ZKA Labs ecosystem, providing a unified access to large language models through a RESTful API. It offers low-latency completions, streaming responses, and flexible authentication, making it suitable for production workloads. ### Key features: - **Unified Access**: ZKA Model allows developers to access a variety of ZKA Labs models. - **Model Layer**: It is part of the ZKA Labs ecosystem, focusing on AI inference. - **Production Ready**: Designed for high-performance, low-latency, and reliable model access. - **Model IDs**: Available models include `zk-v3z`, `zk-v2b`, and `zk-v1b` with different characteristics. - **API Endpoints**: Supports endpoints for chat completions, list models, and health check. - **Authentication**: Requires an API key for secure access, with options for both production and development use cases. ### Tech stack: Node.js, Express, JSON, SSE (Server-Sent Events). ### Why it matters: ZKA Model simplifies the integration of large language models into existing projects by providing a single, unified API interface, ensuring high performance and reliability.

Overview: InferForge is a production-grade serving template designed for vision inference services, offering a streamlined approach to deploying models into HTTP services. Key features: - Agent-first development - Business-facing web service architecture - Reusable inference engines Tech stack: Python Why it matters: InferForge simplifies the process of integrating vision models into production environments, providing a robust and flexible solution for developers. Repository: zjykzj/InferForge Primary language: Python GitHub stars: 0 GitHub description: 🔨 From model to HTTP service — agents forge vision models into production.

### Overview: Fleet is an open-source platform designed to self-host open models on GPUs, providing an OpenAI-compatible gateway with metering and Kubernetes-native deployment. It aims to address the operational challenges of self-hosting open models by offering a fixed-cost fleet model. ### Key Features: - **OpenAI Protocol**: Fleet uses the OpenAI protocol for communication with the inference engine, ensuring compatibility with OpenAI models. - **Fixed Cost Model**: Fleet offers a fixed-cost model, eliminating the need for metered token usage. - **Kubernetes Native Deployment**: Fleet integrates with Kubernetes for efficient model deployment and management. - **Endpoint Selection**: The gateway controls endpoint selection, ensuring only authorized users can call the model. - **Token Counting**: Fleet counts tokens forwarded by the gateway to ensure accurate billing. - **No Vendor SDK**: Fleet does not import any vendor SDKs, adhering to the OpenAI protocol. - **Fixed Cost Pool**: Fleet allocates a fixed cost pool based on GPU hours, ensuring cost predictability. ### Tech Stack: Go ### Why It Matters: Fleet targets the gap in self-hosting open models by providing a fixed-cost model, addressing the operational challenges of metered token usage. It offers a scalable and efficient way to deploy and manage models on GPUs, making it an essential tool for teams looking to self-host open models without the complexity of traditional cloud providers. ### Repository: zlogic-labs/fleet ### Primary Language: Go ### GitHub Stars: 0 ### GitHub Description: Self-host open models on your own GPUs: an OpenAI

Overview: Titanic ML System is an end-to-end MLOps pipeline designed to train, track, and serve a machine learning model for predicting Titanic passengers' survival. Key features: - Data preprocessing - Model tracking (MLflow/DagsHub) - Inference API (Flask) - Traffic simulation generator - Continuous monitoring (Prometheus/Grafana) Tech stack: Jupyter Notebook Why it matters: This system provides a solid foundation for building, training, and deploying machine learning models, making it ideal for applications requiring high performance and reliability. Repository: zuudevs/titanic-system Primary language: Jupyter Notebook GitHub stars: 0 GitHub description: End-to-end MLOps pipeline for training, tracking, serving, and monitoring a Titanic survival prediction model.