Projects

### Overview: This project aims to validate the performance of cross-request KV prefix caching by reproducing Mooncake's published curve on real traces. The study involves 68,266 requests from 393 Claude Code sessions and 23,608 Mooncake requests through a prefix-cache simulator. The key findings reveal that under capacity pressure, most recomputation comes from tool-calling loops, not from sessions idling past a TTL, and the 5-minute TTL never fires. ### Key Features: - **Prefix-Caching Simulator**: A discrete-event simulator of a cross-request prefix cache. - **Hits are Prefix-Contiguous**: A hit is the longest resident prefix of the block chain, not a set intersection. - **Radix Structure Constrains Eviction**: A block with resident children is not evictable. - **In-Flight Chain Must Be Pinned**: Ensures the in-flight chain is pinned to avoid confusion with dead sessions. ### Tech Stack: - Python ### Why It Matters: Under capacity pressure, most recomputation comes from tool-calling loops, not from sessions idling past a TTL. The 5-minute TTL never fires under capacity pressure, indicating that the policy is more effective than previously believed. ### Repository: - **gauravapiscean/agentic-kv-cache** - **GitHub Stars**: 24 - **GitHub Description**: Reproducing agentic KV-cache policy claims on real traces. 68k requests from 393 Claude Code sessions. LRU is harder to beat than the papers suggest.

**Overview:** UpdAPI is a living benchmark designed to measure how quickly frontier coding agents adapt to real API changes. It tracks and evaluates the performance of AI coding models and agents by analyzing and comparing API evolution over time. **Key features:** - Timestamped, provenance-rich records of API and SDK changes. - Evaluation cases for contemporary coding systems, focusing on the workflow of a coding agent. - A public API Freshness Observatory for continuous benchmarking. **Tech stack:** JavaScript. **Why it matters:** UpdAPI provides a valuable empirical measure of how well AI coding systems keep pace with the software ecosystem, helping developers assess the effectiveness of their coding agents and models.

Overview: Programming Generative AI is a comprehensive video course that guides you through the journey from building simple neural networks to creating advanced models like Stable Diffusion. The repository contains the code, slides, and examples from the course. Key features: - Hands-on examples - From VAEs to Stable Diffusion - Text-to-image pipelines - Personalizing large pretrained models Tech stack: Jupyter Notebook Why it matters: This course provides a practical, step-by-step guide to deep generative modeling, making it accessible and engaging for developers looking to learn and apply advanced AI techniques.

Overview: This repository contains the official implementation of the paper "Delving into the Openness of CLIP" presented at ACL 2023. It provides evaluation codes, pre-trained models, and detailed instructions for installation, data preparation, training, and evaluation. Key features: - Evaluation codes for Extensibility and Stability - Pre-trained models for CLIP, CoOp, DeCLIP, SLIP, FILIP, and DeFILIP - Detailed installation and package requirements instructions - Comprehensive documentation for training, evaluation, and reproducibility Tech stack: Python Why it matters: This repository is essential for researchers and developers interested in the openness and extensibility of CLIP, a state-of-the-art visual language model, providing valuable insights and tools for further research and application development.

Overview: This repository contains the official PyTorch implementation and pretrained model weights for the paper "MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations," which has been accepted for publication in the Conference ACM MM 2024 (Poster). The repository includes datasets for EEGCVPR40 and ThingsEEG, as well as scripts for data preprocessing and training. Key features: - Pretrained model weights for MB2C - Preprocessed EEGCVPR40 and ThingsEEG datasets - Scripts for zero-shot classification and image reconstruction tasks using MB2C Tech stack: Python Why it matters: This repository provides a comprehensive solution for researchers and developers interested in multimodal learning and visual representation learning, offering practical tools and pretrained models for zero-shot tasks and image reconstruction.

Overview: This project turns your Readwise library into a blazing-fast, self-hosted semantic search engine, complete with nightly syncs, vector search API, Prometheus metrics, and a streaming MCP server for LLM clients. Key features: - Vector search API - Prometheus metrics - Streaming MCP server for LLM clients Tech stack: Python Why it matters: This project allows users to easily turn their Readwise library into a powerful search engine, leveraging advanced semantic search capabilities, real-time metrics for monitoring, and a streaming MCP server optimized for large-scale language model clients, making it highly useful for developers and researchers alike. Repository: leonardsellem/readwise-vector-db Primary language: Python GitHub stars: 24 GitHub description: Turn your Readwise library into a blazing-fast, self-hosted semantic search engine – complete with nightly syncs, vector search API, Prometheus metrics, and a streaming MCP server for LLM clients.

### Overview: 这是一个结合了ComfyUI自定义节点和ANIMA提示词生成器的项目，旨在为用户提供一个五级NSFW档位闸门的随机提示词生成器。它支持角色库2390位和场景搭配271条，纯本地、零依赖、零联网，运行时零联网，只生成提示词，不生成图片。 ### Key features: 1. **五级档位闸门**：支持从低档到高档的五级档位闸门，低档内容会先从候选池排除，确保SFW档真正干净。 2. **三层权重**：段、字段、选项的权重，从0到3.0，越档选项点到3.0也进不来。 3. **角色库2390位**：构建的只读数据，覆盖284个作品IP，可收藏、编辑、自建、锁官方服装。 4. **场景搭配271条**：场景、特写、姿态三类，每条锁字段+衍生档位+可选变体池。 5. **可复现**：同`seed`设置正文逐字相同，-1才每次重抽。 6. **出图元数据**：把完整正向写进PNG，不改你的工作流。 7. **零依赖零联网**：纯Python标准库+原生JS，不联网、不上传、不调大模型。 8. **可自检**：每次

### Overview: This repository contains code and documentation for optimizing the inference process of large language models (LLMs) using TensorRT and TensorRT-LLM. The project focuses on improving the performance and efficiency of LLM inference by leveraging cutting-edge techniques and optimizations. ### Key Features: - **Prefill Phase**: This phase involves encoding the input prompt and caching the key-value pairs for each layer. This is a single-time computation that is parallelizable. - **Decoding Phase**: This phase is sequential, where each token is generated based on the previous token. It includes two steps: first, calculating the final embedding using the KV Cache, and second, updating the KV Cache with the current token. - **Batch Eviction**: This feature helps in managing memory and reducing the number of tokens processed per batch, which can improve throughput without significantly affecting latency. ### Tech Stack: Python ### Why It Matters: This project is crucial for developers looking to optimize the performance of large language models, which are essential for applications such as language translation, text generation, and more. By improving inference efficiency, developers can enhance the speed and scalability of their models, making them more suitable for real-world applications. ### Repository: Repository: lix19937/llm-deploy Primary language: Python GitHub stars: 24 GitHub description: AI Infra LLM infer/ tensorrt-llm/ vllm ### Inference Optimization Indicators: - **Latency**: Focuses on the service experience, i.e., the time

### Overview: pdf-struct-chunker is a layout-aware PDF chunking tool written in pure Rust. It preserves the document structure by analyzing the actual layout of a PDF, including X/Y coordinates, font sizes, and bold detection. This ensures that chunks are semantically meaningful, without the need for language models or external APIs, making it ideal for RAG pipelines. ### Key Features: - **Layout-Aware**: Preserves document structure by understanding layout details. - **No LLMs, No APIs, No Cloud Dependencies**: Runs offline and in pure Rust. - **Pure Rust Implementation**: Written in Rust, ensuring high performance and low memory footprint. ### Tech Stack: - **Rust**: Pure Rust implementation. ### Why It Matters: - **Preserves Document Structure**: Ensures that chunks retain their original structure, providing context and metadata for vector search pipelines. - **No Dependency on Language Models or External APIs**: Simplifies integration and reduces complexity. - **Fast Processing**: Processes a 100-page PDF in less than 1 second, making it suitable for real-time applications. ### Repository: - **Repository**: matthiasnordwig/pdf-struct-chunker - **GitHub Stars**: 24 - **GitHub Description**: LLM-free, layout-aware PDF chunking for RAG pipelines. Preserves document structure via regex and font heuristics. Written in pure Rust. ### Quick Start: - **Clone Repository**: `git clone https://github.com/matthiasnordwig/pdf-struct-chunker.git` - **Run Sample PDF

Overview: OmniEvalKit is a unified framework designed to evaluate frozen vision-language models across various modalities, including text, image, video, audio, and audio-visual tasks. It simplifies the process of evaluating models without the need for retraining, making it highly useful for researchers and developers. Key features: - Supports training-free evaluation of frozen VLMs. - Unified multimodal evaluation pipeline. - Broad model coverage including Qwen, Gemma, MiniCPM, VILA, and OmniVinci adapters. - Comprehensive benchmark coverage with 118 dataset adapters. - Optional spoken output combining multimodal answers with CosyVoice3 speech synthesis. Tech stack: Python. Why it matters: OmniEvalKit streamlines the evaluation process for researchers and developers, making it easier to assess the performance of vision-language models across different modalities without the need for extensive retraining. This tool is particularly valuable for researchers working on multimodal AI applications and developers looking to evaluate their models efficiently.

Overview: This repository contains the necessary files to run LMStudio within a Docker container, providing a seamless experience through a web interface. Key features: - Dockerfile: Contains instructions for building the Docker image. - Docker Compose file: Manages multiple containers for the application and deployment tools. - Health check: Ensures the application is running properly. - HTTP server configuration: Provides a web interface for interacting with the language model. Tech stack: Dockerfile Why it matters: This setup allows for easy deployment and management of LMStudio, making it accessible on various environments and platforms.

### Overview: Inferna Next is a self-hosted GPU cluster orchestration solution that allows users to deploy and serve open-source AI models on their own hardware. It provides a centralized control plane, model catalog, and automatic engine selection, scheduling, and multi-vendor GPU support. ### Key Features: - **Central Web UI**: Manages multiple GPU clusters, both on-prem and in the cloud. - **Model Catalog**: One-click deploy for popular models like Qwen, Llama, Phi, DeepSeek, BGE, Whisper, and Qwen2.5-VL. - **Automatic Engine Selection**: vLLM and SGLang engines tuned for latency or throughput profiles. - **Automatic Scheduling**: Worker, GPU, and host port chosen server-side with anti-affinity and PostgreSQL-backed concurrency safety. - **Multi-vendor GPU Support**: NVIDIA, AMD, and a mock mode for demos and CI. - **OpenAI-Compatible Gateway**: `/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/transcriptions` with API-key auth, SSE streaming, and per-model request accounting. - **Enterprise Basics**: JWT auth, role-based access control, Prometheus metrics, Grafana dashboards. - **Docker-first**: One `docker compose` command to run the whole stack. ### Tech Stack: - Python 3.12 - Node.js 24 - Docker Compose ### Why It Matters: Inferna Next simplifies the process of deploying AI models on your own hardware, making it accessible to developers and researchers

Overview: This repository contains a pre-trained VGG-16 network fine-tuned for object recognition, specifically designed to identify clothing attributes. Key features: - Fine-tuned VGG-16 network for clothing attribute recognition - Preprocessed dataset for accurate attribute detection - Normalized confusion matrix for visualizing classification performance Tech stack: Python, Keras, Scikit-Learn Why it matters: This model provides a practical solution for developers looking to automate clothing attribute recognition, enhancing the efficiency and accuracy of their applications.

### Overview: A curated list of tools, tricks, news, and regulations related to AI and Large Language Model (LLM) productization, focusing on speeding up the adoption and addressing specific business needs. ### Key features: - Open LLM Models - Full LLM Lifecycle - LLM Prompt Management - LLM Finetuning - Embeddings - Vector Store - LLM Deployment - LLM Boilerplate - LLM Monitoring - Use Cases - General MLOps Tools ### Tech stack: Python, Jupyter Notebooks, Git, Docker, and various open-source libraries and tools. ### Why it matters: This repository aims to provide a comprehensive guide and resources to help developers and businesses navigate the challenges of deploying and productizing Large Language Models (LLMs) effectively. By covering everything from open-source models to deployment strategies, it aims to streamline the process and provide practical solutions for businesses looking to integrate LLMs into their workflows.

Overview: This repository contains reproducible llama.cpp configs for Qwen3.6-27B on a single RTX 4090, including detailed results and quality checks across various task categories. Key features: - Mainline llama.cpp configs for Qwen3.6-27B-Q4_K_M - Per-category quality verification - Winners, dead ends, and the silent-corruption bug Tech stack: Python Why it matters: This repository provides a comprehensive set of configs for Qwen3.6-27B on RTX 4090, including detailed results and quality checks, which can be useful for developers looking to optimize their models or troubleshoot issues related to model performance and quality. Repository: outsourc-e/qwen36-4090-recipes Primary language: Python GitHub stars: 24

### Overview: Turbo MLX is a macOS app that generates images and short videos using Apple Silicon. It's designed to be lightweight, sandboxed, and doesn't require Python or any additional installations. ### Key features: - **Image Generation**: Converts text descriptions into high-quality images and short videos. - **Video Creation**: Generates short videos with sound from a description or starting from an image. - **Upscaling**: Enhances images up to 4 times in resolution. - **Model Support**: Supports Z-Image Turbo, FLUX.2 Klein, Qwen-Image, Qwen-Image Edit, and LTX-2.3. ### Tech stack: Swift ### Why it matters: Turbo MLX simplifies AI image and video generation, making it accessible and efficient for macOS users without the need for Python or additional installations.

Overview: This modular project template provides a structured approach for teams building software with Claude Code, Obsidian, and Figma, suitable for teams of any size and maturity level. Key features: - **Level-based Structure**: Teams can choose their project maturity level (L1-L4) and plug in their specific specifications. - **Bootstrap Script**: A script to quickly set up the project at a chosen level, including initial configuration and project-specific files. - **Project Structure**: A clear directory layout for Claude Code, Obsidian, and Figma integration, including a CLAUDE.md hub for project specifications and a .claude folder for additional configurations and skills. Tech stack: Shell Why it matters: This template offers a practical, opinionated solution for project structure, allowing teams to adapt their workflow to their specific needs without being locked into a specific toolchain. It combines insights from industry frameworks, published research, and hands-on experience to provide a scalable and flexible solution. GitHub stars: 23 Repository: rtazima/claude-proj-blueprint

### Overview: M2M Vector Search is an edge vector search engine designed for resource-constrained devices. It leverages Vulkan GPU acceleration, hierarchical indexing (HRM2), and native LangChain integration to provide efficient similarity search capabilities. The system uses a Gaussian splat-based architecture, enabling probabilistic similarity search and online learning via feedback. ### Key Features: - **Gaussian Splats**: Each vector is represented as a learnable Gaussian, allowing for independent encoding of position, concentration, and importance. - **Online Learning**: Hebbian update rules adapt splat parameters from user feedback after each query, without requiring retraining or re-indexing. - **Energy-Based Model**: Native uncertainty quantification, providing insights into the model's confidence in its predictions. ### Tech Stack: Python ### Why It Matters: M2M Vector Search offers a high-performance solution for vector search on resource-limited devices, making it ideal for applications where computational resources are limited, such as edge devices or resource-constrained servers. Its ability to adaptively learn from user feedback and provide probabilistic similarity scores makes it particularly useful in scenarios where precise and reliable search results are crucial.

Overview: Federated Foundation Models on Heterogeneous Time Series (FFTS) is an implementation of a paper that explores time series foundation models in a federated learning setting, addressing the challenges of statistical heterogeneity and limited generalization across diverse domains. Key features: - Federated learning formulation for time series foundation models - Client-specific local models - Shared knowledge alignment - Dual-side regularization for generalization - Unified adaptation architecture Tech stack: Python, PyTorch, Federated Learning Why it matters: This work is significant because it provides a solution to the challenges faced when training general-purpose time series foundation models across heterogeneous datasets, ensuring more robust and adaptable models. Repository: shengchaochen82/FFTS GitHub stars: 24 GitHub description: The implementation of paper "Federated Foundation Models on Heterogeneous Time Series" | The first work to explore time series foundation models on federated setting.

Overview: This repository contains code for the paper "TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion" by Sophia Tang, Yuchen Zhu, Molei Tao, and Pranam Chatterjee. It builds upon the PepTune repo and MDNS, introducing a framework for discrete diffusion fine-tuning using off-policy RL with tree search and MCTS. Key features: - Tree search for efficient exploration of discrete sequence spaces. - Fine-tuning of pre-trained models on discrete diffusion trajectories. - Integration of off-policy RL with stochastic optimal control theory. - Code for regulatory DNA sequence design and multi-objective peptide design. Tech stack: Python. Why it matters: This framework enhances the performance of off-policy reinforcement learning by leveraging tree search and discrete diffusion fine-tuning, providing a theoretically guaranteed convergence to the optimal distribution. Repository: sophtang/TR2-D2 GitHub stars: 24 GitHub description: Code for TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion.