Projects

Overview: This project is a lightweight implementation of vLLM, focusing on reproducing vLLM v1's scheduler and introducing Chunked Prefill. It offers comparable online inference speeds and offline throughput performance to vLLM v1, with a clean, readable codebase and optimization suite. Key features: - Fast inference - Readable codebase - Optimization suite Tech stack: Python Why it matters: This project provides a practical alternative to vLLM v1, offering similar performance and a clean implementation, making it valuable for developers looking to integrate vLLM into their projects.

### Overview: UPGPT is an advanced universal diffusion model designed for person image generation, editing, and pose transfer. It combines the capabilities of various diffusion models to achieve high-quality results. ### Key Features: - **Universal Diffusion Model**: A versatile model capable of generating, editing, and transferring poses of person images. - **Pose Transfer**: Seamlessly interpolates poses between different images. - **Conditional Generation**: Uses text, visual prompts, and pose information to generate high-quality images. ### Tech Stack: - **Jupyter Notebook**: Primary language used for development and experimentation. ### Why It Matters: UPGPT is a significant advancement in the field of person image generation and pose transfer, offering unparalleled quality and versatility. Its ability to handle multiple input types makes it highly valuable for applications requiring high-fidelity image manipulation. ### Repository: - **Repository**: soon-yau/upgpt - **Stars**: 104 - **Description**: Official GitHub repository for UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfer. ### Publication: - **ICCV Workshop 2023**: Official publication details are available. - **Arxiv**: [arxiv.org/abs/2304.08870](https://arxiv.org/abs/2304.08870) - **BibTeX**: ```latex @InProceedings{Cheong_2023_ICCV, author = {Cheong, Soon Yau and Mustafa, Ar

### Overview: Z-Sampling, proposed in our paper "Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection," has been accepted by ICLR 2025. An upgraded version, "[Weak-to-Strong Diffusion](https://github.com/xie-lab-ml/Weak-to-Strong-Diffusion-with-Reflection)," has also been accepted by ICLR 2026. This guide provides instructions on how to use Z-Sampling, a novel sampling method that leverages the guidance gap to accumulate semantic information step-by-step throughout the entire generation process. ### Key features: - Z-Sampling: A novel self-reflection-based diffusion sampling method that leverages the guidance gap between denoising and inversion to accumulate semantic information step by step. - Weak-to-Strong Diffusion (W2SD): An upgraded version of Z-Sampling, released as our new work "[Weak-to-Strong Diffusion](https://github.com/xie-lab-ml/Weak-to-Strong-Diffusion-with-Reflection)." - Stable Diffusion XL: The inference code supports this method, which can be applied to various diffusion models with limited coding and computational costs. ### Tech stack: Python ### Why it matters: Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, DreamShaper with Z-Sampling can self-improve with the HPSv2 winning rate up to 94% over the original results.

### Overview: ECCV 2024 | EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion ### Key features: - **Scene Graph Diffusion**: A novel method for generating indoor scenes using scene graph diffusion. - **Information Echo**: Echoes information from the scene graph to the latent space for better scene reconstruction. - **Scene Graph**: A graph-based representation of the scene that captures relationships between objects. ### Tech stack: - Python ### Why it matters: ECCV 2024 | EchoScene offers a novel approach to indoor scene generation by leveraging scene graph diffusion and information echo, providing high-quality scene reconstruction that can be applied in various applications such as augmented reality, virtual reality, and more.

### Overview: Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities was presented at CVPR 2024. This project aims to enhance the capabilities of a specific modality's transformer by incorporating irrelevant data from other modalities, such as audio or point cloud datasets. The method leverages the universal sequence-to-sequence modeling abilities of transformers to process data from the target modality, thereby improving performance across various tasks. ### Key Features: - **Cross-Modal Re-parameterization**: Utilizes auxiliary transformer weights to re-parameterize the target modality's transformer, enabling efficient processing of data from the target modality. - **Modality-Specific Tokenizer and Task-Specific Head**: Maintains the standard tokenizer and head for the target modality while reusing the auxiliary transformer's blocks. - **Performance Improvement**: Demonstrated significant and consistent performance improvements on image, point cloud, video, and audio recognition tasks using irrelevant data from other modalities. ### Tech Stack: - **Python**: The primary programming language used for implementing the project. ### Why It Matters: This project bridges the gap between basic Transformer models and multimodal learning, showcasing how irrelevant data from other modalities can be effectively utilized to improve the performance of a specific modality's transformer. The method's success in various tasks highlights the potential of combining different modalities to enhance AI models, particularly in improving the efficiency and generalization capabilities of transformers.

### Overview: This is a comprehensive learning path and practical guide for Generative AI development, focusing on hands-on LangChain implementations and detailed notes. The roadmap includes topics such as foundational knowledge, LLM calling, context engineering, modern RAG, tools and MCP, single-agent engineering, multi-agent orchestration, evaluations and test-driven LLM development, production and economics, fine-tuning and self-hosting, safety, alignment, and real-time voice. The projects and supporting material are organized into a 12-rung skill ladder, with each topic followed by an estimated time for completion. ### Key features: - **Foundations**: 1 week - **Applied LLM Calling**: 1 week - **Context Engineering**: 2 weeks - **Modern RAG**: 3 weeks - **Tools & MCP**: 2 weeks - **Single-Agent Engineering**: 2 weeks - **Multi-Agent Orchestration**: 3 weeks - **Evals & Test-Driven LLM Dev**: 2 weeks - **Production & Economics**: 2 weeks - **Fine-tuning & Self-Hosting**: 2 weeks - **Safety, Alignment & Red-Teaming**: 1 week ### Tech stack: Python ### Why it matters: This roadmap provides a rigorous and file-per-topic approach to becoming a production GenAI engineer, ensuring that every claim is backed by citations and is aligned with the current state of the industry. The projects and supporting material are designed to be practical and hands-on, ensuring that the skills learned can be immediately applied

### Overview: This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through formal/mathematical analysis—proofs, provable guarantees, bounds, expressivity results, convergence analysis, and similar. Papers that also include supporting experiments still count; purely empirical/observational papers do not. ### Key Features: - Theoretical and empirical analysis of language models. - Formal/mathematical analysis of learning behavior, generalization ability, and other properties. - Papers that include both theoretical and empirical contributions. ### Tech Stack: - Python ### Why It Matters: This list provides a comprehensive overview of the theoretical and empirical analysis of language models, focusing on the learning behavior, generalization ability, and other properties of large language models (LLMs). It offers a valuable resource for researchers and practitioners interested in understanding the theoretical underpinnings of language models. ### GitHub Stars: 103 ### Repository: Furyton/awesome-language-model-analysis ### Table of Content: - **Phenomena of Interest** - In-Context Learning - Chain-of-Thought - Hallucination - Reversal Curse - Scaling Laws / Emergent Abilities / Grokking / etc. - Knowledge / Memory Mechanisms - Training Dynamics / Landscape / Optimization / Fine-tuning / etc. - Learning / Generalization / Reasoning / Weak to Strong Generalization - Other

Overview: Idiolect is a hands-free audio development interface for the IntelliJ Platform, designed for visually impaired and RSI users. It supports a simple grammar and provides a wide range of commands for navigation, editing, debugging, and more. Key features: - Voice control and navigation commands - Editing commands for creating, renaming, and deleting files - Debugging commands for context run and coverage - Git commands for managing project files Tech stack: Kotlin Why it matters: Idiolect offers a convenient and accessible way to interact with the IntelliJ Platform, making it easier for users with disabilities to work on their projects without the need for a physical keyboard. Repository: OpenASR/idiolect GitHub stars: 103 GitHub description: A general purpose voice user interface for the IntelliJ Platform, inspired by Tavis Rudd's work. It supports a simple grammar and commands for various tasks, including navigation, editing, debugging, and Git commands. The project is inspired by a JetBrains hackathon and is now a community-supported initiative. Overview: Idiolect is a hands-free audio development interface for the IntelliJ Platform, designed for visually impaired and RSI users. It supports a simple grammar and provides a wide range of commands for navigation, editing, debugging, and more. The project is implemented using Kotlin and is inspired by Tavis Rudd's work. It is a community-supported initiative and is available on GitHub with 103 stars.

Overview: InferSim is a lightweight Python simulator designed for evaluating the performance of large language models (LLMs) like Qwen and Qwen3. It calculates throughput, token throughput, and GPU memory usage based on computational complexity and benchmark data. Key features: - Calculates TTFT, TPOT, and throughput TGS for LLM inference. - Supports multi-GPU and multi-node deployment. - Estimates communication latency for multi-GPU, multi-node scenarios. - Simulates performance bottlenecks and optimization efforts. Tech stack: Pure Python without any third-party dependencies. Why it matters: InferSim provides a practical tool for developers to evaluate the performance of their models, identify bottlenecks, and optimize their inference setups, making it essential for performance tuning and model sysco-design. Repository: alibaba/InferSim Primary language: Python GitHub stars: 103 GitHub description: A Lightweight LLM Inference Performance Simulator README excerpt:

Overview: Collabosm is a local application that allows you to rent a GPU in your own Google Colab account and run Qwen3.8 models on it, including 27B and 125B versions. The app is designed for simplicity and cost transparency, with chat functionality, image integration, and auto-stop features. Key features: - Rent a GPU in your own Colab account - Run Qwen3.8 models on an A100-40G or A100-80G GPU - Chat with the model in a simple app - Start and stop the GPU with a button - See the cost before payment - Use the model in other programs like Codex or ModelDock - Auto-stop, CU ledger, live prefill/decode - Windows, macOS, Linux support Tech stack: Python Why it matters: This app provides a convenient and cost-effective way to access large AI models for research and development, making it accessible to a wide range of users. Repository: architectds/collabosm Primary language: Python GitHub stars: 103 GitHub description: A local app that rents a GPU in your own Colab account and serves Qwen3.8 on it: 27B on an A100-40G, or Flash-Next (125B-A6B MoE) on an A100-80G. The cost is shown before it starts; chat with images; Codex or any Open

Overview: This project is a WebSocket server and client designed for real-time speech recognition using Mozilla's DeepSpeech. It allows easy integration of DeepSpeech models into separate environments, including Windows Subsystem for Linux. Key features: - Server: Supports streaming inference with DeepSpeech v0.2+ and can handle multiple users. - Client: Streams raw audio data from the microphone to the server via WebSocket, with voice activity detection and the option to save each utterance to a separate .wav file. Tech stack: Python with Bottle v0.12.13 and GeventWebSocketServer. Why it matters: This project simplifies the process of integrating DeepSpeech models into different environments, making it easier for developers to test and deploy speech recognition systems.

### Overview: SubForge is a Rust CLI designed for AI subtitle workflows, enabling reproducible and efficient subtitle production. It handles tasks like transcribing, segmenting, translating, evaluating, and burning subtitles into videos. ### Key Features: - **Transcription**: Uses `faster-whisper` for speech recognition. - **Segmentation**: Utilizes SaT-based subtitle segmentation. - **Translation**: Supports Google, Bing, and OpenAI LLMs. - **Quality Estimation**: Uses MAPS-style terminology extraction and GEMBA-MQM quality estimation. - **Subtitle Burning**: Hard-burns subtitles or soft subtitle muxing with ffmpeg. ### Tech Stack: - Rust 1.88+ - Python 3.9+ - FFmpeg ### Why It Matters: SubForge simplifies the complex workflow of subtitle production by making it explicit, repeatable, and inspectable. It is ideal for people who process videos repeatedly and want to avoid the mess of temporary files and manual rework. ### Installation: - **Requirements**: - Rust 1.88+ - Python 3.9+ - FFmpeg - **Installation**: - Install Rust: ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal # Linux / macOS ``` - Install FFmpeg: ```bash sudo apt install ffmpeg # Debian / Ubunt ``` ### Status: SubForge is

### Overview: VisuoThink is a framework designed to enhance Large Vision-Language Models (LVLMs) for reasoning tasks that require slow, visual-verbal thinking. By integrating visuospatial and linguistic domains, VisuoThink enables progressive visual-textual reasoning and supports test-time scaling through look-ahead tree search. This approach significantly improves reasoning capabilities without the need for fine-tuning, achieving state-of-the-art performance in geometry and spatial reasoning tasks. ### Key Features: - **Multimodal Tree Search**: Enables progressive visual-textual reasoning and test-time scaling. - **Inference-Time Scaling**: Enhances reasoning capabilities via inference-time scaling. - **State-of-the-Art Performance**: Achieves leading performance in geometry and spatial reasoning tasks. ### Tech Stack: - **Python**: The primary programming language used for development. ### Why It Matters: VisuoThink addresses the limitations of existing methods by capturing the intricate, interleaved nature of human visual-verbal reasoning processes. By integrating visuospatial and linguistic domains, it provides a more effective way to enhance reasoning capabilities in complex tasks, making it particularly useful in fields requiring slow, deliberate thinking. ### Repository: - **GitHub Repository**: ekonwang/VisuoThink - **GitHub Stars**: 103 - **Primary Language**: Python - **GitHub Description**: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics: VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search ### Quick Start: 1. **Install Dependencies**: ```bash conda create

Overview: FX is a workflow engine for generative asset pipelines, built with Rust, Python, and TypeScript, designed to manage the entire lifecycle of images, audio, video, 3D, and the code around them. Key features: - Priced before it spends, ensuring the whole run is priced upfront. - Never pays twice, caching results and preventing double payments. - Judged, not just generated, allowing steps to judge each other's output. - Switches between different AI providers per step or uses your own subscription. - Runs your own code without any provider, using Grida Gateway for text, images, video, and 3D. Tech stack: Rust, Python, TypeScript. Why it matters: FX simplifies the management of generative asset pipelines, making it easier to create, price, cache, and run workflows efficiently.

### Overview: Active Fine-Tuning is a library designed for automatically selecting data in the fine-tuning phase of large neural networks, enhancing efficiency and performance. ### Key features: - Automatic data selection for active fine-tuning - Supports PyTorch library - Efficiently learns at test-time - Transductive active learning theory and applications ### Tech stack: Python ### Why it matters: Improves the efficiency and effectiveness of fine-tuning large neural networks, leading to better model performance and faster training times.

Overview: Findto is a decentralized search assistant that allows users to search the same term across multiple sources, discover trends, stories, and information, and make safer choices. Key features: - Decentralized Search - Protection (Security, Carbon footprint, Privacy) - Trends - Stories - Autocomplete - Voice Search - Reverse Search - Internationalization - Accessibility Tech stack: TypeScript Why it matters: Findto offers a decentralized search experience that allows users to search the same term across multiple sources, discover trends, stories, and information, and make safer choices. Repository: lucasm/findto Primary language: TypeScript GitHub stars: 103 GitHub description: 🔵🔎 Findto is a decentralized search assistant.

Overview: This repository implements MaMMUT, a simplified vision-encoder text-decoder architecture for multimodal tasks, as described in the Google blog post "MaMMUT: Simple Vision-Encoder Text-Decoder Architecture for Multimodal Tasks". The architecture is based on the CoCa model and is designed to be easily adaptable for various applications. Key features: - MaMMUT architecture for multimodal tasks - Implementation in Pytorch - Adapted from the CoCa model - Cross-attention mechanism for controlling attention placement Tech stack: Pytorch Why it matters: This repository provides a practical implementation of MaMMUT, allowing developers to quickly integrate this architecture into their projects for multimodal tasks. The adaptable cross-attention mechanism offers flexibility in how much cross attention is applied, making it suitable for various applications where different levels of attention are needed. Repository: lucidrains/MaMMUT-pytorch GitHub stars: 103 GitHub description: Implementation of MaMMUT, a simple vision-encoder text-decoder architecture for multimodal tasks from Google, in Pytorch Installation: `pip install mammut-pytorch` Usage: The repository includes examples and usage instructions for training and utilizing the MaMMUT architecture with a vision transformer and text encoder.

Overview: This project is a production-ready ERP intelligent assistant built with Spring AI 2.0, featuring Tool Calling, RAG, chat memory, dynamic SQL tools, multi-tenancy, and ECharts visualization. Key features: - Tool Calling - RAG - Chat Memory - Dynamic SQL Tools - Multi-Tenancy Isolation - Stream Markdown Rendering - Dynamic Tool Management - Tool Hit Tracking - Charge Management Tech stack: Java 17+ Why it matters: This project leverages Spring AI 2.0 to provide a robust ERP intelligent assistant with advanced AI capabilities, including real-time data querying, retrieval of knowledge documents, and dynamic SQL tool management, ensuring seamless integration and scalability for enterprise applications. GitHub stars: 101 Repository: ly1836/spring-ai-rag-demo

Overview: Oracle AI Optimizer and Toolkit provides a streamlined environment for experimenting with Generative Artificial Intelligence (GenAI) and Retrieval-Augmented Generation (RAG) using Oracle AI Database and SQLcl MCP. Key features: - Configuring Embedding and Chat Models - Splitting and Embedding Documentation - Auto-refresh Vector Store from OCI Object Storage buckets - Modifying System Prompts (Prompt Engineering) - Experimenting with LLM Parameters - Testbed for auto-generated or existing Q&A datasets - Managing Oracle Deep Data Security policies Tech stack: Python 3.12–3.14, Streamlit client, and FastAPI/Uvicorn server. Why it matters: Oracle AI Optimizer and Toolkit enables developers and data scientists to explore the potential of GenAI and RAG, integrating Oracle AI Database VectorSearch and SQLcl MCP, to extend the knowledge of existing LLMs through RAG and NL2SQL.

Overview: Deepstory is a project that combines Natural Language Generation, Text-to-Speech, and speech-driven animation to create an interactive video experience. It turns text into a video where the character's voice is animated. Key features: - Text-to-Video Conversion - Interactive Web Interface - Flask Backend for Story Creation - Support for Transformers and PyTorch models - Modern Models Integration Tech stack: Python Why it matters: Deepstory is a pioneering project that showcases the potential of combining AI technologies to create engaging and interactive media experiences, particularly in the realm of storytelling. GitHub stars: 103 Repository: thetobysiu/Deepstory