Projects

### Overview: This project introduces the **Leapfrog Diffusion Model for Stochastic Trajectory Prediction (LED)**, a novel diffusion-based model designed to predict stochastic trajectories efficiently. The model addresses the challenge of expensive inference times in diffusion models by leveraging a trainable leapfrog initializer, which significantly accelerates prediction speed without compromising accuracy. ### Key Features: - **LEapfrog Diffusion Model (LED)**: A diffusion-based model that predicts stochastic trajectories in real-time. - **Efficient Inference**: The leapfrog initializer skips many denoising steps, making inference much faster compared to standard diffusion models. - **Improved Diversity**: The model generates diverse predictions by appropriately allocating correlated samples, enhancing the accuracy of trajectory prediction. - **Real-World Applications**: LED has been tested on datasets including NBA, NFL, SDD, and ETH-UCY, showing significant improvements in ADE and FDE metrics. ### Tech Stack: - **PyTorch**: The primary framework used for implementing the model. ### Why it Matters: LED significantly improves the efficiency and accuracy of stochastic trajectory prediction, making it suitable for real-time applications in various domains, such as robotics, autonomous vehicles, and more. ### Repository: The project is hosted on GitHub under the repository **MediaBrain-SJTU/LED**. It has been starred by 214 developers, indicating its popularity and utility in the field of trajectory prediction. ### Primary Language: The project is written in **Jupyter Notebook**, making it accessible and easy to understand through interactive code execution.

Overview: ChronoMagic-Bench is a benchmark designed for evaluating the capabilities of text-to-time-lapse video generation models. It enables the creation of time-lapse videos with rich physics priors through free-form text prompts. Key features: - Enables the generation of time-lapse videos with rich physics priors - Provides a benchmark for metamorphic evaluation of text-to-time-lapse video generation models - Offers a detailed dataset and benchmark for subject-to-video generation Tech stack: Python Why it matters: ChronoMagic-Bench is significant because it evaluates the capabilities of text-to-time-lapse video generation models in physics, biology, and chemistry by enabling the creation of time-lapse videos with rich physics priors through free-form text prompts. Repository: PKU-YuanGroup/ChronoMagic-Bench GitHub stars: 214 GitHub description: [NeurIPS 2024 D&B Spotlight] ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

### Overview: UniTEX is an innovative, two-stage framework designed to generate high-fidelity textures for 3D shapes. This project addresses the challenges of UV-based inpainting by utilizing a unified 3D functional space approach. It leverages Texture Functions (TFs) to predict textures directly from images and geometry inputs, significantly improving texture quality and consistency. ### Key Features: - **Texture Functions (TFs)**: Lifts texture generation into a 3D space, mapping any 3D point to a texture value based on surface proximity. - **Transformer-based Large Texturing Model (LTM)**: Predicts TFs from images and geometry inputs, enhancing texture quality. - **Flux Lora Checkpoints**: Provides advanced LoRA-based strategies for efficient adaptation of large-scale Diffusion Transformers (DiTs) for multi-view texture synthesis. ### Tech Stack: - Python ### Why It Matters: UniTEX offers a generalizable and scalable solution for automated 3D texture generation, surpassing existing approaches in terms of visual quality and texture integrity. This framework is particularly valuable for applications requiring high-fidelity textures on 3D assets, such as in architectural design, gaming, and virtual reality.

### Overview: SHINE is a novel training-free framework for seamless and high-fidelity image composition that leverages pre-trained customization adapters to guide latent representations for faithful subject representation while preserving background integrity. This approach addresses the limitations of modern text-to-image diffusion models by eliminating the need for latent inversion, ensuring neutralized errors, and improving performance on diverse and challenging conditions. ### Key features: - **Manifold-steered Anchor Loss**: Utilizes pretrained customization adapters to guide latents for faithful subject representation. - **Degradation-suppression Guidance**: Eliminates low-quality outputs and visible seams. - **Adaptive Background Blending**: Enhances background integrity and realism in compositions. - **State-of-the-Art Performance**: Achieves state-of-the-art results on standard metrics and human-aligned scores. ### Tech stack: Python ### Why it matters: SHINE simplifies the process of image composition by eliminating the need for latent inversion, leading to more natural and realistic results. This makes it particularly useful in applications where maintaining the integrity of the background is crucial, such as in artistic or creative contexts. Repository: ZhumingLian/SHINE GitHub stars: 214 GitHub description: [ICLR 2026] "Does FLUX Already Know How to Perform Physically Plausible Image Composition?" (Official Implementation)

Overview: SMITE is an advanced open-source framework designed for temporally consistent video segmentation, leveraging pre-trained diffusion models and few reference images to predict and segment objects across video frames efficiently. Key features: - Utilizes pre-trained diffusion models and a few reference images for segmentation. - Ensures consistent segmentation across video frames through advanced tracking and temporal voting mechanisms. - Offers flexible segmentation at various granularities for tasks requiring different levels of detail. Tech stack: Python Why it matters: SMITE minimizes dataset needs by leveraging pre-trained diffusion models and a few reference images, ensuring efficient and precise segmentation across video sequences, even with occlusions, varying poses, and lighting conditions. Repository: alimohammadiamirhossein/smite GitHub stars: 214 GitHub description: Pytorch Implementation of "SMITE: Segment Me In TimE" (ICLR 2025) README excerpt: SMITE is an advanced open-source framework for temporally consistent video segmentation, designed to predict and segment objects across video frames using one or few reference images. With its ability to track and generalize unseen video sequences based on flexible granularity, it ensures precision and efficiency in segmentation, even when faced with occlusions, varying poses, and lighting conditions.

Overview: This project offers a ComfyUI nodes-based image generation and editing workflow for the MiniMax H3 model, utilizing Qwen for detail refinement. The setup includes two starter workflows, a shared model setup, and optional stackable LoRAs. Key features: - Two starter workflows - Shared model setup - Optional stackable LoRAs - Support for multiple reference images Tech stack: ComfyUI 0.37.0 or newer, including Image Studio Why it matters: This project streamlines the process of generating and editing still images using the MiniMax H3 model with Qwen for refinement, offering a more streamlined workflow compared to previous setups.

### Overview: A research project to add some brrrrrr to Burp, focusing on offensive agent capabilities and automated response capture. ### Key Features: - **Automated Response Capture**: Burp Suite acts as your client monitor, automatically capturing responses that fall within your defined scope. - **API Integration**: Requests and responses are packaged and forwarded to a remote LLM API in JSON format. - **Color-coded Tabs**: Display findings in `critical/high/medium/low/informational` tabs for easy visualization. - **Scanner Analysis**: Provides focused security analysis capabilities, including direct analysis of URLs and OpenAPI specifications. ### Tech Stack: Python ### Why It Matters: This extension aims to enhance the capabilities of Burp Suite by adding an offensive agent, making it more versatile for security testing and engagement.

### Overview: DiLightNet is a novel method for fine-grained lighting control during text-driven diffusion-based image generation, specifically addressing the second stage of the DiLightNet framework. It introduces a ControlNet model that generates a foreground image under the target lighting conditions. ### Key Features: - **Fine-Grained Lighting Control**: The method allows precise control over lighting in generated images. - **Three-Stage Process**: The framework consists of three stages: provisional image generation, foreground synthesis, and background inpainting. - **ControlNet Model**: A neural network model that generates a foreground image under the target lighting conditions, taking a provisional image, a mask, and radiance hints as inputs. ### Tech Stack: Python ### Why It Matters: DiLightNet provides a powerful tool for fine-tuning the lighting in diffusion-based image generation, enhancing the realism and quality of generated images. Its ability to control lighting precisely makes it valuable for applications requiring high-quality, detailed lighting effects in text-driven image synthesis. Repository: iamNCJ/DiLightNet GitHub Stars: 214 GitHub Description: Official Code Release for SIGGRAPH 2024 DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation

### Overview: SceneTok: A Compressed, Diffusable Token Space for 3D Scenes SceneTok is a novel approach that encodes 3D scenes into a compressed, unstructured, and diffusable token representation. This tokenization enables efficient scene reconstruction and generation by leveraging generative models. The method is particularly useful for applications requiring real-time rendering and low-resource scenarios. ### Key Features: - **Compressed Representation**: Scene tokens are compact and can be easily compressed and decompressed. - **Diffusibility**: Tokens can be rendered from novel views using generative models, allowing for flexible scene reconstruction. - **Efficient Reconstruction**: The compressed representation allows for efficient scene reconstruction without the need for high-resolution images. - **Real-Time Rendering**: Tokens can be rendered quickly, making SceneTok suitable for real-time applications. ### Tech Stack: Python ### Why it Matters: SceneTok enables efficient and flexible scene reconstruction and generation, making it suitable for real-time applications and low-resource scenarios. The method's ability to handle large-scale 3D scenes with minimal computational resources is particularly valuable in applications requiring real-time rendering and low-resource environments. Repository: mohammadasim98/scenetok GitHub Stars: 214

### Overview: CoAgents is an official code repository for the paper "Learning to Use Tools via Cooperative and Interactive Agents". This framework combines cooperative and interactive agents to enable effective tool use. ### Key features: - **Cooperative and Interactive Agents Framework**: The ConAgents framework allows agents to work together to generate tool-use instructions, execute selected tools, and adapt iteratively based on feedback from the environment. - **Open-Ended Tool Use**: The agents can generate instructions for any type of tool, making the framework highly flexible and adaptable to various tasks. - **Comprehensive Feedback Loop**: The observing agent extracts relevant values from execution results, ensuring that the agents can adapt their actions based on the environment's feedback. ### Tech stack: Python ### Why it matters: CoAgents provides a robust framework for agents to collaborate and learn from each other, enabling more efficient and effective tool use in various applications. This technology has the potential to revolutionize how tools are used in different environments, making it a valuable contribution to the field of artificial intelligence and robotics. ### Repository: shizhl/CoAgents ### Primary language: Python ### GitHub stars: 214 ### GitHub description: Official code for paper "Learning to Use Tools via Cooperative and Interactive Agents"

### Overview: Starcat is a native macOS app that transforms your GitHub Stars into a searchable, organized, and AI-powered personal knowledge base. ### Key features: - Syncs starred repositories into a local-first desktop workspace. - Renders README content for easy access. - Adds tags, notes, and reading status to enhance organization. - Tracks releases, evaluates repository health, and more. ### Tech stack: Swift ### Why it matters: Starcat provides a comprehensive solution for managing and organizing your GitHub Starred repositories, making it easier to find, understand, and utilize the information you've collected from your contributions.

### Overview: This project, titled "Making Large Language Models Perform Better in Knowledge Graph Completion," focuses on enhancing the performance of Large Language Models (LLMs) in knowledge graph completion (KGC). The research aims to improve the accuracy of LLMs by incorporating structural information from knowledge graphs, thereby enhancing their ability to reason structurally. The project introduces a novel approach called Knowledge Prefix Adapter (KoPA), which leverages structural embeddings to project them into the textual space, enabling LLMs to obtain virtual knowledge tokens as a prefix of the input prompt. ### Key Features: - **KoPA (Knowledge Prefix Adapter)**: A method that integrates structural information from knowledge graphs into LLMs to improve their reasoning ability. - **Structural Embeddings**: Captures the structural information of entities and relations in knowledge graphs. - **LLM Integration**: Enhances the LLMs' ability to reason structurally by incorporating the structural embeddings. ### Tech Stack: - **Python**: The primary programming language used for the project. - **GitHub Stars**: 214, indicating the project has gained significant traction in the developer community. ### Why It Matters: The project addresses a critical gap in the research on LLM-based KGC by focusing on how to effectively utilize the structural information in knowledge graphs. This approach aims to improve the accuracy and effectiveness of LLMs in reasoning about structured data, thereby enhancing their performance in various applications that rely on knowledge graphs. ### Repository: The project is hosted on GitHub at [zjukg/KoPA](https://github.com/z

Overview: This project aims to build a 2.3 million-parameter language model (LLM) using the LLaMA 1 architecture, without requiring a powerful GPU. The approach is detailed and includes explanations of key concepts like RMSNorm, SwiGLU activation, and Rotary Embeddings. Key features: - Building a 2.3 million-parameter LLM from scratch - Using LLaMA 1 architecture - No need for a fancy GPU - Detailed explanation of key concepts Tech stack: Jupyter Notebook Why it matters: This project offers a practical guide for developers interested in creating their own large language models, focusing on a simpler, more accessible approach using a basic dataset and a straightforward dataset preprocessing method. Repository: FareedKhan-dev/create-million-parameter-llm-from-scratch GitHub stars: 213 GitHub description: Building a 2.3M-parameter LLM from scratch with LLaMA 1 architecture.

Overview: This repository contains code for training and evaluating diffusion models with chain-of-thought reasoning, specifically focusing on the paper "Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models." Key features: - Implementation of Diffusion-of-Thought (DoT) model - Integration of DoT with diffusion models for improved reasoning capabilities - Use of Chain-of-Thought (CoT) technique to enhance reasoning ability in autoregressive language models - Flexibility in trade-off between computation and reasoning performance - Self-correction abilities and benefits from existing reasoning-enhancing techniques like self-consistency decoding Tech stack: Python, NVIDIA Apex, PyTorch Why it matters: This project contributes to the understanding and development of reasoning capabilities in diffusion language models, offering a flexible approach to reasoning in diffusion models that can benefit from existing techniques like CoT and self-consistency decoding.

### Overview: Gideon Agent is a personal AI agent platform designed to learn, adapt, and take care of tasks autonomously. It offers a self-hosted edition for developers to control and manage their state, and a hosted service for those who prefer managed offerings. ### Key Features: - **Talk to it, or hand work off.** Chat sessions with streaming replies, tool calls, artifacts, and a transcript for search. - **Let it run unattended.** Loops work across many turns on schedules, turning one-off requests into repeatable tasks. - **Give it a memory.** Layered memory keeps preferences and context between conversations, and the knowledge base holds documents cited from your input. - **Stay in the loop.** Inboxes collect messages from channels like Slack and Gideon itself, with optional voice input and spoken replies. - **Extend it.** The app platform and SDK cover models, channels, search, tools, and dashboards, and skills, prompts, and MCP servers add capability without touching the core. - **Decide what it may touch.** Tool approvals, per-app permissions, credential handling, command screening, and an audit trail. The core is provider-agnostic: integrations live in apps, never in the core package. - **Watch it work.** The console shows sessions, activity, running loops, scheduled jobs, and health, and a terminal for the machine Gideon is working on. ### Tech Stack: Python ### Why It Matters: Gideon Agent is a powerful tool for developers looking to automate and streamline their workflows.

### Overview: World-in-World is a unified closed-loop benchmark and toolkit designed to evaluate visual world models (WMs) based on their embodied utility. It provides a unified online planning strategy, a unified action API, and a comprehensive task suite including Active Recognition (AR), Active Embodied QA (A-EQA), Image-Goal Navigation (IGNav), and Robotic Manipulation. ### Key Features: - **Unified Online Planning Strategy**: Works with various WMs. - **Unified Action API**: Adapts to text, viewpoint, and low-level controls. - **Task Suite**: Covers Active Recognition, Active Embodied QA, Image-Goal Navigation, and Robotic Manipulation. ### Tech Stack: Python ### Why It Matters: Establishing this evaluation framework is essential for tracking genuine progress across the rapidly expanding landscape of visual world models and embodied AI, ensuring that generated worlds actually enhance embodied reasoning and task performance within a closed loop.

### Overview: This project demonstrates a fully managed RAG solution using Knowledge Bases for Amazon Bedrock, a service that offers a choice of high-performing foundation models for building generative AI applications. Amazon Bedrock simplifies the process of experimenting with and evaluating these models, integrating them into enterprise systems and data sources securely. ### Key Features: - **Knowledge Bases for Amazon Bedrock**: Implements the entire RAG workflow from ingestion to retrieval and prompt augmentation, ensuring seamless integration with existing data sources. - **Vector Store**: Manages and retrieves vector representations of documents and content, enabling efficient and accurate queries. - **Data Sources**: Supports integration with Amazon S3 for documents and content from websites, allowing for easy data management and retrieval. ### Tech Stack: JavaScript ### Why It Matters: RAG solutions like this one are crucial for organizations looking to enhance the relevance and accuracy of their language models without the need for costly retraining. By leveraging Amazon Bedrock's capabilities, businesses can experiment with cutting-edge AI models while maintaining control over their data and infrastructure. ### GitHub Stars: 213 ### Repository: aws-samples/amazon-bedrock-rag

Overview: Nosia is a self-hosted platform that allows developers to run AI models on their own data, providing OpenAI-compatible APIs and integrating with the Model Context Protocol (MCP) for seamless connectivity with external tools and services. Key features: - 🔒 Private & Secure: Your data stays on your infrastructure. - 🤩 OpenAI-Compatible API: Drop-in replacement for OpenAI clients. - 📚 RAG-Powered: Augment AI responses with your documents. - 🔌 MCP Integration: Connect AI to external tools and services via Model Context Protocol. - 🤖 Agent Skills: Extend chat with custom LLM-driven and Ruby-based skills. - 🔄 Real-time Streaming: Server-sent events for live responses. - 📄 Multi-format Support: PDFs, text files, websites, and Q&A pairs. - 🚀 Easy Deployment: Docker Compose with one-command setup. - 🔑 Multi-tenancy: Account-based isolation for secure data separation. Tech stack: Ruby Why it matters: Nosia provides a secure, private, and customizable platform for developers to run AI models on their own data, making it ideal for businesses looking to integrate AI into their applications without relying on external services. GitHub stars: 213 GitHub description: Self-hosted AI RAG + MCP Platform

### Overview: Flowcraft is a lightweight, composable, and type-safe workflow engine designed for building complex business processes, data pipelines, ETL workflows, and AI agents. It simplifies the process of defining and executing workflows without requiring any dependencies, ensuring easy integration and scalability. ### Key Features: - **Zero Dependencies**: Lightweight and dependency-free, ensuring easy integration in any runtime. - **Declarative Workflows**: Define workflows as [serializable](https://flowcraft.js.org/guide/core-concepts#workflow-blueprint) objects with [nodes and edges](https://flowcraft.js.org/guide/core-concepts#nodes-edges). - **Unopinionated Logic**: Nodes can be simple [functions](https://flowcraft.js.org/guide/core-concepts#function-based-nodes) or structured [classes](https://flowcraft.js.org/guide/core-concepts#class-based-nodes), supporting any logic. - **Progressive Scalability**: Run workflows in memory or scale to distributed systems using the same blueprint. - **Resilient Execution**: Built-in support for retries, fallbacks, timeouts, and cancellation. - **Advanced Patterns**: Includes batches, loops, subflows, and HITL constructs for complex workflows. - **Extensibility**: Pluggable loggers, evaluators, serializers, and middleware for custom behavior. - **Static Analysis**: Tools to detect cycles, validate blueprints, and generate visual diagrams. - **Type-Safe API**: Fully typed with TypeScript for a robust developer experience. ### Tech Stack: TypeScript ### Why

### Overview: TryOffAnyone is an official repository for the paper "TryOffAnyone: Tiled Cloth Generation from a Dressed Person," focusing on generating tiled cloth images from a dressed person's image. This project utilizes advanced machine learning techniques to create realistic and detailed cloth textures. ### Key Features: - **Cloth Generation**: The repository includes a model that generates tiled cloth images from a dressed person's image. - **Model Compatibility**: The model is compatible with upper clothes similar to those found in the VITON-HD dataset. - **Inference**: Users can easily run the model on any image URL using the provided script. - **Evaluation**: The project includes evaluation scripts for the model's performance on the VITON-HD dataset, including a test evaluation. ### Tech Stack: Python ### Why It Matters: This project is significant for researchers and developers interested in advanced image generation techniques, specifically in the area of cloth generation from dressed persons. It offers a practical tool for creating realistic and detailed cloth textures, which can be applied in various fields such as fashion, design, and virtual reality. ### Repository: - **Repository**: ixarchakos/try-off-anyone - **GitHub Stars**: 212 - **Primary Language**: Python - **GitHub Description**: Official repository of the paper: _"TryOffAnyone: Tiled Cloth Generation from a Dressed Person"_. - **GitHub Logo**: [arXiv](https://img.shields.io/badge/🤗-Models-blue.svg?style=plastic) [