Projects

### Overview Uni-CoT is a unified chain-of-thought reasoning framework designed to extend CoT principles to the multimodal domain, enabling Multimodal Large Language Models (MLLMs) to perform interpretable, step-by-step reasoning across both text and vision. This approach addresses the challenges of reasoning in visual contexts by integrating textual and visual information effectively. ### Key Features - **Unified Reasoning Pipeline**: Decomposes complex multimodal tasks into structured, manageable steps that can be executed sequentially or in parallel. - **Interpretable Reasoning**: Provides clear, step-by-step reasoning that is easy to understand and explain. - **Step-by-Step Execution**: Enables more scalable and systematic reasoning, making it suitable for a wide range of applications. ### Tech Stack Uni-CoT is implemented using Python, leveraging the strengths of the language for efficient development and execution. ### Why It Matters Uni-CoT offers a significant advancement in the field of multimodal reasoning, making it easier and more effective to reason about complex tasks involving both text and vision. This framework is particularly valuable in applications where understanding the reasoning process is crucial, such as in education, decision-making, and more. ### Repository The project is hosted on GitHub at [Fr0zenCrane/UniCoT](https://github.com/Fr0zencr4nE/UniCoT). It has received 237 GitHub stars, indicating its growing interest and importance in the community. ### Contact For more information, you can reach out to the project leader via email: fr0zencr

### Overview: VideoTetris is a novel framework that enables compositional text-to-video generation, enabling precise follow of complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. ### Key features: - **Compositional Text-to-Video Generation**: Enables precise follow of complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. - **Enhanced Video Data Preprocessing**: Equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. - **Successful Examples**: Demonstrations include videos spanning from 10s, 30s to 2 minutes. ### Tech stack: Python ### Why it matters: VideoTetris is a significant advancement in text-to-video generation, enabling precise follow of complex textual semantics and enhancing the quality of video data preprocessing.

Overview: The repository showcases a project that leverages cutting-edge AI technologies to generate talking face avatars. This project uses Leonardo AI for image generation and ElevenLabs for text-to-speech, resulting in a dynamic and lifelike avatar experience. The project is designed to be easily integrated into various applications and is highly customizable. Key features: - Integration with Leonardo AI for image generation - Use of ElevenLabs for text-to-speech - Creation of talking head videos - Enhancements with GFPGAN for improved image quality Tech stack: Jupyter Notebook Why it matters: This project provides a practical solution for developers interested in creating engaging and interactive digital avatars, making it easier to integrate AI-powered features into their applications.

Overview: DeepViewAgg is an official PyTorch implementation of the method presented in "Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation". This project aims to leverage the synergy between images and 3D point clouds for large-scale 3D semantic segmentation, achieving state-of-the-art results without requiring point colorization, meshing, or depth cameras. Key features: - End-to-end learning of 2D+3D features - Attentive multi-view aggregation from viewing conditions - No need for 3D colorization, meshing, depth sensor, synthetic views, or 2D labels - Only requires raw point clouds, images, and poses Tech stack: Python Why it matters: DeepViewAgg provides a powerful solution for large-scale 3D semantic segmentation that is both efficient and effective, making it a valuable tool for researchers and practitioners in the field of computer vision and 3D vision. Repository: drprojects/DeepViewAgg Primary language: Python GitHub stars: 237 GitHub description: Official implementation for "Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation" by Robert, Damien and Vallet, Bruno. Achieves SOTA results without point colorization, meshing, or depth cameras. Requires raw point clouds, images, and poses. README excerpt: [Link to the project page](https://drprojects.github.io/deepviewagg-site), [Link to the video](https://www.youtube.com

Overview: MotionGPT is an official PyTorch implementation of the paper "MotionGPT: Finetuned LLMs are General-Purpose Motion Generators". It leverages pretrained VQ-VAE and LLaMa models to generate realistic human motions from text prompts. Key features: - Text-to-motion generation - Fine-tuned LLaMa model for motion generation - Pretrained VQ-VAE for encoding and decoding motions Tech stack: Python Why it matters: MotionGPT provides a practical solution for generating human motions from text descriptions, making it useful for applications such as motion generation for virtual characters, game animations, and more. Repository: qiqiApink/MotionGPT Primary language: Python GitHub stars: 237 GitHub description: The official PyTorch implementation of the paper "MotionGPT: Finetuned LLMs are General-Purpose Motion Generators".

### Overview: OmniTransfer is an all-in-one framework designed for spatio-temporal video transfer, seamlessly integrating spatial appearance and temporal video transfer tasks. It leverages multi-view information across frames to enhance appearance consistency and exploits temporal cues to enable fine-grained temporal control. ### Key Features: - **Task-aware Positional Bias**: Adaptively leverages reference video information to improve temporal alignment or appearance consistency. - **Reference-decoupled Causal Learning**: Separates reference and target branches to enable precise reference transfer while improving efficiency. - **Task-adaptive Multimodal Alignment**: Dynamically distinguishes and tackles different tasks using multimodal semantic guidance. ### Tech Stack: Python, PyTorch, OpenCV ### Why it Matters: OmniTransfer demonstrates strong generalization across unseen task combinations, making it a versatile tool for flexible, high-fidelity video generation.

### Overview: Railtracks is a Python framework designed to help developers build resilient and customizable agent-based systems. It allows you to assemble your own tools, loops, and context management without relying on YAML or DSLs, making it highly flexible and adaptable. ### Key Features: - **Agent Framework**: Assembles Python objects for creating agents and harnesses. - **Tool Nodes**: Functions and classes that the agent can call. - **Loop Control**: Standard Python control flow for defining when the agent should run. - **Tool Manifests**: Simplifies creating tool nodes for agents. - **MCP Servers**: Allows for context management and permission controls. - **Context Management**: Handles the environment and context of the agent. - **Controls**: Limits and permissions for the agent's actions. - **Record**: Tracks and replayable history of agent activities. ### Tech Stack: - **Python**: The primary language used for development. ### Why It Matters: Railtracks simplifies the process of building and managing agent-based systems, allowing developers to focus on their core problem-solving without the complexity of traditional frameworks. It provides a clean, Pythonic way to manage agents and harnesses, making it ideal for developers looking to build custom, resilient systems.

### Overview: ZSE is a blazing-fast, zero-dependency inference engine for large language models, built entirely in Python with CUDA, ROCm, and Metal support. ### Key Features: - **Zero-Dependency**: Pure Python, no PyTorch, Triton, or bitsandbytes. - **Inference Speed**: Models load in seconds, not minutes. - **Memory Efficiency**: Reduces VRAM usage significantly. - **Cold Start**: 7-second cold start for 7B models. - **GPU Compatibility**: Verified on NVIDIA T4, L4, A10G, A100, and AMD MI300X. - **Model Format**: Runs in `.zse` format, pre-quantized and mmap-able. - **Built-in Features**: Built-in RAG, built-in auth + rate limiting, and LoRA hot-swappable. ### Tech Stack: Pure Python, ctypes, and a kernel compiler emitting CUDA, ROCm, and Metal. ### Why it Matters: ZSE offers unparalleled performance and efficiency, making it a game-changer for LLM inference on GPU architectures.

Overview: This repository contains optimized versions of the Stable Diffusion XL (SDXL) model using various techniques such as BFloat16 precision, scaled dot product attention (SDPA), torch.compile, and combining q, k, v projections for attention computation. These optimizations significantly speed up the generation process and can be applied to other pipelines as well. Key features: - BFloat16 precision - Scaled dot product attention (SDPA) - torch.compile - Combining q, k, v projections for attention computation Tech stack: Python Why it matters: These optimizations can significantly speed up the generation process of image-based models like Stable Diffusion, making them more efficient and faster to run, especially on hardware with limited memory or processing power. GitHub stars: 236 Repository: huggingface/diffusion-fast Primary language: Python GitHub description: Faster generation with text-to-image diffusion models.

Overview: M.I.L.E.S is a GPT-4-Turbo voice assistant designed to adapt its prompts and AI model, offering advanced features such as playing any Spotify song, adjusting system and Spotify volume, performing calculations, browsing the web, searching global weather, delivering date and time, and autonomously choosing and retaining long-term memories. Available for macOS and Windows. Key features: - Spotify integration - Home Assistant integration - Realistic voice - Persistent memory - Built-in calculator - Multi-tasking - Contextual awareness - Weather data - Internet browsing - Image recognition Tech stack: Python Why it matters: M.I.L.E.S offers a highly advanced and realistic voice assistant experience, capable of handling complex tasks and providing a seamless integration with smart home devices and the internet, making it a valuable tool for both personal and professional use.

### Overview: Starwhale is an MLOps/LLMOps platform designed to streamline machine learning operations, providing a unified platform for model development, evaluation, release, and fine-tuning. It abstracts Model, Runtime, and Dataset as first-class citizens, offering robust evaluations, live demos, and efficient fine-tuning capabilities. ### Key Features: - 🚀 Model Evaluation: Implement robust, production-scale evaluations with minimal coding through the Python SDK. - 🌟 Live Demo: Interactively assess model performance through user-friendly web interfaces. - 🌊 LLM Fine-tuning: End-to-end toolchain for efficient fine-tuning and comparative benchmarking, including publishing. ### Tech Stack: Java ### Why It Matters: Starwhale simplifies machine learning operations, making it easier for teams to optimize their workflows around key areas like model building, evaluation, release, and fine-tuning. Its modular design and open-source nature make it a versatile tool for both development and production environments.

Overview: This plugin automates the analysis of decompiled code in IDA Pro using ChatGPT, making reverse engineering more interactive and easier. Key features: - Automatic Code Analysis - Contextual Function Information - Rename Functions - Seamless Integration - Continuous Improvement Tech stack: Python Why it matters: This plugin enhances the reverse engineering process by providing a more interactive and efficient way to analyze code, making it easier and more enjoyable for developers.

Overview: DYffusion is a NeurIPS 2023 paper that introduces a dynamics-informed diffusion model for spatiotemporal forecasting. This model forecasts a sequence of snapshots given initial conditions, similar to standard diffusion models. Key features: - Interpolator network - Forecaster network Tech stack: Python Why it matters: This model offers a novel approach to spatiotemporal forecasting by incorporating dynamics into the diffusion process, potentially improving accuracy and efficiency compared to traditional methods. GitHub stars: 235 Repository: Rose-STL-Lab/dyffusion

### Overview: LucaProt is a novel deep learning framework designed to predict protein function by incorporating amino acid sequence and structural information. This project leverages the power of deep learning to enhance the accuracy of protein function annotation, making it particularly useful in the field of bioinformatics. ### Key Features: - **Model Architecture**: A deep learning model named DeepSequential and Structural Information Fusion Network for Proteins Function Prediction (DeepProtFunc/LucaProt) that integrates sequence and structural information to improve function prediction accuracy. - **Model Input/Output**: The model accepts amino acid sequence input and outputs function labels, which can be either single tags (binary-class classification) or multiple tags (multi-label classification). - **Deployment**: The project includes a deployable web application using Flask, allowing the service to be accessed via a browser on various operating systems (Linux, Mac OS, Windows) with both CPU and GPU configurations. ### Tech Stack: - Python: The primary programming language used for developing the framework. - Anaconda: Used for creating and managing Python environments. - CUDA: Provides support for GPU acceleration in deep learning tasks. ### Why It Matters: LucaProt offers a significant advancement in the field of bioinformatics by integrating sequence and structural information to predict protein function more accurately. This approach can be particularly useful in identifying viral RdRP, which is crucial for understanding the mechanisms of viral replication and drug resistance. ### Repository: The repository is hosted on GitHub at [alibaba/LucaProt](https://github.com/alibaba/LucaProt). It has been starred by 235

### Overview: This Jupyter Notebook serves as an annotated implementation of the **Flow Matching for Generative Modeling** paper by Yaron Lipman et al., focusing on the **Flow Matching** technique applied to **Optimal Transport conditional Vector Fields**. The notebook draws inspiration from the implementation provided by François Rozet and utilizes the Zuko package for solving Ordinary Differential Equations (ODEs) and sampling, aligning closely with the notation and equations presented in the original paper. ### Key features: - Annotated implementation of Flow Matching for Generative Modeling. - Focus on Optimal Transport conditional Vector Fields. - Utilizes the Zuko package for ODE solving and sampling. - Notations and equations from the original paper are maintained. - Implementation inspired by François Rozet's 100 LOC implementation. ### Tech stack: Jupyter Notebook ### Why it matters: This notebook provides a practical and annotated guide to implementing the Flow Matching technique, which is crucial for understanding and applying generative modeling in the context of optimal transport. The use of the Zuko package ensures compatibility with the paper's notation and equations, making it an essential resource for developers and researchers in the field.

### Overview: PenShot 是一个基于 LangChain + LangGraph 的剧本分镜智能体，旨在将剧本自动转换为 Sora、Veo、Runway 等模型可用的连贯文本提示词。通过 LangGraph 和 LlamaIndex 实现的多智能体协作，确保角色、剧情和场景在不同片段间保持一致。 ### Key features: - 智能剧本解析，识别场景、对话和动作指令，支持长文本分段处理。 - 精准时序规划，按镜头粒度智能切分内容，分配合理时长，严格适配 AI 视频生成模型的时长限制。 - 连续性守护，基于任务池优先级排队、多层级记忆（短期/中期/长期）与 Chroma 向量检索，确保相邻分镜间角色状态、场景和情节高度一致。 - 高质量提示词输出，生成详细的中英双语画面描述、负面提示词及音频提示词，开箱即用。 - 多模型兼容，支持 OpenAI、Qwen、DeepSeek、Ollama 等主流 LLM 提供商，可插拔切换。 - 多协议集成，提供 Python SDK、REST API、LangGraph 节点、A2A 协作协议与 MCP 标准接口。 - 健壮性与可追溯，内置自动重试、错误降级机制，每个分镜都记录详细的操作日志。 ### Tech stack: - Python ### Why

### Overview: KSoR (Knowledge System of Record) is an open-source SDK designed to build governed, authoritative knowledge systems for humans and AI agents. It provides a single, governed knowledge record that ensures consistent and reliable AI behavior across enterprises and education institutions. ### Key Features: - **Governed Knowledge Record**: One authoritative knowledge record for both enterprises and education institutions. - **Governance Boundary**: Ensures that knowledge is authoritative and traceable. - **Open Projections**: Allows multiple interpretations of the knowledge, enabling flexibility and adaptability. - **One Book Everyone Must Follow**: A single, unified knowledge system that ensures consistency and predictability. ### Tech Stack: TypeScript ### Why It Matters: An organization cannot expect predictable AI-agent behavior if its agents operate from scattered, conflicting, or outdated knowledge. A Knowledge System of Record (KSoR) provides a single, governed knowledge record that ensures consistent and reliable AI behavior across enterprises and education institutions. KSoR is an open knowledge infrastructure framework that makes governed knowledge usable by people, AI agents, and other knowledge systems without giving any one vendor ownership of the record. It ensures that the knowledge environment is governed, bounded, traceable, and testable, making it a foundational component of an AI-native knowledge platform. ### GitHub Stars: 232 ### Repository: panaversity/ksor

### Overview: fyin is an open-source alternative to Perplexity AI that allows for local running and utilizes parallelism for faster searching and answering. It features local VectorDB for quick search and configurable number of search results. ### Key features: - Run locally using either ollama or the OpenAI API. - Local VectorDB for fast search. - Very quick searching, scraping, and answering due to parallelism. - Configurable number of search results to parse. - Local scraping of websites. ### Tech stack: Rust ### Why it matters: fyin offers a local alternative to Perplexity AI, providing faster answers and an open-source solution.

Overview: This project provides a comprehensive library of prompts for generating high-quality images for e-commerce, including listing photos, model try-on, product showcases, interaction scenes, and social commerce creatives. Key features: - A library of prompts for various e-commerce use cases - Support for generating listing photos, model try-on, product showcases, interaction scenes, and social commerce creatives - Prompts are designed to enhance user engagement and product visibility Tech stack: Python, GPT-3, and various image generation libraries Why it matters: This project is essential for e-commerce businesses looking to improve their online presence and customer experience by leveraging advanced AI techniques to create visually appealing and engaging images.

### Overview: Skimmed CFG is a powerful anti-burn technique for latent diffusion models, enhancing image quality and reducing artifacts. ### Key features: - Skimmed CFG: A version of CFG that skims the conflicting influence, providing better prompt adherence and sharper images. - Skimmed CFG - Replace: Replaces the values within the negative prediction, effectively nullifying the effect of targeted filters. - Skimmed CFG - Linear Interpolation: Performs a linear interpolation between positive and negative values, highly recommended. - Skimmed CFG - Linear Interpolation Dual Scales: Offers two scales, one positive and one negative, with a more intuitive relation to the predictions. - Skimmed CFG - Difference CFG: Adjusts for values that go too far in comparison to the original model. - Skimmed CFG - Timed Flip: Enhances randomness and overall quality, especially useful with normal scales, and reacts well with SDE samplers. - Full Skim Negative: Skims some part of the conflicting influence, enhancing randomness. - Disable Flipping Filter: Provides more control, suitable for use with the Full Skim Negative toggle. - Side-Effects: Improves prompt adherence, sharpens images, reduces conflicts, and sometimes fuses fingers with too low skimming scale and low steps. - Recommanded Skim: 2-3 for maximum antiburn, 5-7 for colorful/strong style, 4 for a cruise scale. - A good negative prompt is a style negative prompt. - To use super high scales, consider cutting the negative before