Projects

### Overview: Video2Music is a cutting-edge AI-powered framework that generates suitable background music from videos using an Affective Multimodal Transformer model. This project is part of the AMAAI-Lab and is designed to streamline the process of creating background music for videos, making it a valuable tool for creators and content creators. ### Key Features: - **Video2Music Model**: Uses an Affective Multimodal Transformer model to generate music from video inputs. - **Cutting-Edge Technology**: Employing advanced Transformer architecture for music generation. - **Versatile Input**: Accepts various video inputs and generates matching music. - **Efficient Solution**: Provides a seamless and efficient way to generate background music. ### Tech Stack: - **Python**: The primary programming language used for the project. - **Hugging Face Spaces**: The repository is hosted on Hugging Face Spaces, providing a live demo and access to the model. ### Why It Matters: - **Seamless Integration**: Automatically generates music that fits the mood and style of the video, enhancing the overall aesthetic of the video. - **Efficient Workflow**: Streamlines the process of creating background music, making it easier for creators to add music to their videos. - **Versatile Use Case**: Suitable for a wide range of video content, from tutorials to music videos, enhancing the overall quality of the video. ### GitHub Stars: - **GitHub Stars**: 197 - **GitHub Description**: Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model - **Repository**: AMAAI

### Overview: This project is a skills.sh-compatible prompt engineering skill for Qwen-Image-2.1, designed to automatically rewrite and optimize text-to-image and multi-image editing prompts. It adheres to Alibaba's official specifications, ensuring high-quality and professional output. ### Key Features: - **Text-to-Image (T2I)**: Converts terse user requests into a single, cohesive, 400-500-word English description. - **Image Editing (Edit)**: Executes high-strength edits with zero attribute leakage, supports dual language protocol, and manages multi-image conditions. - **Dual-Track Multi-Modal Adaptation**: Inspects input images in Vision-enabled environments and infers visual attributes in text-only scenarios. - **Adaptive Output Presentation**: Provides Optimization Breakdown in interactive mode, and outputs in a visually appealing format. ### Tech Stack: Python ### Why It Matters: This skill is essential for developers and AI assistants looking to streamline their workflow by automatically optimizing and rewriting prompts for text-to-image and multi-image editing tasks. It ensures professional and consistent output, enhancing the efficiency and quality of image generation and editing processes. ### Repository: iamyoki/qwen-image-2.1-skill GitHub Stars: 196

Overview: This repository contains the official implementation code of the SIGIR'24 paper "When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications". Key features: - Implementation of the MOELoRA model for parameter-efficient fine-tuning in multi-task medical applications. - Includes dataset handling, model configuration, and training scripts. Tech stack: Python Why it matters: This implementation provides a practical solution for efficiently fine-tuning models in medical applications, showcasing the potential of MOELoRA in reducing computational costs while maintaining performance. Repository: liuqidong07/MOELoRA-peft GitHub stars: 197

Overview: The MongoDB Knowledge Service is a generative AI-powered platform designed to educate users about MongoDB. Key features: - Learning about MongoDB through AI-generated content - Powered by MongoDB and Atlas Vector Search Tech stack: TypeScript Why it matters: This project is a valuable resource for developers looking to enhance their knowledge of MongoDB through AI-driven tutorials and explanations. Repository: mongodb/chatbot Primary language: TypeScript GitHub stars: 197

### Overview: This book provides a comprehensive guide to using Large Language Models (LLMs) in practical AI applications. It starts with the foundational architecture of LLMs, including their training, embedding, and vector database integration. The text then delves into the development of RAG (Revised Attention Graph) applications, discussing everything from the basics of LLMs to advanced topics like multi-modal and agent development. ### Key features: - **LLM Basics**: Explains the foundational architecture of LLMs, including their training, embedding, and vector database integration. - **RAG Applications**: Details the development of RAG applications, including the process from LLMs to the final product. - **LLM Development**: Covers the entire lifecycle of LLMs, from training to deployment and optimization. - **Rama Index (LlamaIndex)**: Discusses the integration of LLMs with AI applications. - **LLM Future**: Explores the future of LLMs, including their ability to recognize and act upon information. ### Tech Stack: - Jupyter Notebook ### Why it Matters: This book is essential for developers looking to adapt to the new paradigm of LLMs, providing both theoretical knowledge and practical insights into how to effectively use and develop applications with LLMs. ### Repository: - Repository: onlybooks/llm - Primary Language: Jupyter Notebook - GitHub Stars: 197 - GitHub Description: LLM을 활용한 실전 AI 애플리케이션 개발 - Readme Excerpt

Overview: This project is an exploration of the design space for the implementation of a matrix multiplication unit for 1-bit Large Language Models, aiming to measure the impact of chip area, precision, and memory bandwidth on the performance of the systolic array and AI accelerators. Key features: - Implementation of a 1.58-bit matrix multiplication unit. - Utilization of a 130 nm eFabless process for fabrication. - Performance results based on simulations, showing a 50% increase in performance with doubled chip area. - Split into compute slices to increase compute power without changing memory bandwidth. Tech stack: Verilog Why it matters: This project aims to provide insights into the design space for 1-bit Large Language Models and evaluate the performance impact of various design parameters, contributing to the development of more efficient and compact AI accelerators. Repository: rejunity/tiny-asic-1_58bit-matrix-mul GitHub stars: 197

Overview: This project demonstrates an end-to-end pipeline for constructing knowledge graphs from news articles, analyzing them through various visualizations, and generating questions for LLMs based on the information provided from the knowledge graph. Key features: - Constructing knowledge graphs from news articles - Analyzing news articles through visualizations - Generating questions for LLMs based on the knowledge graph Tech stack: Neo4j for storing the knowledge graph and Diffbot for data integration. Why it matters: This project showcases the integration of Neo4j for knowledge graph construction and Diffbot for data enrichment, providing a practical example of how to use these tools together to build and analyze information from news articles.

### Overview: OneKE is a flexible Dockerized system for schema-guided knowledge extraction, capable of extracting information from the web and raw PDF books across multiple domains like science and news. It employs a collaborative multi-agent approach and includes a user-customizable knowledge base to enable tailored extraction. Embark on your information extraction journey with OneKE! ### Key features: - Various IE Tasks Support - Various Data Sources Support - Various LLMs Support - Various Extraction Method Support - User-Configurable Knowledge Base ### Tech stack: HTML ### Why it matters: OneKE simplifies and accelerates the process of extracting structured information from diverse sources, making it accessible and efficient for developers and researchers alike.

### Overview: This project focuses on Composed Image Retrieval (CIR) using Contrastive Learning and Task-oriented CLIP-based Features. It addresses the challenge of retrieving images that visually resemble a reference image while incorporating modifications described in a caption. ### Key Features: - Composed Image Retrieval (CIR) using Contrastive Learning and Task-oriented CLIP-based Features. - Task-oriented fine-tuning of CLIP encoders to reduce mismatch between pre-training and downstream tasks. - Training a Combiner network to integrate image-text features effectively. - Contrastive learning in both stages of training. - Baseline using bare CLIP features, showing effectiveness on FashionIQ and CIRR datasets. ### Tech Stack: Python ### Why it Matters: This project demonstrates the efficacy of using large-scale vision and language pretrained models (VLP) in various tasks, specifically in Composed Image Retrieval. The use of CLIP and task-oriented fine-tuning improves the performance of the retrieval system compared to more complex state-of-the-art approaches on popular datasets like FashionIQ and CIRR.

Overview: DiffPose is a state-of-the-art 3D pose estimation model that achieves high reliability and accuracy, making it a valuable contribution to the field of computer vision. Key features: - DiffPose is a reliable and accurate 3D pose estimation model. - It utilizes advanced diffusion processes and GMM-based training strategies. - The code is built on top of DDIM, a popular diffusion model framework. Tech stack: Python, PyTorch, CUDA 11.0 Why it matters: DiffPose offers improved reliability and accuracy in 3D pose estimation, making it a significant advancement in the field of computer vision.

### Overview: Surfkit is a toolkit designed for developers to build and share AI agents that operate on various devices. It simplifies the process of creating, managing, and deploying AI agents, making it accessible to developers of all skill levels. ### Key Features: - **Build Multimodal Agents**: Surfkit allows developers to create agents that can interact with multiple types of data, enhancing their capabilities. - **Share Agents**: Agents can be shared with the community, enabling collaboration and sharing of AI solutions. - **Run Locally or in the Cloud**: Agents can be run either locally on a machine or in the cloud, providing flexibility and scalability. - **Manage Agent Tasks**: Developers can manage and track agent tasks at scale, ensuring efficient execution and monitoring. - **Track Actions**: Surveys the actions of agents, providing insights into their performance and behavior. ### Tech Stack: Python ### Why it Matters: Surfkit simplifies the development and deployment of AI agents, making it easier for developers to create and share AI solutions without extensive technical knowledge. It streamlines the process, making it accessible to a broader audience. ### Repository: agentsea/surfkit ### GitHub Stars: 196 ### Installation: `pip install surfkit` ### Quickstart: ```python from surfkit import solve task = solve( "Search for the most common variety of french duck", agent_type="pbarker/SurfPizza", device_type="desktop", ) task.wait_for_done() result = task.result ``` ### Demo: ![](https://

### Overview: Parts2Whole is a unified reference framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance. It utilizes a semantic-aware appearance encoder and a shared self-attention mechanism to support multi-image conditioned generation. The framework enhances the vanilla attention mechanism by incorporating mask information from reference human images, enabling precise selection of any part. ### Key Features: - **Inference Code and Pretrained Models**: Available for use. - **Evaluation Code**: Available for evaluation. - **Training Code**: Available for training. - **Training Data**: Available for training. - **New Model Based on Stable Diffusion 2-1**: Not yet available. ### Tech Stack: Python ### Why It Matters: Parts2Whole provides a unified framework for generating customized human images from multiple reference images, offering a more controlled and controllable approach to image generation compared to traditional methods. This framework is particularly useful in applications where precise control over the appearance and structure of the generated image is essential, such as in virtual reality, augmented reality, or personalization in the fashion industry. ### Repository: huanngzh/Parts2Whole GitHub Stars: 196 GitHub Description: [TIP 2025] From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation

### Overview: Flexible GraphRAG is an AI context platform designed for document processing and semantic search. It supports a variety of LLMs, ontologies, and property graph databases, making it versatile for different use cases. ### Key Features: - **Python Backend**: Built on Python, using LlamaIndex and LangChain as peer frameworks. - **LlamaIndex Default**: The default framework for each pipeline stage. - **LangChain Option**: Allows selection per stage in environment configuration. - **OpenSearch and Elasticsearch**: Integration with these databases for semantic search. - **14 Data Sources**: Supports 14 data sources with incremental auto-sync. - **15 Property Graph Databases**: Supports 15 property graph databases. - **4 RDF Triple Stores**: Supports 4 RDF triple stores. - **10 Vector Databases**: Supports 10 vector databases. - **10 Auto Sync Data Sources**: Supports 10 auto-sync data sources. - **14 Database Targets**: Targets 14 different databases. - **REST API**: Provides a RESTful API for integration. - **MCP Server**: Includes an MCP server for TypeScript frontends. - **Frontend Options**: Supports Angular, React, and Vue TypeScript frontends. - **Supports Langflow and CocoIndex**: Allows integration with Langflow and CocoIndex. ### Tech Stack: - **Python**: Backend language. - **LlamaIndex**: Default framework for each pipeline stage. - **LangChain**: Option for selection per stage. - **OpenSearch and Elasticsearch**: Integration with these databases.

### Overview: This repository is dedicated to the CVPR 2026 paper titled "First Frame is the Place to Go For Video Content Custimization". It provides a comprehensive solution for customizing video content using the first frame as a starting point. ### Key Features: - **First Frame Customization**: The primary focus is on customizing video content using the first frame as the starting point. - **ComfyUI Workflow Support**: The repository includes support for using the ComfyUI workflow with the trained LoRAs, ensuring all parameters are aligned with the inference code. - **LoRA Adapters**: Available on Huggingface, providing LoRA adapters for the FFGO model. - **Training Data Sample**: A subset of 50 training videos is provided to demonstrate the data format. ### Tech Stack: Python ### Why it Matters: This repository offers a practical solution for video content customization, leveraging the first frame as a starting point. It includes support for the ComfyUI workflow and LoRA adapters, making it a valuable resource for developers and content creators looking to enhance video content using AI techniques.

Overview: This repository provides the implementation for training an AMD-135M language model and includes resources for training on AMD GPUs using TinyLlama architecture. Key features: - Training on AMD MI250 GPUs - Based on LLaMA2 model architecture - Tokenizer compatible with LLaMA2 - Docker image available for training - Preprocessed datasets for pretraining and validation - Pretraining scripts for 4 nodes - Code finetuning script - Evaluation results on popular NLP benchmarks Tech stack: Python Why it matters: This model offers a high-performance training solution for language models on AMD GPUs, making it suitable for applications requiring fast inference and low power consumption.

Overview: MirageStock is an open-source stock asset library for images, audio, video, and 3D assets, designed for developers and creators to easily integrate high-quality assets into their projects. Key features: - High-quality images, audio, video, and 3D assets - Open-source and free to use - Easy integration into projects - Regularly updated with new assets Tech stack: Python Why it matters: MirageStock provides a valuable resource for developers and creators, offering a wide range of assets for their projects, making it an essential tool in the development of applications and projects that require visual assets. Repository: MirageML/MirageStock Primary language: Python GitHub stars: 195

Overview: The `cjkfold` project is a tool designed to fold look-alike CJK glyphs in text extracted from PDFs, ensuring accurate character matching and comparison downstream. Key features: - Automatically folds look-alike CJK glyphs to their correct characters. - Supports Kangxi radicals and CJK radical supplements. - Uses Pure Python 3.9+ with zero runtime dependencies. - Fixes issues with keyword searches, vector retrievers, and deduplication. Tech stack: Pure Python 3.9+. Why it matters: Ensuring accurate character matching and comparison is crucial for applications that rely on text extraction from PDFs, such as document search, knowledge retrieval, and natural language processing tasks. Repository: `cdcailf/cjkfold` GitHub stars: 189 GitHub description: Fold look-alike CJK glyphs (Kangxi radicals, CJK radical supplements) so text extracted from PDFs matches again.

Overview: Auffusion is an official codes and models repository for the paper "Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation". This project aims to create realistic audio outputs from textual prompts using a latent diffusion model, demonstrating superior performance compared to previous TTA approaches. Key features: - Generating realistic audio from textual prompts - Adapted from T2I model frameworks for TTA task - Effective use of generative strengths and precise cross-modal alignment - Pre-trained checkpoints and inference code available for research community Tech stack: Jupyter Notebook Why it matters: The project showcases advancements in text-to-audio generation, providing a practical solution for applications requiring high-quality audio synthesis from textual descriptions, potentially impacting fields such as audio synthesis, speech synthesis, and more. Repository: happylittlecat2333/Auffusion GitHub stars: 195

### Overview: RapidLLM is an LLM inference framework that supports continuous batching, tensor parallelism, data parallelism, and quantized weights. It includes pluggable Triton/CUDA kernels and is designed for efficient inference with acceleration features such as Flash Attention3, W8A16, W4A16, and more. The framework is built on Python 3.13+ and supports various languages including English and Chinese. ### Key features: - Continuous Batching - Tensor Parallelism - Data Parallelism - Quantized Weights - Triton/CUDA Kernel Support - Flash Attention3 - W8A16 (FP8/INT8) - W4A16 (AWQ/GPTQ) - SmoothQuant W8A8 - FP8 KV Cache (2×) - NVFP4 Weight-Only - FP8 W8A8 Fused MoE - TP + CUDA Graph - DP + CUDA Graph - Kernel Autotune - Decode Context Parallelism - Data Parallelism Attention - CPU parallelism - CUDA Graph - Quantization (INT8/FP8/INT4) - Communication overlap - Tile-Signaling - Kernel Autotune - GPU configuration search - Persistent per-device cache ### Tech stack: Python 3.13+ ### Why it matters: RapidLLM is designed for efficient inference and acceleration, making it suitable for various applications requiring high performance and low latency. The framework supports continuous batching, tensor

Overview: This repository contains the official code repository for the paper "Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations" (ICML 2022). It provides a novel score-based generative model for graphs that can generate node features and adjacency simultaneously via the system of SDEs. Key features: - Implements the Graph Diffusion via the System of Stochastic Differential Equations (GDSS) model. - Provides preprocessed datasets for both generic graph generation and molecular graph generation. - Includes dependencies for Python 3.7.0 and Pytorch 1.10.1. - Offers command-line tools for generating, configuring, and training the model, as well as evaluating and generating graphs. Tech stack: Python 3.7.0 and Pytorch 1.10.1. Why it matters: The GDSS model addresses the limitations of previous generative methods by introducing a diffusion process for graphs that can generate node features and adjacency simultaneously, making it a significant advancement in graph generative modeling.