Projects

Overview: This project proposes a novel approach to convert speech audio into a photo-realistic speaking video, focusing on realistic and expressive body dynamics. It utilizes a combination of 3D skeleton movements and conditional generative adversarial networks to achieve this. Key features: - Realistic and expressive body dynamics - 3D skeleton movements from speech audio - Conditional generative adversarial network for video synthesis - Part attention mechanisms for detailed parts of the body Tech stack: Python, PyTorch, TensorFlow, and other deep learning frameworks. Why it matters: This project addresses the challenge of converting speech to video, focusing on realistic and expressive body dynamics, which is significant in fields such as virtual reality, entertainment, and education.

Overview: DeepFish is a visual AI workflow composer platform that allows users to combine models from Fal.ai and Replicate into sophisticated pipelines through a node-based interface. It democratizes access to powerful generative AI models. Key features: - **Visual Composer**: Drag-and-drop node editor to build complex AI pipelines. - **Multi-Modal Support**: Support for Text, Image, Audio, Video, and 3D generation. - **Model Integrations**: First-class support for Fal.ai and Replicate models. - **Workflow Library**: Import, share, and fork community workflows. - **Credit System**: Built-in credit management and Stripe integration for billing. Tech stack: TypeScript, Next.js 15, Zustand + React Query, tRPC, PostgreSQL + Drizzle ORM, Tailwind CSS, ShadCN UI, Clerk, Inngest, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM, PostgreSQL, Drizzle ORM,

Overview: Kaldi-Serve is a server framework for the Kaldi ASR toolkit, designed for easy deployment and optimal performance. Key features: - Real-time streaming audio recognition - Thread-safe concurrent Decoder queue - RNNLM lattice rescoring - N-best alternatives with AM/LM costs, word-level timings and confidence scores - Easy extensibility for custom applications Tech stack: C++ Why it matters: Kaldi-Serve simplifies the deployment of Kaldi ASR tools, making it more accessible and efficient for developers and researchers.

Overview: This repository contains the code for the NeurIPS 2024 paper SF-V: Single Forward Video Generation Model. It presents a novel method for generating high-quality videos by performing a single forward pass during inference. Key features: - Single Forward Video Generation Model - Competitive generation quality with reduced computational overhead - Leverages adversarial training to fine-tune pre-trained video diffusion models Tech stack: Python, PyTorch, and other standard machine learning libraries. Why it matters: This approach offers significant improvements in real-time video synthesis and editing, making it a valuable addition to the field of video generation.

### Overview: This repository serves as a curated collection of papers and code in the field of artificial intelligence, focusing on various aspects of natural language processing (NLP), computer vision (CV), multimodal learning, reinforcement learning, and transformer architectures. ### Key Features: - **NLP**: Papers and code related to Word2Vec, Sequence-to-Sequence (Seq2Seq), Pretraining, Large Language Models (LLM), LLM Applications, LLM Techniques, and Chinese Models. - **CV**: Papers and code related to Computer Vision Applications, Contrastive Learning, Foundation Models, Generative Models (GAN and VAE), Image Editing, Object Detection, Semantic Segmentation, and Video Processing. - **Multimodal**: Papers and code related to Audio, BLIP, CLIP, Diffusion Models, Multimodal LLM, Text-to-Image, Text-to-Video, and Multimodal Survey. - **Reinforcement Learning**: Papers and code related to Reinforcement Learning. - **GNN**: Papers and code related to Graph Neural Networks. - **Transformer Architecture**: Papers and code related to Transformer Architectures. ### Tech Stack: The repository uses Markdown for documentation and GitHub for hosting. ### Why It Matters: This repository provides a comprehensive collection of resources for researchers and practitioners in the field of artificial intelligence, offering insights into the latest advancements and techniques in various subfields of AI.

### Overview: Speech Core is a comprehensive C++17-based speech processing framework designed for on-device applications. It includes voice activity detection, real-time and batch speech-to-text, speaker diarization, text-to-speech, and a voice-agent pipeline. The framework is optimized for Linux, Windows, and Android, ensuring all components run locally without requiring cloud services or Python dependencies. ### Key Features: - **Voice Activity Detection (VAD):** Identifies active speech in audio streams. - **Real-Time Speech-to-Text (STT):** Converts speech to text in real-time. - **Speaker Diarization:** Tracks individual speakers in audio recordings. - **Text-to-Speech (TTS):** Converts text to speech. - **Voice-Agent Pipeline:** Connects all components for a seamless user experience. ### Tech Stack: - **Language:** C++ - **Frameworks/Libraries:** ONNX Runtime, LiteRT, ONNX CLI tools, native libraries for Linux, Windows, and Android. ### Why It Matters: - **Local Processing:** Pure C++ implementation ensures minimal network and platform audio dependencies. - **Versatile Backend:** Supports ONNX Runtime, LiteRT, both, or custom implementation, enabling flexibility in model choices. - **Real-Time Streaming:** Optimized for live agents, including VAD-driven turns, eager STT, and streaming TTS. - **Model-agnostic Core:** The core owns turn detection, audio utilities, conversation state, and tool calls, making it easy to integrate with various applications. ### Tech Stack: - **Language:** C

### Overview: 这是一个面向AI绘画、角色设计和漫剧分镜的五合一中文参考图库，包含服装（写实与动画两版）、脸型、姿势线稿与深度图、二次元条目。图库支持中文检索、多视角浏览、图片复制与下载，保留原始编号、双语索引及素材出处。页面使用原生HTML、CSS、JavaScript，配套Python数据维护脚本与GitHub Pages自动部署，适用于静态站点学习与二次开发。 ### Key features: - **服装库**：收录1,407套写实版和1,320套动画版服装参考，适合角色服装设计和AI绘画。 - **脸型库**：包含1,499张脸型参考，适用于AI创作和角色设计。 - **姿势库**：包含288组线稿，用于人体动作教学和绘画练习。 - **深度库**：包含1,439张深度图，帮助理解姿态和空间遮挡。 - **二次元库**：包含843条目，涵盖发型、眼睛、衣装和姿势，适用于二次元创作。 ### Tech stack: HTML, CSS, JavaScript, Python, GitHub Pages ### Why it matters: 这个图库为AI绘画、角色设计和漫剧分镜提供了丰富的参考资源，帮助创作者快速获取所需信息，提高创作效率。保留原始来源和使用条件，确保素材的合法性和使用规范，促进知识的可持续利用。

Overview: This project aims to create a voice conversion model that can translate any voice into any language, expanding upon the initial voice conversion project by Kyubyong Park and Dabi Ahn. Key features: - Cross-lingual Voice Conversion - Training and Conversion of speech using Triphone PPGs and Griffin-Lim algorithm - Training datasets include TIMIT and CMU ARCTIC SLT - Conversion sample files available for comparison with original speech Tech stack: Python Why it matters: This project offers a valuable tool for individuals interested in creating multilingual voices, potentially aiding in language learning or communication with people from different linguistic backgrounds. Repository: Kyubyong/cross_vc Primary language: Python GitHub stars: 98 GitHub description: Cross-lingual Voice Conversion

### Overview: This repository contains benchmark results and performance data for the Intel Arc Pro B70 GPU (Xe2 / Battlemage), focusing on LLM inference, video generation, and dual-GPU scaling. The data is collected from real runs on a workstation, providing insights into the GPU's capabilities and performance metrics. ### Key Features: - **MoE (Multi-Headed Embedding) Performance:** Qwen 3.6-35B-A3B achieves 54.7 t/s generation and 615 t/s prompt processing on a single card, with an average power consumption of 114 W. - **Dual-GPU Performance:** DeepSeek-R1-Distill-Llama-70B Q4_K_M runs at 11.5 t/s on dual B70 configurations, consuming 79 W on average. - **Biggest Model Run:** Qwen3-Coder-Next 80B-A3B at 43.4 t/s on dual B70, with an average power consumption of 79 W. ### Tech Stack: The repository utilizes the Intel Arc Pro B70 GPU with the Xe2 architecture, featuring 32 GB GDDR6 ECC, and supports both single-card and dual-card configurations. ### Why It Matters: This repository provides valuable insights into the performance and capabilities of the Intel Arc Pro B70 GPU, particularly in the context of large language models (LLMs) and video generation. The data collected is crucial for developers and researchers looking to

### Overview: CapHuman: Capture Your Moments in Parallel Universes This repository contains the source code and checkpoints for the paper *CapHuman: Capture Your Moments in Parallel Universes*. The project aims to address a novel human-centric image synthesis task, where given a single reference facial photograph, the model is expected to generate diverse images with varying head positions, poses, and facial expressions in different contexts. CapHuman leverages a generative model that excels in visual and semantic understanding, identity preservation, and flexible head control. The model is based on pre-trained text-to-image diffusion models and introduces a 3D facial prior to achieve high-fidelity portraits with rich content and varied head renditions. ### Key Features: - **Generative Model**: CapHuman is designed to synthesize images with diverse head positions, poses, and facial expressions. - **Identity Preservation**: The model maintains identity features across different individuals without requiring extensive tuning. - **3D Control**: CapHuman enables flexible and 3D-consistent control over the human head. - **High-Quality Output**: The model produces well-identity-preserved, photo-realistic, and high-fidelity portraits with rich content and varied head renditions. ### Tech Stack: - **Python**: The primary programming language used for the implementation. - **PyTorch3D**: A library for 3D computer graphics and image synthesis, used in the model's architecture. ### Why It Matters: - **Versatile Synthesis**: CapHuman excels in generating diverse and high-quality images with varying head positions and expressions, making it suitable

### Overview: SlotDiffusion is an unsupervised object-centric learning framework that leverages diffusion models to achieve state-of-the-art performance in object segmentation and image/video reconstruction across various datasets. ### Key Features: - **SlotAttention**: A novel object-centric attention mechanism. - **SAVi**: A variant of SlotAttention that enhances performance. - **SLATE**: A method for generating compositional images. - **STEVE**: A model for generating videos. - **SlotDiffusion**: The main unsupervised object-centric learning model. - **Unsupervised Object Segmentation**: Achieves high accuracy on various datasets. - **Image/Video Reconstruction**: Performs well on both images and videos. - **Compositional Generation**: Generates complex images and videos. - **Real-World Applications**: Works on real-world datasets including PASCAL VOC and COCO. ### Tech Stack: PyTorch ### Why It Matters: SlotDiffusion represents a significant advancement in unsupervised object-centric learning, offering state-of-the-art performance across multiple applications.

Overview: This project provides a Flask API for transcribing speech from audio files and generating speech from text using the Tacotron2 and HiFiGAN models. Key features: - Transcription of speech from audio files using Whisper - Generation of speech from text using Tacotron2 and HiFiGAN models Tech stack: Python Why it matters: This project simplifies the process of integrating speech recognition and synthesis capabilities into applications, making it easier for developers to create voice-enabled interfaces and applications.

Overview: This repository contains the code for training and inference with *ShEPhERD*, a generative model that learns the joint distribution over 3D molecular structures, shapes, electrostatics, and pharmacophores for bioisosteric drug design. Key features: - Training and inference code for *ShEPhERD* - Includes dependencies from [shepherd-score](https://github.com/coleygroup/shepherd-score) - Model checkpoints and training data available for older versions Tech stack: Python Why it matters: *ShEPhERD* is a significant contribution to the field of bioisosteric drug design, offering a comprehensive approach to generating new molecules with specific 3D interaction profiles.

Overview: HollowDB Vector is a decentralized vector database designed for building vector search applications. It leverages the Hierarchical Navigable Small Worlds (HNSW) index over HollowDB, providing efficient and scalable search capabilities. Key features: - Efficient vector storage and retrieval - Decentralized architecture for enhanced security and privacy - Support for various data types and vector representations - Integration with HollowDB for seamless data management Tech stack: TypeScript Why it matters: HollowDB Vector simplifies the development of vector search applications by providing a robust and scalable solution that can be easily integrated into existing projects. Its decentralized nature ensures enhanced security and privacy, making it suitable for applications requiring high levels of data confidentiality. Repository: firstbatchxyz/hollowdb-vector Primary language: TypeScript GitHub stars: 98 GitHub description: A decentralized vector database for building vector search applications README excerpt: A detailed guide on how to install, use, and deploy the HollowDB Vector package, including examples of inserting, fetching, and querying vectors.

### Overview: OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer is a state-of-the-art algorithm that addresses the challenges of open-vocabulary multiple object tracking. It introduces a novel perspective, enhancing stability and scalability, making it suitable for real-time applications. ### Key Features: - **End-to-End Architecture**: Implements a unified approach for both tracking and classification, reducing the need for separate models. - **Category Information Propagation (CIP)**: Enhances tracking stability by propagating category information across frames. - **Attention Isolation**: Ensures that tracking and classification operate independently, maintaining open-vocabulary perception and tracking. - **Dual-Branch Decoder**: Empowers the model with strong open-vocabulary perception and multimodal interaction capabilities, eliminating the need for preprocessing. ### Tech Stack: - **Python**: The primary programming language used for implementation and development. - **PyTorch**: Framework used for building and training the model. - **CLIP**: Used for category information propagation and classification tasks. ### Why It Matters: OVTR addresses the limitations of existing methods by providing a more efficient and scalable solution for open-vocabulary multiple object tracking. It offers faster inference speeds and improved stability, making it a significant advancement in the field of computer vision. ### Repository: - **Repository**: jinyanglii/OVTR - **Stars**: 98 - **Description**: [ICLR 2025] 🏄 OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer ### Installation: - **Virtual Environment

Overview: Speech-to-text system based on wav2letter architecture, designed for transfer learning. Key features: - Supports CTC loss, KenLM, and greedy decoding. - Compatible with transfer learning between different languages. - Training available for English with LibriSpeech corpus, and German with manual data download. - Minimal requirements for Python 3.4+ and TensorFlow. Tech stack: Python, TensorFlow. Why it matters: This system is designed for efficient speech recognition, leveraging transfer learning to improve performance across various languages and dialects.

Overview: Aimybox Android SDK is a voice assistant SDK written in Kotlin, providing a comprehensive solution for integrating intelligent voice assistants into Android applications. It includes modular components, customizable speech-to-text and text-to-speech implementations, and a wide range of NLU providers. Key features: - Ready-to-use UI components for fast app development - Independent from speech-to-text and text-to-speech vendors - Supports various speech-to-text and text-to-speech libraries like Android platform speechkit, Kaldi, Pocketsphinx, Yandex Cloud speechkit, Google Cloud speechkit, Houndify, and Snowboy wake word trigger - Compatible with Aimylogic, Dialogflow, and Rasa NLU providers - Fully customizable, allowing integration with any speech-to-text, text-to-speech, and NLU services - Open source under Apache 2.0, written in pure Kotlin - Embeddable into any Android application or device - Supports any local device services and local networks Tech stack: Kotlin Why it matters: Aimybox Android SDK simplifies the integration of intelligent voice assistants into Android applications, providing a robust, customizable, and open-source solution for developers looking to enhance their applications with voice-based functionalities.

Overview: OasisDB is a lightweight vector database designed for simplicity and ease of use, providing efficient vector similarity search through various index types and a minimal resource footprint. Key features: - Multiple vector index types including HNSW, IVFFLAT, IVFPQ, and Flat - Ultra-lightweight architecture with standalone deployment and zero internal network communication - Embedding service integration for seamless vector generation - Developer-friendly RESTful API and multi-language SDKs Tech stack: Go Why it matters: OasisDB simplifies vector search by offering a lightweight and easy-to-use database, making it perfect for beginners and developers looking to quickly integrate vector search into their applications.

### Overview: Zorro is an implementation of the Masked Multimodal Transformer (Zorro) in Pytorch, designed by Deepmind. This project aims to showcase the effectiveness of a special masking strategy within a transformer for achieving state-of-the-art results on multimodal benchmarks. ### Key features: - Zorro is a masked multimodal transformer implementation. - It includes support for audio and video inputs. - The model can be configured with various parameters such as dimensions, depth, attention heads, and more. ### Tech stack: Pytorch ### Why it matters: Zorro demonstrates the potential of masked multimodal transformers, showing how a specific masking strategy can significantly improve performance on multimodal tasks, as claimed in the research paper [1]. [1] Adrià Recasens, Jason Lin, Joao Carreira, Drew Jaegle, Luyu Wang, Jean-Baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, Andrew Zisserman. Zorro: the masked multimodal transformer. 2023. Repository: lucidrains/zorro-pytorch Primary language: Python GitHub stars: 98

Overview: BlueTTS is a fast, multilingual text-to-speech (TTS) model that leverages ONNX Runtime for high-performance inference. It supports Hebrew, English, Spanish, Italian, and German, making it versatile for various language needs. Key features: - Multilingual support across Hebrew, English, Spanish, Italian, and German - Uses ONNX Runtime for efficient and fast inference - Supports voice conversion using the `codec_encoder` and `style_encoder` graphs - Provides a reference encoder for conditioning and guidance - Ships pre-trained models and voice JSONs in the `voices/` directory Tech stack: Python Why it matters: BlueTTS is designed to be fast and efficient, making it suitable for real-time applications and high-performance speech synthesis tasks. Repository: maxmelichov/BlueTTS GitHub stars: 97 GitHub description: Fastest Open Source TTS Model README excerpt: <p align="center"> <img src="assets/logo.png" alt="BlueTTS" width="280"> </p> <p align="center"> <b>Multilingual text-to-speech on ONNX Runtime</b><br> Hebrew &middot; English &middot; Spanish &middot; Italian &middot; German </p> <p align="center"> <a href="https://huggingface.co/spaces/notmax123/BlueV2"><img src="https://img.shields.io/badge/%F0%9F%A4%