Projects

Overview: LN3Diff is a high-quality 3D object mesh generation framework that achieves real-time 3D object mesh creation from text within 8 seconds. Key features: - Scalable Latent Neural Fields Diffusion - Real-time 3D object mesh generation - Achieves high-quality results within seconds Tech stack: Python Why it matters: LN3Diff is a significant advancement in the field of 3D object mesh generation, offering real-time capabilities that were previously unattainable.

Overview: This repository contains code for performing various experiments with Stable Diffusion, utilizing the diffusers library. The codebase is supported by the diffusers library and includes instructions for installation, model weights, and experiment setup. Key features: - Single Inference - Visualize Diffusion - Random Walk - Interpolation - Diffevolution - Outpaint Walk Tech stack: Python Why it matters: This project provides a comprehensive toolkit for experimenting with Stable Diffusion, allowing users to explore various aspects of the model and experiment with different configurations and techniques.

Overview: The Kaldi Offline Transcriber is an offline transcription system designed for Estonian speech recognition using Kaldi, a powerful toolkit for automatic speech recognition. Key features: - Uses a better speech activity detector - Filters out non-Estonian utterances before decoding - Provides shorter subtitles and turns in TRS files - Supports Docker images with Estonian models - Updated speaker ID models - Improved robustness of speech-non-speech detection - Uses chain models for acoustic models - Faster and more accurate than previous versions Tech stack: Python Why it matters: This system is crucial for Estonian language processing and speech recognition, offering improved accuracy and efficiency in offline transcription tasks.

Overview: This repository provides a comprehensive explanation of the Transformer architecture, including the full architecture, attention mechanisms, positional embeddings, and layers inside a Transformer. Key features: - Full Transformer architecture explained - Self-attention mechanism - Multi-head attention - Positional embeddings - Feed-forward networks - Residual connections - Layer normalization Tech stack: Python Why it matters: This guide is essential for developers looking to understand how modern Large Language Models like BERT and GPT operate, providing a detailed breakdown of the Transformer architecture and its components. Repository: amitshekhariitbhu/transformers-explained GitHub stars: 227

Overview: vid2cleantxt is a Python API and command-line tool designed to transcribe speech-based video files into clean, readable text using an advanced model. This project aims to improve the efficiency of video content by allowing users to quickly access relevant information without the need to watch the entire video. Key features: - Transcription of speech-based video files into clean text - Uses a transformers-based pipeline for robust speech transcription - Supports both Python package installation and installation from source Tech stack: Python Why it matters: This tool simplifies the process of extracting valuable information from videos, making it easier for users to quickly find and understand the most relevant parts of a video without having to watch the entire thing. This is particularly useful in fields where dense or technical information is conveyed, such as in scientific presentations or educational videos. Repository: pszemraj/vid2cleantxt Primary language: Jupyter Notebook GitHub stars: 229 GitHub description: Python API & command-line tool to easily transcribe speech-based video files into clean text

### Overview: Simulatrex Engine is a powerful tool designed to enable decision-making based on simulations. Leveraging LLMs, it provides a robust platform for making informed decisions through data-driven insights. ### Key Features: - **Decision-Making Automation**: Simulatrex Engine automates the process of decision-making by analyzing simulations and providing insights. - **Data-Driven Insights**: It leverages LLMs to generate accurate and relevant data-driven insights, ensuring that decisions are based on the most up-to-date and comprehensive information. - **Integration with Discord**: Simulatrex Engine integrates with Discord, allowing developers to engage with the community and request features directly. ### Tech Stack: Python ### Why it Matters: Simulatrex Engine empowers developers to make data-driven decisions, enhancing the accuracy and reliability of their projects. By integrating LLMs and providing a seamless interface, it streamlines the decision-making process, making it easier for teams to implement and trust AI-driven solutions. Repository: simulatrex/simulatrex-engine GitHub Stars: 229 GitHub Description: Enable decision-making based on simulations README Excerpt: <div align="center"> <img src="./docs/assets/header.png" alt="Simulatrex" width="100%" /> </div> <br/> <h4 align="center"> <a href="https://discord.gg/YxdTpQxrbS"> <img src="https://img.shields.io/static/v1?label=Chat%20on&message=Discord&color

Overview: This project is an auto-transcription tool that uses the Whisper model from OpenAI to generate subtitles for video or audio content. It supports 99 languages and offers high accuracy, ease of use, and GPU acceleration for improved performance. Key features: - Automatically generates subtitles - Translates content to English - Supports 99 languages - Offers high accuracy and ease of use - Provides support for GPU acceleration and CLI mode - Includes a one-click installer - Increased time precision from 1 to 0.01 seconds - Supports YouTube integration - Preview subtitles in video - Works fine even during heavy metal live performances - Supports long files, 3-hour files have been tested - Resolves the issue of subtitle repetition - Supports batch processing Tech stack: Python Why it matters: This tool simplifies the process of generating subtitles, making it accessible and efficient for users who may not have extensive knowledge of video editing or programming. It's particularly useful for creators who want to add subtitles to their videos quickly and easily. Repository: tomchang25/whisper-auto-transcribe Primary language: Python GitHub stars: 229 GitHub description: Auto transcribe tool based on whisper README excerpt: <div id="top"></div>

### Overview: This repository is a comprehensive survey of Remote Sensing Spatio-Temporal Vision-Language Models (RS-STVLMs) for IEEE GRSM 2025. It aims to provide a detailed overview of recent advancements in RS-STVLMs, including methods, datasets, and applications. ### Key features: - **Change Captioning** - **Multitask Learning of Change Detection and Change Captioning** - **Change Question Answering** - **Text-driven Temporal Images Retrieval** - **Change Grounding** - **Text-driven Temporal Images Generation** ### Tech stack: Python ### Why it matters: This repository serves as a valuable resource for researchers, practitioners, and students interested in the field of RS-STVLMs. It provides a comprehensive understanding of the latest developments and methodologies in this area, making it an essential tool for anyone working on remote sensing applications involving temporal and spatial information. Repository: Chen-Yang-Liu/Awesome-RS-SpatioTemporal-VLMs

Overview: mPLUG-2 is a unified multi-modal foundation model that addresses the challenges of modality entanglement by introducing a modular design. This model achieves state-of-the-art or competitive results across various multi-modal tasks, including image-text and video-text understanding and generation, and uni-modal tasks such as text-only, image-only, and video-only understanding. Key features: - Modularized multi-modal paradigm - Shared universal modules for modality collaboration - Disentangled modality modules to tackle modality entanglement - Flexible selection of different modules for various tasks Tech stack: Python Why it matters: mPLUG-2 offers a flexible and effective solution for multi-modal pretraining and downstream tasks, making it a valuable addition to the field of multimodal AI. Repository: X-PLUG/mPLUG-2 GitHub stars: 228 GitHub description: mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video (ICML 2023) README excerpt: [https://arxiv.org/abs/2302.00402](https://arxiv.org/abs/2302.00402) ## Introduction We present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement. In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG

Overview: jetson-voice is a deep learning inference library for NVIDIA Jetson devices, providing Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Text-to-Speech (TTS) capabilities. It supports Python and JetPack 4.4.1 or newer, and is optimized with TensorRT for enhanced performance. Key features: - Automatic Speech Recognition (ASR) - Natural Language Processing (NLP) - Text-to-Speech (TTS) Tech stack: Python Why it matters: jetson-voice simplifies the development of ASR, NLP, and TTS applications on Jetson devices, making it easier to integrate these functionalities into edge computing environments. The library's optimized performance and support for JetPack ensure efficient deployment and reduced latency, making it suitable for real-time applications in autonomous driving, smart homes, and other edge-based solutions.

### Overview: theDAW is a free, full-featured DAW, DJ, and VJ app that runs on your own computer. It interoperates with Ableton, Reaper, Resolume, and more, allowing you to generate tracks from text prompts, edit and mix them, turn them into sheet music, sing along to them with timed lyrics, DJ them, and play them live. All processing stays on your machine, and there is no need for an account or subscription. ### Key Features: - **Stable Audio 3**: Generates tracks from text prompts. - **Magenta RT2**: Converts recorded audio into LoRA models. - **Suno API**: Integrates with various audio and visual effects. - **Chimera Track Fusion**: Combines multiple audio tracks into a single, cohesive track. - **Demucs Stems**: Separates audio stems for better mixing. - **MIDI Generate/Notate**: Converts MIDI to sheet music. - **Drawing → Music**: Translates visual elements into music. - **VST3 & .gan Plugins**: Accessible plugins for advanced effects. - **Automix & Key-Lock**: Automatic mixing and key-locking features. - **GLSL Shaders**: Customizable shaders for visual effects. - **Volumetric Video**: Adds immersive visuals. - **Quest 3 XR Interface**: Companion app for virtual reality. - **MIDI Auto-Map**: Automatically maps MIDI to the track. - **RAG Assistant**: AI-powered assistant for creative tasks. ### Tech Stack: - TypeScript ### Why

### Overview: ComfyUI-fal-API is a production-ready ComfyUI custom nodes solution that integrates 1,400+ fal.ai models, providing auto-updated image, video, audio, LLM, and VLM APIs. It features a persistent result cache, zero-I/O fal→fal chaining, and native media support. ### Key Features: - **Auto-Generated Nodes**: Covers every live public model on fal.ai, including image, video, audio, LLMs, and VLMs. - **Persistent Result Cache**: Ensures identical fal calls are served from disk, offering free and instant results across restarts. - **Spend Guard**: Prevents submitting past a session budget or below a balance floor, ensuring financial safety. - **URL Passthrough**: Allows a fal node's URL output to be used as an input for the next node, keeping uploaded media deduplicated. - **Platform Utilities**: Includes Fal Submit, Fal Collect, Fal Result by Request ID, and Fal Cost Estimator for various platform utilities. ### Tech Stack: Python ### Why it Matters: ComfyUI-fal-API simplifies integrating a vast array of fal.ai models into ComfyUI, ensuring seamless integration and efficient use of resources, while maintaining financial and performance controls.

Overview: This project is a fine-tuning model of KoGPT2, a Korean version of the GPT-2 language model, designed for fine-tuning on Korean lyrics data. It leverages pre-trained Korean data from SKT-AI and fine-tunes it for various genres, including pop, jazz, and more. The project also integrates Google Drive and Dropbbox for seamless data handling and learning. Key features: - Fine-tuning of KoGPT2 on Korean lyrics data - Integration of Google Drive and Dropbbox for data management - Support for multiple genres and song types - Sample lyrics for demonstration - Fine-tuning parameters and usage of Colab for easy execution Tech stack: Python Why it matters: This project is significant for anyone working on Korean language models, as it demonstrates the ability to fine-tune models on specific data sets and provides a practical tool for generating and analyzing Korean lyrics.

Overview: ToolEmu is an LM-based emulation framework designed to identify and assess risks associated with language model (LM)-based agents, specifically focusing on the risks of leaking private data and causing financial losses due to complex tool use. It provides a scalable and automated approach to evaluate LM agents across various tools and scenarios. Key features: - Runs specific test cases in an emulation sandbox. - Evaluates LM agents using safety and helpfulness evaluators. - Curates toolkits and test cases for scalable risk assessment. Tech stack: Python Why it matters: As LM agents become more sophisticated, identifying and mitigating risks becomes increasingly important. ToolEmu simplifies the process of testing and evaluating these agents, making it easier to develop safer and more reliable language models. Repository: ryoungj/ToolEmu GitHub stars: 227 GitHub description: [ICLR'24 Spotlight] A language model (LM)-based emulation framework for identifying the risks of LM agents with tool use README excerpt: # ToolEmu: Identifying the Risks of LM Agents with an LM-Emulated Sandbox <div align="center"> <img src="assets/figures/ToolEmu.png" width="50%"> </div> <div align="center" style="font-size: 28px;"> [<a href="https://arxiv.org/abs/2309.15817">📃 Paper</a>] &nbsp;&nbsp;&nbsp; [<a href="https://toolemu.com">🌐 Website</a

### Overview: LLaVA-MoD is an efficient framework for training small-scale Multimodal Language Models by distilling knowledge from larger models. ### Key features: - **Efficient Training**: LLaVA-MoD distills knowledge from larger models to train small-scale MLLMs. - **Sparse MoE Architecture**: Enhances computational efficiency while maintaining model expressiveness. - **Two-Stage Transfer Strategy**: Mimic Distillation and Preference Distillation for better knowledge transfer. - **Significant Performance Improvement**: Outperforms existing models across multimodal benchmarks. ### Tech stack: Python ### Why it matters: LLaVA-MoD significantly reduces the computational cost and parameter count of MLLMs, making them more accessible and efficient for various applications.

### Overview: KVzip is a query-agnostic KV cache compression and eviction method that significantly reduces memory usage and decoding latency while maintaining performance. ### Key Features: - **Query-Agnostic Compression**: KVzip optimizes the KV cache for any query, ensuring consistent performance across various inputs. - **3-4× Reduction in KV Cache Size**: Achieves substantial memory savings by compressing the KV cache. - **2× Decrease in Decoding Latency**: Reduces decoding time by 200% compared to traditional methods. - **Context-Dependent Eviction**: Automatically evicts KV pairs based on importance scores, improving efficiency. - **Context-Independent Optimization**: Enhances head-level KV compression using minimal forward passes, reducing performance impact. ### Tech Stack: - Python 3.10 - CUDA 12.1 ### Why It Matters: - **Memory Efficiency**: KVzip reduces the memory footprint of the KV cache by up to 3-4 times, making it ideal for applications with limited memory constraints. - **Latency Reduction**: The method decreases decoding latency by 200%, enhancing the overall user experience. - **Performance Consistency**: Ensures consistent performance across different queries and models, making it suitable for a wide range of applications. ### Repository: - **GitHub Repository**: [snu-mllab/KVzip](https://github.com/snu-mllab/KVzip) - **Stars**: 227 - **Description**: A NeurIPS 2025 Oral Presentation that introduces

Overview: Geometry Forcing (GF) is an innovative method that combines video diffusion models with 3D representation to generate consistent and temporally coherent videos. This approach enhances the ability of diffusion models to accurately reconstruct 3D geometry, making it a significant advancement in video modeling. Key features: - GF enhances video diffusion models by integrating geometric features from VGGT. - It generates more temporally and geometrically consistent videos compared to baseline methods. - GF learns features that enable accurate 3D reconstruction, whereas baseline methods struggle to reconstruct meaningful 3D geometry. Tech stack: Python Why it matters: GF is a crucial development that bridges the gap between video diffusion models and 3D representation, leading to more accurate and consistent video modeling. Its ability to handle complex geometric transformations makes it particularly useful in applications requiring high precision in video analysis and synthesis.

Overview: This project aims to develop a self-improving agentic RAG pipeline that evolves itself through a collaborative team of specialist agents, a multi-dimensional evaluation system, and an SOP architect agent. The healthcare domain presents unique challenges, requiring multiple possibilities to be considered while final decisions remain in human hands. Key features: - Self-improving agentic RAG pipeline - Collaborative team of specialist agents - Multi-dimensional evaluation system - SOP architect agent - Full test run of the guild graph - Custom evaluator for each parameter - Aggregate LangSmith evaluator - Managing guild configurations - Building director-level agents - Running full evolutionary loop - 5D Pareto-based analysis Tech stack: Jupyter Notebook Why it matters: This project is crucial for automating the process of optimizing RAG systems in a healthcare domain, where human decisions are essential. By automating the optimization process, it can significantly reduce the time and effort required for manual tuning, leading to more efficient and effective solutions.

Overview: WTM is a fast, lightweight Whisper transcription library built on MLX, designed for efficient speech-to-text tasks. Key features: - Supports multilingual transcription - Offers timestamps and quick transcription options - Compatible with FFmpeg for faster audio decoding Tech stack: Python Why it matters: WTM simplifies the process of using Whisper for transcription, making it faster and more lightweight, ideal for applications requiring high performance and low resource usage. Repository: JosefAlbers/wtm GitHub stars: 227 GitHub description: Blazing fast whisper turbo for ASR (speech-to-text) tasks

### Overview: Morpheus is a decentralized network designed to power smart agents, connecting LLMs, AI agents, wallets, Dapps, and smart contracts. It leverages the Morpheus token to incentivize community participation, ensuring a fair and sustainable ecosystem for the development and operation of smart agents. ### Key Features: - **Smart Agent Integration**: Morpheus connects LLMs and AI agents to wallets and smart contracts, enabling seamless communication and execution of tasks. - **Token Incentives**: The Morpheus token rewards users, developers, capital providers, and compute providers for their contributions to the network. - **Community Participation**: The Morpheus network is built on a community-driven approach, fostering collaboration and innovation. - **Scalable Infrastructure**: The network is designed to be scalable and resilient, ensuring long-term stability and growth. ### Tech Stack: - **JavaScript**: The primary programming language used to develop the Morpheus network and smart agents. ### Why It Matters: The Morpheus network represents a significant step towards democratizing access to smart agents and decentralized finance (DeFi). By leveraging the Morpheus token, the network aims to incentivize a diverse range of contributors, including users, developers, capital providers, and compute providers, to ensure a sustainable and growing ecosystem. This approach not only enhances the functionality of smart agents but also promotes decentralization and competition, contributing to the long-term scalability and stability of the network.