Projects

### Overview: LiteASR is a low-rank compression scheme designed for automatic speech recognition (ASR) models. It leverages the low-rank properties of activation values in the encoder to significantly reduce inference costs while maintaining high transcription accuracy. This method, introduced in the preprint [arXiv:2502.20583](https://arxiv.org/abs/2502.20583), achieves up to 50% compression of OpenAI's Whisper encoder size, matching Whisper medium's size with improved accuracy, establishing a new Pareto-optimal frontier. ### Key Features: - **Low-Rank Compression**: Utilizes principal component analysis (PCA) with a small calibration dataset to approximate linear transformations with chain of low-rank matrix multiplications. - **Self-Attention Optimization**: Further optimizes self-attention to work in the reduced dimension, enhancing performance without significant loss of accuracy. - **Efficient Inference**: Reduces computational intensity by up to 50%, making it suitable for real-time applications. ### Tech Stack: - **Python**: The primary programming language used for developing and implementing the compression scheme. - **Triton**: A high-performance inference engine used for optimizing the inference process on hardware accelerators. ### Why It Matters: - **Efficiency**: Establishes a new Pareto-optimal frontier in terms of efficiency and performance, offering a significant improvement in inference costs. - **Accuracy**: Maintains high transcription accuracy even with significant compression, ensuring reliable and accurate speech recognition. - **Versatility

### Overview: This repository contains code for a tutorial/review paper on RL-based fine-tuning of diffusion models for biological sequences, focusing specifically on the design of DNA (enhancers) and RNA (UTRs). ### Key Features: - Obtain raw data and preprocess. - Train conditional and unconditional diffusion models (Score-based Diffusion over Simplex). - Evaluate the performances of pre-trained diffusion models. - Implement fine-tuning with direct reward backpropagation and PPO. - Compare generated RNA sequences and DNA sequences before and after fine-tuning. ### Tech Stack: Jupyter Notebook ### Why it Matters: This code provides a practical implementation of RL-based fine-tuning of diffusion models for biological sequence design, offering insights into the optimization of biological sequences for specific applications.

Overview: Persephone is a tool designed for automatic phoneme transcription, particularly useful for low-resource language documentation. It aims to make state-of-the-art phonemic transcription accessible to linguists involved in language documentation. Key features: - Automatic phoneme transcription - Easy-to-use command-line interface - Extensible architecture Tech stack: Python/Tensorflow Why it matters: Persephone simplifies the process of phoneme transcription for low-resource languages, making it accessible to linguists and language documentation projects.

Overview: CharXiv is a dataset designed to evaluate models' performance on realistic chart understanding tasks within Multimodal Large Language Models (MLLMs). It aims to address the limitations of existing datasets by providing a more diverse and challenging benchmark. Key features: - Comprehensive evaluation of MLLMs on real-world chart understanding tasks. - Includes a wide range of charts and chart types, covering various domains such as scientific papers and financial reports. - Provides detailed leaderboard for comparison of model performance. - Supports evaluation on multiple MLLMs, including Qwen2-VL, InternVL2.5, and others. Tech stack: Python Why it matters: CharXiv offers a more realistic and challenging benchmark for evaluating the performance of MLLMs in chart understanding tasks, providing valuable insights into the current state of the art and areas for improvement.

### Overview: TextCraft is an AI-powered Microsoft Word add-in that seamlessly integrates cutting-edge AI tools, including text generation and proofreading, without requiring an internet connection. Built for offline use, it ensures privacy and a smarter, sleeker alternative to Microsoft® Copilot™️. ### Key Features: - **AI Integration**: Seamless integration of AI tools directly into Microsoft® Word. - **Offline Capabilities**: Works perfectly without an internet connection. - **Customization**: Tailor settings for your needs via environment variables. ### Tech Stack: C# ### Why It Matters: TextCraft offers a smarter, sleeker alternative to Microsoft® Copilot™️, ensuring privacy and a more efficient workflow without the need for an internet connection. It is designed to work effortlessly with OpenAI-compatible API endpoints, making it fully compatible with all display languages supported by Office, ensuring a seamless experience across different language preferences.

### Overview: 立正的公开资料库是一个基于已发表原文、媒体内容和公共知识的源地数据集，旨在为用户提供可检索、可引用、可继续开发的数据。立正的数据结构包括文章、课程、书籍、视频字幕、对谈原稿、来源目录、阅读地图、框架、判断卡、校验、导入和目录生成等，为用户提供全面的资料检索和引用支持。 ### Key features: - 文章、课程和书籍文字、视频字幕、对谈原稿、来源目录、阅读地图、框架、判断卡、校验、导入和目录生成。 - 人物、来源和参与关系的结构化事实，按人物和来源发现资料。 - 个人网站与问问立正消费同一份版本。 - 《真本事》的公开资料。 - 《科学能解释的东西很少》的公开资料。 - 《项目复盘》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 - 《立正》的公开资料。 -

### Overview: Voice typing with whisper.cpp offers a state-of-the-art, private, offline voice typing solution for Linux and Windows. This project uses a tiny bash script to enable seamless voice typing in terminals, eliminating the need for graphical operating systems. The script is free, open-source, and promotes peace through efficient communication. ### Key Features: - **Privacy-Focused**: Uses Whisper.cpp for offline speech recognition, ensuring all data stays under your control. - **Organic Software**: Free, open-source, and unrestrictive license. - **Peaceful Resolution**: Promotes peace by expediting communications and resolving conflicts. - **Hands-Free**: Utilizes `sox` for rudimentary sound-level detection and `ydotool` for typing into active windows. - **Low Memory Requirements**: Resources are freed between each spoken interaction. - **Windows Compatibility**: Works with GUI X, Wayland, and other environments, but does not require X. - **Linux Users**: Scripts are available for customization and are around 50-100 lines of code. - **Voice Keyboard Integration**: Compatible with Windows 11 voice keyboard, but users can switch to the Linux version with the Windows-H key. ### Tech Stack: - Shell Script ### Why It Matters: - **Privacy and Control**: Ensures all data is kept private and under your control. - **Efficient Communication**: Promotes peaceful resolutions through quick and efficient communication. - **Resource-Efficient**: Uses minimal resources and frees them between each interaction. - **Versatile Compatibility**: Works seamlessly across Linux and Windows

### Overview: Embodied AI systems, which integrate physical interaction with artificial intelligence, present significant challenges in safety. This survey aims to provide a comprehensive overview of the risks, attacks, and defenses in embodied AI, covering 500+ key works across perception, cognition, planning, interaction, and agentic systems. It serves as a unified safety framework, offering valuable insights and practical solutions for researchers and practitioners. ### Key Features: - **Unified Safety Framework**: A comprehensive survey and unified safety framework for embodied AI. - **500+ Papers**: Includes 500+ key works across perception, cognition, planning, interaction, and agentic systems. - **ArXiv Publication**: [arXiv:2605.02900](https://arxiv.org/abs/2605.02900) - **Website**: [Project Page](https://x-zheng16.github.io/Awesome-Embodied-AI-Safety/) - **Latest Screened Papers**: [Screened Papers](https://x-zheng16.github.io/Awesome-Embodied-AI-Safety/screened.html) - **Maintained**: Yes, with active maintenance and contributions. - **GitHub Stars**: 160 ### Tech Stack: Python ### Why it Matters: Embodied AI systems are increasingly integrated into various applications, making safety an essential concern. This survey provides a critical analysis of risks, attacks, and defenses, offering practical solutions and a unified framework for researchers and practitioners to address

Overview: This repository contains code for the paper "Parallel Sampling of Diffusion Models" by Andy Shih et al., presented at NeurIPS 2023 Spotlight. It implements a parallel diffusion generative model sampler using PyTorch. Key features: - Parallel sampling of diffusion models - Utilizes PyTorch for efficient computation - Implements a scheduler for parallel execution on multiple GPUs Tech stack: Python Why it matters: This implementation accelerates the sampling process of diffusion models on multiple GPUs, providing a significant speedup for large-scale applications, making it particularly useful for research and production environments where efficiency is crucial. Repository: AndyShih12/paradigms Primary language: Python GitHub stars: 159

### Overview: HybridClaw is an enterprise-ready self-hosted AI assistant runtime designed for practical business use. It offers compact models for Salesforce, HubSpot, Google Analytics, and more, while also treating agents as networked coworkers and enabling secure, approved channel access. ### Key Features: - **Compact Models**: Implement tested business skills with structured rails for enterprise use. - **Secure Credentials**: Secrets live in the runtime store, resolved by SecretRef-backed tools. - **Approval and Channels**: Agents can communicate securely and route work through approved channels. - **Structured Setup**: A guided onboarding process with setup links, welcome emails, and structured audit events. - **HybridAI Cloud**: Launch HybridClaw on HybridAI Cloud in minutes. ### Tech Stack: TypeScript ### Why It Matters: HybridClaw provides a compact yet powerful AI assistant runtime that integrates seamlessly with various enterprise tools, ensuring business workflows remain efficient and secure. ### Repository: HybridAIOne/hybridclaw ### GitHub Stars: 159 ### Business Skills That Work on Smaller Models: HybridClaw's compact models can operate a variety of enterprise tools like Salesforce, HubSpot, Google Analytics, and more, making them highly practical for businesses. ### Tech Stack: TypeScript ### Why HybridClaw: HybridClaw simplifies the integration of enterprise tools with compact models, secure credentials, and approved channels, ensuring business workflows remain efficient and secure.

Overview: This is the official implementation of the CVPR 2024 paper "Prompt Highlighter: Interactive Control for Multi-Modal LLMs". It allows users to control the generation of multi-modal language models by highlighting their prompts, making it compatible with both LLMs and VLMs. Key features: - Interactive control over multi-modal language models - Compatible with LLMs and VLMs - Provides a training-free inference pipeline - Compatible with both LLaVA and Vicuna (LLaMA-based LLMs) - Compatible with InstructBLIP and InternLM-VLComposer Tech stack: Python Why it matters: This tool enables users to interactively control the generation of multi-modal language models, making it easier to fine-tune and customize the output based on specific needs.

### Overview: Zero to AI Mastery This repository offers a comprehensive, step-by-step guide to learning AI and machine learning using Python, deep learning, large language models (LLMs), retrieval-augmented generation (RAG), AI agents, prompt engineering, fine-tuning, and MLOps. The 950+ Jupyter notebooks provide a practical, hands-on approach to mastering the fundamentals of AI and machine learning. ### Key Features: - **Python Fundamentals**: Essential Python programming skills. - **Deep Learning**: Implementing neural networks and architectures. - **LLMs and RAG**: Exploring language models and retrieval-augmented generation. - **AI Agents**: Building conversational AI agents. - **Prompt Engineering**: Crafting effective prompts for AI models. - **Fine-Tuning**: Training models on specific datasets. - **MLOps**: Deploying AI models in production environments. ### Tech Stack: Jupyter Notebook The primary technology used is Jupyter Notebook, which allows for interactive coding and data analysis. ### Why It Matters: - **Practical Learning**: Hands-on experience through Jupyter notebooks. - **Comprehensive Coverage**: From basics to advanced topics. - **Community Support**: Community-driven content and discussions. - **Career Preparation**: Ideal for beginners and professionals looking to enhance their AI skills. ### Repository: PavanMudigonda/zero-to-ai - **GitHub Stars**: 158 - **GitHub Description**: A free AI/ML course with 950+ Jupyter notebooks — Python, deep learning, LLM

### Overview: ComfyUI-sol-attn is a project that introduces Sol-Attn, a specialized attention mechanism for ComfyUI, leveraging NVIDIA's Sol-Attn Triton kernel. This project aims to enhance the efficiency and performance of ComfyUI by reducing memory usage and peak VRAM consumption, specifically targeting the H3 node. ### Key Features: - **Memory-Efficient Attention**: Sol-Attn uses zero-copy MiniMax H3 nodes to achieve significant memory savings. - **Scheduled Sparsity**: Sol-Attn supports scheduled sparsity, allowing for efficient memory usage. - **Graph Preview**: Sol-Attn includes a graph preview feature for better understanding of the attention mechanism. - **Peak VRAM Reduction**: Sol-Attn achieves up to 1.44× performance improvements over SageAttention and reduces MLP peak VRAM by 37%. ### Tech Stack: - **Python**: The primary programming language used for this project. ### Why It Matters: - Sol-Attn offers a significant performance boost compared to SageAttention, achieving up to 1.44× performance improvements. - It reduces peak VRAM usage by 37%, making it more efficient for high-performance applications. ### Repository: - **Repository URL**: [Saganaki22/ComfyUI-sol-attn](https://github.com/Saganaki22/ComfyUI-sol-attn) - **Stars**: 159 - **Description**: A project that integrates Sol-Attn with ComfyUI, focusing on NVIDIA's

### Overview: FrozenBiLM is a zero-shot video question answering model that leverages a frozen bidirectional language model to perform exceptionally well in settings with no manual annotation (zero-shot) or limited training data (few-shot). This innovative approach excels in scenarios requiring minimal human intervention or data, making it highly valuable for applications where data is scarce or expensive to obtain. ### Key Features: - **Zero-Shot Performance**: Performs competitively even when trained on standard datasets without any manual annotation. - **Few-Shot Learning**: Succeeds in tasks with limited training data, demonstrating strong generalization capabilities. - **Cross-Modal Training**: Integrates video and language modalities effectively, enhancing understanding and answering accuracy. ### Tech Stack: The project utilizes Python and the PyTorch framework for implementing the model and training scripts. It includes environment setup, data preprocessing, feature extraction, and pre-trained checkpoints for various downstream tasks such as VideoQA, downstream fully-supervised, few-shot, and zero-shot VideoQA, and video QA demos. ### Why It Matters: FrozenBiLM offers a significant improvement in video question answering by providing a solution that is robust and adaptable to a wide range of scenarios, including those with minimal data or annotation. This makes it particularly useful in applications where obtaining high-quality training data is challenging or expensive, such as in remote or under-resourced settings.

Overview: Aeiva is a general AI agent framework designed to be adaptable to various tasks and environments, focusing on human-centered growth and continuous improvement. Key features: - Terminal chat for direct local use - Web dialogue UI for multimodal interaction - LifeRPG dashboard for structured self-modeling and growth tracking - Slack and WhatsApp gateways for real-world messaging channels Tech stack: Python Why it matters: Aeiva aims to help people experience deep, high-agency growth in the RPG of life by providing a human-centered, lifelong AI partner that augments human potential across different moments of life. GitHub stars: 159

### Overview: Uni-3DAR is an autoregressive model designed to unify various 3D tasks, including generation and understanding of microscopic and macroscopic structures. It offers improvements in handling multiple 3D data types, supporting diverse tasks, high efficiency, and high accuracy, making it a powerful tool in the field of 3D data processing. ### Key Features: - **Unified Handling of Multiple 3D Data Types:** The model can seamlessly handle various 3D data types such as molecules, proteins, and crystals, expanding its applicability. - **Support for Diverse Tasks:** Uni-3DAR naturally supports a wide range of tasks, including both generation and understanding, making it versatile for different applications. - **High Efficiency:** It uses octree compression-in combination with 2-level subtree compression-to represent the full 3D space using only hundreds of tokens, significantly reducing the computational complexity compared to a full-size grid. - **High Accuracy:** Building on octree compression, Uni-3DAR tokenizes fine-grained 3D patches to maintain structural details, achieving substantially better generation quality than diffusion-based models. ### Tech Stack: The project is implemented using Python. ### Why It Matters: Uni-3DAR is a significant advancement in the field of 3D data processing, offering a unified approach to various 3D tasks, improved efficiency, and enhanced accuracy, making it a valuable tool for researchers and developers in the field of 3D data analysis and generation.

### Overview: Eion is a shared memory storage designed for multi-agent systems, enabling unified knowledge graph capabilities across various AI deployment scenarios, from single LLM applications to complex multi-agency systems. ### Key Features: - **Unified Knowledge Graph**: Eion provides a shared memory storage that integrates knowledge graph capabilities, facilitating collaboration and knowledge sharing among AI agents. - **Context Storage**: It connects AI agents through shared memory, enabling seamless communication and data exchange. - **Agent Applications**: Supports both LLM applications and AI agent applications, adapting to different deployment scenarios. - **Multi-Agent Systems**: Offers solutions for sequential and concurrent live agency systems, ensuring efficient data flow and real-time updates. - **External Guest Agent Access**: Allows external agents to access internal agency systems with controlled access. ### Tech Stack: Go ### Why it Matters: Eion simplifies the integration and collaboration of AI agents, making it a valuable tool for multi-agent systems, especially in complex environments where data sharing and knowledge management are crucial. Repository: <https://github.com/eiondb/eion> GitHub Stars: 159 GitHub Description: Shared Memory Storage for Multi-Agent Systems

### Overview: VideoGrain is a zero-shot method for class-level, instance-level, and part-level video editing, enabling the manipulation of objects within the same class, each individual instance, and adding new objects or modifying existing attributes at the part-level. It achieves this without the need for training or fine-tuning. ### Key features: - **Multi-grained Video Editing**: Supports class-level, instance-level, and part-level video editing. - **Training-Free**: Does not require any training or fine-tuning. - **One-Prompt Multi-region Control & Deep investigations about cross/self attn**: Modulates cross-attention for multi-region control and modulates self-attention for feature decoupling, providing visualizations and clustering for deep investigations. ### Tech stack: Python ### Why it matters: VideoGrain is a significant contribution to the field of video editing, offering a zero-shot method that simplifies complex video manipulation tasks, making it accessible and efficient for various applications.

Overview: This project involves training a Chinese speech-to-text model using the Wavenet neural network, which is based on the THCHS30 open Chinese speech database. Key features: - Uses the Wavenet neural network for speech recognition. - Trained on a small dataset of 10,000 samples. - Achieved a CTC loss of 0.2768 after 124 epochs. - Training time on a GTX 1080 GPU is 15 hours. Tech stack: Python3.6, Keras2.02, Tensorflow v1.1 backend, Cuda8.0, Cudnn6.0. Why it matters: This project showcases the potential of using neural networks for speech recognition, particularly in the Chinese language, and demonstrates the effectiveness of the Wavenet architecture in achieving high recognition accuracy with a small dataset.

### Overview: Manor AI is a self-hosted AI workspace designed for small businesses and lean teams. It provides a shared context, durable tasks, scoped tools, and human approval gates, ensuring operational data stays inside infrastructure controlled by the team. ### Key features: - **Self-hosted AI workspace**: Runs agents against team knowledge and scoped tools, requiring human approval before sensitive actions. - **BYOK (Bring Your Own Key)**: Agents can run against team knowledge and scoped tools, and operational data stays inside infrastructure controlled by the team. - **Workflow and goal-based automation**: Agents operate through goals, tasks, workspace context, tool permissions, and human approval gates. Important automation leaves durable status, evidence, and review points instead of disappearing into a one-off chat transcript. - **Governance and accountability**: Rules are visible to operators and mapped to enforcement patterns in the runtime, making automation inspectable before it touches critical workflows. ### Tech stack: Python ### Why it matters: Manor AI provides a comprehensive AI workspace that ensures operational data stays within the team's infrastructure, enabling governance and accountability in the AI-driven workflow.