Projects

Overview: h-Edit is a finetuning-free diffusion-based editing method that leverages Doob’s h-Transform for effective and flexible image editing. Key features: - Theoretical Guarantee - Training-Free, Simple, General - Strong Performance, SOTA on PieBench - Flexible, Compatible, Plug-and-Play - Text-Guided Editing - Face Swapping - Combined Text-Guided & Style Editing Tech stack: Python Why it matters: Provides both explicit and implicit forms with unique features, enabling smarter and more efficient image editing without the need for fine-tuning.

Overview: Nomi is designed to simplify computer usage by enabling users to interact with their computers through text and voice inputs, offering a versatile AI runtime and privacy-focused capabilities. Key features: - Versatile AI Runtime - Privacy-Focused Data Management - Multi-Modal Interface - Provider Integration - Conversation Management - Prompt Engineering - Code Interpreter - Voice Interaction - Terminal Experience Tech stack: Go Why it matters: Nomi simplifies the interaction with computers, making advanced AI accessible and user-friendly for everyone, democratizing AI and enhancing data privacy. GitHub stars: 80 Repository: nomihq/nomi-computer-use Primary language: Go GitHub description: Nomi enable people to use computer more simply.

Overview: Tadpole Studio is an easy-to-use AI music generation UI that allows developers to generate, remix, and manage local AI models for a superior user experience. Key features: - Text-to-Music (Simple Mode) - Custom Mode - Remix (Cover) - Repaint - Extract - Lego - Complete - AutoGen Tech stack: TypeScript Why it matters: Tadpole Studio simplifies the process of AI music generation, making it accessible and efficient for developers, allowing them to focus on their projects without the complexity of advanced AI tools.

Overview: This repository contains the official project for Recursion's accepted spotlight paper at the NeurIPS 2023 Generative AI & Biology workshop, focusing on masked autoencoders for microscopy and cellular morphology. Key features: - Masked Autoencoders for Microscopy: Scalable Learners of Cellular Biology - CVPR 2024 Spotlight Paper: Spotlight full-length paper on Masked Autoencoders for Microscopy - NeurIPS 2023 Workshop Paper: Spotlight workshop paper on Masked Autoencoders for Microscopy - CVPR 2024 Poster Page: Video presentation of the CVPR 2024 poster - CVPR 2024 Conference Page: Link to the conference page - NeurIPS 2023 Workshop: Link to the workshop page - CVPR 2024 Poster Page: Video presentation of the CVPR 2024 poster - CVPR 2024 Conference Page: Link to the conference page - CVPR 2024 Poster Page: Video presentation of the CVPR 2024 poster - CVPR 2024 Conference Page: Link to the conference page Tech stack: Jupyter Notebook Why it matters: This repository provides the official project for Recursion's accepted papers at the NeurIPS 2023 Generative AI & Biology workshop, showcasing the effectiveness of masked autoencoders in learning cellular morphology from microscopy data. GitHub stars: 80 Repository:

Overview: This project involves creating a production-ready trip planner system that integrates multiple AI agents to provide users with detailed and accurate destination information. The system is designed to be modular and scalable, ensuring future upgrades and cost efficiency. Key features: - Multi-AI agent support - Modular architecture for scalability and future upgrades - Integration with Taskflow AI for efficient tool-calling and reasoning capabilities Tech stack: Jupyter Notebook Why it matters: This project showcases the integration of multiple AI agents to create a robust and efficient travel planning system, demonstrating the capabilities of Taskflow AI in modular design and multi-agent integration.

Overview: This repository allows fine-tuning LLaMA, Open LLaMA, RedPajama, Falcon, and StableLM models on consumer hardware using QLoRA, a method for accelerating language models. It provides a straightforward application of QLoRA PEFT to these models and supports training on various datasets and models. Key features: - Fine-tuning LLaMA, Open LLaMA, RedPajama, Falcon, and StableLM models - Supports training on consumer hardware - Uses QLoRA for accelerating model training - Provides examples for specific model training - Supports training on datasets like Dolly and CodeAlpaca Tech stack: Python Why it matters: This tool simplifies the fine-tuning process for large language models, making it accessible for users with limited computational resources.

### Overview: This repository is a curated list of speech recognition learning resources, including courses, books, models, tutorials, toolkits, and papers. It serves as an invaluable resource for developers interested in the field of speech recognition, providing a comprehensive guide to the latest techniques and tools. ### Key features: - **Courses**: A selection of recommended courses on automatic speech recognition, including lectures from prestigious institutions like the University of Edinburgh and Columbia University. - **Books**: Comprehensive guides to speech recognition theory, algorithms, and system development, covering foundational texts and advanced topics. - **Models**: Pre-trained and foundation models for speech recognition, including models from Alibaba and other leading research institutions. - **Tutorials**: Practical tutorials and guides to help developers understand and implement speech recognition technologies. - **Toolkits**: Comprehensive toolkits and libraries for speech recognition, including models and utilities for various languages and dialects. ### Tech stack: Python, C++, and various machine learning libraries. ### Why it matters: This repository is crucial for developers looking to advance their skills in speech recognition. It provides a blend of theoretical knowledge and practical implementation, making it an essential resource for anyone interested in the field.

### Overview: This project introduces Image Neural Field Diffusion Models, a novel approach to image generation and diffusion, presented at CVPR 2024. The implementation is based on the paper "Image Neural Field Diffusion Models" by Yinbo Chen et al. ### Key Features: - **Image Generation and Diffusion**: The project focuses on generating high-quality images using neural fields, showcasing advancements in diffusion models. - **Model Training**: Provides detailed instructions for training on FFHQ dataset and FFHQ-6K-Mix settings, including configurations and environment setup. - **Custom Datasets**: Allows training on custom datasets by modifying the configuration files. - **Pretrained Models**: Offers pretrained models for FFHQ and FFHQ 6K-Mix datasets, with pFID scores provided for comparison. ### Tech Stack: Python ### Why it Matters: This project represents significant progress in the field of image generation and diffusion, showcasing the capabilities of neural fields and their applications in generating high-quality images. The detailed implementation and pretrained models make it a valuable resource for researchers and developers in the field. ### Repository: yinboc/infd ### GitHub Stars: 80 ### Primary Language: Python ### GitHub Description: Official implementation of the paper "Image Neural Field Diffusion Models" by Yinbo Chen et al.

Overview: Awesome Multimodal Assistant is a curated list of multimodal chatbots/conversational assistants that utilize various modes of interaction, such as text, speech, images, and videos, to provide a seamless and versatile user experience. Key features: - MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning - GPT-4 - Visual Instruction Tuning - MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models - mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality - LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model - Video-LLaMA: An Instruction-Finetuned Visual Language Model for Video Understanding - LMEye: An Interactive Perception Network for Large Language Models - MultiModal-GPT: A Vision and Language Model for Dialogue with Humans - X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages - Otter: A Multi-Modal Model with In-Context Instruction Tuning Tech stack: Python, PyTorch, TensorFlow, Hugging Face Transformers, and various open-source libraries for multimodal processing and model training. Why it matters: This curated list aims to provide a comprehensive overview of multimodal chatbots, highlighting the latest advancements in multimodal instruction tuning, vision-language models, and other cutting-edge technologies that enhance the capabilities of conversational assistants.

### Overview: LoRA-Torch is a PyTorch implementation of LoRA, a low-rank adaptation technique for large language models. This project aims to provide a comprehensive reimplementation of LoRA, including support for nn.MultiheadAttention from the OpenCLIP model. ### Key features: - **Linear Layers:** Both `loralib` and `loratorch` compute the linear layer outputs using the pre-trained weight matrix and LoRA matrix. However, `loratorch` merges the pre-trained weight and LoRA weight before applying the linear transformation, making it more extensible. - **Complex Layers:** LoRA is not directly extendable to complex layers like `nn.MultiheadAttention` due to the lack of a clear relationship between the input and the output of such layers. `loratorch`, on the other hand, simplifies the process by merging weights first and then applying the original layer's forward method. ### Tech stack: Python ### Why it matters: LoRA-Torch provides a more flexible and extensible implementation of LoRA, making it easier to extend the technique to complex layers and other types of layers in PyTorch models. This project serves as a valuable resource for developers interested in applying LoRA to their projects.

Overview: HermesFlow is a multimodal understanding and generation framework designed for NeurIPS 2025, focusing on closing the gap between multimodal data understanding and generation. Key features: - Seamless integration of multimodal data understanding and generation - Use of self-play iterative optimization with Pair-DPO for efficient learning - Implementation of homologous preference data curation for better model performance - Comprehensive evaluation using VQA and image generation methods Tech stack: Python Why it matters: HermesFlow provides a practical solution for improving the alignment between multimodal understanding and generation in large language models, making it easier to bridge the gap in AI applications. Repository: Gen-Verse/HermesFlow GitHub stars: 79

### Overview: AI短劇系統，用于Codex、Claude Code及兼容Skills客户端，提供故事与连播状态的管理，包括微集规划、角色一致性、每集爽点与追更线索，以及连续长镜与宫格瞬间的路由。 ### Key features: - **微集规划与连播状态管理**：规划微集内容，确保角色一致性与每集的爽点与追更线索。 - **模型-aware生产**：记录模型时长、参考额度，锁定角色三视图、全局风格与逐镜时间线。 - **连续长镜与宫格瞬间路由**：在连播单集、完整微短剧、爆款一镜、宫格瞬间与连续长镜之间进行路由。 - **生产输出**：输出schema-valid的生产包、固定entity ID、逐镜生成brief与剪辑手交。 ### Tech stack: Python ### Why it matters: 简化了短剧的制作流程，提高了效率，同时保持了高质量的连贯性和角色一致性。

### Overview: Rose is a stateless PyTorch optimizer designed for efficient optimization without the need for historical statistics. It uses instantaneous per-slice range normalization to adaptively scale gradients, reducing memory usage and improving training dynamics. ### Key Features: - **Stateless Optimization**: No momentum, variance estimates, or step counters, leading to minimal memory footprint. - **Improved Generalization**: Training loss is higher while validation loss is lower, demonstrating implicit regularization rather than memorization. - **Accurate and Fast**: Matches or exceeds AdamW in observed accuracy and convergence across tested workloads. - **Gradient Centralization**: Removes shared directional bias, improving stability and generalization. - **CV Trust Gating**: Automatically detects noisy per-slice ranges and falls back to a robust global estimate. - **Decoupled Weight Decay**: Standard or schedule-coupled weight decay; the schedule-coupled option prevents late-training decay from overpowering vanishing learning rates. - **BF16 Stochastic Rounding**: Eliminates systematic truncation drift, significantly improving low-precision training fidelity. ### Tech Stack: Python 3.10+ ### Why It Matters: Rose offers a more efficient and accurate way to optimize models without the overhead of historical statistics, making it ideal for resource-constrained environments and improving the generalization of models.

### Overview: This repository is a curated list of sources and channels from where Large Language Models (LLMs) can be learned. It includes historical perspectives on the development of NLP, key milestones in the evolution of language processing, and modern advancements in the field. ### Key Features: - **Historical Perspectives**: Detailed insights into the journey of NLP, including the contributions of key figures like Warren Weaver, Chomsky, and others. - **Milestones in NLP**: A comprehensive list of significant events and breakthroughs in the field, from early grammar theories to modern LLMs. - **Modern Advancements**: Up-to-date information on the latest developments in LLMs, including their architecture, training methods, and applications. ### Tech Stack: - **Python**: Used for implementing and experimenting with various LLM models. - **TensorFlow**: For building and training neural networks. - **PyTorch**: For developing and training deep learning models. ### Why It Matters: Understanding the history and evolution of LLMs is crucial for anyone looking to master this technology. It provides a foundation for grasping the current state of the art and the challenges that still need to be overcome. The insights into past successes and failures can also help in identifying potential areas for future innovation and improvement. ### Repository: ParthaPRay/LLM-Learning-Sources - **Stars**: 79 - **Description**: Contains a list of channels and sources from where LLMs should be learned. - **Primary Language**: Unknown - **GitHub Description**: "This repo

### Overview: This repository is a curated list of open-source tools, frameworks, and resources aimed at securing autonomous AI agents. It covers various aspects of AI agent security, including runtime protection, red teaming, static analysis, sandboxing, compliance, benchmarks, and identity authentication. ### Key features: - **Agent Firewalls & Gateways (Runtime Protection):** - **[AgentGateway](https://github.com/agentgateway/agentgateway)**: A Linux Foundation project providing an AI-native proxy for secure connectivity (A2A & MCP protocols). It adds RBAC, observability, and policy enforcement to agent-tool interactions. - **[Envoy AI Gateway](https://gateway.envoyproxy.io/)**: An Envoy-based gateway that manages request traffic to GenAI services, providing a control point for rate limiting and policy enforcement. - **[Immunity Agent](https://github.com/PrismorSec/immunity-agent)**: Security-focused AI agent runtime for scanning prompt injection, MCP risks, unsafe package installs, and dangerous agent actions before execution. - **Red Teaming & Vulnerability Scanners:** - **[Strix](https://github.com/usestrix/strix)**: An autonomous AI agent designed for penetration testing. It runs inside a docker sandbox to actively probe applications and generate verified exploit capabilities. - **[PyRIT](https://github.com/Azure/PyRIT)**: Microsoft's open-source red teaming framework for generative AI. It automates multi-turn adversarial attacks to test if an

Overview: TransFusion is an open-source project that focuses on transcribing speech using Multinomial Diffusion. It includes code for training, inference, and scoring of ASR models from a paper titled "TransFusion: Transcribing Speech with Multinomial Diffusion". Key features: - Transcription of speech using Multinomial Diffusion - Code for training, inference, and scoring of ASR models - Available checkpoints for quickstart Tech stack: Python Why it matters: TransFusion provides a practical solution for speech transcription, leveraging advanced diffusion models to improve accuracy and efficiency in ASR tasks. Repository: RF5/transfusion-asr GitHub stars: 79 GitHub description: Transcribing Speech with Multinomial Diffusion, training code and models.

### Overview: This repository contains a collection of sample code demonstrating various use cases with SAP BTP, presented during the SAP BTP AI Foundation Use Cases series. Each numbered folder includes the code for a specific use case, which can be run by following the README.md within the folder. ### Key Features: - **Generative AI Hub**: Central cockpit for creating, operating, monitoring, and orchestrating generative AI scenarios. - **SAP AI Core**: The central hub for AI capabilities in SAP BTP. - **SAP S/4HANA Cloud**: The cloud-based platform for SAP's financial applications. - **SAP HANA Cloud**: The cloud-based platform for SAP's business applications. - **SAP Build Apps**: The platform for building SAP applications. - **Joule**: A tool for code development. - **CAP**: The platform for Cloud Platform Application Programming. - **LLMs (Large Language Models)**: Popular and commercially viable open-source LLMs such as Phi 3, LLaMa 3, Mistral, Mixtral, Gemma, LlaVA, etc. - **Open-Source LLMs**: Popular and commercially viable open-source LLMs such as Ollama, llama.cpp, vLLM, Transformers, Infinity, SAP Generative AI Hub, SAP Generative AI Hub SDK, SAP AI Core. ### Tech Stack: - **SAP BTP Services**: Various SAP BTP services including Generative AI Hub, SAP AI Core, SAP S/4HANA Cloud, SAP HANA Cloud, CAP

Overview: QuickVideo is a system-algorithm co-design that significantly speeds up the understanding of long videos by reducing the time from 70 seconds to 20 seconds for 1-hour videos while using 50% less memory. Key features: - Parallelized CPU-based decoder - Group-based prefilling for memory-efficient activation handling - KV-cache pruning to retain only essential tokens - Intelligent scheduling to minimize end-to-end latency Tech stack: Python Why it matters: QuickVideo enables real-time video understanding, making it crucial for applications requiring quick responses to video content, such as meeting summarization, video surveillance, and educational lecture analysis.

### Overview: MAFDET is a chat interface that reads your message as you type, starting to think about it before you send. This allows the model to prefill the prompt and begin reasoning immediately, significantly reducing the wait time. ### Key Features: - **Reading Ahead**: Reads your message as you type, prefilling the prompt. - **Thinking Ahead**: Starts reasoning about your message while you type, keeping that reasoning even after you change the text. - **Immediate Response**: Reduces the wait time after pressing Enter, improving user experience. ### Tech Stack: - Python ### Why It Matters: - **Efficiency**: Reduces the time the model spends waiting for your input. - **Accuracy**: Improves the accuracy of the model's responses by starting to think about your message before you send it. - **User Experience**: Enhances the overall user experience by reducing the wait time and improving the efficiency of the model's responses. ### Repository: - **TheLastOxygen/mafdet** - **GitHub Stars**: 76 - **Primary Language**: Python - **GitHub Description**: You type. It already knows. Local LLMs that read and think while you type.

Overview: Tempus is a C++ framework designed for developing multimodal path planning requests. It includes components such as a core framework, data loaders, WPS servers, Python clients, GeoJSON servers, and a Python API, among others. Key features: - A core framework written in C++ - Data loaders (OSM loader) - WPS server - Python client - GeoJSON server - Python API - QGIS plugin - Set of demos Tech stack: C++ Why it matters: Tempus provides a comprehensive solution for developing and deploying multimodal path planning applications, making it an essential tool for developers working in the field of geographic information systems and route planning.