Projects

Overview: This repository is a comprehensive guide and resource for the FLUX 3 video API and image API, covering prompts, parameters, and examples from Black Forest Labs' unified text-to-video, image-to-video, image, and audio generation model. Key features: - Prompts and parameters for FLUX 3 video API and image API - Multimodal workflows and production-ready examples - Integration with MuAPI for seamless API access Tech stack: MuAPI Why it matters: This guide provides essential information for developers and creators looking to leverage Black Forest Labs' cutting-edge FLUX 3 model for various applications, including video generation, image creation, and audio synthesis, with a focus on creating high-quality content through a unified multimodal approach. Repository: Anil-matcha/awesome-flux-3-api-prompts GitHub stars: 147

### Overview: LiveWorld is an open-source project that implements the paper "LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models". The project focuses on enabling persistent out-of-sight dynamics in generative video world models by decoupling world evolution from observation rendering. It includes inference code, training code, and a demo pipeline, aiming to provide a comprehensive solution for simulating dynamic scenes in video world models. ### Key Features: - **Persistent Out-of-Sight Dynamics**: The project simulates out-of-sight dynamics by explicitly decoupling world evolution from observation rendering. - **Monitor Registration**: Monitors are registered to autonomously fast-forward the temporal progression of active entities in the background. - **State-Aware Renderer**: The state-aware renderer projects continuously evolved world states to synthesize the final observation, ensuring dynamic events progress naturally even when entities are completely out of view. ### Tech Stack: - **Python**: The primary programming language used for the project. - **CUDA 12.8**: Used for accelerating GPU computations. - **PyTorch 2.8**: Used for training and inference. ### Why It Matters: LiveWorld is significant because it addresses the challenge of simulating dynamic scenes in generative video world models where entities are out of the observer's view. By decoupling world evolution from observation rendering, it ensures that dynamic events progress naturally even when entities are completely out of the observer's view, making the simulation more realistic and engaging. ### Repository: The repository is hosted on GitHub: [ZichengDuan/Live

### Overview: This repository is a curated list of watermarking schemes for generative AI models across various content types, including images, video, speech, audio, text, code, and 3D content. It covers watermarking during training, generation, and post-generation, as well as attacks and evaluation methods. ### Key features: - **Model Training and Adaptation:** Mark examples before training and test for watermark detection. - **Model Training:** Modify generator weights or adapt model training to include watermarking. - **Generation Procedure:** Modify sampling, logits, or prompt instructions to include watermarking. - **Post-Generation:** Transform completed content to include watermarking. ### Tech stack: Python and JavaScript ### Why it matters: This repository provides a comprehensive overview of watermarking techniques for generative AI, helping developers understand and implement effective watermarking strategies for their projects.

Overview: ialacol is a lightweight replacement for OpenAI's APIs, designed for easy deployment on Kubernetes clusters. It supports various models and is compatible with OpenAI's LangChain. Key features: - OpenAI API compatibility - Lightweight deployment - Streaming support - Optional CUDA acceleration - Compatible with Github Copilot Tech stack: Python Why it matters: ialacol simplifies the deployment of AI models on Kubernetes, making it accessible and easy to use for developers. Repository: chenhunghan/ialacol Primary language: Python GitHub stars: 147 GitHub description: 🪶 Lightweight OpenAI drop-in replacement for Kubernetes

### Overview: StockChat is an open-source project that offers a real-time chat interface integrated with AI-powered stock analysis capabilities. Built for developers, it provides a financial NLP solution. ### Key features: - **Real-time Stock Analysis**: StockChat leverages AI to provide real-time stock analysis and trading signals. - **Open-Source Implementation**: Built using Python, FastAPI, React, and Docker, StockChat is an open-source alternative to Perplexity Finance. - **Docker-Driven Development**: The project includes a backend service and a frontend application, allowing for easy deployment and development. - **Extensible Data Ingestion**: Market data can be processed and ingested through an extensible data framework. - **SQLite-Based Analysis Cache**: Users can save their analysis results for later reference. - **Technical Analysis Integration**: Built with TA-Lib for professional technical analysis calculations. ### Tech stack: - Python - FastAPI - React - Docker - SQLite - TA-Lib ### Why it matters: StockChat offers a practical solution for developers looking to implement financial NLP solutions, providing a robust and scalable framework for real-time stock analysis and trading signals.

Overview: APS is a personal toolkit designed for single/multi-channel speech recognition, enhancement, and separation, aiming to simplify the training and evaluation process and facilitate experiments with neural network-based methods. Key features: - Speech recognition - Speech enhancement - Speech separation Tech stack: Python Why it matters: APS simplifies the process of speech-related tasks, making it easier and more flexible for developers to experiment with neural network-based methods. Repository: funcwj/aps GitHub stars: 147 GitHub description: A personal toolkit for speech related tasks, such as single/multi-channel speech enhancement and separation, and speech recognition.

Overview: This project provides local speech recognition capabilities on an ESP32, allowing for hot-word detection and speech recognition directly on the device, enhancing the integration with Home Assistant. Key features: - Hot-word detection - Speech recognition - Integration with Home Assistant - Customizable speech commands using MQTT Tech stack: C Why it matters: This project offers a practical solution for integrating voice commands directly into smart home devices, improving the user experience and automation capabilities. GitHub stars: 147

Overview: Infino is a fast, efficient, and scalable embedded retrieval library built on Parquet, designed for cost-effective and efficient querying of large datasets. It supports full-text, vector, and hybrid search, as well as SQL queries over a single table, all stored in ordinary Parquet format. Key features: - Supports full-text, vector, and hybrid search - Optimized for cost, with Parquet storage on local disk or object storage - Simple, scalable, and optimized for performance - Ingest throughput up to 10M docs per second Tech stack: Rust Why it matters: Infino is essential for developers looking to efficiently query and retrieve large datasets, especially in environments where cost and performance are critical considerations. Its ability to handle full-text, vector, and hybrid search, as well as SQL queries, makes it a versatile tool for a wide range of applications, from search engines to recommendation systems.

### Overview: GEMS: Agent-Native Multimodal Generation with Memory and Skills is a project that leverages agent-native multimodal generation techniques, combining memory and skills to enhance the generation of images and other multimodal content. This project aims to provide a robust and versatile framework for generating high-quality images and other multimedia content using a variety of skills and agents. ### Key Features: - **Agent-Native Multimodal Generation**: GEMS integrates skills and agents to generate high-quality images and other multimodal content. - **Memory and Skills Integration**: The project utilizes skills to enhance the generation process, ensuring that the generated content is contextually relevant and skillful. - **Multi-Task Support**: GEMS supports multiple downstream tasks, making it versatile for various applications. ### Tech Stack: - **Python**: The core implementation and project management are done using Python, leveraging its flexibility and extensive libraries for machine learning and web development. ### Why It Matters: GEMS offers a powerful tool for generating high-quality images and other multimedia content, making it useful in various applications such as creative design, content creation, and more. By integrating skills and agents, it provides a more contextually relevant and skillful approach to generating content, enhancing user experience and efficiency. ### Repository: The project is hosted on GitHub at [lcqysl/GEMS](https://github.com/lcqysl/GEMS). It has received 147 GitHub stars, indicating its growing popularity and usefulness. ### Quick Start: To get started with GEMS, clone the repository and install the required dependencies: ```

### Overview: This repository is a curated list of tools and resources for red-teaming large language models (LLMs), focusing on vulnerability research and reverse engineering techniques. It includes playgrounds for hands-on experimentation, frameworks for systematic red teaming, attack generation tools, obfuscation utilities, and support for traffic interception and defenses. ### Key Features: - **Playgrounds and Practice Targets:** - **Folly:** Open-source LLM prompt injection and jailbreak playground. - **DamnVulnerableMathLLM:** Intentionally vulnerable maths LLM app for prompt injection and logic exploits. - **DamnVulnerableShoppingLLM:** Intentionally vulnerable shopping assistant scenario for data leakage and unsafe tool use. - **RedTeam Arena:** Competitive jailbreak challenges that train you to bypass model guardrails under time pressure. - **MyLLM Bank:** WithSecure CTF where you exploit a multi-LLM banking app to learn real-world prompt injection chains. - **MyLLM Doctor:** WithSecure medical chatbot challenge showing multi-stage prompt injection and policy evasion. - **Gandalf by Lakera:** Public challenge where you try to extract a secret from an aligned model despite protections. - **Red Teaming Frameworks and Agent Harnesses:** - **PyRIT:** Microsoft framework for systematic, repeatable red teaming of generative AI systems with automation support. - **Rigging:** Lightweight agent and tool-calling harness for orchestrating offensive LLM experiments in controlled environments. - **Attack Generation and Jailbreak

### Overview: CoEdIT is an advanced text editing system designed to improve the grammaticality and readability of text based on user-provided instructions. ### Key features: - **Model Training**: CoEdIT is a large language model fine-tuned specifically for text editing tasks. - **Task-Specific Instructions**: Users provide instructions to improve the text, such as "Make the sentence simpler" or "Write it in a more neutral style." - **Instruction Tuning**: The model adjusts its output to better align with the provided instructions, enhancing the quality of the edited text. ### Tech stack: Shell ### Why it matters: CoEdIT offers a powerful tool for text editors and writers, providing a high-quality output by leveraging advanced language models and user-specific instructions.

### Overview Salient object detection (SOD) is a crucial technique in computer vision, focusing on identifying and locating the most important objects within an image or video. This curated list aims to provide a comprehensive overview of resources and techniques for salient object detection, particularly emphasizing multi-modal SOD, including RGB-D and RGB-T SOD. ### Key Features - **RGB-D SOD**: Techniques and resources for detecting salient objects in RGB-D images. - **RGB-T SOD**: Similar to RGB-D SOD but focusing on RGB-T images. - **V-D-T SOD**: Multi-modal SOD focusing on RGB, RGB-D, and RGB-T images. - **Image SOD**: Basic techniques and resources for salient object detection in single images. - **Appendix**: Additional resources and supplementary information. ### Tech Stack - **Python**: The primary programming language used for implementing various SOD techniques. ### Why It Matters - **Enhanced Object Understanding**: Multi-modal SOD techniques can provide a more comprehensive understanding of the scene by integrating multiple modalities, such as RGB, RGB-D, and RGB-T. - **Improved Accuracy**: Techniques like V-D-T SOD can enhance the accuracy of salient object detection by leveraging the complementary information from different modalities. - **Versatility**: Resources in this list are designed to cater to various applications, from research to practical use, making it a valuable resource for developers and researchers alike. ### Repository - **Visionxiang/awesome-salient-object-detection**: The primary repository for this curated list,

Overview: The Ark CLI is a command-line tool that simplifies the integration of Volcengine Ark into your terminal, enabling a seamless workflow for generating media, multimodal answers, and deploying endpoints directly from prompts. Key features: - Model selection and invocation - Image and video generation - Multi-modal understanding - Plan management - Agent integration - Model production - Endpoint deployment - Usage, billing, and monitoring Tech stack: The Ark CLI is built using JavaScript (jq) and is designed to be used with the Volcengine Ark MaaS platform. Why it matters: The Ark CLI streamlines the process of integrating Volcengine Ark into your development environment, making it easier to deploy models, generate media, and manage resources without the need for API glue code. This tool allows developers to focus on their core applications while leveraging the full capabilities of the Volcengine platform.

## Overview: Whisrs is a voice-enabled desktop application for Linux that supports dictation, read-aloud functionality, and voice-driven commands using Rust. It provides a seamless experience across various desktop environments and compositor window managers, including Wayland, X11, Hyprland, Sway, Niri, GNOME, and KDE. With features like pluggable backends and GPU acceleration, it offers fast, private, and open-source speech-to-text and text-to-speech capabilities. ## Key Features: - **Dictation**: Record and transcribe speech directly into text. - **Read-Aloud**: Speak selected text aloud. - **Voice Commands**: Execute commands by speaking them. - **Cloud Transcription**: Supports cloud-based transcription using models from Groq, Deepgram, and OpenAI. - **Offline Transcription**: Transcribes speech locally using whisper.cpp. - **System Audio**: Records both system audio and mic input. - **Timestamped Transcripts**: Keeps a timestamped Markdown transcript of your recordings. ## Tech Stack: Whisrs is written in Rust, utilizing the Rust programming language for its core functionality. It supports Wayland, X11, and various compositor window managers, making it compatible with a wide range of Linux desktop environments. ## Why It Matters: - **Native Dictation**: Provides a native dictation tool for Linux, which is not available in other desktop environments. - **Unified Desktop Experience**: Supports multiple desktop environments and compositor window managers, ensuring a consistent user experience. - **Fast and Private**: Utilizes whisper.cpp for

### Overview: This repository contains the course materials and assignments for "Machine Learning and having it deep and structured" (MLDS) at NTU in Spring 2018. The course covers four homeworks focusing on deep learning theory, sequence-to-sequence models, deep generative models, and deep reinforcement learning. ### Key features: - Deep Learning Theory - Sequence-to-sequence Model - Deep Generative Model - Deep Reinforcement Learning ### Tech stack: Python ### Why it matters: This course provides a comprehensive overview of deep learning techniques and models, essential for understanding modern machine learning. The assignments cover a wide range of topics, from deep learning theory to reinforcement learning, making it a valuable resource for anyone interested in deep learning and artificial intelligence.

### Overview: KOALA, a fast text-to-image model, was developed by compressing SDXL's U-Net and distilling knowledge from SDXL into a more efficient U-Net. This approach allows KOALA to generate 1024x1024 images in 0.66 seconds on an NVIDIA 4090 GPU, significantly faster than SDXL, which is more than 4x faster. ### Key Features: - **KOALA-Lightning-700M**: Generates 1024x1024 images in 0.66 seconds on an NVIDIA 4090 GPU, making it 4x faster than SDXL. - **KOALA-Lightning-700M**: Can be used as a cost-effective alternative between SDM and SDXL in limited resources. - **KOALA-Turbo**: Uses a compact U-Net (1B) for faster inference. - **KOALA-Lightning-700M**: 4x faster than SDXL while maintaining satisfactory generation quality. ### Tech Stack: The project is built using Python and utilizes the Lightning framework for efficient training and inference. ### Why It Matters: KOALA models are designed to reduce inference costs and explore the generative capabilities of T2I models using publicly available datasets and open-source models. By focusing on self-attention and high-resolution image generation, KOALA offers a practical solution for researchers and users in resource-constrained environments, providing faster and more efficient text-to-image synthesis.

Overview: SoM-LLaVA is a new dataset and learning paradigm designed to enhance the capabilities of multimodal language models, specifically in handling visual reasoning tasks. Key features: - Incorporates Set-of-Mark prompting to improve visual reasoning ability. - Provides a complementary dataset for training multimodal language models. - Includes model weights in both LLaVA format and HF format. Tech stack: Python Why it matters: SoM-LLaVA offers a significant enhancement in the performance of multimodal language models by adding 30k SoM data to the visual instruction tuning stage of LLaVA, resulting in 1% to 6% relative improvements across various benchmarks.

### Overview: Build5Nines SharpVector is a lightweight, in-memory, local, Semantic Search, and Text Vector Database library designed for .NET applications. It allows you to store, search, and manage text data using vector representations. ### Key features: - **Vectorized Text Storage**: Store and retrieve text data using vector representations. - **Customizable Vector Comparison Methods**: Support for different vector comparison methods. - **Extensible**: Extensible for different vectorization strategies and preprocessing techniques. - **Local Embeddings**: Supported for local vector embeddings. - **OpenAI Embeddings**: Supported for OpenAI embeddings. - **Ollama Embeddings**: Supported for Ollama embeddings. - **Generative AI**: Ready for integration with Generative AI solutions. ### Tech stack: C# ### Why it matters: While there are many large databases available for building vector databases, there are not many lightweight options that can be embedded into any .NET application. Build5Nines SharpVector simplifies the process by providing a lightweight, local, in-memory Text Vector Database, making it an ideal choice for implementing semantic search into any .NET application.

### Overview: EvoTune is a framework designed to discover new algorithms by combining evolutionary search over LLM-generated Python programs with reinforcement learning to fine-tune the search operator based on performance scores. ### Key Features: - **Combines Evolutionary Search and Reinforcement Learning**: EvoTune uses evolutionary search to explore the space of LLM-generated Python programs and reinforcement learning to fine-tune the LLM operator. - **Efficient Algorithm Discovery**: The framework is optimized for discovering new algorithms by leveraging the strengths of both evolutionary search and reinforcement learning. - **Versatile Task Support**: EvoTune supports various tasks such as bin packing, flat-pack, and TSP, making it suitable for a wide range of algorithm discovery scenarios. ### Tech Stack: Python ### Why it Matters: EvoTune is a powerful tool for algorithm discovery, combining the strengths of evolutionary search and reinforcement learning to efficiently explore and fine-tune the space of LLM-generated programs. This makes it an attractive solution for researchers and developers looking to discover new algorithms quickly and effectively.

Overview: This repository contains the official code and data for the paper "Dissecting Adversarial Robustness of Multimodal Language Model Agents" presented at ICLR 2025. The project is focused on analyzing the adversarial robustness of multimodal language model agents. Key features: - Installation - Additional Setup - Usage Tech stack: Python Why it matters: This project provides a comprehensive framework for understanding how adversarial attacks affect multimodal language model agents, contributing to the field of adversarial robustness in AI. Repository: ChenWu98/agent-attack Primary language: Python GitHub stars: 146 GitHub description: [ICLR 2025] Dissecting adversarial robustness of multimodal language model agents README excerpt: <div align=center> <img src="docs/attack-vwa.png" align="middle"> </div> <br> ## Contents - [Dissecting Adversarial Robustness of Multimodal LM Agents](#adversarial-attacks-on-multimodal-agents) - [Contents](#contents) - [Installation](#installation) - [Install VisualWebArena](#install-visualwebarena) - [Install this repository](#install-this-repository) - [Additional Setup](#additional-setup) - [Setup API Keys](#setup-api-keys) - [Setup experiment directory](#setup-experiment-directory) - [Usage](#usage) - [Run attacks](#