Projects

### Overview 本仓库收集了多模态对话系统相关的论文和笔记，涵盖了多模态数据集、多模态预训练模型、多模态生成以及经典论文等内容。这些资源旨在帮助开发者和研究人员更好地理解和应用多模态对话技术。 ### Key Features - **多模态数据集**：提供了多模态对话数据集的概述，包括多模态数据集的论文、笔记状态、年份、会议/期刊、CCF级别和研究机构。 - **多模态预训练模型**：收录了多模态预训练模型的论文、笔记状态、年份、会议/期刊、CCF级别和研究机构。 - **多模态生成**：包含了多模态生成的论文、笔记状态、年份、会议/期刊、CCF级别和研究机构。 ### Tech Stack 本仓库使用了Python进行数据处理和分析，以及Markdown进行文档编写。 ### Why it Matters 多模态对话系统在自然语言处理、情感分析、机器翻译等领域具有广阔的应用前景，能够提高对话系统的理解和生成能力，为用户提供更加自然、流畅的交互体验。

### Overview: SDPO is a novel reinforcement learning framework designed to align few-step diffusion models with downstream objectives. This paper, "Aligning Few-Step Diffusion Models with Dense Reward Difference Learning," was accepted by IEEE TPAMI and has been published on arXiv. The repository contains a PyTorch implementation of SDPO, which addresses the limitations of existing RL methods in low-step regimes with limited state spaces and suboptimal sample quality, by introducing a dual-state trajectory sampling mechanism and a dense reward prediction strategy. ### Key Features: - **SDPO Framework**: A novel reinforcement learning framework tailored for few-step diffusion models. - **Dual-State Trajectory Sampling**: Tracks both noisy and predicted clean states at each step to provide dense reward feedback. - **Latent Similarity-Based Dense Reward Prediction**: Minimizes costly dense reward queries by leveraging latent similarities. - **Reward Difference Learning Objective**: Optimizes a dense reward difference learning objective for more frequent and granular policy updates. - **Refinements**: Additional refinements like stepwise advantage estimates, temporal importance weighting, and step-shuffled gradient updates enhance long-term dependency, low-step priority, and gradient stability. ### Tech Stack: - Python ### Why It Matters: SDPO consistently delivers superior reward-aligned results across diverse few-step settings and tasks, making it a valuable tool for researchers and practitioners in the field of reinforcement learning and diffusion models.

Overview: Llama2-CodeGen leverages the Llama2-7B model to generate context-aware Python code from natural language descriptions, utilizing a custom dataset scraped from over 25 GitHub repositories. The project achieved an initial training accuracy of 89% and was later optimized through increased epochs and black formatting. Key features: - Custom dataset from over 25 GitHub repositories. - Initial training on 1000 samples for 100 epochs, achieving 89% accuracy. - Gradual increase in epochs for enhanced model performance. - Integrated black formatting for cleaner output. - Deployed on Vercel with a React-based frontend. Tech stack: Jupyter Notebook Why it matters: This project demonstrates the potential of fine-tuning large language models for practical applications, showcasing how natural language can be transformed into executable code.

### Overview: ARLE is a local inference server for coding agents built with pure Rust, designed for Apple Silicon and NVIDIA systems. It supports Anthropic and OpenAI APIs and uses a KV cache to survive across turns, making turns 20 start as fast as turn 2. The model is trained on OPD and can be run on the same runtime as Claude. ### Key Features: - Pure Rust implementation - One binary for Apple Silicon and NVIDIA - Supports Anthropic and OpenAI APIs - KV cache survives across turns, making turns 20 start as fast as turn 2 - Training on OPD and can be run on the same runtime as Claude ### Tech Stack: Rust ### Why It Matters: ARLE is designed to be fast and efficient, making it ideal for real-time coding interactions. Its ability to cache and re-prefill KV data across turns ensures that subsequent turns start as quickly as the initial turn, significantly reducing latency. This makes it particularly useful for developers who need to quickly respond to code snippets or tool outputs. ### Repository: Repository: acupof-ai/arle GitHub Stars: 22 GitHub Description: The local inference server for coding agents. Pure Rust, one binary, Apple Silicon and NVIDIA. Anthropic and OpenAI APIs; the KV cache survives across turns, so turn 20 starts as fast as turn 2. OPD training on the same runtime. ### Quick Start: 1. **Install**: - **Apple Silicon (Homebrew)**: `brew install cklxx

### Overview: QilbeeDB is a Rust-based cloud-agnostic high-performance graph database designed for autonomous AI agents, real-time memory, and enterprise automation. ### Key features: - **Scoped Retrieval and Evidence-Driven Memory**: Store and retrieve knowledge with explicit graph relations and evidence validation. - **Durable Agent Memory**: Persistent, authorized, and revision-controlled storage for AI agents. - **Graph-Assisted Retrieval**: Efficiently retrieve knowledge using graph-assisted evidence. - **Context Freshness**: Ensure knowledge is up-to-date with checkpoints and eligibility checks. - **Procedural Knowledge**: Versioned procedures with evaluation outcomes and applicability bound to current evidence. - **Graph Consolidation**: Leased work and fenced publication of externally inferred relations. - **Administration**: Company resource discovery and agent display names. ### Tech stack: Rust ### Why it matters: QilbeeDB provides a robust and efficient solution for storing and retrieving knowledge, making it ideal for applications requiring autonomous AI agents, real-time memory, and enterprise automation.

### Overview: Superfluid is a lightweight, one-served daemon designed to facilitate local LLM inference across different runtimes and machines. It simplifies the interaction with large language models like Qwen3.8-27B, providing a seamless experience without the need for multiple servers or complex orchestration. ### Key Features: - **Single Daemon for Multiple Machines**: One daemon serves multiple machines, ensuring that no agent gets in the way of another. - **Session Persistence**: Sessions persist across crashes and restarts, maintaining the state of the conversation. - **Efficient Token Management**: Prefills are shared across multiple requests, reducing the number of prefill operations. - **Performance**: Measured on an Apple M5 Pro, Superfluid matches the performance of llama-server, Ollama, and MLX_lm.server, with a serving scenario where eight agents can prefill and serve a chat in real-time. ### Tech Stack: Rust ### Why It Matters: - **Immediate Response**: An interactive request preempts batch work, ensuring immediate responses. - **Session Persistence**: Every committed token is in a write-ahead log, preventing data loss. - **Efficient Prefills**: Prefills are shared across multiple requests, reducing the number of prefill operations. ### Use It: - **Call the Server**: Use the server from any API that speaks OpenAI, Anthropic, or Ollama. - **Launch the Server**: Use `superfluid launch` to start the server, and `superfluid node join` to add another machine to the cluster. ### How It

Overview: This project is an implementation of Llama-2 inference in C for the PlayStation Portable (PSP), based on the work by ytmytm and Karpathy's original Llama2.c. Key features: - Llama-2 inference in C on the PSP - Supports TinyStories model with 260K parameters - Includes screenshots of running on a physical device and PPSSPP emulator Tech stack: C++ Why it matters: This project demonstrates the potential of using existing models on alternative hardware platforms, showcasing the versatility of language models across different devices.

### Overview: ZARA is a training-free, evidence-grounded LLM agent framework designed for motion time-series reasoning. It leverages statistically grounded domain knowledge, multi-sensor retrieval-augmented evidence, and hierarchical multi-agent reasoning to classify motion sensor data with transparent, human-readable rationales. ### Key features: - **Training-Free HAR Inference**: ZARA performs human activity recognition without task-specific fine-tuning or classifier training, enabling plug-and-play deployment in parameter-frozen settings. - **Evidence-Grounded Reasoning**: Instead of relying on black-box projections, ZARA grounds LLM decisions in retrieved signal evidence and statistically derived feature priors. - **Pairwise Knowledge Base**: ZARA builds an activity-pair knowledge base that captures discriminative motion properties as interpretable linguistic priors. - **Multi-Sensor Retrieval**: Placement-specific retrieval and rank fusion provide class-balanced, query-relevant evidence across heterogeneous wearable sensors. - **Hierarchical Agentic Workflow**: Specialized agents perform feature selection, evidence pruning, and final decision reasoning to produce concise, evidence-backed rationales. - **Strong Generalization**: ZARA achieves state-of-the-art performance across 8 HAR benchmarks, with robust transfer across unseen subjects and heterogeneous sensor domains. ### Tech stack: Jupyter Notebook ### Why it matters: ZARA is a groundbreaking solution for motion time-series reasoning, offering a training-free, evidence-grounded approach that simplifies the implementation and deployment of LLM agents. Its transparent and interpretable reasoning methods make it highly valuable in various applications where clear and understandable explanations are crucial.

Overview: This project provides AI media generation capabilities for Claude Code, including transcription, image generation, text-to-speech, and OCR. It supports up to 20x cheaper inference via deAPI. Key features: - Transcribe YouTube videos - Generate AI images - Convert text to speech - Extract text from images (OCR) Tech stack: Python Why it matters: This skill allows Claude Code to handle AI media generation tasks efficiently and at a lower cost, making it a valuable tool for developers and content creators.

Overview: This repository contains agent skills for the deAPI platform, allowing developers to transcribe recordings, generate images, create speech, music, and more. The skills are compatible with various agents and can be integrated into applications. Key features: - Transcribe YouTube videos and YouTube links - Generate and edit images - Create speech and music - Read text from images - Remove backgrounds - Upscale and compute embeddings - Verify deAPI webhook signatures Tech stack: Python Why it matters: These skills enable developers to easily integrate AI capabilities into their applications, enhancing functionality and user experience. Repository: deapi-ai/skills GitHub stars: 22 Primary language: Python GitHub description: Agent skills for deAPI: images, video, speech, music, transcription, OCR, embeddings - via native API, MCP or OpenAI-compatible API. Formerly claude-code-skills.

Overview: Semantic AI is an open-source framework designed for Retrieval-Augmented Systems (RAG) that utilizes semantic search to retrieve relevant results and generate human-readable conversational responses using Large Language Models (LLM). Key features: - Semantic search for precise result retrieval - Integration with various connectors and indexers - Support for different LLM models and APIs Tech stack: Python Why it matters: Semantic AI provides a robust solution for developers looking to enhance their applications with semantic search capabilities, improving the accuracy and relevance of their conversational responses.

Overview: This project demonstrates the process of fine-tuning the Flan-T5-Base model for the NVIDIA question-answering task, providing beginners with hands-on experience in the fine-tuning process. Key features: - Fine-tuning of Flan-T5-Base model for NVIDIA question-answering task - Environment setup for Conda and pip - Detailed steps for data preparation, tokenization, model training, and evaluation Tech stack: Jupyter Notebook Why it matters: This project serves as an educational resource for beginners to understand and practice the fine-tuning process of large language models, making it easier to step into the world of Large Language Models.

### Overview: Daydream is an agentic code-review CLI that records each review/fix/test run trajectory, building a corpus for fine-tuning open-weight models. ### Key features: - Records review, fix, and test run trajectories. - Builds a corpus for fine-tuning open-weight models. - Supports multiple backend options: Claude, Codex, Pi, Osprey. ### Tech stack: Python ### Why it matters: Daydream helps researchers fine-tune open-weight models by providing a corpus of code-review trajectories, which can be used to improve the accuracy and efficiency of code reviews.

### Overview: A modular framework for building and deploying Retrieval-Augmented Generation (RAG) systems, providing comprehensive monitoring, evaluation, and customization options. ### Key features: - Seamless extraction from multiple data sources (Confluence, Notion, PDF) - Support for numerous embedding and language models - Efficient vector search using vector stores - User-friendly interactive chat interface - Performance monitoring and evaluation metrics - Easy setup and flexibility ### Tech stack: - Python - LlamaIndex - Chainlit - Langfuse - RAGAS ### Why it matters: This framework simplifies the development of RAG applications, offering complete control over performance, resource usage, and evaluation capabilities, making it suitable for both lightweight knowledge bases and enterprise-grade applications.

Overview: slp is a framework for fast and reproducible development of multimodal models, focusing on NLP. Key features: - Extensive logging and experiment management - Fast development through convention over configuration - Separation of clean and dirty code - Support for PyTorch and PyTorch Lightning - Python 3.8 compatibility Tech stack: Python, PyTorch, PyTorch Lightning, Hugging Face Transformers, Wandb Why it matters: slp provides a structured and efficient way to develop and test NLP models, ensuring reproducibility and ease of use.

Overview: Foresight-RL-Core is a config-driven library designed for fine-tuning language models using reinforcement learning methods. It supports training on local GPUs, AWS SageMaker, and HyperPod. Key features: - Supports GRPO, DPO, and SFT fine-tuning methods - Config-driven approach with YAML files for easy configuration - Runs evaluation, optimization, and load tests after training Tech stack: Python Why it matters: This library simplifies the process of fine-tuning language models using reinforcement learning, making it accessible for developers and researchers. Repository: https://github.com/intuit/foresight-rl-core GitHub stars: 14 GitHub description: A config-driven library for reinforcement-learning fine-tuning.

### Overview: RT-X Net is an RGB-Thermal cross attention network designed for low-light image enhancement. This project introduces a novel approach to improve low-light image quality by leveraging RGB and thermal information. ### Key Features: - **Cross Attention Mechanism:** RT-X Net employs a cross attention mechanism to enhance both RGB and thermal images simultaneously. - **Low-Light Image Enhancement:** The network excels in enhancing low-light images, making them more visible and clearer. - **Real-Time Processing:** The implementation is designed to be efficient and capable of real-time processing. ### Tech Stack: - **Python:** The primary programming language used for the project. - **TensorFlow/Keras:** For implementing the deep learning models. - **PyTorch:** For training the models. ### Why It Matters: RT-X Net offers a significant improvement in low-light image enhancement, making it particularly useful in applications where low-light conditions are common, such as surveillance, night vision, or medical imaging. The ability to enhance both RGB and thermal images simultaneously makes it a versatile solution for various low-light scenarios. ### Repository: jhakrraman/rt-xnet - **Primary Language:** Python - **GitHub Stars:** 22 - **GitHub Description:** Official implementation of RT-X Net: RGB-Thermal cross attention network for Low-Light Image Enhancement ### Model Architecture: The model architecture is illustrated in the provided image, showcasing the cross attention mechanism that integrates both RGB and thermal information. ### Qualitative Results: The project includes qualitative results from various datasets, demonstrating the effectiveness of RT-X Net

Overview: EmergentDB is a self-optimizing vector database that uses MAP-Elites evolutionary algorithm to automatically discover the optimal index configuration for your workload. It is 51x faster than ChromaDB and 82x faster than LanceDB, achieving 100% recall on real embeddings. Key features: - Auto-evolves optimal index configs via Quality Diversity - Stores vectors and finds the most similar ones at blazing speed - 51-82x faster than competitors - 100% recall on real embeddings Tech stack: Rust Why it matters: EmergentDB provides a solution to the challenges of traditional vector databases by automatically optimizing index configurations, achieving maximum search speed while ensuring a 99% recall floor.

### Overview: BaraDB is a highly efficient, native, and zero-dependency multimodal database engine written in Nim. It combines document, graph, vector, columnar, and full-text search storage in a single engine with a unified query language (BaraQL). BaraDB compiles to a single 3.3MB binary with no runtime dependencies, making it highly scalable and performant. ### Key Features: - **Native Multi-Engine Storage**: Supports document, graph, vector, columnar, and full-text search storage. - **Built-in Vector Search**: Uses HNSW/IVF-PQ for vector search. - **Hybrid RAG Search**: Combines vector search with FTS and RRF reranking. - **Graph Algorithms**: Includes BFS, DFS, Dijkstra, PageRank, and Louvain algorithms. - **Graph SQL Integration**: Supports CREATE GRAPH, GRAPH_TABLE(), and SQL-native integration. - **Full-Text Search**: Built-in BM25 + TF-IDF for full-text search. - **Unified Search Engine**: Supports HNSW + inverted index + boolean + phrase + facets + stemmers. - **AI Agents / NL→SQL**: Built-in `nl_to_sql()` and `schema_prompt()` for AI agents. - **MCP Server**: Supports STDIO JSON-RPC for AI tools. - **LangChain Integration**: Native Vector Store (Python + JS) for LangChain integration. - **Embedded Mode**: Supports both SQLite-like embedded mode and distributed mode. ### Tech Stack: Nim ### Why It

### Overview: MIFNet is a novel approach to multimodal image matching that leverages modality-invariant features learned from a pretrained Stable Diffusion model. This method aims to improve the generalization ability of image matching across different modalities, specifically focusing on improving the matching accuracy and robustness. ### Key Features: - **Semantic Features from Stable Diffusion**: Introduces semantic features from Stable Diffusion to enhance multimodal matching. - **Cross-Modal Hybrid Aggregation**: Combines low-level geometric features with high-level semantic guidance to create a hybrid feature representation. - **Lightweight GNN Backbone**: Utilizes a lightweight Graph Neural Network (GNN) to perform semantic-aware feature aggregation, making the model efficient and scalable. ### Tech Stack: - Python ### Why it Matters: MIFNet addresses the challenge of large appearance and texture discrepancies across modalities in multimodal image matching by learning modality-invariant features. This approach demonstrates strong generalization on various unseen multimodal datasets, making it a valuable contribution to the field of computer vision and image processing.