Projects

### Overview: NeMAR is an unsupervised multi-modal image registration method that leverages geometry-preserving image-to-image translation for accurate alignment of images from different modalities. ### Key Features: - **Unsupervised Multi-Modal Image Registration**: NeMAR uses geometry-preserving image-to-image translation to register images from different modalities without requiring paired training data. - **Geometry-Preserving Translation**: The model ensures that the transformed images maintain geometric properties, ensuring accurate alignment. - **Affine and Deformation Field Registration Networks**: Supports both affine-based and deformation field-based registration networks for flexibility. ### Tech Stack: Python ### Why it Matters: NeMAR offers a novel approach to unsupervised multi-modal image registration, providing accurate alignment without the need for paired training data, making it particularly useful in applications where paired data is difficult or impossible to obtain.

Overview: The How-2 dataset is a large multimodal dataset comprising 80,000 instructional videos, 2,000 hours of English subtitles, and Portuguese summaries. It includes audio features, speech summaries, and visual action features for speech recognition and text summarization tasks. Key features: - Audio features (300 hours) - Speech summaries (2000 hours) - Visual action features (2000 hours) - English subtitles - Portuguese summaries - Object grounding features Tech stack: Python Why it matters: The How-2 dataset is crucial for researchers and developers looking to advance multimodal language understanding, speech recognition, and text summarization. It provides a comprehensive resource for training and evaluating models across various tasks. Repository: srvk/how2-dataset Primary language: Python GitHub stars: 193

Overview: Open-Extract simplifies the extraction and processing of unstructured data for AI Agents, allowing seamless integration into workflows using predefined schemas. Key features: - Extract relevant information from documents and websites with a single API call. - Support for multiple schemas and document types. - Built-in caching for rapid repeat extractions. - Vendor lock-in flexibility. Tech stack: Python Why it matters: Open-Extract streamlines the process of extracting structured data, making it easier for developers to integrate AI agents into their workflows and improve the efficiency of their applications.

### Overview: This repository is a comprehensive survey paper that provides an overview of conditional image synthesis techniques using diffusion models. The survey covers various methods and approaches for integrating conditions into the denoising network and the sampling process, highlighting the underlying principles, advantages, and challenges of different conditioning mechanisms. The paper aims to help researchers and practitioners understand the core concepts and core ideas of conditional image synthesis with diffusion models, and to identify critical open problems for future research. ### Key Features: - **Paper Structure**: The survey paper is structured to cover the integration of conditions into the denoising network and the sampling process, with a focus on popular applications. - **Condition Integration in Denoising Networks**: - **Training Stage**: Discusses conditional models for text-to-image (T2I), image restoration, and other synthesis scenarios. - **Re-purposing Stage**: Highlights re-purposed conditional encoders, condition injection, and backbone fine-tuning. - **Specialization Stage**: Focuses on condition injection and testing-time model fine-tuning. - **Condition Integration in the Sampling Process**: - **Inversion**: Introduces the concept of inversion in the sampling process. - **Attention Manipulation**: Explains how attention mechanisms can be manipulated to integrate conditions. - **Noise Blending**: Discusses blending techniques to integrate conditions into the sampling process. ### Tech Stack: The primary language of the repository is unknown. ### Why It Matters: This survey paper provides a valuable resource for researchers and practitioners in the field of conditional image synthesis using diffusion models.

Overview: This repository, named LLMs4OL, is dedicated to the study of large language models (LLMs) for ontology learning (OL). It aims to explore whether the capability of LLMs to capture intricate linguistic relationships can effectively translate to the construction of ontologies, which primarily rely on the automatic extraction and structuring of knowledge from natural language text. Key features: - Task A: Term Typing - Task B: Type Taxonomy Discovery - Task C: Type Non-Taxonomic Relation Extraction Tech stack: Python Why it matters: This repository serves as a platform for researchers and developers to experiment with LLMs for ontology learning, providing a structured framework for tasks and datasets that can be used to evaluate the performance of different models and techniques in this domain.

Overview: AI Movie Studio 2 is a model-agnostic AI filmmaking workstation designed to democratize filmmaking by providing a tool that treats the craft as a fundamental part of the process rather than a budget constraint. It includes 3D storyboard, ComfyUI local + Fal/Replicate cloud, multi-reference continuity, and timeline export capabilities, aiming to replace traditional software like Storyboarder and LTX Studio. Key features: - 3D storyboard - ComfyUI local + Fal/Replicate cloud integration - Multi-reference continuity - Timeline export Tech stack: TypeScript Why it matters: AI Movie Studio 2 aims to bridge the gap between talent and the industry, providing a platform where anyone can bring their vision to life without the need for expensive equipment or industry connections. This tool seeks to empower independent filmmakers by treating filmmaking as a craft rather than a financial burden, making it accessible to a wider audience.

### Overview: ComfyUI-piFlow is a collection of custom nodes for ComfyUI that implement the AsymFlow and pi-Flow model families. These nodes enable the generation of realistic images in pixel space and fast few-step generation, respectively. ### Key features: - **AsymFlow Nodes**: Implement AsymFlow model families for generating realistic images in pixel space. - **pi-Flow Nodes**: Implement pi-Flow model families for fast few-step generation. ### Tech stack: Python ### Why it matters: ComfyUI-piFlow provides a set of nodes and workflows for AsymFlow and pi-Flow, enabling users to leverage the strengths of these models for various applications, including image generation and text-to-image editing. The collection is designed to work seamlessly with the latest ComfyUI version, ensuring compatibility and performance improvements.

## Overview: ThinkDiff is a novel alignment paradigm designed to enable multimodal in-context understanding and reasoning capabilities in diffusion models by integrating the capabilities of vision-language models (VLMs). It leverages the shared input feature space between the large language model (LLM) decoder and diffusion decoders, making it feasible to align VLMs with the LLM decoder instead of the diffusion decoder. This approach simplifies the alignment process and eliminates the need for complex reasoning-based multimodal datasets, resulting in improved performance on challenging benchmarks. ## Key features: - **Multimodal In-Context Reasoning Generation**: ThinkDiff enables the transfer of reasoning capabilities from VLMs to diffusion models by aligning the generated tokens with the diffusion decoders. - **Single Image + Text for Video**: Generates videos by aligning single images with text inputs. - **Single Image + Text**: Aligns single images with text inputs. - **Two Images**: Aligns two images with text inputs. - **Two Images + Text**: Aligns two images with text inputs. ## Tech Stack: ThinkDiff is built using Python, leveraging the Hugging Face framework for model training and inference. ## Why it Matters: By simplifying the alignment process and eliminating the need for complex reasoning-based multimodal datasets, ThinkDiff significantly improves performance on challenging benchmarks, such as the CoBSAT benchmark, where it achieves a notable improvement in accuracy from 19.2% to 46.3% with only 5 hours of training on 4 A100 GPUs.

Overview: DDM2 is an official repository for self-supervised diffusion MRI denoising using generative diffusion models, presented at ICLR 2023. The repository includes detailed configurations, dependencies, and usage instructions for reproducible experiments. Key features: - Configurable training and inference stages for different MRI datasets. - Support for various MRI datasets such as Sherbrooke and Stanford HARDI. - Detailed configuration files and scripts for reproducibility. - Python-based framework for efficient data processing and model training. Tech stack: Python Why it matters: DDM2 provides a comprehensive solution for MRI denoising, offering reproducible results and a user-friendly interface for researchers and developers in the field of medical imaging. Repository: StanfordMIMI/DDM2 GitHub stars: 192 GitHub description: [ICLR2023] Official repository of DDM2: Self-Supervised Diffusion MRI Denoising with Generative Diffusion Models

Overview: VL-Rethinker is an innovative project aimed at enhancing the slow-thinking capabilities of vision-language models through reinforcement learning. This approach is designed to surpass the performance of fast-thinking models by incentivizing self-reflection, leading to significant improvements in various benchmarks such as MathVista, MathVerse, and MathVision. Key features: - **Selective Sample Replay (SSR)**: A technique to address the vanishing advantages problem in reinforcement learning. - **Forced Rethinking**: An approach that explicitly enforces a self-reflection reasoning step to encourage slow-thinking. - **Comprehensive Dataset**: The ViRL39K dataset, covering 39K queries across eight categories with fine-grained model-capability annotations. Tech stack: Python Why it matters: By leveraging reinforcement learning to enhance slow-thinking capabilities, VL-Rethinker aims to bridge the performance gap between vision-language models and fast-thinking systems, potentially leading to breakthroughs in various applications that require nuanced understanding and reflection.

### Overview: RAG SaaS is an end-to-end SaaS solution designed for Retrieval-Augmented Generation (RAG) and Agentic applications, providing developers with a streamlined and efficient way to ship RAG solutions quickly. ### Key Features: - 🔐 Basic Authentication: Securely authenticate users with basic credentials. - 📝 Chat History Tracking: Monitor and track chat history for improved user engagement. - 🧠 Multiple RAG Variations: - Basic RAG - Two additional configurations - 👨‍💼 Admin Dashboard: - Data Ingestion - Monitoring - Observability - RAG Configuration Switching - 📁 S3 Integration for PDF uploads - 🐳 Easy Deployment with Docker / Docker Compose ### Tech Stack: - 🦙 LlamaIndex: For building and orchestrating RAG pipelines - 📦 MongoDB: As both a normal database and a vector database - ⚡ FastAPI: Backend API framework - ⚛️ Next.js: Frontend framework - 🔍 Qdrant: Vector database for efficient similarity search - 👁️ Arize Phoenix: Observability Platform to monitor and evaluate your RAG system ### Why RAG-SaaS? - Setting up reliable RAG systems can be time-consuming and complex. RAG-SaaS simplifies the process, allowing developers to focus on refining and developing their RAG pipeline without worrying about packaging it into a usable application. - Built on top of [create-llama](https://

### Overview: This repository contains a collection of pre-trained models for audio and speech processing. These models have been trained on various datasets and can be used as starting points for similar applications. The models are implemented using different frameworks such as TensorFlow, Keras, PyTorch, and Caffe. ### Key Features: - **TensorFlow**: Models like Wavenet, Lip Reading, MusicGenreClassification, and Audioset. - **Keras**: Models like Ultrasound nerve segmentation. - **PyTorch**: Models like espnet, TTS, Neural Sequence labeling model, waveglow, deepvoice3_pytorch, and deepspeech2. ### Tech Stack: - **TensorFlow** - Implementations of generative neural networks and deep learning models. - **Keras** - Models for ultrasound image segmentation. - **PyTorch** - Implementations of deep learning models for speech processing and text-to-speech. ### Why It Matters: These pre-trained models can be used as a starting point for various audio and speech processing tasks, potentially reducing the time and effort required to build models from scratch. They are valuable resources for researchers and developers looking to leverage existing knowledge and expertise in the field.

Overview: PyAutoSRT is a PySimpleGUI-based desktop application designed to automatically generate subtitle and translated subtitle files for any video or audio file using Google Speech Recognition and Google Translate APIs. Key features: - Auto-generates subtitle and translated subtitle files for videos and audio files - Uses Google Speech Recognition API for subtitles and Google Translate API for translations - Supports multiple languages for subtitles and translations - Compatible with Python 3.10 and above - Supports streamlink module for improved compatibility Tech stack: Python Why it matters: PyAutoSRT simplifies the process of generating subtitles for videos and audio files, making it easier for developers and content creators to translate their content efficiently and quickly.

### Overview: oxidize-pdf is a pure Rust library designed for generating, parsing, modifying, extracting, and validating PDFs. It simplifies PDF workflows without the need for C bindings, making it suitable for AI/RAG applications. ### Key Features: - **Generate PDFs with text, graphics, images, and tables**: Use the `Document` struct. - **Parse PDFs and extract text**: Utilize the `PdfDocument` struct. - **Split, merge, rotate, or reorder pages**: Employ the `operations` module. - **Validate PDF/A or inspect signatures**: Access PDF/A validation and signature inspection. - **Build an AI/RAG ingestion pipeline**: Utilize the `rag_chunks` method for AI/RAG applications. ### Tech Stack: - Pure Rust implementation. ### Why It Matters: - **One toolkit**: Simplifies PDF processing and certificate verification. - **Pure Rust deployment**: No native crypto library, ensuring Rust implementations. - **AI-ready when needed**: Provides structured chunks for AI/RAG applications. - **Explicit scope**: No page rasterization, use a renderer like Pdfium for bitmap output. - **Auditable capability claims**: Records source, version, verification evidence, and limits. ### Structure-aware RAG - **Chunks**: Text, pages, bounding boxes, element types, heading context, token estimates. - **Generated PDFs**: 6,311 one-page documents/sec and 732 ten-page reports/sec. ### Structure-aware RAG Pipeline ```rust use oxidize_pdf::parser::PdfDocument; fn

### Overview: Rust implementation of OpenFST, optimized for performance and ease of use. ### Key features: - Construct, combine, optimize, and search weighted finite-state transducers (FSTs). - Python binding available for easy integration with other Python projects. - Supports deterministic and non-deterministic FSTs. - Optimized for performance, making it suitable for applications like speech recognition and machine translation. - Supports various types of FSTs including acceptors and transducers. - Provides a variety of iterators for easy iteration over state IDs, transitions, paths, and more. ### Tech stack: Rust ### Why it matters: Rustfst is a high-performance library for constructing, optimizing, and searching weighted finite-state transducers, making it ideal for applications requiring fast and efficient string processing and pattern matching.

### Overview: TaiSu is a large-scale Chinese multimodal dataset, containing 166 million high-quality images and corresponding captions. It is designed for large-scale Chinese vision-language pre-training, providing a rich resource for researchers and developers in the field. ### Key Features: - **High-Quality Images and Captions:** TaiSu includes 166 million images with corresponding captions, ensuring high-quality input for training models. - **Text-Based Filtering:** The dataset undergoes text-based filtering to enhance the relevance and quality of the images. - **Image-Text Retrieval-Based Filtering:** Images are filtered using image-text retrieval to further refine the dataset. - **Image-Captioning-Based Text Augmentation:** Captions are generated using image captioning techniques to augment the dataset with more diverse and contextually relevant examples. ### Tech Stack: Python ### Why It Matters: TaiSu offers a comprehensive and high-quality dataset for researchers and developers working on Chinese vision-language pre-training, enabling advanced models to learn more effectively from diverse and contextually rich data.

### Overview: This open-source Slack AI app allows users to summarize threads and channels on demand, providing valuable insights into conversations. It leverages OpenAI models to generate detailed summaries and overviews, making it a powerful tool for managing Slack communities. ### Key Features: - **Thread Summaries**: Generate a detailed summary of any Slack thread using GPT-6 Luna. - **Channel Overviews**: Generate an outline of the channel's purpose based on the extended message history. - **Channel Summaries Since**: Generate a detailed summary of a channel's messages since a given point in time. - **Full Channel Summaries**: Generate a detailed summary of a channel's extended history, powered by GPT-6 Luna. ### Tech Stack: The app is built using Python, leveraging OpenAI models and ensemble NLP techniques. ### Why It Matters: This open-source alternative to Slack AI's premium features offers a flexible and accessible solution for managing Slack communities, providing valuable insights and summaries without the limitations of the official product. ### Repository: The repository is hosted on GitHub and has received 192 stars, indicating its popularity and usefulness in the community. ### Primary Language: The project is primarily written and developed in Python. ### GitHub Stars: The repository has received 192 stars on GitHub, indicating its popularity and usefulness in the community. ### GitHub Description: An open source alternative to some of Slack AI's premium features. Summarizes channels and threads any time you want. ### Table of Contents: - **Getting Started**: Instructions for setting up and running the project. - **

### Overview: GoWhisper is a unified speech-to-text and translation service that provides a single API for multiple providers, including local models using GPU acceleration and commercial models using OpenAI APIs. ### Key Features: - **Command Line Interface**: Downloadable CLI for communicating with the server for audio processing. - **Download and Realtime Transcription**: Transcription and translation with JSON, SRT, VTT, and text output formats. - **HTTP API Server**: RESTful API for transcription and translation service. - **Docker Support**: Pre-built GPU-enabled containers for easy deployment of the service. - **GPU Support**: CUDA, Vulkan, and Metal (macOS) acceleration for local models. - **Model Management**: Download, cache, and manage models locally. ### Tech Stack: Go ### Why it Matters: GoWhisper simplifies the process of speech-to-text and translation by providing a unified API, making it easier to integrate with various applications and services. The use of GPU acceleration and commercial models ensures high performance and accuracy, making it suitable for real-time applications and professional use cases.

Overview: TAgent 是一个基于 Java 17、Spring Boot、Spring AI 和 DDD 分层架构构建的企业级 AI Agent 工程实践项目。它覆盖了从接入、路由、运行时装配、规划执行、RAG、记忆、MCP 工具治理、人工审批、交互通知、执行中干预到 SSE 流式输出和全链路观测的完整过程。 Key features: - 全面覆盖 AI 请求生命周期 - 采用 DDD 架构进行工程化落地 - 支持多种模型连接与密钥管理 - 提供个人工作区、知识库、运行轨迹、产物和反馈功能 - 实现系统总览和功能演示 Tech stack: Java 17, Spring Boot, Spring AI, DDD Why it matters: TAgent 是一个全面覆盖 AI 请求生命周期的企业级工程框架，旨在帮助开发者系统学习 Java/Spring AI Agent 工程化落地，MCP 工具治理，动态工具补挂，Agentic RAG，多层记忆，流式 SSE 输出，全链路观测等完整链路的解决方案。

### Overview: PaperLens is an AI-powered research assistant designed to streamline the process of academic paper discovery, analysis, and citation intelligence. It combines the fragmented nature of academic research with a single, efficient workflow, offering a seamless experience from finding papers to understanding their relevance and citing them effectively. ### Key Features: - **43 Databases**: Searches PubMed, Scopus, ScienceDirect, arXiv, CNKI, and more, ensuring comprehensive coverage. - **300+ Universities**: Accessible via CARSI, no need for a VPN, allowing access to ScienceDirect, Scopus, and JSTOR from anywhere. - **100% Local-First**: All data stays on your machine, running fully offline with Ollama, and no account required. - **AI-Driven**: Automatically extracts key information from papers, understanding research intent beyond keyword matching. - **Multi-Paper Comparison**: Identifies contradictions, research gaps, and complementary approaches between papers. - **Citation Graph**: Reveals connections and emerging research clusters, providing a comprehensive view of the research landscape. ### Tech Stack: - **Python 3.9+**: The primary programming language used for development. - **PaperLens Community License**: Ensures the project is open and accessible to the community. - **46 Data Sources**: Accessible through CARSI, ensuring comprehensive coverage of academic databases. - **AI Models**: Local-first approach, ensuring data stays on your machine and runs fully offline. - **Universities**: 300+ universities via CARSI, providing a wide range