Projects

### Overview: This project introduces SPO (Step-by-Step Preference Optimization), a method for improving aesthetics in post-training diffusion models. The paper proposes a novel approach that discards the propagation strategy of existing DPO methods, focusing on fine-grained image details and fine-tuning the diffusion model to enhance aesthetics without sacrificing image-text alignment. ### Key Features: - **Step-by-Step Preference Optimization (SPO):** Discards the propagation strategy of DPO, focusing on fine-grained image details. At each denoising step, it samples a pool of candidates, finds a suitable win-lose pair to supervise the diffusion model, and randomly selects one to initialize the next step. - **Fine-Grained Visual Differences:** Ensures that diffusion models focus on subtle, fine-grained visual differences rather than layout aspects. - **Aesthetic Enhancement:** Significantly enhances aesthetics by accumulating improved minor differences. ### Tech Stack: - Python ### Why It Matters: - Economically improves aesthetics without sacrificing image-text alignment compared to vanilla models. - Converges faster due to more accurate preference labels provided by the step-aware preference model. ### Repository: - Repository: [RockeyCoss/SPO](https://github.com/RockeyCoss/SPO) - GitHub Stars: 271 - GitHub Description: [CVPR 2025] Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization ### Gallery: -  - ![teaser example 0](assets/teaser/

Overview: Openjourney is an open-source clone of the MidJourney web interface, featuring real AI image and video generation powered by Google's Gemini SDK. It includes Imagen 4 for image generation, Veo 2 and 3 for video creation, and supports interactive features like download, image-to-video conversion, and professional animations. Key features: - **AI Image Generation**: Integration with Imagen 4 for high-quality image generation. - **AI Video Generation**: Veo 2 and 3 for image-to-video conversion with text-to-video functionality. - **Interactive Features**: Download, image-to-video conversion, and professional animations. Tech stack: TypeScript, Next.js 15, Tailwind CSS v4, Framer Motion, ShadCN UI, Google GenAI SDK, Radix UI. Why it matters: Openjourney provides a high-fidelity, user-friendly interface for generating images and videos using AI, making it accessible and powerful for developers and designers alike. Repository: ammaarreshi/openjourney Primary language: TypeScript GitHub stars: 270 GitHub description: Open-source clone of the MidJourney web interface featuring real AI image and video generation powered by Google's Gemini SDK. Use Imagen 4 to generate images and Veo 2 and 3 for image and text to video with audio.

### Overview: AI-WEBUI is a universal web interface designed to facilitate AI creation across various media types, including image, audio, and video processing. This project offers basic functionalities such as image segmentation, object tracking, image restoration, speech recognition, speech synthesis, and advanced features like chatbot, video translation, and watermark removal, significantly enhancing the efficiency of short video creation. ### Key features: - Image Segmentation - Object Tracking - Image Restoration - Speech Recognition - Speech Synthesis - Video Translation - Video Watermark Removal ### Tech stack: Jupyter Notebook ### Why it matters: AI-WEBUI provides a user-friendly interface for creating and processing AI-generated content, making it a valuable tool for both beginners and professionals in the field of AI and video editing.

### Overview: YourMemory is an AI memory tool that leverages the principles of human memory to improve recall and retention. Unlike traditional memory systems, YourMemory works by consolidating related facts into clean summaries, using a biological forgetting curve to prioritize important information, and linking memories through shared entities to provide contextually relevant recall. ### Key Features: - **Consolidation**: When enough related facts accumulate, they're compressed into one clean summary, making memory sharper over time. - **Biological Decay**: Every memory ages on an Ebbinghaus forgetting curve, with stale, unused facts fading away, while important and frequently recalled ones persist. - **Entity Graph**: Memories link by shared people, places, and concepts, providing contextually relevant recall. - **Survives Context Resets**: When the context window compacts, YourMemory hands the working context back, no re-reading files to figure out where you were. - **Tamper-Evident Audit Trail**: Every read / write / delete is logged in a hash-chained ledger, ensuring alterations can be traced. - **Team Memory Pools**: Role-based shared memory, allowing a whole team's agents to draw on the same institutional knowledge, with private memories kept private. - **Data Rights Built In**: Right to access and right-to-forget commands, with SOC 2-aligned controls. - **MCP Native & Local First**: Works with Claude, Cursor, Cline, Windsurf, or any MCP client, running entirely on your machine with no API key or system dependencies. ### Tech Stack: Python ### Why It Matters:

Overview: This project aims to compare three self-supervised models, Wav2vec, HuBERT, and WavLM, for phoneme recognition across various languages. By leveraging pretrained models on English speech, it explores the impact of language choice, feature extraction, and model training data on phoneme recognition performance. Key features: - Modularity between state-of-the-art models in self-supervised speech processing - Ability to select any language from CommonVoice hosted on HuggingFace - Visualization tool using W&B for better understanding of models Tech stack: Python, PyTorch, PyTorch Lightning, HuggingFace Why it matters: This project addresses the challenges of limited annotated data in speech processing by exploring the effectiveness of different self-supervised models and their ability to transfer knowledge across languages. It provides valuable insights into the performance of these models and helps in improving phoneme recognition for various languages.

### Overview: ASMR Dubber 是一个多功能音频处理工具，能够将音频文件中的原声转换为另一种语言，同时保持音色和空间跟随效果。它支持日语、英语和中文等多种语言的音频和视频处理，适用于配音和字幕制作。用户可以轻松地将音频文件转换为双语版本、替换版、配音轨、翻译字幕，甚至制作视频。此外，它还具有空间跟随、人声分离和音色克隆等功能，使得处理过程更加灵活和高效。 ### Key features: - **双语配音与字幕**：将音频文件中的原声转换为另一种语言，保持音色和空间跟随效果。 - **RTF空间处理**：处理音频文件中的空间线索，使原声和配音位置保持一致。 - **人声分离**：分离音频文件中的原声和背景，保留背景音效和配音，制作“替换版”。 - **音色克隆**：从原声中选择一句作为参考，配音则使用这个声音说另一种语言。 - **逐句修改**：识别错误时可以修改原文，翻译错误时可以修改翻译，只影响当前项目。 - **批量处理**：将项目文件夹中的所有音频文件处理完毕，生成一个文件或视频。 - **字幕导出**：无需识别，直接导出字幕；无需配音，只导出字幕。 - **参数开放**：200多个参数，需要时展开，平时折叠。 ### Tech stack

Overview: Cevahir AI is an open-source project designed to combine tokenizer infrastructure, a trainable language model core, and cognitive systems, all built specifically for Turkish language. It includes a BPE pipeline, a configurable PyTorch Transformer decoder, training and checkpoint management, generation, cognitive workflows, conversational memory, and application services. Key features: - A trainable language model core - A configurable PyTorch Transformer decoder - Tokenizer infrastructure for Turkish - Training and checkpoint management - Generation and cognitive workflows - Conversational memory - Application services Tech stack: Python Why it matters: Cevahir AI provides a comprehensive solution for building language models, making it accessible to developers and educational institutions. It offers a solid foundation for understanding and developing language representations, model learning, and model use in a single extensible system.

### Overview: Statewave is an open-source memory runtime designed for AI agents, providing reproducible and provenance-tagged context instead of query-time retrieval. It offers a structured and durable memory system that ensures context is retained across sessions and decisions. ### Key Features: - **Compiled Context Bundles**: Ranked, token-bounded, and deterministic per subject and task. - **Provenance Tagging**: Every memory traces back to its source episodes, providing clear lineage. - **Self-Hosted on PostgreSQL + pgvector**: Runs on self-hosted PostgreSQL with pgvector support, ensuring data integrity and scalability. - **Python + TypeScript SDKs**: Provides a Python and TypeScript SDK for integration and development. ### Tech Stack: - Python - PostgreSQL + pgvector ### Why It Matters: Statewave offers a robust and reliable memory management system for AI agents, ensuring that context is preserved and retrievable with high fidelity, making it an essential tool for developers looking to build more robust and user-friendly AI applications. Repository: smaramwbc/statewave Primary Language: Python GitHub Stars: 270 GitHub Description: Open-source memory runtime for AI agents — reproducible, provenance-tagged context bundles instead of query-time retrieval. Apache-2.0, self-hosted on Postgres + pgvector, Python + TypeScript SDKs.

### Overview: Jarvis is a fully functional voice-based natural language interface that empowers developers to create intelligent applications through its robust API. ### Key Features: - **Voice Activation**: Responds to voice commands for quick access to information and tasks. - **Natural Language Processing**: Understands and executes commands in a human-like manner. - **API Integration**: Extensive API support for integrating with various backend services. ### Tech Stack: - **Python** - The primary programming language used for development. ### Why It Matters: Jarvis simplifies the development of voice-based applications, making it easier for developers to create intelligent and user-friendly interfaces without needing extensive knowledge of natural language processing or machine learning. ### Repository: Thevickypedia/Jarvis - **Stars**: 270 - **Language**: Python - **Description**: Fully Functional Voice Based Natural Language UI ### Platform Support: - **MacOS** - Tested on Mojave, Catalina, Big Sur, Monterey, and Ventura ### Language Stats: - **Code Coverage**: 100% - **Code Size**: 100 KB ### Deployments: - **Pypi**: Downloads, Formats, Status, and Pages - **Actions**: PyPI, Resolver, CodeQL, Release Notes, Pages, Stale ### Code Stats: - **File Count**: 100 - **Code Size**: 100 KB - **Code Line**: 100 ### Activity: - **Created**: 2023-01-01

### Overview: This repository is a comprehensive survey of state space models in medical image analysis, focusing on their applications and advancements in the computation-efficient era. ### Key Features: - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **State Space Models for Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) - **Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis** (arXiv:2406.03430) -

### Overview: This repository contains a collection of recent diffusion models for various text-image generation tasks, including document restoration, font generation, visual text generation, text removal, image super resolution, text editing, and more. ### Key features: - **Document Restoration**: Predicts the original appearance of damaged historical documents. - **Font Generation**: Models for high-quality and wide-coverage CJK character generation. - **Visual Text Generation**: Automatic generation of complex glyph posters with diffusion models and large language models. - **Text-to-Image (Visual Text Generation)**: Accurate multilingual visual text rendering. - **Text Removal**: Removes text from images. - **Image Super Resolution**: Enhances text image quality. - **Text Editing**: Edits text in images. - **Inpainting**: Reconstructs missing parts of images. - **Handwritten Generation**: Generates handwritten text. - **Scene Text Recognition**: Recognizes text in scenes. - **Scene Text Detection**: Detects text in scenes. ### Tech stack: Python, PyTorch, Hugging Face Transformers, and custom diffusion models. ### Why it matters: This repository provides a comprehensive collection of state-of-the-art diffusion models for various text-image generation tasks, making it a valuable resource for researchers and developers looking to improve their text-image generation capabilities.

### Overview: LLM Zoo is a project dedicated to collecting information about various open- and closed-source Large Language Models (LLMs). It serves as a comprehensive resource for developers and researchers interested in NLP. ### Key Features: - Release time - Model size - Languages supported - Domain - Training data - Links to resources: GitHub, HuggingFace, Demo, Paper, Official blog ### Tech Stack: Python, Markdown, Git ### Why it Matters: LLM Zoo provides a centralized repository for tracking the latest developments in the field of language models, ensuring that developers have access to the most up-to-date and comprehensive information.

### Overview: LLaVAR is an enhanced visual instruction tuning model designed for text-rich image understanding. It builds upon the existing LLaVA model to improve performance in tasks involving textual instructions and rich visual content. ### Key Features: - **Improved Visual Instruction Tuning**: Enhances the model's ability to interpret and respond to textual instructions in complex image scenarios. - **Text-Rich Image Understanding**: Specifically designed for tasks requiring both textual and visual information. - **Versatile Training Data**: Utilizes pretraining and finetuning datasets from LAION, providing robust training for various image and instruction types. ### Tech Stack: - **Python**: The primary programming language used for developing and training the model. ### Why It Matters: LLaVAR significantly enhances the model's capability to understand and respond to textual instructions in image-based tasks, making it particularly useful in applications requiring detailed visual understanding and textual guidance. ### Repository: - **Repository**: SALT-NLP/LLaVAR - **Stars**: 269 - **Description**: Code/Data for the paper "LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding" - **Arxiv Link**: https://arxiv.org/abs/2306.17107 ### Environment Setup: - **Model Weight Delta**: Available on Google Drive and Huggingface. Merge with LLaVA's model weight. - **Training Data**: Contains pretraining and finetuning datasets from LAION, including images and instructions. - **Evaluation Data**: 50 instruction-following questions and

Overview: Aegis is a self-hardening firewall designed specifically for large language models to protect against adversarial attacks such as prompt injections, prompt leakage, and toxic language. Key features: - Prompt injection detection - Toxic language detection - PII detection - Attack signature learning Tech stack: Python Why it matters: Aegis provides a robust solution for safeguarding large language models from potential security threats, ensuring the integrity and reliability of model outputs.

### Overview: Intelligent Grimm is an open-ended visual storytelling project that leverages latent diffusion models for generating dynamic and interactive narratives. This repository provides a PyTorch implementation of the StoryGen framework, which was presented at CVPR 2024. It offers a comprehensive set of tools and data for creating engaging and interactive visual stories. ### Key Features: - **StoryGen Framework**: A PyTorch implementation of the StoryGen framework, which is designed for generating open-ended visual stories. - **Latent Diffusion Models**: Utilizes diffusion models to generate dynamic and interactive narratives. - **Dataset**: Includes a dataset of keyframes from YouTube videos, metadata, and corresponding masks and descriptions. - **Data Processing Pipeline**: A detailed data processing pipeline that includes extracting keyframes, removing duplicates, segmenting text and people, inpainting images, and generating paired image-text samples. - **GitHub Repository**: The repository contains the official implementation of the project, including data, scripts, and documentation. ### Tech Stack: - Python 3.8 or higher - PyTorch 1.12 or higher - xformers, diffusers, accelerate, transformers ### Why It Matters: Intelligent Grimm offers a powerful tool for creating interactive and dynamic visual stories, making it a valuable resource for anyone interested in the field of visual storytelling and artificial intelligence. The project's open-source nature and comprehensive documentation make it accessible and easy to use, making it a significant contribution to the field of visual storytelling.

Overview: Phosphene is a free, open-source app for Apple Silicon Macs that generates AI videos, sounds, and images, leveraging LTX-2.5, Hailuo H3, and Qwen-Image-Edit. It allows users to create short videos, scenes, music, and images with local processing and offline capabilities. Key features: - Video with LTX-2.5: Turn text or a picture into video with sound. - Scenes with Hailuo H3: Create scenes with dialogue. - Music Studio: Generate songs with vocal training. - Images: Edit images with words. - Storyboard, One Shot, and Editor: Build films from prompts or shots. - Train a character: Create a face and voice LoRA. - Scripts and agents: Queue renders through the local HTTP API. - Prompt help: Enhance with Google's Gemma. - Hardware: Requires Apple Silicon M1 or newer, 16-64 GB RAM. Tech stack: Python, MLX by Mr Bizarro. Why it matters: Phosphene offers a powerful and free solution for creating engaging videos, music, and images on Apple Silicon Macs, making it ideal for creators and content producers looking to enhance their projects locally without the need for cloud-based solutions. Repository: mrbizarro/Phosphene GitHub stars: 269 GitHub description: Free, open-source app for Apple Silicon Macs: local AI video with sound, songs and images. Includes LTX-2.5 and Hailuo H3.

### Overview: AutoTimes is an official implementation that converts Large Language Models (LLMs) into autoregressive time series forecasters. Unlike traditional methods, it can handle arbitrary-length lookback and predictions, making it versatile and adaptable to various time series forecasting tasks. ### Key features: - **Time Series Forecasting**: Converts LLMs into autoregressive time series forecasters, enabling multi-step forecasting and accommodating arbitrary-length lookback and predictions. - **Zero-Shot Forecasting**: Demonstrates good performance without the need for downstream samples, showcasing the effectiveness of LLMs for time series forecasting. - **In-Context Forecasting**: Proposes in-context forecasting, where time series prompts can be incorporated into the input context to enhance forecasting accuracy. - **Easy-to-Use**: Compatible with any decoder-only LLMs, demonstrating generality and proper scaling behavior. ### Tech stack: Python ### Why it matters: AutoTimes leverages the inherent autoregressive property of LLMs to fully revitalize them as foundational models for time series forecasting, achieving state-of-the-art performance with 0.1% trainable parameters and over 5x training/inference speedup compared to advanced LLM-based forecasters.

Overview: LynnReal-Omni is a unified framework for generating videos across various tasks, including text-to-video, image-to-video, human and hand pose guided generation, structural control, omni-reference generation, style transfer, video editing, degraded-video restoration, and streaming long-video generation. It achieves four-step fast generation and accepts heterogeneous inputs. Key features: - Text-to-video - Image-to-video - Human and hand pose guided generation - Structural control - Omni-reference generation - Style transfer - Video editing - Degraded-video restoration - Streaming long-video generation Tech stack: Python Why it matters: LynnReal-Omni simplifies video generation by integrating multiple tasks into a single framework, making it easier to create videos for various applications, such as game recordings, 3D renders, and more.

### Overview: X-Dyna is a zero-shot, diffusion-based pipeline for generating expressive dynamic human image animations. It leverages facial expressions and body movements from a driving video to create realistic, context-aware dynamics. ### Key Features: - **Dynamics Adapter**: A lightweight module that integrates reference appearance context into the diffusion backbone while preserving motion synthesis capabilities. - **S-Face ControlNet**: A local control module that captures identity-disentangled facial expressions for accurate expression transfer. - **Temporal Consistency**: Ensures temporal coherence in the generated dynamics. ### Tech Stack: PyTorch ### Why It Matters: X-Dyna offers a novel approach to human image animation by integrating facial expressions and body movements from a driving video, resulting in lifelike and context-aware dynamics. This method addresses the loss of dynamic details in traditional human pose control approaches, enhancing the realism of animations.

### Overview: FeedGen is an open-source tool that optimizes product feeds by leveraging Generative AI models to improve titles and descriptions, addressing quality issues in feeds. ### Key Features: - **Generative AI Model Integration**: FeedGen utilizes Google Cloud's Vertex AI models to enhance product titles and descriptions. - **Zero-Shot and Few-Shot Inference**: The tool supports both zero-shot and few-shot prompting to adapt to specific feed needs. - **Easy Configuration**: Users can customize the model parameters and settings to suit their specific needs. - **Scalability**: The tool is designed to handle up to 30,000 items efficiently, making it suitable for both small and large-scale operations. ### Tech Stack: TypeScript ### Why it Matters: FeedGen simplifies the process of optimizing product feeds by leveraging Generative AI, making it easier for merchants and advertisers to surface and fix quality issues in their feeds.