Projects

### Overview: FaithDiff is a powerful tool designed for enhancing classic films, old photos, and social media images. It leverages diffusion priors to achieve faithful image super-resolution, making it a valuable asset in the field of image restoration and enhancement. ### Key Features: - FaithDiff is a versatile tool that can rejuvenate classic films, restore old photos, and enhance social media images. - It supports high-resolution image restoration, including ultra-high-resolution (8K and above) on powerful GPUs. - The project has been integrated into the Diffusers library, making it accessible to researchers and developers. - The repository includes a pre-trained model and testing code, as well as a real-world enhancement dataset of 238 images. ### Tech Stack: Python ### Why it Matters: FaithDiff offers a unique approach to image restoration by leveraging diffusion priors, making it highly effective for enhancing classic films, old photos, and social media images. Its integration into the Diffusers library and availability of a pre-trained model make it a valuable addition to the field of image restoration and enhancement. ### Repository: JyChen9811/FaithDiff ### GitHub Stars: 273 ### To Do: The project is still in development, with plans to create FaithDiff-SD3-Large and release the training and testing codes.

### Overview: The OfflineLLM is a fully offline, private AI chat app for Android that runs any GGUF model locally using llama.cpp. It eliminates the need for network permissions, supports GPU acceleration, and offers advanced features like fast multi-turn chat, performance controls, and streaming responses. ### Key Features: - **100% Offline**: No internet permissions required, no phone home. - **On-Device Inference**: Runs GGUF models via llama.cpp; runtime CPU dispatch picks the best kernel set for your SoC. - **GPU Acceleration (Vulkan)**: Opt-in toggle in settings with per-layer offload control and automatic CPU fallback; detects and names your GPU. - **Fast Multi-Turn Chat**: Incremental prompt processing: only the new message is fed into the KV cache. - **Performance Controls**: CPU thread count, prompt-phase threading, memory-mapped loading, RAM lock, experimental quantized KV cache. - **Streaming Responses**: Token-by-token output as the model generates. - **Import Any Model**: Bring your own GGUF at runtime via file picker. - **Multiple Conversations**: Auto-titled, renameable, searchable. - **Translator**: 75+ languages. - **Advanced Sampling**: Temperature, Top-P, Top-K, Min-P, Repeat Penalty. ### Tech Stack: Kotlin ### Why It Matters: The OfflineLLM is the first Android LLM app that runs inference locally without internet permissions, making it ideal for privacy-conscious users. Its GPU acceleration feature, combined with fast multi-turn chat and advanced sampling,

### Overview: BoxDiff is a text-to-image synthesis technique that leverages training-free box-constrained diffusion for generating high-quality images from textual prompts. This method significantly reduces the need for extensive training, making it highly efficient and suitable for real-time applications. ### Key Features: - **Box-Constrained Diffusion**: Utilizes a box-constrained diffusion process to ensure accurate and detailed image generation. - **Training-Free**: Achieves state-of-the-art results without the need for extensive training, making it highly efficient. - **Versatile Applications**: Can be integrated into various image generation pipelines, including Stable Diffusion, to enhance spatial control and detail. ### Tech Stack: - **Python**: The primary programming language used for implementing and testing the BoxDiff technique. - **Stable Diffusion**: The base model used for generating images, with modifications to incorporate spatial control using BoxDiff. ### Why It Matters: BoxDiff offers a significant improvement in image generation quality and efficiency, making it a valuable tool for applications requiring high-resolution and detailed images from textual prompts. Its training-free nature makes it particularly suitable for real-time or low-resource environments. ### Repository: - **Repository**: showlab/BoxDiff - **GitHub Stars**: 275 - **Description**: [ICCV 2023] BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion ### Integration in Diffusers: - **BoxDiff-SD-XL**: An implementation of BoxDiff based on the SD-XL model. - **Integration**: Thanks to [@zjyst

Overview: TerminalGPT is a terminal-based ChatGPT personal assistant app designed to provide optimized, tailored answers for your machine's terminal. It supports various models and offers features like efficient context management and local storage of conversations. Key features: - Supports multiple models: gpt-3.5-turbo, gpt-4-turbo, gpt-4o, o1, o1-mini, gpt-5-mini. - Tailored answers based on your machine's operating system, distribution, and chip-set architecture. - Does not use your conversation data for training the model. - Local storage of conversations for security and privacy. - Efficient context management for cost-effective API usage. - Available with your IDE terminal, eliminating the need to switch between browser and IDE. - Fast, reliable, lightweight. - Quick one-shot question mode. Tech stack: Python Why it matters: TerminalGPT offers a convenient and secure way to interact with language models, tailored to your specific environment, and provides efficient and reliable assistance without storing your conversation data. Repository: adamyodinsky/TerminalGPT Primary language: Python GitHub stars: 274 GitHub description: TerminalGPT - Terminal-based ChatGPT personal assistant app. Provides optimized, tailored answers for your machine's terminal. README excerpt: # TerminalGPT <!--- [](https://github.com/adamyodinsky/TerminalG

Overview: This repository fine-tunes CLIP models, a popular image-text retrieval model, to improve their performance on specific tasks. Key features: - Fine-tuning CLIP models for improved performance - Utilizes Python for implementation - Optimized for various image-text retrieval tasks Tech stack: Python Why it matters: This project is essential for developers looking to enhance the capabilities of CLIP models for applications requiring high accuracy in image-text matching, such as search engines, virtual assistants, and more.

Overview: Cobra is a lightweight voice activity detection engine designed for on-device applications, offering high accuracy and efficiency. Key features: - Python demo - .NET demo - C demo - Android demo - iOS demo - Web demo - Node.js demo Tech stack: Python Why it matters: Cobra is a crucial tool for developers looking to integrate voice activity detection into their applications, ensuring reliable and accurate voice recognition without the need for cloud-based services.

Overview: This project aims to provide a scene-aware storyboard for DeepSeek Harness, compressing video into a single image with timestamps and transcripts, along with a video zoom tool for detailed analysis. Key features: - Scene-aware storyboard - Video zoom tool - Supports multiple tools: video_probe, video_storyboard, video_zoom, and video_locate Tech stack: JavaScript Why it matters: This project simplifies the process of video analysis by combining visual and textual information into a single, concise representation, making it easier for developers to quickly understand and interpret video content. Repository: SAOS-3624/dsh-video-storyboard GitHub stars: 186 GitHub description: Scene-aware storyboard for DeepSeek Harness: 9-16 numbered cells with timestamps + transcript + an audio envelope, packed into ONE image (384 tokens). Two tools: video_storyboard and video_zoom. Installation: From zero, the complete steps (including dependencies, self-check, and troubleshooting) are provided in [docs/INSTALL.zh.md]. The plugin is not packaged with ffmpeg and yt-dlp, only the installed version is used. The installation guide is provided if the plugin cannot be found. Usage: - To use, simply provide the video URL or path. - For local files, run the script with the video path and desired parameters. Parameters: - Default input is the video path or URL. - Parameters include: from/to, cells, sheets, scanGap, pick, burn, minGap, transcript, detail, cellRange. Output: A single

### Overview: This repository contains the official implementation of "Consistency Flow Matching: Defining Straight Flows with Velocity Consistency," a paper that introduces a novel method called Consistency Flow Matching (Consistency-FM) to generate high-quality samples with fewer function evaluations. The method directly defines straight flows starting from different times to the same endpoint, imposing constraints on their velocity values, and proposes a multi-segment training approach to enhance expressiveness and achieve a better trade-off between sampling quality and speed. ### Key features: - Consistency Flow Matching (Consistency-FM): A novel method for defining straight flows with velocity consistency. - Multi-segment training approach: Enhances expressiveness and improves the trade-off between sampling quality and speed. - Explicit self-consistency in the velocity field: Directly defines straight flows starting from different times to the same endpoint. - Significantly improves training efficiency: Converges 4.4x faster than consistency models and 1.7x faster than rectified flow models while achieving better generation quality. ### Tech stack: Python ### Why it matters: This method offers a more efficient and effective way to generate high-quality samples by reducing the number of function evaluations and improving the trade-off between sampling quality and speed, making it a valuable contribution to the field of flow matching and image generation.

### Overview: Grid is an open-source project that enables you to network your existing computers for AI inference and training. It creates a single, OpenAI-compatible endpoint for all your devices, allowing them to communicate with each other seamlessly without requiring any additional software or accounts. ### Key Features: - **Network Your Computers:** Grid connects your existing computers behind one OpenAI-compatible endpoint, enabling them to communicate with each other for AI tasks. - **Local by Default:** Grid operates in local mode by default, ensuring no additional accounts or relays are required. - **Inference and Training:** Grid can be used for both inference and training, making it versatile for various AI applications. - **Remote Mode:** If you already have Ollama, vLLM, or LM Studio running, Grid can network them together without replacing them. - **CLI Reference:** Detailed command-line interface (CLI) reference for easy integration and management. - **Contributing:** Open-source project, encouraging contributions for further development and improvements. ### Tech Stack: - **Python:** The primary language used for development and configuration. - **OpenAI-compatible Endpoint:** Grid leverages OpenAI's API for seamless integration with AI models. ### Why It Matters: Grid simplifies the process of AI integration by allowing you to leverage your existing hardware for AI tasks without the need for additional software or accounts. This makes it an ideal solution for developers and businesses looking to enhance their AI capabilities without the complexity of setting up new infrastructure.

### Overview: This repository is a comprehensive survey of forging vision foundation models for autonomous driving, focusing on challenges, methodologies, and opportunities. It aims to provide researchers with a quick reference for reading relevant papers and implementing associated methods. ### Key Features: - **Challenges**: Discusses the technical and practical challenges faced in forging vision foundation models for autonomous driving. - **Methodologies**: Outlines various methodologies used to address these challenges, including generative adversarial networks (GANs), diffusion models, NeRFs, and 3D Gaussian splatting. - **Opportunities**: Highlights potential future developments and research directions in the field. ### Tech Stack: Python, Jupyter Notebook, and Markdown. ### Why It Matters: This survey is crucial for researchers and practitioners in the field of autonomous driving, as it provides a comprehensive overview of the current state of the art and potential future directions. It serves as a valuable resource for understanding the complexities and opportunities in the development of vision foundation models for autonomous driving. Repository: zhanghm1995/Forge_VFM4AD GitHub Stars: 273 GitHub Description: A comprehensive survey of forging vision foundation models for autonomous driving, including challenges, methodologies, and opportunities. README Excerpt: This is the partner repository for the survey paper [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://arxiv.org/abs/2401.08045). The repository will be continuously updated to track the progress of forging VFMs for AD. We hope this

Overview: TinkerBird is a client-side vector database designed for efficient storage and retrieval of high-dimensional vectors, leveraging HNSW indexes for fast vector retrieval. It eliminates the need for server-side interactions, benefiting from local storage and reduced reliance on external systems. Key features: - Efficient storage and retrieval of high-dimensional vectors using HNSW indexes - Eliminates the need for server-side interactions, benefiting from local storage and reduced reliance on external systems - Uses IndexedDB as its storage layer, built upon Blobs and LevelDB, with benefits from IndexedDB's adoption, stability, and familiarity as a native choice for offline-first workflows Tech stack: TypeScript Why it matters: TinkerBird is a client-side vector database that simplifies vector search by eliminating the need for server-side interactions, making it ideal for applications requiring local storage and efficient vector retrieval without the associated costs, compliance, and security risks. Repository: 0xnyn/tinkerbird Primary language: TypeScript GitHub stars: 272 GitHub description: Client Side Vector Database

### Overview LongVT is an end-to-end agentic framework designed to enable "Thinking with Long Videos." It addresses the vulnerability of large multimodal models (LMMs) to hallucinations, particularly in processing long-form videos where evidence is sparse and temporally dispersed. Inspired by human reasoning, LongVT first skims the video globally and then examines relevant clips for details, ensuring more accurate and contextually relevant outputs. ### Key Features - **Global Skimming**: LongVT starts by skimming the video to understand the overall context. - **Contextual Clipping**: It then clips relevant parts of the video to focus on specific details. - **Incentivized Tool Calling**: By incentivizing the calling of native tools, LongVT addresses the "Tool Prior Paradox" (Format Fragility + Tool Necessity Gap) of native-RL agentic video reasoning. ### Tech Stack - **Python**: The primary programming language used for implementing and training LongVT. ### Why It Matters LongVT provides a more robust and contextually relevant approach to video reasoning, making it a valuable tool for applications that require accurate and detailed understanding of long-form videos. By addressing the vulnerabilities of LMMs, it opens up new possibilities for leveraging these models in complex and challenging scenarios. ### GitHub Repository The repository for LongVT is hosted on GitHub at [EvolvingLMMs-Lab/LongVT](https://github.com/EvolvingLMMs-Lab/LongVT). It has been starred by 272 developers, indicating its growing interest and adoption

Overview: This project introduces a novel approach called Min-SNR Weighting Strategy (Min-SNR-$\gamma$) for efficient diffusion training, addressing slow convergence issues in image generation models. Key features: - Min-SNR-$\gamma$ loss weights timesteps based on clamped signal-to-noise ratios, balancing optimization conflicts. - Significantly improves convergence speed by 3.4x compared to previous weighting strategies. - Achieves a new record FID score of 2.06 on ImageNet 256x256 benchmark using smaller architectures. Tech stack: Python Why it matters: This method offers a practical solution to the slow convergence problem in diffusion models, enabling faster training and better performance on various image generation tasks.

Overview: AthenaDB is a simple, serverless, distributed vector database that can be used as an API, written with Cloudflare Workers, D1, and Vectorize. It provides straightforward endpoints for various database operations, ensuring high availability and scalability. Key features: - Simple API Endpoints - Distributed Nature - Built-In Data Replication - Scalability - Serverless Architecture Tech stack: TypeScript Why it matters: AthenaDB simplifies vector database management, making it accessible and efficient for developers of all skill levels, while ensuring high availability and scalability through automatic data replication and serverless architecture.

### Overview: This project involves training a language model (LLM) using Group Relative Policy Optimization (GRPO) to create schedules from a list of events and priorities. The model aims to maximize the total duration of selected events, with higher priority events receiving a weight of 2. ### Key features: - Train a language model using GRPO to generate schedules. - Use events with priorities and durations to create a schedule. - The model is trained to optimize the schedule based on the provided priorities and events. ### Tech stack: Jupyter Notebook ### Why it matters: This experiment explores the potential of LLMs in solving complex scheduling problems without explicit examples, focusing on the effectiveness of GRPO in generating schedules that maximize priority events' durations.

### Overview: Lighthouse is a user-friendly library designed for reproducible video moment retrieval and highlight detection. It supports seven models, four features (video and audio features), and six datasets for various applications. The library also includes an inference API and a Gradio demo for developers to easily integrate state-of-the-art methods into their projects. ### Key Features: - **Video Moment Retrieval (MR-HD)**: Supports seven models and four features (video and audio). - **Audio Moment Retrieval (AMR)**: A task to identify relevant moments from audio inputs based on text queries. - **Inference API**: Available for both CPU and GPU environments, including support for the Hugging Face wrappers for the Clotho dataset. - **Gradio Demo**: Provides a user-friendly interface for developers to test and integrate the library’s features. ### Tech Stack: - Python 3.9 - PyTorch 2.1.0 - TorchVision 0.16.0 - Torchaudio 2.1.0 ### Why It Matters: Lighthouse simplifies the process of integrating advanced moment retrieval and highlight detection techniques into projects, making it accessible to developers without deep technical knowledge. The library's user-friendly interface and support for various datasets and models make it an ideal choice for researchers and developers looking to enhance their projects with cutting-edge moment retrieval techniques.

### Overview: DashInfer is a high-performance LLM inference engine optimized for various hardware architectures, including CUDA, x86, and ARMv9. It aims to deliver industry-leading performance with minimal dependencies and efficient integration. ### Key Features: - **Lightweight Architecture**: Minimal third-party dependencies and static linking for C++ and Python interfaces. - **High Precision**: Rigorous testing ensures consistent accuracy with PyTorch and GPU engines. - **High Performance**: Optimized kernels for high-performance LLM serving and standard inference techniques. - **Continuous Batching**: Supports immediate insertion of new requests and streaming outputs. - **Paged Attention**: Efficient acceleration of attention operator using paged attention and KV cache quantization. - **Prefix Cache**: Highly efficient Prefix Cache for prompts, accelerating standard LLMs and MMLMs. - **Quantization Support**: Weight-only quantization acceleration without fine-tuning, improving deployment efficiency. - **Asynchronous Interface**: Request-based asynchronous interfaces for individual control over generation parameters and request status. ### Tech Stack: C++ ### Why it Matters: DashInfer is designed to deliver production-level implementations optimized for various hardware architectures, ensuring high performance and minimal dependencies, making it suitable for integration into existing systems and serving with fastChat for OpenAI-compatible API servers.

Overview: The repository "awesome-large-action-model" contains a collection of models designed to facilitate large language models in their tasks, offering a range of tools and resources for developers. Key features: - CogAgent: A visual language model for GUI agents. - Gorilla: A large language model connected with massive APIs. - ToolLLM: Facilitating large language models to master 16,000+ real-world APIs. - Empowering LLM to use smartphones for intelligent task automation. - MetaTool Benchmark: Deciding whether to use tools and which to use. - T-Eval: Evaluating the tool utilization capability step by step. Tech stack: The repository utilizes various open-source tools and models, including CogVLM, Gorilla, ToolBench, ToolLLaMA-2-7b-v1, and MetaTool. Why it matters: This repository provides a comprehensive set of models and tools that can help developers enhance the functionality and efficiency of large language models, making them more versatile and effective in real-world applications.

Overview: VectorDBZ is an open-source desktop application designed for exploring, managing, and analyzing vector databases. It allows users to connect to local or cloud instances, explore collections, run vector and hybrid searches, and visualize embeddings in 2D/3D. Key features: - Vector and hybrid search capabilities - Native desktop app for seamless integration - Supports multiple vector databases including Qdrant, Weaviate, Milvus, ChromaDB, Pinecone, PostgreSQL, Elasticsearch, RedisSearch, and Ahnlich - Includes a step-by-step guide for adding new database integrations Tech stack: TypeScript Why it matters: VectorDBZ simplifies the process of managing and analyzing vector databases, making it easier for developers to integrate and utilize vector data in their projects. Repository: vectordbz/vectordbz GitHub stars: 271 GitHub description: A modern desktop application for exploring, managing, and analyzing vector databases

Overview: This library provides a universal format for vector datasets, enabling seamless export, import, backup, re-embedding, and access of vector data across various vector databases and repositories. Key features: - Comprehensive vector data tooling - Universal interface for vector databases, datasets, and RAG platforms - Easy export and import of data - Support for multiple vector databases including Pinecone, Qdrant, Milvus, and more - Import and export options for Azure AI Search, Weaviate, MongoDB Atlas, OpenSearch, Apache Cassandra, SQLite-VSS, and pgvector - In progress support for Azure AI Search, Weaviate, Redis Search, ClickHouse, and Rockset - Not supported by Vespa, Marqo, Elasticsearch, Redis Search, ClickHouse, USearch, Rockset, Epsilla, Activeloop Deep Lake, ApertureDB, CrateDB, Meilisearch, MyScale, Neo4j, Nuclia DB, OramaSearch, Typesense, Anari AI, Vald, and Apache Tech stack: Jupyter Notebook Why it matters: This library simplifies the process of working with vector data across different platforms and databases, making it easier for developers to integrate vector datasets into their projects and workflows.