Projects

Overview: This repository contains code for multimodal (visual) instruction tuning of the Llama 3 language model, fine-tuning it on a dataset of textual instructions and image URIs. Key features: - Fine-tuning of Llama 3 model on a multimodal dataset - Uses Hugging Face Transformers library for training - Provides a script to download and prepare the dataset - Saves the fine-tuned model in the output directory Tech stack: Python Why it matters: This repository offers an easy-to-use starting point for multimodal instruction tuning, making it accessible for developers looking to fine-tune their own Llama 3 models on multimodal datasets.

Overview: A cutting-edge live English speech recognition system running on the ESP32-P4 microcontroller, featuring a 65.7M-parameter Kroko ASR model optimized for streaming and 2/4-bit weights, achieving 4.34% WER on LibriSpeech test-clean and 10.26% WER on test-other with beam search, all while processing at 0.70× real time. Key features: - Live English speech recognition - 65.7M-parameter Kroko ASR model - 2/4-bit weights optimized for streaming - 26.65 MB weights on Hugging Face - 0.70× real time processing - 2 s delay per word - 32 MB PSRAM and 121 KB internal RAM - Bit-exact computation - Open vocabulary - Six voice-control demos Tech stack: C++ Why it matters: This project represents a significant advancement in real-time speech recognition, enabling hands-free control of devices using the ESP32-P4 microcontroller, making it ideal for IoT applications and voice-activated assistants. Repository: [BoumedineBillal/esp32-pie-asr](https://github.com/BoumedineBillal/esp32-pie-asr) GitHub stars: 49

Overview: DGX Monarch is a tool that allows developers to generate images and videos on two NVIDIA DGX Sparks from a single ComfyUI window, enabling faster rendering times, parallel processing, and efficient model management. Key features: - Distributed image and video generation - Split render between GPUs - Run separate prompts at once - Shard supported model weights to fit larger checkpoints Tech stack: Python Why it matters: DGX Monarch simplifies the process of managing and rendering models on multiple GPUs, allowing developers to work more efficiently and effectively in their projects. Repository: Deen-Media/dgx-monarch GitHub stars: 51 GitHub description: Distributed image and video generation in ComfyUI for two NVIDIA DGX Sparks.

Overview: DeployQL's LintDB is a vector database designed for Gen AI, supporting late interaction and token-level embeddings. Key features: - Multi-vector support - Bit-level compression - Embedded functionality - Full support for PLAID and ColBERT - Filtering capabilities Tech stack: C++ Why it matters: LintDB simplifies vector database management, making it easy to integrate with Python applications and supports advanced retrieval models like ColBERTv2. Repository: DeployQL/LintDB GitHub stars: 54

Overview: 轻语输入是一款专注于中文语音输入的工具，旨在通过按下快捷键说话，将用户的口语整理成结构化、去口头语的文本，并自动插入到当前光标位置。它支持用户词典、中文转英文和自选 ASR/LLM，适用于多种场景，如日常沟通、工作任务、英文输出等。 Key features: - 中文语音输入 - 低延迟链路 - 轻度润色 - 清晰结构 - 正式表达 - 格式规范 - 用户词典 - 历史记录 - 剪贴板兜底 Tech stack: Rust Why it matters: 轻语输入为需要快速获得可直接使用的文字的用户提供了便捷的解决方案，特别适用于需要在聊天、写作、工作说明、任务拆解或英文输出中快速获取文字的场景。 Repository: EthanYoQ/whisper-input Primary language: Rust GitHub stars: 54 GitHub description: 中文 AI 语音输入工具：按快捷键说话，在任意应用把口述变成去口头语、可分段和结构化的文字；支持用户词典、中文转英文和自选 ASR/LLM。

### Overview: This project is a codebase for the paper "Patched Diffusion Models for Unsupervised Anomaly Detection" presented at MIDL23. It focuses on generative models for creating anatomically consistent MRI scans of healthy brains, aiming to improve unsupervised anomaly detection in brain MRI scans. ### Key features: - **Generative Models for Anatomically Consistent MRI Scans**: The project proposes a method to reformulate the generation task of diffusion models as a patch-based estimation of healthy brain anatomy. - **Unsupervised Anomaly Detection**: The approach uses spatial context to guide and improve reconstruction, offering a relative improvement of 25.1% compared to existing baselines. - **Preprocessed Data Sets**: The project utilizes the IXI, BraTS21, and MSLUB data sets for experiments, applying several preprocessing steps including resampling, skull stripping, registration, and N4 Bias correction. ### Tech stack: The project is written in Python. ### Why it matters: This project offers a novel solution for unsupervised anomaly detection in brain MRI scans, improving performance and eliminating the need for patching, making it a significant contribution to the field of medical imaging and machine learning.

### Overview: BenchBoard is a comprehensive source-linked collection of published AI model benchmark scores, providing detailed results for current AI models. It includes every score with its model release, benchmark variant, evaluation setting, and original source attached, ensuring accuracy and reliability. ### Key Features: - **Model Scores**: Indexed results for various AI models, including scores from different benchmarks and versions. - **Source Coverage**: Every score retains its original source, ensuring data integrity. - **Cross-Benchmark Separation**: Results from different harnesses or versions remain separate, preventing cross-benchmark composite scores. ### Tech Stack: JavaScript ### Why It Matters: BenchBoard serves as a reliable resource for developers and researchers, providing a clear and comprehensive view of AI model performance across various benchmarks and settings.

### Overview: ML4CO-Bench-101 is a comprehensive benchmark framework designed to evaluate and compare neural combinatorial optimization (NCO) methods. It categorizes these methods based on their paradigms, model designs, and learning strategies, providing a unified and transparent evaluation protocol for a wide range of combinatorial optimization problems. ### Key Features: - **Unified Benchmarking:** ML4CO-Bench-101 categorizes NCO solvers into three unified solvers: Global Prediction (GP), Local Construction (LC), and Adaptive Expansion (AE) mannered neural solvers. - **Comprehensive Datasets:** It includes 34 datasets across various scales for 7 mainstream combinatorial optimization problems (TSP, ATSP, CVRP, MIS, MCl, MVC, MCut). - **Modular Framework:** The framework is modular and streamlined, providing a structured way to evaluate the performance of different NCO methods. - **Transparent Evaluation:** ML4CO-Bench-101 evaluates the applicability and generalization of different NCO approaches across a broad range of problems, offering universal insights that can be transferred across various domains. ### Tech Stack: Python ### Why it Matters: Under this unified benchmark, an up-to-date replication of typical ML4CO methods is maintained, providing a convenient reference and insightful guidelines for both engineering development and academic exploration of the ML4CO community in the future.

Overview: DisC-Diff is a multi-contrast brain MRI super-resolution method based on denoising diffusion probabilistic models, designed to improve model interpretation under various conditions of multi-contrast inputs. Key features: - Disentangled multi-stream network - Improves model interpretation - Validates effectiveness on IXI and clinical datasets Tech stack: Python Why it matters: DisC-Diff offers a novel approach to MRI super-resolution by leveraging disentangled networks, enhancing model interpretability and performance across different contrast conditions.

### Overview: Seedream 5.0 Pro Prompts: 8 Tested Recipes With Outputs ### Key Features: - **Model ID:** `seedream-5-0-pro` - **Price:** $0.0146 per image - **Aspect Ratios:** 1:1, 4:3, 8:5, 3:4, 2:1, 1:1.5, 1:2, 1:3 - **Example Renders:** From commercial product photography to food photography, each recipe is crafted to ensure the best possible output. ### Tech Stack: - Python ### Why It Matters: Seedream 5.0 Pro Prompts offer a comprehensive set of tested recipes that can be used to create high-quality images for e-commerce, food photography, and more. The cost is estimated at $0.0146 per image, making it an economical choice for developers looking to integrate Seedream 5.0 Pro into their projects.

Overview: DubWorld is a local-first AI video dubbing pipeline that translates, synthesizes speech, and aligns subtitles to the original video, allowing users to dub videos into another language on their own computer. Key features: - Transcription and translation of speech - Speech synthesis with timing alignment - Retiming of the video to match the dubbed speech - Subtitles for the dub Tech stack: Python, FFmpeg, OpenRAIL-M, Opus-MT, Supertonic 3 voice model, and other language-specific models. Why it matters: DubWorld simplifies the process of dubbing videos, making it accessible to a wide audience without requiring advanced programming skills. It provides a user-friendly interface and supports multiple languages, making it a valuable tool for language learners and language enthusiasts alike. Repository: alexturkey/dubworld Primary language: Python GitHub stars: 46 GitHub description: Local-first AI video dubbing pipeline: transcription, translation, speech synthesis and timing alignment.

Overview: deepx 是一种以统一存算面为底座的深度学习计算框架，支持模型函数的数学表达和自动分布式训练/推理。 Key features: - Model function can be expressed as a mathematical form and written into a unified storage space for execution. - Unified storage space for tensors, where tensor information and computation graphs are organized and managed. - Support for multi-language SDK and heterogeneous hardware, including CPU, GPU, and other accelerators. Tech stack: C++ Why it matters: deepx simplifies the complex process of distributed training and inference by abstracting tensor computations and storage, making it easier to develop and maintain efficient and scalable deep learning models.

Overview: pyjsgf is a Python package designed for working with JSpeech Grammar Format (JSGF) grammars, including compilation, matching, and parsing. Key features: - Compilation of JSGF grammars into strings or files - Matching of speech strings to grammar rules - Support for finding matching rules given a speech hypothesis - Parsers for grammars, rules, and rule expansions Tech stack: Python Why it matters: pyjsgf simplifies the process of working with JSGF grammars, making it easier to develop speech recognition applications in Python.

### Overview: The Vaccine project is a code repository for implementing the "Vaccine: Perturbation-aware Alignment for Large Language Models" paper, which was presented at NeurIPS 2024. The paper focuses on developing a perturbation-aware safety alignment technique to address security risks in user fine-tuning. The project includes key features such as a customized trainer, package requirements, data preparation, and an example command for running experiments. ### Key features: - Customized trainer implementation - Package requirement installation via `conda` and `pip` - Data preparation scripts for fine-tuning tasks - Example command for running experiments using Slurm ### Tech stack: The tech stack is primarily implemented using Shell scripting and Python. ### Why it matters: This project is crucial for researchers and developers interested in improving the safety and security of large language models (LLMs) by providing a perturbation-aware safety alignment technique. The Vaccine project offers a practical solution to mitigate the risks associated with user fine-tuning and aligns with the growing importance of ensuring the robustness and ethical use of AI models. ### Repository: The repository is named `git-disl/Vaccine` and has 54 GitHub stars, indicating its growing popularity and utility in the field of AI safety and security. ### Primary language: The primary language used in the project is Shell. ### GitHub description: This repository contains the official code for the "Vaccine: Perturbation-aware Alignment for Large Language Models" paper, which was presented at NeurIPS 2024. The project aims

Overview: EasyEspnet simplifies the installation, learning, and use of ESPNet, making it easier to integrate into Python projects. Key features: - Wraps ESPNet for easier Python usage - Provides a Docker image for quick, easy installation - Supports ASR tasks and feature extraction Tech stack: Python Why it matters: By making ESPNet more accessible and user-friendly, EasyEspnet streamlines the development process, enabling developers to focus on their core tasks rather than dealing with the complexities of ESPNet installation and configuration.

Overview: This repository implements a two-stage iterative refinement pipeline for generating aesthetic QR codes, leveraging a pretrained ControlNet for improved scanning robustness. The project was presented at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2025. Key features: - Iterative refinement pipeline - Pretrained ControlNet for aesthetic QR code generation - Scanning robustness guided refinement - Support for various parameters such as module size, padding, negative prompt, and more Tech stack: Python Why it matters: This project showcases the potential of diffusion models in generating aesthetically pleasing QR codes, demonstrating the effectiveness of iterative refinement techniques and the integration of pretrained models for improved scanning robustness.

Overview: Canvas Realm Studio is a self-hosted AI image generator workbench designed for teams, offering features like text-to-image, image-to-image, templates, quotas, history, and an admin dashboard. Key features: - Text-to-image - Image-to-image - Templates - Quotas - History - Admin dashboard Tech stack: TypeScript Why it matters: It provides a centralized, self-hosted solution for image generation, making it suitable for teams looking to streamline their workflow and manage resources efficiently. Repository: laolin5564/canvas-realm-gpt-image-2-studio Primary language: TypeScript GitHub stars: 54 GitHub description: Self-hosted GPT Image 2 / Image-2 workbench for teams: text-to-image, image-to-image, templates, quotas, history and admin dashboard.

Overview: This project is a minimal code base designed for creating AI apps that perform Question Answering over PDF documents, implemented entirely in JavaScript. It utilizes Qdrant for vector database storage, Langchain for LLM orchestration, and FastAPI for backend logic. Key features: - Question Answering over PDF documents - Local inference capabilities - Minimalistic architecture Tech stack: JavaScript Why it matters: This project simplifies the development process for AI applications by providing a straightforward way to integrate vector databases, language models, and backend logic, making it ideal for developers looking to create AI projects locally. Repository: mallahyari/ai-starter-kit GitHub stars: 54

Overview: Modelship is a self-hosted, OpenAI-compatible inference solution that provides a unified backend for reasoning, tool calling, and Responses API, embedding, speech, and image models. It supports multiple GPUs and is designed to be easily deployable, with a high score on the Open Responses conformance suite. Key features: - Unified backend for reasoning, tool calling, and Responses API - OpenAI-compatible inference - Supports multiple GPUs - Deployed on hardware controlled by the user - Scores 17/17 on the independent Open Responses conformance suite Tech stack: Python Why it matters: Modelship simplifies the process of deploying and managing large language models, providing a consistent and efficient way to integrate them into various applications.

Overview: Auto-Ollama is a toolkit designed to simplify the inference or Quantization of Large Language Models (LLMs) directly on your local environment. It supports both direct usage and conversion of models into an efficient format for local deployment. Key features: - Direct usage of LLMs - Conversion of models into GGUF format for local deployment - Support for quantization with Auto-QuantLLM - AutoGGUF utility for converting Hugging Face models to GGUF format - Quick tour script for deploying LLMs with Auto-Ollama Tech stack: Shell Why it matters: Auto-Ollama provides a streamlined and user-friendly approach to deploying and quantizing large language models (LLMs) locally, making it accessible for developers and researchers without the need for complex setup or extensive knowledge of machine learning models.