Projects

Overview: HugNLP is a unified and comprehensive NLP library based on Hugging Face Transformer, awarded the CIKM 2023 Best Demo Paper. Key features: - CIKM 2023 Best Demo Paper Award - Unified and comprehensive NLP library - Supports various NLP tasks and models - Includes pre-built KP-PLM and task-specific models - Provides fine-tuning, few-shot learning, and parameter-free learning tools - Includes uncertainty estimation and prediction calibration Tech stack: Python Why it matters: HugNLP simplifies NLP research and development, making it easier and more effective for researchers to apply state-of-the-art techniques to their projects.

### Overview: Star Attention is a novel block-sparse attention mechanism designed to enhance the efficiency of long sequence inference in transformer-based language models. This method improves inference time by up to 11x while preserving 97-100% accuracy, making it compatible with most LLMs without additional training or fine-tuning. ### Key Features: - **Block-Sparse Attention**: Segments context into blocks for efficient processing. - **Compatibility**: Operates seamlessly with existing Transformer-based LLMs. - **Performance**: Improves inference time by up to 11x while maintaining 97-100% accuracy. ### Tech Stack: PyTorch ### Why it Matters: Star Attention significantly enhances the efficiency of long sequence inference in LLMs, offering up to 11x faster inference times while maintaining high accuracy, making it a valuable optimization for models like Llama3.1-8B-Instruct and Llama-3.1-70B-Instruct. Repository: NVIDIA/Star-Attention Primary language: Python GitHub stars: 392 GitHub description: Efficient LLM Inference over Long Sequences

### Overview: This project is an official implementation of the research paper "Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances" presented at ICLR 2025. The code aims to evaluate and enhance the robustness of watermarking methods against various image editing techniques, including image regeneration, global editing, local editing, and image-to-video generation. ### Key features: - **VINE (Watermarking Inference Network)**: A watermarking method that significantly enhances robustness against image editing techniques. - **W-Bench (Watermarking Benchmark)**: A comprehensive benchmark designed to evaluate the robustness of watermarking methods against a wide range of image editing techniques. - **SDXL-Turbo**: A large-scale pretrained diffusion model adapted for the watermarking task to achieve more imperceptible and robust watermark embedding. ### Tech stack: - Python ### Why it matters: This project provides a practical solution for evaluating and improving the robustness of watermarking methods against advanced image editing techniques, ensuring better copyright protection against sophisticated image editing tools.

### Overview Reconstructing human vision from brain activities has been an appealing task that helps to understand our cognitive process. Despite recent research successes in reconstructing static images, work on recovering continuous visual experiences in the form of videos is still limited. In this work, we propose MinD-Video, a framework that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. We show that high-quality videos of arbitrary frame rates can be reconstructed using adversarial guidance. The recovered videos achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. ### Key Features - **Masked Brain Modeling**: Learns spatiotemporal information from continuous fMRI data. - **Multimodal Contrastive Learning**: Incorporates spatiotemporal attention for better reconstruction. - **Co-Training with Augmented Stable Diffusion**: Biologically plausible and interpretable model. ### Tech Stack Python ### Why It Matters MinD-Video provides a novel approach to video reconstruction from brain activity, offering high-quality videos with improved semantic and pixel-level metrics compared to previous methods. This work is significant for understanding cognitive processes and advancing the field of brain-computer interaction. ### Repository Repository: jqin4749/MindVideo ### Primary Language

### Overview: 本仓库记录了 LLMs 算法工程师相关的顶会论文研读笔记，涵盖了多模态、PEFT、小样本QA问答、RAG、LMMs可解释性、Agents、CoT 等主题。这些笔记旨在帮助工程师们深入理解这些领域的最新研究成果。 ### Key features: - 多模态篇：介绍了一种新的多模态模型系列，Gemini，在图像、音频、视频和文本理解方面具有非凡的能力。 - PEFT 系列篇：评估了 GPT4-V 在结构化推理任务上的表现。 - GPT 系列篇：介绍了一种将 Clip 作为图像编码器的方法，用于将图像特征映射到语言模型特征空间中。 - RAG 系列篇：介绍了一种名为 ProTIP 的渐进式工具检索框架，用于复杂的多步骤规划任务。 - Prompt 系列篇：介绍了一种名为 ProTIP 的渐进式工具检索框架，用于复杂的多步骤规划任务。 - LMMs 可解释性篇：介绍了一种名为 LLaVA 的经典多模态大模型。 - LLMs4KG 篇：介绍了一种将 Clip 作为图像编码器的方法，用于将图像特征映射到语言模型特征空间中。 - LLMs Agents 篇：介绍了一种名为 ProTIP 的渐进式工具检索框架，用于复杂的多步骤规划任务。 - Attention 篇：介绍了一种名为

### Overview: `srai` is a Python library designed for geospatial machine learning, focusing on creating embeddings for various downstream tasks. It provides tools for downloading and processing vector data, regionalization, and embedding regions into vector spaces. ### Key Features: - **OSM / OvertureMaps Data Download**: Downloads OpenStreetMap/Overture Maps data for a given area using different sources. - **Vector Data Processing**: Extracts useful information from acquired vector data, such as road networks, buildings, and POIs. - **GTFS Processing**: Extracts features from GTFS data. - **Regionalization**: Splits a given area into smaller regions using algorithms like Uber's H3, Voronoi, etc. - **Embedding**: Embeds regions into a vector space based on different spatial features and uses PyTorch models like hex2vec. - **Datasets**: Provides prepared datasets for downstream tasks. - **Visualization Utilities**: Offers tools for visualizing and processing spatial data. ### Tech Stack: Python ### Why It Matters: `srai` is essential for geospatial machine learning tasks, providing a robust toolkit for embedding regions into vector spaces and processing geospatial data efficiently. Its focus on vector geometries and spatial features makes it particularly useful in various applications, including data visualization and downstream tasks.

Overview: Automatically generate YouTube shorts with a simple script that automates the entire process, including creation, editing, and voiceover, without the need for tedious tasks or endless hours. Key features: - Automatically generate YouTube shorts - Text-to-Speech functionality - Video editing using MoviePY Tech stack: Python Why it matters: This script provides a convenient and efficient way to create YouTube shorts, saving users time and effort.

### Overview: Learn to deploy Stable Diffusion models on cloud infrastructure with Lightning Apps, a framework designed for easy orchestration and management of microservices. ### Key features: - Load-balancing - GPU inference - Performance testing - Micro-services orchestration ### Tech stack: Python ### Why it matters: This app demonstrates how to efficiently deploy Stable Diffusion models in a production environment, showcasing advanced features like load balancing, GPU inference, and micro-services orchestration using the Lightning Apps framework.

Overview: MACE introduces a finetuning framework for mass concept erasure in diffusion models, aiming to prevent models from generating images that embody unwanted concepts. Key features: - Mass Concept Erasure in Diffusion Models - Leveraging closed-form cross-attention refinement and LoRA finetuning - Successfully scaling the erasure scope up to 100 concepts - Achieving an effective balance between generality and specificity Tech stack: Jupyter Notebook Why it matters: MACE addresses the growing concerns of large-scale text-to-image diffusion models, particularly their potential misuse in creating harmful or misleading content, by preventing models from embedding undesirable concepts in their output.

Overview: Reflexion: an autonomous agent with dynamic memory and self-reflection is an innovative approach that enhances decision-making through dynamic memory and self-reflection. Key features include dynamic memory and self-reflection capabilities, making it a powerful tool for autonomous agents. Key features: - Dynamic Memory - Self-Reflection Tech stack: Python Why it matters: Reflexion is a valuable technique for enhancing decision-making processes, making it a useful addition to various applications where autonomous agents are involved. Repository: noahshinn/reflexion-draft Primary language: Python GitHub stars: 391

Overview: Overeasy is a zero-shot computer vision framework that simplifies the creation of powerful custom solutions without requiring large training datasets. Key features: - 🧩 Workflows: Define a sequence of specialized image processing tasks. - 🧑 Agent: Specialized tools for specific image processing tasks. - 📦 Models: Pre-trained zero-shot models for various tasks. - 🎨 Visualization: Visualize the image processing pipeline. Tech stack: Python, PyTorch, and various pre-trained models. Why it matters: Overeasy makes it easy to combine pre-trained zero-shot models to build powerful custom computer vision solutions, making it accessible for a wide range of applications.

### Overview: Super-Rag is a high-performance RAG (Relevance-Augmented Generation) pipeline designed specifically for AI applications. It offers a simple REST API that supports summarization, retrieval, reranking, and code interpretation, making it an ideal tool for developers looking to integrate AI capabilities into their projects. ### Key features: - **Supports Multiple Document Formats and Vector Databases**: Easily ingest documents from various formats and vector databases. - **Production-Ready REST API**: Provides a robust and ready-to-use API for integration. - **Customizable Splitting/Chunking**: Allows for flexible document splitting and chunking options. - **Data Encoding Options**: Supports encoding data using proprietary and open-source models. - **Code Interpreter Mode**: Includes a built-in code interpreter for computational question and answer scenarios. - **Session Management**: Enables session management through unique IDs for caching purposes. ### Tech Stack: Python ### Why it Matters: Super-Rag simplifies the integration of AI capabilities into applications by providing a comprehensive and efficient solution. Its ability to handle multiple document formats, vector databases, and customizable features makes it a versatile tool for developers looking to enhance their applications with AI-driven insights and functionalities.

Overview: This repository serves as a personal blog, documenting the author's journey of growth. Key features: - Blogging platform for sharing thoughts and experiences - Gitee mirror for Chinese users - Detailed documentation on various programming languages, such as Python, JavaScript, and TypeScript - Comprehensive guides on front-end development, including Vue.js, Webpack, Rollup, and Vite - Algorithm and data structure knowledge - Vue.js and VuePress source code analysis - TypeScript basics and type challenges - Automated testing for Vue applications Tech stack: Shell Why it matters: This repository provides valuable resources for developers looking to improve their skills in various programming languages, front-end development, and algorithmic thinking.

### Overview: BERT-CH-NER 是一个基于BERT的中文命名实体识别模型，旨在通过修改TensorFlow官方代码来实现有效的中文命名实体识别任务。该模型使用了BERT作为基础模型，并结合了LSTM和CRF层以提高识别精度。通过在搜狐比赛和中文数据集上的实验，展示了BERT在中文命名实体识别任务中的优越性能。 ### Key features: - **BERT作为基础模型**：使用了预训练的BERT模型作为核心，增强了模型的语义理解和上下文建模能力。 - **LSTM和CRF层**：结合LSTM和CRF层来处理序列标注任务，提高了识别的准确性和灵活性。 - **中文数据集和搜狐比赛**：在中文数据集和搜狐比赛上进行了基准测试，展示了模型在实际应用中的有效性。 - **代码示例**：提供了从数据预处理到模型训练和测试的完整代码示例，方便开发者快速上手。 ### Tech stack: - **Python**: 3.6版本，适用于TensorFlow 1.13环境。 - **TensorFlow**: 1.13版本，用于模型训练和预测。 ### Why it matters: BERT-CH-NER模型通过结合BERT的预训练能力和LSTM+CRF的序列标注技术，显著提高了中文命名实体识别的准确率。其在搜狐比赛和中文数据集上的表现证明了其在实际应用中的强大性能和广泛适用性。

### Overview: SkyText 是由奇点智源发布的中文 GPT3 预训练大模型，支持多种任务如聊天、问答、中英翻译、内容续写、诗词对联等。 ### Key features: - 聊天 - 问答 - 中英翻译 - 内容续写 - 诗词对联 - 古诗创作 - 菜谱生成 - 第三人称转述 - 创建采访问题 ### Tech stack: Python ### Why it matters: SkyText 是一个强大的中文预训练模型，能够处理多种任务，包括聊天、问答、翻译、续写、诗词对联等，极大地提高了语言处理的效率和准确性。 ### Repository: GitHub Repository: [SkyWorkAIGC/SkyText-Chinese-GPT3](https://github.com/SkyWorkAIGC/SkyText-Chinese-GPT3) ### GitHub stars: 390

### Overview: This repository contains a collection of BentoML example projects focused on deploying different diffusion models, specifically Stable Diffusion (SD) family models. It serves as a practical guide for developers to deploy and manage Stable Diffusion models using BentoML. ### Key features: - **SDXL Turbo**: A demonstration of deploying Stable Diffusion models with Turbo features. - **SDXL Large**: A model optimized for large-scale inference. - **SDXL Medium**: A medium-sized model for various applications. - **SDXL Large Turbo**: A Turbo version of the Large model for enhanced performance. - **SDXL Light**: A lightweight model for faster inference. - **SDXL**: The base model for Stable Diffusion. - **ControlNet**: A model for controlling and manipulating images. ### Tech stack: Python ### Why it matters: This repository provides a comprehensive guide for developers to deploy and manage Stable Diffusion models using BentoML, making it easier to integrate and use these models in various applications.

### Overview: ST-MoE is an implementation of the ST-MoE model, a state-of-the-art method for mixture of experts, specifically designed for autoregressive transformers. It leverages the Pytorch framework, making it accessible and efficient for developers. ### Key Features: - **ST-MoE Model Implementation**: The repository contains the complete implementation of the ST-MoE model, based on the official Mesh Tensorflow implementation. - **Mixture of Experts**: ST-MoE is an advanced version of the Mixture of Experts (MoE) model, designed to improve performance in autoregressive transformers. - **Gating Mechanism**: The model uses gating mechanisms to decide which experts to route tokens to, ensuring efficient computation. - **Thresholds for Gating**: The thresholds for routing tokens to the second and third experts can be adjusted for optimal performance. - **Capacity Factor**: The capacity factor for experts is set to ensure sufficient capacity for gating, which is crucial for the effectiveness of the model. ### Tech Stack: The repository uses Python, specifically Pytorch, for its implementation. ### Why it Matters: ST-MoE is expected to be state-of-the-art for mixture-of-experts models in autoregressive transformers. It is particularly useful for improving the performance of models like GPT4, which is rumored to use 16 experts with top2 gating. ### Repository: The repository is named `lucidrains/st-moe-pytorch` and has received 390 GitHub stars, indicating its popularity and utility in

Overview: sklearn-genetic-opt is a scikit-learn-compatible optimization toolkit that uses evolutionary algorithms for hyperparameter tuning and feature selection, offering a modern alternative to GridSearchCV and RandomizedSearchCV. Key features: - Genetic algorithm for evaluating complete parameter configurations - GAFeatureSelectionCV for feature subset selection with cross-validation - Support for scikit-learn estimators and scoring - Adaptive search with feedback from earlier evaluations - Joint feature selection and tuning - Mixed search spaces for different types of parameters - Smart initialization techniques - Early stopping callbacks for automatic convergence detection - Adaptive schedules for evolving search strategies - Optimization history and plots for visual analysis Tech stack: Python Why it matters: sklearn-genetic-opt provides a powerful and flexible alternative to traditional hyperparameter tuning methods, making it easier and faster to find optimal parameter configurations for scikit-learn models. Its evolutionary approach offers a fresh perspective on hyperparameter optimization, potentially leading to better performance and faster convergence compared to traditional methods.

Overview: LastChat is a feature-rich AI assistant application designed for Android, built on Kotlin and built with a focus on privacy and personalization. It is a fork of RikkaHub, featuring advanced AI capabilities, tools and integrations, multiple personas, and a modern, user-friendly interface. Key features: - Multi-Provider Support - RAG Memory - Multi-Modal Inputs - Python Built-in Engine - JavaScript Built-in Engine - Web Search Integration - MCP Server Support - Multiple Personas - Tagging System - Import/Export Assistant Configurations - Image Generation Interface - Text-to-Speech (TTS) Support - Local-First Storage - Secure WebDAV Backup Tech stack: Kotlin, Jetpack Compose, Koin, Room, DataStore, WorkManager, AlarmManager, QuickJS, Python, JavaScript, Markdown, LaTeX, WebDAV, Material You 3 Expressive, Rich Rendering. Why it matters: LastChat offers a highly personalized and privacy-focused AI chat experience, tailored for Android users, with advanced AI capabilities, robust tools, and a modern, user-friendly interface.

Overview: Learn LLM Inference Engineering step by step, from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs. Key features: - Autoregressive Models - Prefill vs Decode: LLM Inference Optimization - Prefill-Decode Disaggregation Tech stack: Markdown Why it matters: This series provides a comprehensive guide to understanding and optimizing the inference process of large language models (LLMs), covering essential topics like KV cache, PagedAttention, continuous batching, and advanced techniques like vLLM, SGLang, and GPU optimization. Repository: amitshekhariitbhu/llm-inference-engineering GitHub stars: 386