Projects

Overview: Youku-mPLUG is a 10 million large-scale Chinese video-language pre-training dataset, collected from Youku with strict criteria of safety, diversity, and quality. It includes 3 different downstream multimodal video benchmark datasets for measuring pre-trained model capabilities. Key features: - 10 million high-quality videos in 20 super categories distributed into 45 categories - 3 downstream benchmarks: Video Category Prediction, Video-Text Retrieval, Video Captioning - Public dataset for researchers and developers to evaluate pre-trained models Tech stack: Python Why it matters: Youku-mPLUG provides a comprehensive dataset for pre-training models, enabling researchers to test and evaluate the capabilities of their models across various downstream tasks, thus advancing the field of video-language pre-training.

Overview: The Second Brain AI Agent Project is a cutting-edge initiative aimed at transforming how individuals manage and access their vast amounts of information. This project leverages artificial intelligence to provide a seamless, intuitive interface for indexing, searching, and interacting with various types of content, including markdown files, videos, web pages, and PDFs. Key features: - **AI-Driven Indexing**: Automatically categorizes and indexes all collected content for quick and accurate retrieval. - **Search Functionality**: Precise search capabilities to find specific information within the vast repository. - **Interactive Content Engagement**: Allows users to interact with content, such as highlighting, annotating, and sharing insights directly within the platform. - **Personalized Learning Paths**: Tracks user preferences and behavior to recommend relevant content for continuous learning and growth. Tech stack: The project utilizes Python, leveraging libraries such as TensorFlow, PyTorch, and NLTK for natural language processing and machine learning tasks. Other technologies include Flask for web development, and various APIs for integrating with external services. Why it matters: The Second Brain AI Agent Project offers a revolutionary solution for anyone looking to enhance their knowledge management skills. By providing a personalized and efficient way to access and interact with information, it empowers users to stay organized, learn effectively, and collaborate more effectively with their colleagues and students.

### Overview: This repository contains a list of English stopword lists from various sources, including Sphinx, EBSCOhost, CoreNLP, Ranks NL, Lucene/Solr, and MySQL. The default list is English stopwords, and it serves as a common step in preprocessing text for various purposes. ### Key Features: - **Sphinx:** A list extracted from the Sphinx search server. - **EBSCOhost:** Stopwords used in EBSCOhost medical databases. - **CoreNLP:** Hardcoded in the Stanford NLP library. - **Ranks NL (Google):** A short list based on Google's stopwords. - **Lucene/Solr:** An unmodifiable set of common English words. - **MySQL (InnoDB):** A word used by default as a stopword for FULLTEXT indexes on InnoDB tables. ### Tech Stack: Python ### Why it Matters: This repository provides a comprehensive list of English stopword sources, useful for preprocessing text in various applications.

### Overview: This project demonstrates the implementation of a Transformer-based NLP model for emotion prediction in the browser, using Edge Inference with ONNX Runtime Web. The model is fine-tuned on the GoEmotions dataset, which is a multi-label text categorization problem, and hosted on Huggingface. ### Key Features: - **Cloudflare Pages**: Deployed React app and model via CDN. - **ONNX Runtime Web**: Inference engine for model execution in the browser. - **Huggingface Model Hosting**: NLP model hosted and trained using the Transformer library. - **Google Colab**: Model training using GPU instances. - **Fine-tuned Model**: Fine-tuned version of the pre-trained language model `microsoft/xtremedistil-l6-h384-uncased`. - **Dataset**: GoEmotions dataset, a fine-grained emotion dataset with 58k Reddit comments. - **Model Inference**: Multi-threaded inference with specific HTTP headers for cross-origin isolation. ### Tech Stack: - **React**: Frontend framework. - **ONNX Runtime Web**: Inference engine. - **Huggingface**: Model hosting and training. - **Google Colab**: Model training. - **Cloudflare Pages**: Deployment and CDN. ### Why It Matters: This project showcases the practical application of Edge Inference in the browser, leveraging advanced NLP models hosted on Huggingface for real-time emotion prediction. It demonstrates how to fine-tune models on large datasets and optimize for performance in the browser environment, addressing the challenges of model

Overview: 这个项目是一个基于云端语音识别的智能音箱设备，类似于天猫精灵和小爱同学，采用STM32F407、WM8978和ESP8266芯片，旨在提供类似亚马逊Echo和小爱同学的功能。 Key features: - 基于云端语音识别技术 - STM32F407、WM8978和ESP8266芯片 - 高度可拓展的播放音乐功能 - 简单的人机交互界面 Tech stack: C语言 Why it matters: 该项目展示了如何在单片机上实现类似智能家居设备的功能，具有较高的技术价值和实际应用潜力。

Overview: The RAG Experiment Accelerator is a tool designed to expedite and facilitate experiments and evaluations using Azure Cognitive Search and RAG pattern. Key features: - Experiment Setup - Integration with Azure AI Search, Azure Machine Learning, MLFlow, and Azure OpenAI - Rich Search Index - Multiple Document Loaders (Azure Document Intelligence, LangChain) - Custom Document Intelligence Loader - Query Generation - Multiple Search Types - Sub-Querying - Re-Ranking - Metrics and Evaluation Tech stack: Python Why it matters: This tool simplifies the process of conducting experiments and evaluations, making it easier and faster to test different search engine parameters, evaluate the performance of various search strategies, and fine-tune parameters, all while generating detailed reports and visualizations from experiment results.

Overview: Sherpa-Rs is a Rust library that provides bindings to the Sherpa-onnx project, enabling developers to leverage the capabilities of the Sherpa-onnx library in Rust applications. Key features: - Spoken language detection - Speaker embedding (labeling) - Speaker diarization - Speech to text - Text to speech - Text punctuation - Voice activity detection - Audio tagging - Keyword spotting Tech stack: Rust Why it matters: Sherpa-Rs simplifies the integration of the Sherpa-onnx library into Rust projects, making it easier to leverage the advanced speech processing capabilities of the Sherpa-onnx project within Rust environments. Repository: thewh1teagle/sherpa-rs GitHub stars: 314 GitHub description: Rust bindings to https://github.com/k2-fsa/sherpa-onnx Primary language: Rust Tech stack: Rust Why it matters: Sherpa-Rs provides a Rust interface to the Sherpa-onnx project, making it accessible and easier to integrate into Rust-based applications.

### Overview: MicroLens is a large-scale dataset designed for micro-video recommendation, featuring diverse modalities including text, audio, and videos. ### Key Features: - **Micro-Video Generation**: Includes 100 randomly sampled videos. - **Video Understanding**: Covers various tasks such as general video understanding and video recommendation. - **Foundation Models**: Supports tasks like short video generation and video recommendation. - **Dataset**: Contains 100 videos, 100K pairs, and 1M pairs for different modalities. ### Tech Stack: Python ### Why it Matters: MicroLens provides a comprehensive dataset for researchers and developers to evaluate and improve micro-video recommendation systems, including generation, understanding, and recommendation tasks. ### Repository: westlake-repl/MicroLens ### GitHub Stars: 312 ### GitHub Description: A Large Short-video Recommendation Dataset with Raw Text/Audio/Image/Videos (Talk Invited by DeepMind).

### Overview: Relay Diffusion is an innovative framework designed to unify diffusion processes across different resolutions for image synthesis. It achieves state-of-the-art results in various benchmarks, including CelebA-HQ and ImageNet-256, with a FID score of 1.87. ### Key Features: - **Unified Diffusion Process**: Transfers low-resolution images or noise into equivalent high-resolution ones through blurring diffusion and block noise. - **State-of-the-Art Performance**: Achieves the best FID on CelebA-HQ and sFID ImageNet-256. - **Efficient Training**: Reduces training cost by 15% using `xformers.ops.memory_efficient_attention`. - **Versatile Setup**: Compatible with PyTorch, supports both CPU and GPU environments, and can be run on servers with Nvidia A100s. - **Easy-to-Use Dataset**: Preprocessed and organized datasets for CelebA-HQ and ImageNet, ready for training at resolutions 64x64 or 256x256. ### Tech Stack: Python ### Why it Matters: Relay Diffusion offers a more efficient and unified approach to diffusion-based image synthesis, surpassing existing methods in terms of quality and performance. Its ability to seamlessly integrate into existing models and frameworks makes it a valuable tool for researchers and practitioners in the field of computer vision and image synthesis.

Overview: MinImagen is a minimal implementation of the Imagen text-to-image model, designed for educational purposes to demonstrate the implementation of diffusion models in text-to-image applications. Key features: - Noise Conditioning Augmentation - Dynamic Thresholding Tech stack: Python Why it matters: This project serves as an educational tool to showcase how diffusion models are implemented and integrated into text-to-image models, providing insights into the underlying principles of Imagen. Repository: AssemblyAI-Community/MinImagen Primary language: Python GitHub stars: 312 GitHub description: MinImagen: A minimal implementation of the Imagen text-to-image model README excerpt: # MinImagen ### A Minimal implementation of the [Imagen](https://imagen.research.google/) text-to-image model. <br/> <p align="center"><img src="./images/model_structure.png?raw=True" width="700"/></p> <br/> ### See [Build Your Own Imagen Text-to-Image Model](https://www.assemblyai.com/blog/build-your-own-imagen-text-to-image-model/) for a tutorial on how to build MinImagen. ### See [How Imagen Actually Works](https://www.assemblyai.com/blog/how-imagen-actually-works/) for a detailed explanation of Imagen's operating principles. <br/> Given a caption of an image, the text-to-image model **Imagen** will generate an image that reflects the scene described by the caption. The model is a [cascading diffusion model](https://arxiv.org/abs/2106.1

Overview: This repository accompanies a Medium article that provides a hands-on tutorial on implementing custom Graph Convolutional layers and building Graph Neural Networks using PyTorch Geometric on the YooChoose dataset. Key features: - Implementing custom Graph Convolutional layers - Preparing data for training Graph Neural Networks - Building custom Graph Neural Networks Tech stack: Jupyter Notebook Why it matters: This tutorial offers practical insights into implementing graph neural networks using PyTorch Geometric, making it valuable for developers looking to enhance their skills in this area.

Overview: SDGym is a benchmarking framework designed to evaluate and compare synthetic data generation methods across various techniques and datasets. Key features: - Integrates with the Synthetic Data Vault ecosystem, allowing use of SDV's synthesizers, datasets, and metrics. - Supports classical statistics, deep learning, and custom machine learning models as synthesizers. - Measures performance, memory usage, synthetic data quality, and privacy through various metrics. - Provides a detailed performance, memory, and quality evaluation across multiple datasets and synthesizers. Tech stack: Python Why it matters: SDGym serves as an essential tool for developers and researchers in the field of synthetic data generation, enabling them to benchmark and compare different techniques and datasets effectively. Repository: sdv-dev/SDGym Primary language: Python GitHub stars: 312 GitHub description: Benchmarking synthetic data generation methods.

### Overview: SummarizePaper is a comprehensive AI tool designed for researchers, offering summaries, chat capabilities, and collections of papers across various sources including arXiv, bioRxiv, medRxiv, and journals. It provides daily digest, saved searches, rising topics, and citation alerts, making it a valuable resource for staying updated and informed in the field. ### Key features: - **Summaries**: Detailed summaries of papers, including key points, lay summaries, and blog articles. - **Chat with Papers**: Answers to questions from the full text of papers, with passages linked to the original text. - **Collections**: Papers organized and compared side by side, with open questions and open-source comparison tables. - **Daily Digest**: New papers of your field every morning, ranked by research profile and votes. - **Saved Searches**: Papers matching specific searches, ranked and available via email or RSS. - **Rising Topics**: Topics rising each week, field-specific. - **Weekly Letters**: Articles on papers that matter beyond their abstracts. - **Scoop Watch**: Describing a research project in progress. - **Citation Alerts**: Notifications on papers that cite your work. ### Tech Stack: Python ### Why it Matters: SummarizePaper is a valuable resource for researchers, providing summaries, chat capabilities, and collections of papers across various sources. It helps researchers stay updated and informed in their field, making it an essential tool for staying current in their research. Repository: summarizepaper/summarizepaper Primary language: Python GitHub stars: 312

### Overview: This repository contains a comprehensive survey on Multimodal Large Language Models (MLLMs) for autonomous driving, including background information, existing tools and datasets, and key challenges in applying MLLMs in autonomous driving systems. ### Key Features: - **MLLMs for Perception & Planning & Control for Autonomous Driving**: Explore the use of LLMs in perception, planning, and control for autonomous driving. - **Existing MLLM Tools for Driving, Transportation, and Map Systems**: Overview of existing MLLM tools for driving, transportation, and map systems. - **Existing Datasets and Benchmarks**: Information on datasets and benchmarks used in the field of MLLM for autonomous driving. ### Tech Stack: Python, Jupyter Notebook, and various machine learning libraries. ### Why It Matters: Understanding the key challenges and opportunities in applying MLLMs in autonomous driving systems is crucial for the development and future endeavors of LLM-driven autonomous driving technologies.

Overview: RLHF-V is a framework designed to enhance the trustworthiness of Multimodal Large Language Models (MLLMs) by aligning their behavior through fine-grained correctional human feedback. This approach reduces hallucination rates and improves the overall quality of the models. Key features: - Fine-grained correctional human feedback data collection - Reduction of hallucination rate by 34.8% on a base model - Experimentation with Muffin and UniMM-Chat models - Available at Hugging Face datasets Tech stack: Python Why it matters: RLHF-V offers a practical solution for improving the reliability and trustworthiness of MLLMs by leveraging human feedback to correct model hallucinations, thus enhancing the model's performance and credibility. Repository: RLHF-V/RLHF-V Primary language: Python GitHub stars: 311 GitHub description: [CVPR'24] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

### Overview: AudioStory is a unified framework designed to generate long-form narrative audio using large language models, addressing the limitations of current text-to-audio generation in handling long-form audio synthesis. ### Key Features: - **Decoupled Bridging Mechanism**: AudioStory separates the collaboration of language models and diffusion into two distinct components—bridging queries for intra-event semantic alignment and consistency queries for cross-event coherence preservation. - **End-to-End Training**: It integrates instruction comprehension and audio generation within a single end-to-end framework, enhancing synergy between components and eliminating the need for modular training pipelines. - **Benchmark**: AudioStory-10K is a benchmark dataset that includes diverse domains such as animated soundscapes and natural sound narratives. ### Tech Stack: The project is built using Jupyter Notebook. ### Why it Matters: AudioStory offers a unified approach to generating long-form narrative audio, addressing the shortcomings of current text-to-audio generation methods. It excels in instruction-following reasoning and maintains temporal coherence and emotional tone consistency, making it a valuable tool for audio storytelling.

### Overview: Cumulative Reasoning with Large Language Models Cumulative Reasoning (CR) is a structured framework designed to enhance Large Language Models (LLMs) by simulating human-like iterative and cumulative thought processes. CR orchestrates LLMs in three distinct roles—Proposer, Verifier(s), and Reporter—to systematically decompose tasks, validate intermediate steps, and compose them into a final solution. ### Key Features: - **Game of 24**: Achieves 98% accuracy, a +24% absolute improvement over Tree-of-Thoughts (ToT). - **MATH Dataset (No Code Interpreter)**: Attains 58% accuracy with GPT-4, outperforming Progressive-Hint Prompting (PHP) by +4.2%. - **MATH Dataset (Hardest Problems)**: Shows a 43% relative improvement on Level 5 problems (from 22.4% to 32.1%). - **MATH Dataset (with Code Interpreter)**: The CR Agent reaches 72.2% accuracy, surpassing PAL (PoT) by +20.2%. ### Tech Stack: Python The project is implemented in Python 3.10+. ### Why It Matters: Cumulative Reasoning (CR) enhances LLMs' ability to solve complex reasoning problems by emulating human-like iterative and cumulative thought processes, leading to improved accuracy and efficiency in various applications. ### Repository: iiis-ai/cumulative-reasoning The primary language of the

Overview: 啊哈自然语言处理包提供包括分词、依存句法分析、语义角色标注、自动摘要、语义相似度计算、LDA 主题预测、词云等服务。 Key features: - 分词类 - 标准分词 - NLP 分词 - 分词断句 - 命名实体识别 - 句法分析类 - 依存句法分析 - 摘要类 - TextRank 摘取关键词 - TextRank 摘取关键句和自动摘要 - 语义类 - Word2Vec 词语相似度 - Word2Vec 句子相似度 - 语义角色标注 - LDA 主题预测 - 附加功能 - 简繁转换 - WordCloud 绘制词云 Tech stack: Java Why it matters: 提供了全面的中文自然语言处理能力，适用于多种应用场景，包括文本摘要、主题建模、关键词提取等，提升了文本处理的效率和准确性。 Repository: jsksxs360/AHANLP GitHub stars: 310

Overview: BNLP is a comprehensive natural language processing toolkit designed specifically for Bengali language, offering a variety of functionalities such as tokenization, embedding, POS tagging, named entity recognition, text cleaning, and more. Key features: - Tokenization: Supports basic, NLTK, and SentencePiece tokenizers. - Embeddings: Provides Word2Vec, FastText, and GloVe embeddings. - Part of Speech Tagging: Uses CRF-based POS tagging. - Named Entity Recognition: Supports CRF-based NER. - Text Cleaning: Includes cleaning for Bengali NLP purposes. - Corpus: Offers access to Bengali letters, vowels, punctuation, and stopwords. - Command Line Interface (CLI): Simplifies text processing without Python code. - Pipeline API: Chains multiple NLP operations for complex tasks. Tech stack: Python, Jupyter Notebook, and various NLP models. Why it matters: BNLP streamlines the process of natural language processing for the Bengali language, making it easier to analyze and understand text data in Bengali, which is crucial for applications such as language translation, sentiment analysis, and more. Repository: sagorbrur/bnlp Primary language: Jupyter Notebook GitHub stars: 310 GitHub description: BNLP is a natural language processing toolkit for Bengali Language. Overview: BNLP is a comprehensive natural language processing toolkit designed specifically for Bengali language, offering a variety of functionalities such as tokenization, embedding, POS tagging, named entity recognition, text cleaning, and more.

### Overview: Este curso de Data Science é uma ferramenta educacional que oferece uma visão completa e prática sobre o campo de ciência de dados. ### Key features: - **Machine Learning**: Introdução a técnicas de aprendizado de máquina, incluindo regressão, classificação, e redes neurais. - **Data Visualization**: Práticas de visualização de dados para melhor compreensão e interpretação. - **Python**: Fundamentos em Python para manipulação de dados e implementação de algoritmos. - **Jupyter Notebook**: Prática de desenvolvimento de código e análise de dados em um ambiente colaborativo. - **Projeto Prático**: Implementação de um modelo de Machine Learning em um conjunto de dados real. ### Tech stack: - Python - Jupyter Notebook ### Why it matters: Este curso é essencial para aqueles que desejam ingressar ou aprimorar suas habilidades em ciência de dados, oferecendo uma combinação de teoria prática e prática prática.