Projects

Overview: Mference is a Swift + Metal runtime designed for models exceeding RAM, providing bounded-memory streaming and pinned checkpoints for efficient inference on Apple Silicon devices. Key features: - Pinned checkpoints for models exceeding RAM - Bounded-memory streaming with measured ports - Supports multiple expert families with bounded working sets Tech stack: Swift, Metal, macOS 15 or later Why it matters: Mference enables efficient inference of models larger than available RAM, making it ideal for edge devices and high-performance computing environments.

### Overview: This project presents a unified approach to tokenization and generation using a single-stage training method called UNITE. The code is based on the paper "End-to-End Training for Unified Tokenization and Latent Denoising" and aims to achieve state-of-the-art performance without the use of adversarial losses or pretrained encoders. ### Key Features: - **Unified Tokenization and Generation**: UNITE combines tokenization and generation into a single model, leveraging shared parameters to jointly optimize both tasks. - **Single-Stage Training**: The model is trained in a single stage, optimizing both tokenization and generation simultaneously. - **Latent Space Alignment**: The Generative Encoder is used to align the latent space, enabling a "common latent language" for both tokenization and generation. - **Performance**: UNITE achieves near state-of-the-art performance across image and molecule modalities, with FID scores of 2.12 and 1.73 for the Base and Large models on ImageNet 256 × 256. ### Tech Stack: - **Python**: The primary programming language used for the project. ### Why It Matters: UNITE provides a more efficient and effective way to handle tokenization and generation tasks by combining them into a single model, potentially reducing the complexity and computational requirements compared to traditional staged training methods.

### Overview: This repository is dedicated to exploring and advancing large recommendation models, focusing on their scalability, performance, and application in sequential recommendations and large language models for recommendation systems. ### Key Features: - **Scalability Analysis**: Studies the scalability of large recommendation model architectures, leveraging popular Transformers such as HSTU, Llama, GPT, and SASRec. - **Comprehensive Study**: Conducts an extensive ablation study and parameter analysis on HSTU, uncovering the origins of scaling laws. Enhances the scalability of traditional Transformer-based sequential recommendation model, SASRec, by integrating effective modules from scalable large recommendation models. - **Complex User Behavior**: First study to assess the performance of large recommendation models on complex user behavior sequence data, pinpointing areas for improvement in modeling intricate user behaviors. - **Ranking Tasks Evaluation**: Comprehensive evaluation of large recommendation models on ranking tasks, demonstrating their scalability, with a focus on datasets and hyperparameters. ### Tech Stack: Python ### Why It Matters: This repository provides valuable insights into the latest advancements in large recommendation models, helping developers and researchers optimize their models for better scalability and performance.

Overview: DocumentGPT is a web application designed for seamless interaction with research documents, leveraging OpenAI's chat API and vector databases for semantic search and contextual information. Key features: - Upload and view PDF documents within the web app. - Engage with an AI chatbot powered by OpenAI's chat API. - Retrieve answers from vector databases for semantic search. - Access web results for additional context. - Perform Google/YouTube searches within the app. Tech stack: Python Why it matters: DocumentGPT simplifies the process of interacting with research documents by providing a user-friendly interface, enhancing the efficiency and accuracy of research by leveraging advanced AI and vector databases. Repository: aju22/DocumentGPT Primary language: Python GitHub stars: 122 GitHub description: DocumentGPT is a web application that allows you to chat over your research document using OpenAI's chat API and perform semantic search using vector databases. This tool provides a seamless interface for interacting with your research document, exploring search results, and engaging in a conversation with an AI chatbot. README excerpt: DocumentGPT is a web application that allows you to chat over your research document using OpenAI's Chat API and perform semantic search using Vector Databases. This tool provides a seamless interface for interacting with your research document, exploring search results, and engaging in a conversation with an AI chatbot. ## Features - Upload your PDF document and view its contents within the web app. - Chat with an AI chatbot powered by OpenAI's chat API, using the content of your research document

### Overview: GraphTranslator is a project that aims to align graph models with large language models for open-ended tasks. It involves pre-training a graph model, using LLM to summarize node/neighbor information, and training a translator to align the graph model with the LLM. The project is part of a paper presented at the Web Conference 2024 and is implemented using Python, CUDA, and Hugging Face libraries. ### Key Features: - **Pre-training Phase**: Employing link prediction as the self-supervised task for pre-training the graph model. - **Producer Phase**: Utilizing LLM to summarize node/neighbor information. - **Translator Training Phase**: Training the translator for GraphModel-Text alignment and GraphModel-LLM alignment. - **Translator Generate Phase**: Generating predictions with the pre-trained translator model. ### Tech Stack: Python, CUDA, Hugging Face ### Why it Matters: GraphTranslator leverages the strengths of graph models and large language models to achieve open-ended tasks, making it a valuable tool for researchers and developers looking to enhance their models' capabilities in various applications.

### Overview: Emmy is an optimized GPU compiler designed for high-performance inference of Large Language Models (LLMs). It supports compiling PyTorch models into CUDA kernels, benchmarking LLM inference, and deploying models on various GPUs. Emmy simplifies the process of optimizing models and provides comprehensive tools for performance analysis and community sharing. ### Key features: - **Compile**: Converts PyTorch operations into CUDA kernels for faster inference. - **Tune**: Optimizes models using kernel fusion, autotuning, and advanced scheduling. - **Deploy**: Runs benchmarks across different GPU types and configurations, and deploys models on local or cloud GPUs. - **Benchmark**: Compiles and benchmarks LLM inference performance. - **Serve**: Provides a user-friendly interface for deploying and managing models. - **Recipes**: Offers a list of optimized recipes for different models and configurations. ### Tech stack: Python ### Why it matters: Emmy simplifies the process of optimizing LLM models and provides a streamlined workflow for deployment and performance analysis, making it an essential tool for developers working with large language models.

### Overview: This repository is maintained to release dataset and models for multimodal puzzle reasoning. It features two novel datasets: **PuzzleVQA** and **AlgoPuzzleVQA**, designed to challenge the performance of large language models (LLMs) in understanding and solving puzzles. ### Key Features: - **PuzzleVQA**: A dataset of abstract visual patterns crafted to test the reasoning capabilities of LLMs. - **AlgoPuzzleVQA**: A more complex dataset that presents advanced puzzles requiring algorithmic solutions. ### Tech Stack: Python ### Why It Matters: The performance of LLMs on these datasets is notably deficient, highlighting the need for significant improvements in their ability to reason across multiple modalities. ### Repository: - **Repository**: declare-lab/LLM-PuzzleTest - **GitHub Stars**: 122 - **GitHub Description**: This repository is maintained to release dataset and models for multimodal puzzle reasoning. - **Primary Language**: Python ### Environment Setup: - **conda create -n puzzle-eval python=3.10 -y** - **conda activate puzzle-eval** - **pip install -r requirements.txt** ### API Setup: - **GPT-4-Turbo**: `{"engine": "gpt-4-turbo", "key": "<API_KEY>", "location": "westus", "endpoint": "<ENDPOINT>", "api_version": "2024-08-01-preview"}` - **GPT-4o**: `{"engine": "

### Overview: Prompt Alchemist is a sophisticated meta-prompting orchestration system that optimizes large language model reasoning pipelines by imbuing raw instructions with dimensional awareness, constraint framing, and intent amplification. ### Key Features: - **Intent Resonance Mapping**: Maps input goals to a vector space of task archetypes. - **Constraint Weaving**: Infers and interleaves boundary conditions, success criteria, and failure modes. - **Epistemic Layering**: Assigns confidence levels to different parts of the instruction. - **Temporal Anchoring**: Embeds time-aware context markers. - **Recursive Self-Correction Injection**: Bakes in meta-instructions for reflection. ### Tech Stack: HTML ### Why It Matters: Prompt Alchemist transforms raw instructions into context-aware, semantically-rich prompt architectures optimized for large language model reasoning pipelines, ensuring clear and effective communication of reasoning processes.

Overview: PhotonInfer is a high-performance LLM inference engine designed for advanced batching capabilities, supporting models like Llama-3.2 and Qwen3. It delivers production-grade inference performance through vLLM-Style Continuous Batching, with a speedup of up to 1.63× compared to llama.cpp. Key features: - **vLLM-Style Continuous Batching**: Token-level dynamic scheduling, two-phase scheduler, request state tracking, and perfect production support. - **GPU-Optimized Kernels**: Batched Paged Attention, vectorized memory access, fused operations, INT8 quantization, and optimized softmax. - **Modern C++20 Architecture**: Type-safe error handling, zero-copy design, device agnostic interface, and concepts & ranges. Why it matters: PhotonInfer is optimized for real-time inference services, providing high concurrency and real-time responsiveness, making it suitable for production environments. Repository: lumia431/photon_infer Primary language: C++ GitHub stars: 124

Overview: GRAPE is a unified framework for positional encoding that combines multiplicative and additive mechanisms, ensuring exact relative position laws and efficient streaming cacheability. Key features: - Efficient training procedures optimized for large-scale datasets and multi-GPU setups. - Compatible with popular datasets like Fineweb-Edu-100B and OpenWebText. - Integrated with lm-evaluation-harness for standardized benchmarking. Tech stack: Python 3.12+ Why it matters: GRAPE simplifies the implementation of position encoding, unifying multiplicative and additive mechanisms and ensuring exact relative position laws, making it a valuable tool for sequence modeling in Transformers.

### Overview: vale-ai-tells is a comprehensive, cutting-edge package designed to enhance technical documentation by detecting and correcting linguistic patterns commonly associated with AI-generated prose. This package is particularly useful for writers aiming to improve the clarity and directness of their technical content, making it an invaluable tool for ensuring clean, professional writing. ### Key Features: - **AI-generated commit messages detection**: Identifies and corrects patterns in AI-generated commit messages, including self-referential preambles, trailing justification clauses, buzzword adjective-noun pairs, and gitmoji patterns. - **Opt-in structural rules**: Provides 15 rules specifically designed for commit messages, separate from the prose rules, allowing for a more focused approach to improving commit message quality. ### Tech Stack: The package is built using Shell scripting, leveraging Vale's powerful linting capabilities to enforce stylistic and structural rules. ### Why It Matters: For technical documentation, where precision and clarity are paramount, vale-ai-tells offers a significant enhancement. It helps writers maintain a consistent and professional tone, ensuring that AI-generated content is not disguised as human-written. This tool is particularly useful in environments where AI-generated content is expected to be treated as genuine technical documentation. ### GitHub Stars: The repository has been starred by 122 developers, indicating its growing popularity and utility in the technical documentation community. ### Installation: To use vale-ai-tells, add the package to your `.vale.ini` configuration file, specifying the `ai-tells` style and optionally the `ai-tells-commits` style for commit

### Overview: MachineSoM is a project exploring the collaboration mechanisms of large language model agents from a social psychology perspective. It aims to understand how different traits and strategies influence the effectiveness of these agents in complex tasks. ### Key features: - The Society of Mind (SoM): intelligence emerges from collaborative and communicative computational modules. - Societies of LLM agents with traits: easy-going and overconfident. - Collaboration processes: debate and self-reflection. - Interaction strategies: when to interact and with whom. ### Tech stack: Python ### Why it matters: This project provides a comprehensive framework for understanding how LLM agents collaborate and interact, offering insights into the social psychology of AI agents.

### Overview This repository provides comprehensive samples for building Voice Assistants hosted in your Azure subscription. It includes sample clients for C#, C++, and Java, along with a tool for end-to-end regression testing. Voice Assistants can be keyword activated using models in this repository or custom ones. The repository also includes tutorials for integrating Voice Assistants into Bot-Framework bots and a client tool for testing Voice Assistant systems. ### Key Features - **C# Clients**: Sample clients for UWP and WPF platforms. - **C++ Clients**: Sample client for Smart Speaker devices. - **Java Clients**: Sample client for Android and iOS platforms. - **Regression Testing Tool**: Tool for end-to-end testing of Voice Assistant systems. ### Tech Stack - **Language**: C#, C++, Java - **Platform**: UWP, WPF, Android, iOS, Smart Speaker ### Why It Matters - **Integration with Bot-Framework**: Easily integrate Voice Assistants into existing Bot-Framework bots. - **Keyword Activation**: Use models provided in this repository or create custom ones for keyword activation. - **Robust Testing**: Comprehensive testing tools for Voice Assistant systems. ### GitHub Stars - **GitHub Stars**: 121 ### Repository Description Welcome to the Microsoft Cognitive Services Voice Assistant samples repository! This repository provides samples to help you get started building Voice Assistants for your bot or Custom Commands service. You can deploy a working Voice Assistant to your own Azure subscription using the provided samples. ### Sample Code - **Clients**: Sample clients for UWP, WPF, Android

Overview: Generative models learn the underlying data distribution in an unsupervised manner, but many applications require specific sampling or even distributional control. This project introduces **Generative Visual Prompt (PromptGen)**, a framework that allows efficient control over pre-trained generative models using knowledge from off-the-shelf models. Key features: - CLIP-guided text-to-image synthesis - Inverse graphics model-guided image synthesis - Classifier-guided image synthesis - De-biasing generative models with the moment constraint - Iterative control Tech stack: Python Why it matters: PromptGen provides a general and modular approach to controlling generative models, enabling efficient and accurate sampling without the need for training data. It also offers iterative controls, addressing the issue of bias in controlled distributions. Repository: ChenWu98/generative-visual-prompt Primary language: Python GitHub stars: 121 GitHub description: NeurIPS 2022 paper on **Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models**.

Overview: 这个项目旨在为学习者提供一个系统性的指南，帮助他们从理论知识到实际操作，再到应用落地，全面掌握大模型的训练和应用过程。该项目涵盖了从理论基础到工业框架，再到智能体和应用的完整学习路线，特别强调了在单张消费级显卡上实现预训练、对齐、微调和落地应用。 Key features: - 理论基础：包括CS336中文精讲，涉及分词、Transformer、注意力变体、MoE、GPU/并行、Scaling Law、推理优化、对齐与GRPO。 - 手写训练：使用原生PyTorch和Triton进行训练，包括BPE分词器、RMSNorm/RoPE/GQA/SwiGLU等组件，并进行数值自检。 - 工业框架：源码级走读LLaMA-Factory四层链路，对齐、PEFT、量化选型笔记，双卡DDP/ZeRO分布式实测，垂直领域微调方案。 - Agent工程：终端编程Agent：ReAct + Harness、Skill路由、记忆闭环、分层上下文压缩、主从多智能体、分层安全。 - RAG应用：医学文献混合检索问答，BM25+稠密双路、加权融合、Rerank、HyDE/QE、带引文生成、ReAct编排、Streamlit。 Tech stack: Python Why it matters: 该项目为学习者

### Overview: `micapipe` is a semi-flexible and robust framework designed for processing MRI images and generating ready-to-use modality-based connectomes. Developed by the MICA-lab at McGill University, it utilizes a set of known software dependencies, brain atlases, and software developed in the lab. ### Key Features: - **Microstructure Profile Covariance**: A method for analyzing the covariance of microstructural profiles. - **Multiple Parcellations (18 x 3)**: The pipeline supports 18 parcellations with 3 layers each. - **Cerebellum and Subcortical Areas**: The pipeline includes the cerebellum and subcortical areas. - **Surface Based Analysis**: The pipeline performs surface-based analysis. - **Latest Software Dependencies**: The pipeline uses the latest versions of software dependencies. - **Ready-to-Use Outputs**: The outputs are standardized in the BIDS format. - **Easy to Use**: The pipeline is designed to be easy to use. - **Standardized Format (BIDS)**: The outputs are in the BIDS format. ### Tech Stack: - Shell ### Why It Matters: The `micapipe` pipeline is essential for processing multimodal neuroimaging data, providing a standardized and flexible framework for researchers and clinicians to analyze and visualize brain connectomes.

Overview: This project focuses on designing multi-agent systems (MAS) using zero supervision, aiming to create autonomous and self-reinforcing systems that can learn and improve without human intervention. Key features: - Implementing zero supervision in multi-agent systems - Reinforced self-play reasoning for autonomous learning - Designing a multi-agent system with self-reinforcement capabilities Tech stack: Python Why it matters: This project is significant because it addresses the challenge of creating autonomous systems that can learn and improve without human oversight, which is crucial for various applications in robotics, game development, and more. Repository: SalesforceAIResearch/MAS-Zero GitHub stars: 121 GitHub description: Designing Multi-Agent Systems with Zero Supervision

Overview: FollowBench is a benchmark designed to evaluate the instruction-following capability of large language models (LLMs) across various levels of difficulty, including Content, Situation, Style, Format, and Example constraints. It provides a comprehensive set of fine-grained constraints and a multi-level mechanism to handle diverse instruction-following challenges. Key features: - Comprehensive fine-grained constraints: Includes five types of constraints (Content, Situation, Style, Format, and Example). - Multi-level mechanism: Adds constraints incrementally at each level to ensure precise estimation. - Constraint-evolution paths: Used to prompt strong LLMs to handle challenging open-ended instructions. - 13 popular LLMs evaluated on FollowBench: Highlights weaknesses and points to future work. Tech stack: Python Why it matters: FollowBench offers a valuable tool for evaluating the instruction-following capabilities of LLMs, providing insights into their strengths and weaknesses, and guiding future research directions. Repository: YJiangcm/FollowBench GitHub stars: 121 GitHub description: [ACL 2024] FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models GitHub URL: [https://github.com/YJiangcm/FollowBench](https://github.com/YJiangcm/FollowBench)

Overview: SparkWork 是一款面向专业开发者与重度知识工作者的本地优先桌面 Agent 工作台，提供任务编排、协作、智能路由、自定义工具和审查功能。它运行于 macOS、Windows 和 Linux 并支持多种编程语言。 Key features: - 可编排工作流 - 会话内挂载工作流 - 智能路由 - 自定义工具与应用 - 可审查的执行 Tech stack: TypeScript Why it matters: SparkWork 是一款高效、本地优先的 Agent 工作台，专为专业软件开发和复杂工作设计，提供从任务编排到代码审查的全面解决方案，适用于 macOS、Windows 和 Linux 平台。

Overview: This project provides advanced document extraction and chunking techniques for retrieval augmented generation that is aware of the layout of documents. It enhances knowledge retrieval accuracy and allows for better control over retrieved knowledge context management. Key features: - Amazon Textract Layout feature for extracting document text while maintaining its layout and reading format. - Integration with Amazon SageMaker to deploy embedding and text generation models. - Use of IAM for fine-grained access control. Tech stack: Jupyter Notebook Why it matters: This solution improves Retrieval Augmented Generation (RAG) performance by enhancing the accuracy of knowledge retrieval and providing better context management for retrieved information. Repository: aws-samples/layout-aware-document-processing-and-retrieval-augmented-generation GitHub stars: 121