Projects

### Overview: This repository contains the code for the paper "Blue noise for diffusion models" presented at SIGGRAPH 2024. The paper investigates the impact of noise correlation in diffusion models and introduces a novel time-varying noise model transitioning from Gaussian white to blue noise for improved denoising in diffusion models. Additionally, it presents a fast method for generating blue noise using matrix operations and tiling, and evaluates the framework on various tasks including image generation, super-resolution, and latent diffusion. ### Key features: - A novel time-varying noise model transitioning from Gaussian white to blue noise for diffusion models. - A fast method for generating blue noise using matrix operations and tiling. - Evaluations on image generation, super-resolution, and latent diffusion. ### Tech stack: - Python ### Why it matters: This work provides a significant contribution to the field of diffusion models by introducing a more effective noise model and efficient noise generation method, leading to improved performance in various applications.

Overview: MultiEYE is an official implementation of a novel setting for OCT-enhanced disease recognition from fundus images, presented at the IEEE TMI 2024 conference. This dataset and benchmark aim to improve the accuracy of eye disease diagnosis using unpaired multi-modal data. Key features: - Multi-modal multi-disease classification dataset, MultiEYE, assembled from 12 public fundus datasets and 4 OCT datasets. - Large multi-modal multi-class dataset for eye disease diagnosis, MultiEYE. - OCT-assisted Conceptual Distillation Approach (OCT-CoDA) for extracting disease-related knowledge from OCT images and leveraging it into the fundus model. - Novel setting for OCT-enhanced disease recognition from fundus images, mimicking real clinical circumstances. Tech stack: Python, PyTorch, Transformers, GPT-4. Why it matters: This dataset and approach provide a valuable resource for researchers and practitioners in the field of ocular imaging, aiming to enhance the accuracy and efficiency of disease diagnosis through the integration of OCT and fundus images.

### Overview: This project is an implementation of the paper "Use Your INSTINCT: Instruction Optimization Using Neural Bandits Coupled with Transformers" presented at ICML 2024. It provides all the codes for the experiments, including instruction induction and improving chain-of-thought instructions. The code is based on the APE and InstructZero repositories and uses neural bandits to optimize instructions for black-box LLMs, significantly improving performance. ### Key features: - Instruction induction - Improving chain-of-thought instructions ### Tech stack: Python ### Why it matters: This project is significant because it demonstrates how neural bandits can be used to optimize instructions for LLMs, leading to improved performance in various tasks.

Overview: DiffPhyCon is a novel method for controlling complex physical systems using generative models, presented at NeurIPS 2024. It aims to minimize the learned generative energy function while specifying control objectives. Key features: - Minimizes the learned generative energy function - Specifies control objectives - Enhances DiffPhyCon with prior reweighting Tech stack: Jupyter Notebook Why it matters: DiffPhyCon offers a new approach to control complex physical systems by leveraging generative models, making it particularly useful for applications requiring precise control and minimal disturbance. Repository: AI4Science-WestlakeU/diffphycon Primary language: Jupyter Notebook GitHub stars: 52 GitHub description: [NeurIPS2024] DiffPhyCon uses generative models to control complex physical systems. [Paper](https://openreview.net/forum?id=MbZuh8L0Xg) | [arXiv](https://web3.arxiv.org/abs/2407.06494) | [Project page](https://ai4s.lab.westlake.edu.cn/projects/diffphycon/)

### Overview: Logic-RL-Lite is a lightweight replication study of the DeepSeek-R1-Zero framework, focusing on pure reinforcement learning (RL) without supervised fine-tuning (SFT) to post-train base models for reasoning capabilities. It leverages the RL Framework [verl], RL Algorithms REINFORCE++ and GRPO, and the RL Dataset Knights and Knaves (K&K) Logic Puzzle Dataset. The project investigates the impact of base model selection and finds that Qwen2.5-3B is the smallest model capable of learning reasoning, with Llama3.2-3B succeeding in larger models. The findings also reveal the absence of "Aha Moment" during pure RL and the importance of base model selection. ### Key features: - RL Framework: verl - RL Algorithms: REINFORCE++ and GRPO - RL Dataset: Knights and Knaves (K&K) Logic Puzzle Dataset - Base Models: Qwen2.5-3B and Llama3.2-3B ### Tech stack: Python ### Why it matters: The findings highlight the importance of base model selection in reinforcement learning tasks and the limitations of pure RL in certain scenarios. This project contributes to the understanding of how different models perform in reasoning tasks and provides insights into the effectiveness of instruction tuning in reinforcement learning.

Overview: This decentralized expense tracker application allows users to securely manage and track their income and expense transactions, providing visualizations and the ability to export data to Excel. It utilizes blockchain technology for secure storage and decentralized hosting. Key features: - Data Security - Graphical Visualization - Top Diversifications - Exporting Data to Excel - Voice Recognition Tech stack: TypeScript, Next JS, React, Bootstrap, Chart.js, Solidity, Ethereum, Hardhat, Fleek, Goerli Etherscan, Alchemy. Why it matters: This application offers a secure and decentralized way to manage financial transactions, ensuring data integrity and privacy.

### Overview: This repository contains the official code and data for the paper "Dataset Enhancement with Instance-Level Augmentations" by Orest Kupyn and Christian Rupprecht. The repository includes an augmented version of the COCO dataset, focusing on various classes such as cars, people, and saliency detection, among others. The augmented datasets are designed to improve the diversity and match the scene composition of the original datasets, making them suitable for training and evaluation in computer vision tasks. ### Key features: - **COCO Augmented Car**: Focuses on the "car" class. - **COCO Augmented People**: Focuses on the "person" class. - **COCO Augmented**: An augmented version of the entire COCO dataset. - **DUTS Augmented**: An augmented version of the DUTS dataset. - **DUTS SDXL (Experimental)**: A larger, augmented version of the DUTS dataset. - **SHA512 Checksums**: A file containing the SHA512 checksums for the augmented datasets, ensuring integrity. ### Tech stack: Python 3.8 ### Why it matters: This repository is essential for researchers and developers looking to enhance their datasets for computer vision tasks. The instance-level augmentations provided in this repository can significantly improve the diversity and realism of training datasets, making them more effective for training models that require accurate object recognition and segmentation.

Overview: S3OD is a large-scale fully synthetic dataset designed for salient object detection and background removal, featuring 140K high-quality images generated using diffusion models. The model trained on this dataset achieves state-of-the-art performance on real-world benchmarks and enables effective single-step background removal from images in high resolution. Key features: - 140K high-quality images with detailed salient object masks - 1,000+ ImageNet categories - Variable resolution during training - Parquet format storage for efficient dataset management Tech stack: Python Why it matters: S3OD provides a comprehensive dataset for researchers and practitioners to evaluate and improve salient object detection models, enabling more accurate and efficient background removal in real-world applications. Repository: KupynOrest/s3od Primary language: Python GitHub stars: 51 GitHub description: [ICLR 2026] Official repo for S3OD: Towards Generalizable Salient Object Detection with Synthetic Data

### Overview: This repository demonstrates running Ollama with Intel GPUs, utilizing the `ipex-llm` backend for accelerated inference, compatible with both Intel iGPUs and dedicated GPUs. The provided `docker-compose.yml` includes a patched version of Ollama for Intel acceleration, along with an Open WebUI interface for convenience. ### Key Features: - Intel iGPU acceleration - Dedicated GPU support - Open WebUI interface for easy access - Docker and Docker Compose integration ### Tech Stack: Python ### Why it Matters: This setup allows for efficient and accelerated inference of Ollama models, making it suitable for both CPU and GPU environments, particularly beneficial for tasks requiring high performance and low resource usage. ### Repository: NikolasEnt/ollama-webui-intel - Primary language: Python - GitHub stars: 52 - GitHub description: Ollama with Intel GPUs - Readme excerpt: Demonstrates running Ollama with Intel GPUs, including Docker and Open WebUI interface, with benchmark results. Requires Intel firmware and Docker for setup. ### Quick Start: 1. Install Intel firmware and update initramfs. 2. Build and run Docker Compose with the provided `docker-compose.yml` file. 3. Access Open WebUI at `http://127.0.0.1:18080`. ### Parameters: - Adjust volumes, memory limits, and service configurations as needed. - Use `cpuset` to pin the `ipex_ollama` service to specific CPU cores for optimal performance

### Overview: RoboPaintBench is a benchmark and playground for multimodal language models designed to test their ability to interpret and execute painting goals through text prompts and camera feedback. ### Key Features: - **Reference Image Painting:** Replays of models using shared photographs and text instructions. - **Text-Prompt Painting:** Models respond to written prompts without reference images. - **Custom Painting:** Allows users to input their own prompts and reference images. ### Tech Stack: AgentArk ### Why It Matters: RoboPaintBench provides a comprehensive environment for evaluating and benchmarking multimodal language models, focusing on their capability to interpret and execute painting goals through text and camera feedback. ### Repository: P90-RushB/robopaint-bench - Primary Language: Jupyter Notebook - GitHub Stars: 52 - GitHub Description: A robot-painting benchmark and playground for multimodal language models: reference images, text prompts, and custom goals. Built on AgentArk.

Overview: Engram-PEFT is an unofficial implementation of the DeepSeek Engram architecture, designed to inject high-capacity conditional memory into large language models (LLMs) using sparse retrieval parameter-efficient fine-tuning (PEFT). Key features: - Injects conditional memory into any Transformer-based LLM - Supports stacked-adapter and full-finetuning workflows - Uses sparse retrieval to efficiently store and access factual information without increasing inference FLOPs - Automatically detects and configures NPU settings for optimal performance Tech stack: Python Why it matters: Engram-PEFT provides a high-performance, paper-aligned implementation of the DeepSeek Engram architecture, allowing for efficient integration of conditional memory into LLMs without significant performance overhead. Repository: QingGo/engram-peft GitHub stars: 52

### Overview: Fast-paced lectures designed to introduce new students to the fundamentals of robotics, including perception, navigation, control, and learning. ### Key features: - **Lectures**: 11:00 AM – 1:00 PM each day (Sundays off) - **Hands-on Sessions**: 2:00 PM onwards with discussions and Q&A over Slack - **Location**: H-101/102 (Himalaya Block) - **Mode**: Primarily in-person (hybrid for remote participants) ### Tech stack: - Jupyter Notebook ### Why it matters: The summer school provides a comprehensive introduction to robotics, essential for students and researchers interested in the field.

Overview: This project provides comprehensive documentation and annotated code for using the Chinese-LLaMA-Alpaca model, including detailed instructions and core code annotations. Key features: - Comprehensive documentation for using the Chinese-LLaMA-Alpaca model. - Core code annotations for the model's implementation and usage. - Detailed explanations of the model's architecture and functionality. - A step-by-step guide for data preparation and model conversion. Tech stack: Jupyter Notebook Why it matters: This project serves as a valuable resource for developers looking to integrate the Chinese-LLaMA-Alpaca model into their projects, offering clear instructions and annotated code to ensure a smooth and effective implementation. Repository: WangRongsheng/Chinese-LLaMA-Alpaca-Usage GitHub stars: 52 GitHub description: 对Chinese-LLaMA-Alpaca进行使用说明和核心代码注解 README excerpt: <center><img src="./assets/images/home.png" height="50%"></center> 本项目基于[Chinese-LLaMA-Alpaca V3.1](https://github.com/ymcui/Chinese-LLaMA-Alpaca/tree/v3.1) 进行使用说明。[Chinese-LLaMA-Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca) 开创了基于LLaMA的中文扩充改进，在原版LLaMA的基础上扩充了中文词表并使用了中文数据进行二次预训练，进一步提升了中文基础语义理解能力

Overview: This repository provides an unofficial PyTorch implementation of the LIMoE model, a multimodal contrastive learning approach based on the paper "Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts". Key features: - Multimodal contrastive learning - LIMoE model architecture - PyTorch framework Tech stack: PyTorch Why it matters: This implementation offers a practical tool for developers interested in multimodal contrastive learning, providing a straightforward way to experiment with and potentially improve upon the LIMoE model.

### Overview: Llama3.cu is a lightweight CUDA implementation of the LLaMA3 architecture designed for efficient causal language modeling on Nvidia GPUs. This project leverages the transformer principles from seminal papers, enabling scalable parallel processing and optimized inference performance. ### Key Features: - CUDA-native implementation of LLaMA3 architecture - Custom CUDA kernel definitions for transformer core - Parameter weights stored as BF16 in .safetensor format - Compatible with HuggingFace models and supported by CUDA Toolkit 7.5+ ### Tech Stack: CUDA C ### Why it Matters: This project offers a highly efficient and scalable solution for language model inference, particularly beneficial for applications requiring real-time processing on GPUs with sufficient VRAM.

### Overview: Kling 3.0 Motion Control prompts offer tested image prompt recipes for various applications, including commercial product photography, cinematic scenes, and more. This repository provides a set of 4 recipes with example renders, aspect ratios, and the cost per image on the APIMart route. ### Key Features: - **Model ID:** `kling-v3-motion-control` - **Recipes:** 4 (available in `data/prompts.json` and `PROMPTS.md`) - **Effective Price:** $0.1029 per second — estimated cost of the set: $0.4116 ### Tech Stack: Python ### Why It Matters: - **Key:** Same Base-URL and `Authorization`-Header for multiple models and pricing. - **Minimum Cost:** $1 per call, pay-as-you-go. - **Real-Time Pricing:** Cost is reflected in the API response. - **Asynchronous:** Task ID is required, and calls can be batched and retried. ### What Separates a Usable Recipe from a Wasted Call: - **Material vs. Adjective:** "Frosted glass" vs. "beautiful" or "high quality". - **Light Direction:** "Single softbox from the left" vs. "golden hour backlight". - **Framing:** "3:4, medium format look" vs. "focus-stacked studio light". ### Prompt Structure Cheat-Sheet: | Slot | What to Write | Example | | --- | --- | --- | | **Subject** |

Overview: OAKMower is a project that utilizes the OpenCV Spatial AI Competition 3rd prize winner, aiming to enable automatic lawn mower navigation using multi-modal obstacle and limit detection with object detection and depth estimation and classification simultaneously. Key features: - Automatic lawn mower navigation - Multi-modal obstacle and limit detection - Object detection and depth estimation and classification Tech stack: Python Why it matters: OAKMower showcases the potential of combining object detection and depth estimation for obstacle and limit detection, demonstrating the effectiveness of multi-modal AI in real-world applications like lawn mower navigation. Repository: alemamm/oakmower Primary language: Python GitHub stars: 52 GitHub description: OpenCV Spatial AI Competition 3rd prize winner project "Automatic Lawn Mower Navigation" README excerpt:

Overview: This project implements a meta-transfer learning approach for code-switched speech recognition, specifically for ASR and LM tasks. The method leverages high-resource monolingual datasets to improve the recognition of mixed-language speech in low-resource settings. Key features: - Meta-transfer learning for ASR and LM - Implementation using PyTorch - Data from SEAME Phase II, HKUST, and CommonVoice v2 Tech stack: Python Why it matters: This project addresses the challenge of building speech recognition systems for code-switching, which is crucial for multilingual communication. The meta-transfer learning approach makes it possible to achieve better performance with limited resources, making it valuable for applications where data is scarce. Repository: audioku/meta-transfer-learning Primary language: Python GitHub stars: 52 GitHub description: Implementation of meta-transfer-learning for ASR and LM (ACL 2020)

### Overview: TEMU-VTOFF is a dual-DiT architecture designed for generating clean, in-shop images of garments worn by individuals. It addresses the Virtual Try-Off task by combining a pretrained feature extractor with a text-enhanced generation module, enabling the generation of multiple garment categories and handling occlusions and ambiguous appearances. ### Key Features: - **Unified Framework for Multi-Category Try-Off**: Handles multiple garment types (upper-body, lower-body, and full-body clothes) without category-specific pipelines. - **Multimodal Hybrid Attention**: Integrates garment textual descriptions into the generative process using a novel attention mechanism that links text to the generation. ### Tech Stack: Python ### Why it Matters: TEMU-VTOFF offers a comprehensive solution for generating realistic, multi-category virtual try-on images, addressing the challenges of garment variety, category ambiguity, and occlusions. This method enhances the authenticity and usability of virtual try-on experiences, making it a valuable tool for e-commerce and fashion retail.

### Overview: LoRANN is an approximate nearest neighbor search library that leverages reduced-rank regression for extremely fast queries, minimal memory usage, and rapid indexing on modern embedding datasets. ### Key Features: - **Query Speed Matching State-of-the-Art**: Achieves comparable performance to state-of-the-art methods but with significantly reduced memory usage. - **Tiny Memory Usage**: Efficiently handles datasets with dimensions greater than 100, optimizing for modern CPU architectures. - **Rapid Indexing**: Fast indexing and search capabilities, optimized for modern CPU and GPU architectures. - **GPU Batch Queries (Experimental)**: Experimental support for GPU batch queries to enhance performance. - **Supported Data Types**: Float32, Float16, BFloat16, Uint8, Binary. - **Supported Distances**: Inner Product, Euclidean Distance, Cosine Distance. - **Index Serialization**: Support for index serialization for easy data migration. ### Tech Stack: Header-Only C++17 Library with Python Bindings ### Why It Matters: LoRANN is optimized for modern high-dimensional embedding datasets, providing extremely fast queries and minimal memory usage, making it ideal for applications requiring high performance and efficient data handling.