Projects

## Overview: This repository provides an implementation of the NeurIPS 2023 paper "Spontaneous Symmetry Breaking in Generative Diffusion Models". It showcases the phenomenon of spontaneous symmetry breaking in diffusion models, which divides the generative dynamics into two distinct phases: a linear steady-state dynamics around a central fixed-point and an attractor dynamics directed towards the data manifold. This split is marked by a significant change in the potential well's shape, leading to a shift in the dynamics and effectively illustrating these two generative phases. ## Key features: - Symmetry Breaking in Diffusion Models: A detailed explanation of the phenomenon of spontaneous symmetry breaking in diffusion models. - Video Paper Presentation: A video explanation of the paper. - One-Dimensional Case: Demonstrates the symmetry-breaking event in the one-dimensional case. - Gaussian Late Initialization Scheme: A proposed solution to improve model performance by significantly improving model performance, achieving up to 3x FID improvements on fast samplers, while also increasing sample diversity. ## Tech stack: Jupyter Notebook ## Why it matters: Our findings challenge the current dominant conception that the generative process of diffusion models is essentially comprised of a single denoising phase. In particular, we show that an accurate simulation of the early dynamics does not significantly contribute to the final generation, since early fluctuations are reverted to the central fixed point. This insight can significantly improve model performance and reduce bias in fast samplers. ## Repository: - Repository: gabrielraya/symmetry_breaking_diffusion_models - GitHub stars: 4

### Overview: MedMO is an advanced multimodal foundation model designed for comprehensive medical image understanding and grounding. It has been trained on over 26 million diverse medical samples across 45 datasets, achieving state-of-the-art performance across various medical imaging tasks. ### Key Features: - **Visual Question Answering (VQA)**: Answers complex medical image questions. - **Text-Based Medical QA**: Clinical reasoning and medical knowledge question answering. - **Radiology Report Generation**: Generates detailed radiology reports. - **Disease Localization with Bounding Boxes**: Precise localization of pathological findings. - **Anatomical Grounding**: Spatial localization and grounding of anatomical structures. - **Clinical Reasoning**: Step-by-step diagnostic reasoning and clinical decision support. - **Diagnostic Classification**: Multi-class disease classification. - **Spatial Object Detection**: Fine-grained detection in microscopy, pathology slides, and cellular imaging. - **Medical Report Summarization**: Extracts and summarizes key clinical findings from complex medical reports. ### Tech Stack: Python ### Why it Matters: MedMO-8B-Next is the latest and most powerful iteration of the MedMO family, achieving the highest average scores on both medical VQA and Text QA benchmarks, surpassing strong baselines including Lingshu-7B and Fleming-VL-8B.

Overview: Booster is an alignment stage defense mechanism designed to mitigate harmful fine-tuning attacks on large language models. It enhances the robustness of the aligned model by simulating and attenuating harmful perturbations from alignment and harmful datasets. Key features: - Simulates harmful perturbations at the alignment stage to prevent forgetting of alignment knowledge. - Attenuates the impact of harmful perturbations on the aligned model. - Implements a customized trainer on top of the Hugging Face Trainer to achieve Booster. Tech stack: Shell Why it matters: Booster is crucial for ensuring the safety and reliability of large language models by preventing harmful fine-tuning attacks, thereby maintaining the integrity and effectiveness of the model in real-world applications.

Overview: This repository aims to train Stable Diffusion 3 from scratch using multiple GPUs. Key features: - Training scripts - Sample images of model outputs at 512 resolution Tech stack: Python Why it matters: The scripts could be useful for others to replicate the training process and potentially improve upon it. GitHub stars: 43

### Overview: Neptune-X is a novel framework designed to address the challenges of maritime object detection, particularly focusing on the scarcity of annotated maritime data and poor generalization across various maritime attributes. The project aims to enhance training effectiveness by leveraging synthetic data generation with task-aware sample selection, aiming to improve detection accuracy in challenging and previously underrepresented settings. ### Key features: - Neptune-X is a data-centric generative-selection framework. - X-to-Maritime: A multi-modality-conditioned generative model that synthesizes diverse and realistic maritime scenes. - Bidirectional Object-Water Attention module: Captures boundary interactions between objects and their aquatic surroundings to improve visual fidelity. - Attribute-correlated Active Sampling: Dynamically selects synthetic samples based on their task relevance to enhance downstream tasking performance. ### Tech stack: Python ### Why it matters: Neptune-X sets a new benchmark in maritime scene synthesis, significantly improving detection accuracy, particularly in challenging and previously underrepresented settings.

Overview: This project provides an uncensored version of DeepSeek-V4.1 Flash running on a single Google Colab GPU, offering up to 100 tokens per second and no server requirement. It utilizes Google AI Pro or Colab Pro subscriptions for optimal performance. Key features: - Uncensored model - 100 tokens per second - Runs on a Google AI Pro subscription - Supports image input and text-only requests - Context size up to 131K tokens - Decoding speed of 105.6 tok/s - Image input option available - Context per request of 262,144 tokens - First token time of 9.9 s - Follow-up turns time of 1 s Tech stack: Python Why it matters: This project offers a high-performance, low-latency endpoint for natural language processing tasks, ideal for developers and researchers who need a powerful, open-source alternative to commercial AI solutions.

Overview: This project implements Frame-level Speech Signal-to-Noise Ratio Estimation using deep learning, providing a PyTorch implementation based on the research paper from ISCA. Key features: - Distributed training - Trained on LibriSpeech dataset - Includes noise collected from various sources Tech stack: PyTorch Why it matters: This project offers a practical solution for estimating the Signal-to-Noise Ratio (SNR) of speech signals, which is crucial in various applications such as speech recognition and noise reduction.

### Overview: SynthVision is a comprehensive synthetic medical VQA dataset and VLM fine-tuning pipeline. It leverages 119,000 medical images annotated by two frontier VLMs (Qwen 3.5 and Kimi K2.5) to generate a 110,000 training dataset. Fine-tuning three small models (2-3B params) significantly improves various benchmarks, including +15.0% average exact match. ### Key Features: - **Dataset Generation**: 119,000 medical images from four open datasets (ROCO, MultiCaRe, PathVQA, VQA-RAD). - **Annotation Pipeline**: Multi-turn clinical annotation using a VLM API. - **Training Data Preparation**: Merge and validate annotations, then convert to ShareGPT JSONL format. - **Fine-tuning**: LoRA fine-tuning with assistant-only label masking, supporting three model families. - **Benchmarking**: Evaluates on VQA-RAD, PathVQA, and SLAKE using vLLM batched inference. ### Tech Stack: - Python ### Why It Matters: SynthVision provides a robust and scalable solution for medical VQA research, offering improved accuracy and benchmark performance through fine-tuning on multiple model families. ### Repository: - **Repository**: openmed-labs/synthvision - **GitHub Stars**: 43 - **GitHub Description**: Synthetic medical VQA dataset generation and VLM fine-tuning pipeline. ### Setup: - **Requirements**: Python

### Overview: VoxIntel Engine is a real-time multi-modal speech telemetry and acoustic DSP engine built with FastAPI and Python 3.11. It analyzes technical presentations, executive pitches, and mock interview speeches across multiple parallel acoustic and semantic streams, providing insights into fundamental pitch, RMS energy, silence/hesitation pauses, and lexical pacing. The system is designed for high throughput and asynchronous processing, making it suitable for applications requiring real-time analysis and feedback. ### Key Features: - **Acoustic DSP Analyzer:** - **Fundamental Pitch (F0):** Estimates vocal pitch register using normalized autocorrelation over Fourier transform frames. - **Intonation Variety & Pitch Jitter ($\sigma_{F_0}$):** Measures standard deviation around mean pitch to detect flat/monotone delivery vs expressive presentation style. - **RMS Energy & Volume (dB):** Measures loudness and dynamic contrast in decibels. - **Micro-Pause Segmentation:** Automatically detects hesitation gaps and natural rhythmic pauses. - **Lexical & NLP Metrics:** - **Speaking Pace (WPM):** Classifies cadence against standard presentation benchmarks. - **Filler Word Density:** Measures the frequency of filler words in a speech. ### Tech Stack: Python 3.11 with FastAPI, NumPy, Docker, and GitHub Actions. ### Why It Matters: VoxIntel Engine provides a comprehensive solution for analyzing and optimizing speech presentations, making it invaluable for professionals in the fields of communication and education. Its ability to detect and analyze various aspects

Overview: This project is an implementation of the DDPM and DDIM diffusion models using PyTorch. The code aims to provide a comprehensive tool for understanding and experimenting with these models. Key features: - DDPM (Denoising Diffusion Probabilistic Models) implementation - DDIM (Denoising Diffusion Implicit Models) implementation - Support for CIFAR-10 and CelebA-HQ datasets - Inference capabilities with pre-trained model checkpoints Tech stack: PyTorch Why it matters: This project serves as a practical resource for developers and researchers interested in diffusion models, offering a detailed implementation and relevant performance metrics. Repository: taehoon-yoon/Diffusion-Probabilistic-Models Primary language: Python GitHub stars: 43 GitHub description: PyTorch implementation for DDPM & DDIM README excerpt: # DDPM & DDIM PyTorch ### DDPM & DDIM re-implementation with various functionality This code is the implementation code of the papers DDPM ([Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)) and DDIM ([Denoising Diffusion Implicit Models](https://arxiv.org/abs/2010.02502)). <img src="./images_README/cifar10_128_ex1.png" height="350" width="350"> &nbsp; &nbsp; &nbsp; <img src="./images_README

Overview: `arrowspace` is a Rust library designed for vectors analytics and search, utilizing dispersion models. It provides graph analysis, vector search, and energy-distribution statistics, all packaged into a single package. Key features: - Supports vectors with a graph representation and key-value store. - Focuses on AI search capabilities, graph characterization analysis, and indexing of high-dimensional vectors. - Implements a novel scoring method called `λτ` (lambda-tau) indexing, which combines Rayleigh and Laplacian scoring for better spectral-awareness. - Provides complete DevLog and examples for developers. Tech stack: Rust 1.90+ (edition 2024). Why it matters: `arrowspace` offers advanced vector similarity and search capabilities, graph analysis, and efficient indexing of high-dimensional vectors, making it a valuable tool for developers and researchers in the field of data analytics and search. Repository: `tuned-org-uk/arrowspace-rs` GitHub stars: 43 GitHub description: A database for vectors supported by a graph representation and a key-value store. Designed for AI search capabilities, graph characterization analysis, and indexing of high-dimensional vectors. Implements a novel scoring method called `λτ` (lambda-tau) indexing, which combines Rayleigh and Laplacian scoring for better spectral-awareness. Provides complete DevLog and examples for developers.

### Overview: This project is a demonstration for simple isolated Chinese speech word recognition using GMMHMM in Python. It utilizes the `hmmlearn` library to model audio files in 10 categories, specifically digits from 0 to 9. The project includes a demo script that can be run on Python 2.x, requiring the installation of `scikits.talkbox`, `hmmlearn`, and `scipy` packages. The demo provides recognition results on test sets and includes a workaround for a tricky bug in `hmmlearn`. ### Key features: - Simple isolated speech word recognition - Utilizes GMMHMM model from `hmmlearn` - Models audio files in 10 categories (digits 0-9) - Includes a demo script for running recognition results ### Tech stack: Python ### Why it matters: This project is useful for developers interested in speech recognition and machine learning, providing a practical example of isolated speech word recognition using GMMHMM in Python.

### Overview: StyleT2I is a framework designed to achieve compositional and high-fidelity text-to-image synthesis results. It addresses the issue where previous methods fail to generate attributes with good image quality when the input text contains underrepresented compositions of attributes. ### Key features: - **Compositional and High-Fidelity Text-to-Image Synthesis**: StyleT2I generates images that are both compositionally coherent and high in fidelity. - **Pretrained StyleGAN2 Model**: Utilizes a pre-trained StyleGAN2 model for efficient training. - **Cross-Dataset Generation**: Capable of generating images across different datasets, including CelebA-HQ, CUB, FFHQ, and NABirds. ### Tech stack: Python ### Why it matters: StyleT2I offers a significant improvement in text-to-image synthesis by ensuring better compositionality and high fidelity, making it a valuable tool for researchers and practitioners in the field of image synthesis and style transfer.

### Overview: IVM is a versatile visual grounding model designed for diverse multimodal tasks, including VQA and embodied robotic control, that utilizes instruction-guided visual masking to enhance model performance. ### Key features: - Instruction-guided Visual Masking (IVM) for enhancing model performance on complex instructions. - Compatible with LMM and robot models. - IVM-Mix-1M dataset for diverse multimodal tasks. - Discriminator Weighted Supervised Learning (DWSL) for prioritizing high-quality data samples. ### Tech stack: Jupyter Notebook ### Why it matters: IVM significantly boosts the performance of diverse multimodal models by focusing on task-relevant image regions, making it a versatile tool for various applications.

Overview: This comprehensive guide provides a universal process for preparing speech datasets, training and fine-tuning text-to-speech models, running inference, packaging models, and troubleshooting common problems. Key features: - Data Preparation - Training Setup - Model Training - Inference - Packaging and Sharing - Troubleshooting and Resources Tech stack: Python Why it matters: This guide offers a straightforward and accessible method for developers to create their own text-to-speech models, making it useful for both beginners and advanced users.

Overview: This repository is a curated collection of recent research papers on controllable generative models, focusing on UNet-based diffusion models and Transformer-based architectures. Ideal for researchers and developers exploring controllable synthesis. Key features: - 🧭 Control mechanisms: Papers on condition injection, adapters, and differentiable routers. - 👁️ Attention interpretation: Understanding what diffusion models focus on. - 🎛️ Frequency-based control: Using spectral domain knowledge for generation. - 🔁 Alignment & knowledge transfer: Enhancing coherent and faithful synthesis. - 🧑‍🎨 Image-to-image (I2I) editing: Flexible, structure-preserving transformation across domains. Tech stack: GitHub repository. Why it matters: This list aims to track the state-of-the-art in controllable generation and provide a well-organized knowledge map for newcomers and researchers building on diffusion models. Repository: Angusliuuu/Awesome-Controllable-Generative-Models-Papers GitHub stars: 42 GitHub description: A curated and continuously updated collection of recent research papers on controllable generative models, focusing on UNet-based diffusion models and Transformer-based architectures.

### Overview: This project proposes a Voice-based Email System utilizing AI, designed to make email communication accessible to visually challenged individuals and contribute to societal inclusivity. ### Key Features: - Voice-based Email System - AI-driven accessibility - Python-based technology stack - Django framework for backend development ### Tech Stack: - Python - Django - AI (Voice commands) ### Why it Matters: Accessibility is the primary concern while developing this system, ensuring that visually challenged individuals can communicate effectively and easily. ### Repository: Candida18/Virtual-Assistance-For-The-Blind ### GitHub Stars: 42 ### GitHub Description: The proposed Voice-based Email System aims to make email communication accessible to visually challenged people and contribute to societal inclusivity.

### Overview: DiffusionNAG is an official PyTorch implementation of "DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models" (ICLR 2024). This project introduces a novel predictor-guided Diffusion-based NAG (Neural Architecture Generation) framework, DiffusionNAG, which efficiently and effectively generates architectures that satisfy specific objectives. ### Key features: - DiffusionNAG leverages diffusion models to generate neural architectures. - It incorporates predictors into the generation process to enhance both the generative model and the evaluation of architectures. - The framework is designed to be efficient and effective, reducing the need for extensive search spaces and passive property predictors. ### Tech stack: - Python - PyTorch ### Why it matters: - DiffusionNAG addresses the inefficiencies and limitations of existing NAS (Neural Architecture Search) methods by integrating predictors into the generation process. - It offers superior utilization of predictors for both NAG and evaluation purposes, making it more adaptable across various tasks. - The project is based on the paper "DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models" (ICLR 2024). ### GitHub stars: 42 ### Repository: CownowAn/DiffusionNAG ### Primary language: Python ### GitHub description: Official PyTorch implementation of "DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models" (ICLR 2024) ### Environment Setup: - Requires Python 3.7.2 and Py

### Overview: DeepEnlighten is a lightweight study of pure reinforcement learning (RL) for post-training social reasoning capabilities, focusing on the DeepSeek-R1-Zero framework. Leveraging RL, it aims to generalize social reasoning from EQ to IQ tasks without fine-tuning. ### Key Features: - **RL Framework**: Verl - **RL Algorithms**: REINFORCE++ - **RL Dataset**: Social IQa - **Base Models**: Qwen2.5, Llama3.2 - **Math Evaluation**: DeepSeek-Math ### Tech Stack: Python ### Why it Matters: DeepEnlighten explores the generalization of social reasoning capabilities from EQ to IQ tasks, leveraging RL without fine-tuning, and demonstrates its effectiveness in mathematical reasoning tasks. ### GitHub Stars: 42 ### Repository: DolbyUUU/DeepEnlighten

Overview: This project is a QA bot designed to accelerate development within the Hugging Face ecosystem by leveraging Hugging Face's LLM and Retrieval Augmented Generation (RAG) techniques. It supports various interfaces including an API, Discord bot, and Gradio app, and provides access to detailed Hugging Face documentation. Key features: - Integrates with Hugging Face's LLM and RAG for intelligent question answering. - Operates as an API, Discord bot, or Gradio app. - Supports multiple interfaces for easy integration. - Offers links to the used documentation for each answer. - Provides context from the documentation for answers. - Includes sources of retrieved documents as context. - Enables conversational experience through chat history. - Allows for debugging and logging for development purposes. Tech stack: Python Why it matters: This QA bot streamlines the development process within the Hugging Face ecosystem by providing quick and accurate answers to questions about documentation, enhancing efficiency and reducing the time spent on research. Repository: KonradSzafer/hugging-face-qa-bot GitHub stars: 42 GitHub description: QA Bot for Hugging Face documentation to accelerate development within the ecosystem.