Projects

Overview: This repository is a 3-part series that explores using multimodal foundation models hosted on Amazon Bedrock and Amazon SageMaker to talk to your slide decks. It includes solutions for converting slides into embeddings, storing these embeddings in a vector store, and using LLaVA 1.5 to respond to user queries. Key features: - Implementing a solution design that converts slides into embeddings and stores them in Amazon OpenSearch Serverless. - Generating and storing LLaVA 1.5 inferences in a vector store for response to user queries. Tech stack: The repository uses Amazon Bedrock and Amazon SageMaker for hosting foundation models and Amazon OpenSearch Serverless for vector store storage. Why it matters: This series provides practical examples of how to leverage foundation models to interact with slide decks, showcasing the integration of Bedrock, SageMaker, and OpenSearch Serverless in a real-world application.

Overview: This project is an implementation of the Autoregressive Diffusion Model for graph generation, based on the research paper by Lingkai Kong et al. 2023. Key features: - An implementation of the Autoregressive Diffusion Model for graph generation. - The model corrupts the original graph by absorbing nodes and then generates the graph by sampling nodes in reverse order. - The project includes a forward diffusion chain and a reverse diffusion chain for generating graphs. Tech stack: Python Why it matters: This project provides a practical implementation of an advanced graph generation technique, which can be useful in various applications such as graph-based data processing, network analysis, and more. Repository: caio-freitas/GraphARM Primary language: Python GitHub stars: 46

Overview: This project utilizes the Gemini API to create a multimodal chat application, integrating text, images, and other media for enhanced user interaction. Key features: - Integrates multimodal chat functionality using LangChain and Google/Gemini API. - Streamlit-based interface for easy deployment and testing. - Supports Google/Gemini API for advanced text generation and multimodal capabilities. Tech stack: Python Why it matters: This project showcases the potential of integrating multiple modalities in chat applications, enhancing user engagement and interaction through multimodal communication. Repository: elizabethsiegle/gemini-multimodal-chat Primary language: Python GitHub stars: 46

### Overview: This collection features 252 curated prompts for HY Image 3.5, grouped into 9 categories, with preview images to help you choose the right one before running it. ### Key Features: - **Portrait & Avatar**: 28 prompts focusing on editorial studio and candid street portraits. - **Product & E-commerce**: 28 prompts for product and e-commerce scenarios. - **Poster & Typography**: 28 prompts for poster and typography designs. - **Infographic & Diagram**: 28 prompts for infographics and diagrams. - **Anime & Illustration**: 28 prompts for anime and illustration styles. - **3D & Material**: 28 prompts for 3D and material designs. - **Food & Lifestyle**: 28 prompts for food and lifestyle scenarios. - **Architecture & Interior**: 28 prompts for architecture and interior designs. - **Nature & Landscape**: 28 prompts for nature and landscape scenes. ### Tech Stack: The collection is built using HY Image 3.5, a text-to-image tool known for its flexible dimensions. ### Why It Matters: This collection provides a practical and curated set of prompts that work well, making it easier for developers to create professional-looking images without having to experiment with different settings.

Overview: HuggingBench is an extensible, open-source MLOps tool designed to help find the optimal model serving configuration for HuggingFace models. It generates multiple model serving configurations, deploys models, performs load testing, and collects metrics to determine the most efficient model serving. Key features: - Generates multiple model serving configurations - Deploys models using Docker containers - Performs load testing and collects metrics - Supports PyTorch, ONNX, and Triton Inference Servers Tech stack: Python 3.9 or 3.10 Why it matters: HuggingBench simplifies the process of finding the best model serving configuration, ensuring optimal performance and understanding of latency/throughput and hardware needs. Repository: huggingbench/huggingbench GitHub stars: 46 GitHub description: Find the optimal model serving solution for 🤗 Hugging Face models 🚀

### Overview: Unifi3D is a unified framework designed to evaluate the performance of 3D representations across various criteria such as quality, computational efficiency, and generalization. It compares these representations based on multiple benchmarks, including standard model testing, and aims to derive best practices for the entire 3D generation pipeline, from preprocessing to mesh reconstruction and autoencoder compression. ### Key features: - **Unified Framework**: Evaluates 3D representations across multiple criteria. - **Benchmarking**: Standard model testing and best practices derived for the entire pipeline. - **Loss Functions**: Loss functions including diffusion and autoencoders. - **Neural Models**: Relevant neural models and submodules. - **Data**: Torch dataset and sampler classes. - **Losses**: Loss functions including diffusion and autoencoders. - **Models**: Relevant neural models and submodules. - **Utils**: Useful functions including data, evaluation, rendering, and model components. - **Configurations**: Hydra configs for data, debug, experiment, extras, hparams_search, local, logger, model, paths, trainer, default.yaml, eval.yaml, benchmark.yaml, sample_diffusion.yaml, train.yaml. - **Checkpoints**: Symbolic links to saved checkpoints in the format of *.safetensors. ### Tech stack: Python ### Why it matters: Reconstruction and generation errors significantly impact overall performance, highlighting the need to evaluate these processes jointly.

### Overview: SteadyText is a deterministic text generation and embedding tool that requires no configuration. It leverages open-source AI workflows from Julep to ensure consistent and reliable outputs across different runs and environments. ### Key Features: - **Deterministic Output**: Ensures consistent results across different runs and environments. - **Zero Configuration**: No need for manual configuration or setup, making it ideal for developers and teams. - **Diverse Models**: Supports various model sizes, including small (4B) and large (30B) models. - **Streaming**: Supports streaming generation, providing real-time output. - **Temperature Control**: Allows for controlled randomness with adjustable temperature values. - **Model Switching**: Supports switching between different models for varying output quality and speed. ### Tech Stack: - Python: The primary language used for development and integration. - PostgreSQL: The backend database for storing and managing model data. ### Why It Matters: SteadyText simplifies the process of generating deterministic text and embeddings, ensuring consistent and reliable outputs. This is particularly useful in environments where reproducibility and reliability are critical, such as testing, tooling, and continuous integration/continuous deployment (CI/CD) pipelines.

### Overview: MedSegFactory is a medical image synthesis framework that generates high-quality paired medical images and segmentation masks. It aims to serve as an unlimited data repository, enhancing existing segmentation tools by providing image-mask pairs. The core of MedSegFactory is a dual-stream diffusion model, where one stream synthesizes medical images and the other generates corresponding segmentation masks. To ensure precise alignment, Joint Cross-Attention (JCA) is introduced, enabling a collaborative denoising paradigm. ### Key Features: - **Dual-Stream Diffusion Model**: One stream synthesizes medical images, while the other generates segmentation masks. - **Joint Cross-Attention (JCA)**: Dynamic cross-conditioning between streams to enhance semantic consistency. - **User-Defined Prompts**: Specify target labels, imaging modalities, anatomical regions, and pathological conditions for on-demand generation. - **High-Quality Data**: Generates superior quality and usability paired medical images and segmentation masks, achieving competitive or state-of-the-art performance in various segmentation tasks. ### Tech Stack: - **Python** - **Stable Diffusion Model** - **Hugging Face** - **nnUNet** ### Why It Matters: MedSegFactory unlocks on-demand generation of paired medical images and segmentation masks through user-defined prompts, facilitating seamless integration into diverse medical imaging workflows. This new paradigm of medical image synthesis enhances efficiency and accuracy, addressing data scarcity and regulatory constraints. Extensive experiments demonstrate superior quality and usability, achieving competitive or state-of-the-art performance in 2D and 3D segmentation tasks.

### Overview: NoiseKit is a Python library designed to generate degraded speech datasets for benchmarking noise-robust Automatic Speech Recognition (ASR) systems. It provides a variety of real-world degradation presets and atomic noise scenarios, along with compound multi-condition scenarios, to simulate various real-world noise conditions. ### Key Features: - **Atomic Presets**: Six atomic noise and codec degradation presets, including telephony, wideband codec compression, ambient noise, clipping distortion, and far-field reverb. - **Compound Scenarios**: Composite multi-condition scenarios that combine multiple atomic presets. - **Real-World Degradations**: Simulates real-world noise conditions programmatically. - **Benchmarking Metrics**: Scores each output using PESQ, SNR, and NISQA, providing a JSONL manifest ready for noise-robustness benchmarking. - **Supported Datasets**: Compatible with HuggingFace datasets, allowing for easy integration and benchmarking. ### Tech Stack: - **Python 3.10+**: The library is built using Python 3.10 and above. - **UV**: A library for audio processing and manipulation. - **audiomentations**: A library for applying real-world noise and degradation presets to audio files. ### Why It Matters: - **Real-World Simulations**: The library simulates various real-world noise conditions, ensuring that benchmarks are as close to production as possible. - **Versatile Degradation**: Offers a wide range of atomic and compound noise scenarios, making it suitable for a variety of ASR benchmarking needs. - **Benchmark

Overview: ChemLIFT is a language-interfaced fine-tuning project for chemistry, enabling predictions using large language models. Key features: - Few-shot learning: Provides examples for model training. - Fine-tuning: Updates model weights on a specific dataset. Tech stack: Jupyter Notebook Why it matters: It facilitates the integration of large language models into chemical data analysis, making predictions more accessible and powerful. Repository: lamalab-org/chemlift GitHub stars: 46 GitHub description: Language-interfaced fine-tuning for chemistry

Overview: This repository contains the official implementation of the paper "Diffusion Augmentation for Sequential Recommendation", focusing on the DiffuASR model for sequential recommendation tasks. Key features: - Diffusion Augmentation for Sequential Recommendation - Model Implementation - Data Preprocessing and Augmentation Tech stack: Python Why it matters: This implementation provides a practical way to apply diffusion augmentation techniques to sequential recommendation tasks, potentially improving the performance of recommendation systems. Repository: liuqidong07/DiffuASR Primary language: Python GitHub stars: 46

### Overview: This repository is a curated and auto-updated collection of the latest research papers, projects, and resources related to Video Diffusion Models and Video Generation. It covers a wide range of topics including text-to-video, image-to-video, controllable generation, world models, video editing, and audio-video generation. The repository is maintained by the Longxiang-AI team and is regularly updated to ensure the latest and most relevant information. ### Key features: - **Interactive Paper Map**: An interactive site built from all daily snapshots, providing a monthly trends overview and topic network. - **Resilient Scheduled Updates**: Temporary arXiv rate limits and server errors are preserved to ensure the latest valid data. - **Broader and More Accurate Video Indexing**: Expanded coverage to 1,000 relevant papers across various diffusion, flow matching, autoregressive generation, world models, editing, enhancement, and audio-video generation categories. - **Unified CLI**: A single entry point `python main.py` with subcommands: `init`, `search`, `suggest`, `export-bib`, `readme`. - **Custom Time Range Filtering**: Support for relative periods (`6m`, `1y`, `2y`) and absolute date ranges. - **Smart Link Extraction**: Automatically extracts and classifies GitHub, project page, dataset, video, demo, and HuggingFace links from paper abstracts. - **BibTeX Export**: Fetch BibTeX from arXiv and export to `.bib` files with category/date filters. - **LLM Keyword S

### Overview: This repository contains the code for the O'Reilly book "Hands-On RAG for Production," authored by Ofer Mendelevitch and Forrest Sheng Bao. It provides a comprehensive guide to building and deploying retrieval-augmented generation (RAG) applications, covering all phases from parsing to vector search and generation. ### Key Features: - **Code Examples**: The repository includes 35 Jupyter notebooks that can be run locally or in Google Colab. - **End-to-End Pipeline**: It walks through the entire process of building a RAG application, including parsing, chunking, embeddings, vector search, and generation. - **Real-World Applications**: The notebooks cover various RAG techniques such as agentic RAG, multimodal RAG, and GraphRAG, providing practical examples. - **Open Source**: The code is available under the MIT License, allowing users to run it locally or in Google Colab. ### Tech Stack: - **Programming Languages**: Python, Jupyter Notebook - **Tools and Libraries**: LangChain, PyMuPDF, python-docx, Claude, PostgreSQL, SQLite-vec, pgvector, LLMs ### Why it Matters: This repository is essential for developers looking to build and deploy production-ready RAG applications. It provides a step-by-step guide on how to implement RAG, including best practices and real-world applications. The code examples and practical examples make it a valuable resource for anyone interested in integrating RAG into their projects.

### Overview: Lattice is a Rust-based inference engine for transformer models, designed for Apple Silicon and Raspberry Pi. It includes a native macOS app, Lattice Studio, for fine-tuning and managing models. ### Key features: - Pure Rust compute: Hand-written SIMD kernels (AVX2/NEON) for optimal performance. - Metal GPU backend: Native Apple Silicon acceleration via Metal shaders, with fallback to WGPU for cross-platform compatibility. - Generation models: Supports Qwen3.5-0.8B, 2B, Qwen3.6-27B, and 35B-A3B models. - Embedding models: Covers 9 models including BGE, E5, MiniLM, and Qwen3-Embedding families. - Tokenizers: Supports WordPiece, SentencePiece, Qwen-style BPE, and Gemma BPE. ### Tech stack: Rust ### Why it matters: Lattice provides a compact and efficient solution for running AI models on Apple Silicon and Raspberry Pi, eliminating the need for external ML runtimes and Python dependencies.

Overview: This is the official PyTorch implementation of the paper "Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis." The project presents a novel approach for enhancing text-image correspondence by leveraging available semantic layouts, specifically a Gaussian-categorical diffusion process (GCDP). This method aims to guide text-to-image generation models to generate semantic labels for each pixel, thereby improving text-image correspondence in specific domains. Key features: - GCDP: Gaussian-categorical diffusion process for text-image generation. - Implementation: Available in `imagen_pytorch/joint_imagen.py`. - Pretrained models: Released on August 16, 2023. - Updates: Project page updated on July 30, 2023. Tech stack: Python. Why it matters: This project addresses a significant challenge in text-to-image generation by improving text-image correspondence in specific domains, such as urban scenes, medical images, and faces, without relying on web-scale text-image datasets.

### Overview: VortexDB is a lightweight, vector database built from scratch in Rust. Designed for efficient similarity search, it can be used as the backbone for AI-powered applications. ### Key features: - **Flat Indexing**: Brute-force exact search for smaller datasets. - **KD-Tree Indexing**: Space-partitioning for low-dimensional vectors. - **HNSW Graphs**: Fast approximate nearest neighbor search for large-scale datasets. - **Supports**: Euclidean, Manhattan, Hamming, and Cosine similarity metrics. ### Tech stack: Rust ### Why it matters: VortexDB offers a flexible and efficient solution for vector databases, making it suitable for applications requiring high-speed similarity search capabilities.

### Overview: React.ai is a comprehensive, production-ready React machine learning API boilerplate project that leverages various cutting-edge technologies including React.js, Redux, GraphQL, and Machine Learning APIs such as DialogFlow and apiai. It provides a robust foundation for building server-side rendered full-stack applications, including speech recognition, text-to-speech, and AI chatbots. ### Key Features: - **Speech Recognition and AI Chatbot**: Recognizes speech and trains an AI bot to respond, suitable for customer service and personal assistant roles. - **Server-Side Rendering (SSR)**: Provides SEO-Ready, Production-Ready, and SSR capabilities with Next.js. - **Hot Reloading and CSS-in-JS**: Instantly reflect changes to CSS and JS without refreshing the page, and use styled-components for composable CSS. - **Redux and Redux Saga**: Unidirectional data flow and change logging for predictable state management. - **ESNext Features**: ES6 template strings, object destructuring, arrow functions, and JSX syntax. - **Offline First**: Offers availability without a network connection. - **SEO and i18n**: Supports SEO and internationalization with `react-intl`. - **Command Line Setup**: Includes AppVeyor and TravisCI for automatic test runs on Windows and Unix. ### Tech Stack: - **JavaScript**: Main programming language used. - **React.js**: Front-end framework. - **Redux**: State management library. - **GraphQL**: API for data fetching. - **DialogFlow**: Machine Learning API for speech recognition. - **apiai

Overview: DiffBlender is a scalable and composable multimodal text-to-image diffusion model designed to synthesize complex combinations of input modalities, offering customized generation aligned with user preferences. Key features: - Intuitive extension to additional modalities - Low training cost through partial update of hypernetworks - Scalable and composable design Tech stack: Python Why it matters: DiffBlender enables flexible manipulation of conditions and provides customized generation aligned with user preferences, making it a valuable tool for various applications in text-to-image synthesis.

Overview: This project focuses on the task of item alignment, specifically identifying similar items within a dataset. The primary goal is to develop a model that can accurately classify pairs of items as either being the same item or not, which is crucial for building a knowledge graph of items and their relationships. Key features: - Item alignment model using various modalities: text, image, and graph. - Model ensemble approach combining different modalities for improved performance. - Configuration options for interaction type, classification method, similarity measure, and loss function. Tech stack: Python, Jupyter Notebook, TensorFlow, PyTorch, and other relevant libraries for data preprocessing, model training, and evaluation. Why it matters: This project is essential for the development of a robust knowledge graph for items, which can be applied in various e-commerce scenarios, such as product recommendations, inventory management, and more. The ability to accurately identify similar items is critical for enhancing user experience and improving operational efficiency in e-commerce platforms.

Overview: llmff is an FFmpeg-shaped command-line and library tool designed for LLM inference pipelines. It focuses on typed pipeline graphs, reproducible YAML manifests, backend adapters, local retrieval, JSON validation and repair, and JSONL traces. Key features: - Typed pipeline graph - Reproducible YAML manifests - Backend adapters - Local retrieval - JSON validation and repair - JSONL traces Tech stack: Rust Why it matters: llmff simplifies and standardizes LLM inference workflows, making them more reliable and easier to debug. It supports a wide range of backend providers and is compatible with various operating systems, ensuring flexibility and ease of use. Repository: syndicalt/llmff