Projects

Overview: MEGA-Bench is an evaluation suite designed to scale multimodal evaluation to over 500 real-world tasks, addressing the diverse and rich set of tasks and their output formats. Key features: - 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators - Wide range of output formats including numbers, phrases, code, LaTeX, coordinates, JSON, free-form, etc. - Over 40 metrics developed to evaluate these diverse tasks - Fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill) - Interactive visualization of model capabilities Tech stack: Python Why it matters: MEGA-Bench provides a comprehensive and diverse evaluation framework for vision-language models, enabling optimized model performance across various dimensions of tasks and outputs. Repository: TIGER-AI-Lab/MEGA-Bench Primary language: Python GitHub stars: 83 GitHub description: This repo contains the code for "MEGA-Bench Scaling Multimodal Evaluation to over 500 Real-World Tasks" [ICLR 2025] README excerpt: # MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks [ICLR2025] [](https://arxiv.org/abs/2410.

### Overview: SafeDiffuser is an implementation of our ICLR 2025 paper, SafeDiffuser: Safe Planning for Diffusion Probabilistic Models. It focuses on providing safe and efficient planning for diffusion probabilistic models. ### Key features: - **Diffusion Model Implementation**: Implements the diffusion model from our paper. - **Visualization Tools**: Provides tools for visualizing diffusion outputs. - **Pre-trained Models**: Offers pre-trained diffusion models and value functions for various tasks. - **Training from Scratch**: Allows training diffusion models and value functions from scratch. ### Tech stack: Python ### Why it matters: SafeDiffuser provides a robust and efficient framework for planning and decision-making in diffusion probabilistic models, ensuring safety and reliability in complex applications.

Overview: This repository contains a MXNet implementation of Baidu's DeepSpeech architecture, designed to facilitate the creation of scalable speech-to-text models using convolutional neural networks, fully connected networks, recurrent neural networks, and other components. Key features: - MXNet implementation - DeepSpeech architecture - Rich functionalities for building STT models - Batch-normalization and dropout for training efficiency - Warp CTC for loss calculations - Support for different types of recurrent neural networks (CNNs, RNNs, LSTMs, GRUs) - Audio preprocessing using SoundFile library - Installation instructions for TensorBoard - Example configuration files for both DeepSpeech2 and original DeepSpeech2 Tech stack: Python Why it matters: This repository provides a comprehensive and user-friendly solution for developers looking to build speech recognition models using MXNet, making it easier and more efficient than previous examples.

### Overview: Aniemore is an open-source library for real-time emotion recognition from audio and text files, specifically designed for Russian language. ### Key Features: - **Emotion Recognition from Audio and Text**: Recognize emotions in speech and text. - **Russian Language Support**: Works with the Russian language. - **High-Quality Data**: Uses the Russian Emotional Speech Dialogues (RESD) dataset, containing over 3000 audio clips from 200 different people. - **Model Performance**: Models can recognize emotions in 3-second audio clips, with a 30% WER and an overall accuracy of 75%. - **Wide Range of Emotions**: Recognizes 7 emotions: anger, disgust, fear, joy, interest, sadness, and neutral. - **Real-Time Processing**: Models respond within 5 seconds. - **Data Integration**: Supports the RESD dataset and other datasets of varying sizes. ### Tech Stack: Python ### Why It Matters: Aniemore simplifies the process of integrating emotion recognition capabilities into applications, making it easier for developers to implement real-time emotional analysis in their projects. The library's focus on Russian language and high-quality datasets ensures robust performance and accurate results, making it a valuable tool for developers working in the Russian-speaking market.

Overview: This repository provides a Nano Banana Pro (official) API endpoint, compatible with OpenAI, for generating image previews. It offers flexible pricing models and supports both text and image inputs, with a focus on token billing and real-time billing updates. Key features: - Supports both text and image inputs - Offers flexible pricing models - Real-time billing updates - Compatible with OpenAI's API Tech stack: Python Why it matters: This API is crucial for developers looking to integrate image generation capabilities with their applications, offering a cost-effective and flexible solution for generating image previews. GitHub stars: 80 Repository: apimarte80m2k/gemini-3-pro-image-preview-official-api-ja

Overview: This repository contains step-by-step demos of Google's Gemma models running locally on a Jetson Orin Nano, showcasing voice assistants with RAG, multi-agent NPCs, live EN-JA translation, and a Gemma 4 voice + vision agent that decides when to look through the webcam. Key features: - Gemma4 (VLA demo) - Gemma3 (Ollama implementation) - Gemma2 (Original LLaMA.cpp implementation) Tech stack: Python Why it matters: These demos provide a comprehensive overview of how to set up and run Gemma models locally on a Jetson Nano, demonstrating practical applications of AI and machine learning in edge computing environments.

Overview: This project is a genAI WhatsApp app that leverages Amazon Bedrock to process images, voice notes, videos, and documents, and stores conversation history using DynamoDB sessions. It integrates with AWS CDK and Python for seamless integration. Key features: - Multi-language conversations with Claude 3.5 and LangChain - Voice note transcription with Amazon Transcribe - Image analysis via Claude 3.5 Sonnet - Session-based conversation history stored in DynamoDB - Supports LangChain-based and direct Bedrock API text agents Tech stack: Python, AWS CDK, Amazon Bedrock, Amazon DynamoDB, Amazon API Gateway, Amazon Transcribe, Amazon S3, AWS End User Messaging Why it matters: This project showcases the integration of genAI capabilities with WhatsApp, providing a powerful tool for businesses and developers looking to enhance their customer interactions and AI assistant solutions.

### Overview: LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model (CVPR2024) ### Key features: - **Reference-guided Image Synthesis**: LeftRefill efficiently stitches reference and target images together as a single input, enabling reference-based image synthesis. - **Generalized Text-to-Image Diffusion Model (T2I)**: Utilizes a generalized T2I model to learn both structural and textured correspondence between reference and target without additional encoders or adapters. - **Cross-Attention Modules**: Injects task and view information through cross-attention modules in T2I models, enabling multi-view reference ability. - **Unified Framework**: Provides a simple yet unified framework for addressing reference-guided synthesis tasks, including reference-guided inpainting and novel view synthesis. ### Tech stack: Python ### Why it matters: LeftRefill offers an innovative and efficient approach to reference-guided image synthesis, leveraging large T2I models and generalized text-to-image diffusion techniques to achieve consistent results without the need for fine-tuning or modifications. This method is particularly useful in applications where reference images are available, such as in reference-guided inpainting and novel view synthesis.

### Overview: Semantic Genesis is an innovative deep learning model designed to enhance the semantic understanding of medical images. This project leverages self-supervised learning techniques to create a pre-trained 3D model that can be seamlessly integrated into existing representation learning methods. By improving the semantic enrichment of medical image representations, it addresses a significant challenge in general-purpose representation learning. ### Key features: - **Self-supervised Learning Framework**: Employs self-discovery, self-classification, and self-restoration to learn semantically enriched visual representations. - **Pre-trained Model**: Provides a pre-trained 3D model that can be used as an add-on to existing representation learning methods. - **Transferable Visual Words**: Explores the semantics of anatomical patterns for self-supervised learning, enhancing the transferability of visual word embeddings. ### Tech stack: Python ### Why it matters: This project introduces a novel approach to improving the semantic understanding of medical images, making it a valuable contribution to the field of representation learning. By enhancing the semantic enrichment of medical image representations, it opens up new possibilities for general-purpose representation learning, potentially leading to more accurate and effective image analysis.

Overview: **Speech** is an online tool designed to help non-native English speakers practice and improve their pronunciation through voice recording and speech recognition. Key features: - Voice recording and playback - Pronunciation feedback - Speech recognition using Vosk - Automatic audio encoding to WAV format Tech stack: Vue Why it matters: **Speech** offers a practical solution for English language learners looking to enhance their pronunciation skills, providing immediate feedback and improved pronunciation tracking.

Overview: This repository contains inference-only code for our work, SIM, a cutting-edge approach for distilling pre-trained diffusion models into efficient one-step generators. It achieves impressive performance metrics, including an FID of 2.06 for unconditional generation and 1.96 for class-conditional generation on the CIFAR10 dataset. Additionally, SIM has been applied to a state-of-the-art transformer-based diffusion model for text-to-image generation, achieving an aesthetic score of 6.42 and outperforming existing one-step generators. Key features: - Inference-only implementation - Distilling pre-trained diffusion models into efficient one-step generators - Achieving high-quality sample generation without training samples for distillation - Effective computation of gradients for various score-based divergences - Applied to a state-of-the-art transformer-based diffusion model for text-to-image generation Tech stack: Python Why it matters: SIM is a significant advancement in the field of diffusion models, offering high-quality sample generation without the need for training samples, making it a valuable tool in various applications, including text-to-image generation and other diffusion-based tasks.

### Overview: Patter is a speech-to-text framework built using PyTorch, designed for developers looking to integrate speech recognition capabilities into their projects. It includes features such as configurable corpora definitions, a highly configurable DeepSpeech model, beam decoding with KenLM support, and support for various dataset augmentations. ### Key features: - **Configurable Corpora**: File-based configuration for repeatability. - **DeepSpeech Model**: Highly configurable with options for RNN types, activation functions, and CNN frontends. - **Augmentation**: Support for speed, gain, time shift, and noise addition. - **Tensorboard Integration**: For visualization and debugging. - **gRPC Model Server**: Built for efficient model serving. ### Tech stack: Python ### Why it matters: Patter provides a robust and flexible framework for developers to integrate speech recognition into their applications, leveraging PyTorch for efficient training and inference. Its configurable nature and comprehensive feature set make it a valuable tool for speech recognition projects. Repository: ryanleary/patter GitHub stars: 83 GitHub description: speech-to-text in PyTorch Primary language: Python Installation: Manual installation of dependencies Dataset Definitions: JSON-Lines files for defining audio and transcription paths Training: Built-in command-line trainer with corpus, model, and training configuration files Dataset Augmentation: Support for speed, gain, time shift, and noise addition gRPC Model Server: Built for efficient model serving

Overview: LLM-Client-SDK is a Python SDK designed to simplify integration with various Generative AI Large Language Models (LLMs), providing both async and sync versions for seamless and powerful integration. Key features: - Async and sync versions for seamless integration - Support for multiple LLM providers including OpenAI, Google, AI21, HuggingFaceHub, and more - Base interface for easy integration with LLMs - ChatMessage class for handling chat messages - LLMAPIClientConfig for configuration options - Async methods for text completion, token count, chat completion, and embeddings Tech stack: Python Why it matters: LLM-Client-SDK simplifies the integration process with LLMs, making it easier for developers to leverage the capabilities of different providers without losing flexibility.

### Overview: InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization is a project that focuses on enhancing the text-to-image diffusion models by optimizing the initial noise configuration. The paper was presented at CVPR 2024 and aims to improve the generation of high-fidelity images by guiding the initial noise into a valid region, resulting in more accurate and aligned images. ### Key features: - **Initial Noise Optimization (InitNO)**: A method to optimize the initial noise configuration for better image generation. - **Comparison with SD**: The project compares the effectiveness of different initial noise configurations with the Stable Diffusion model. - **Visualization of Attention Maps**: Provides attention map visualization for better understanding of the model's attention distribution. ### Tech stack: Python ### Why it matters: This project addresses a significant challenge in text-to-image diffusion models by optimizing the initial noise configuration, leading to more accurate and aligned images. The insights gained from this work can be applied to improve the performance of other diffusion models, potentially leading to better results in various applications.

Overview: This project is a sample chatbot using Google's Gemini models, featuring both Gemini-Pro and Gemini-Pro-Vision. Key features: - Multi-turn conversations - Supports both text and image inputs - Optimized for text-and-image input Tech stack: Kotlin Why it matters: This project demonstrates the capabilities of Google's latest large language models, showcasing their potential for various applications beyond text-based interactions. Repository: yveskalume/gemini-chat GitHub stars: 83 GitHub description: a sample project using Gemini, the Google's latest family of large language models.

Overview: DecisionNCE is an innovative framework that addresses decision-making tasks by extracting both local and global task progression features through implicit preference learning. It provides an embodied representation learning solution that ensures temporal consistency and instruction grounding, showcasing its effectiveness in diverse downstream policy learning tasks. Key features: - Multimodal joint encoding - Temporal consistency enforced through implicit time contrastive learning - Instruction grounding via multimodal joint encoding Tech stack: Python Why it matters: DecisionNCE has been accepted by ICML2024 and selected as an outstanding paper at MFM-EAI workshop@ICML2024, highlighting its versatility and practical application in unified representation and reward learning for decision-making tasks. Repository: 2toinf/DecisionNCE Primary language: Python GitHub stars: 82 GitHub description: [ICML 2024] The offical Implementation of "DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning"

### Overview: This repository is a curated list of papers on 3D generation, focusing on various representations such as Mesh, SDF, Point Cloud, and more. It includes papers that demonstrate valuable techniques and insights for 3D generation research, as well as papers about Articulated 3D, Mesh, Implicit Shape Representations, Point Cloud, NeRF, CAD, Voxel, Part-based 3D, 3D Editing, Automatic Rigging, and other related topics. ### Key Features: - Papers on 3D generation with visual previews. - Organized by different 3D representations: Mesh, SDF, Point Cloud, NeRF, and more. - Papers that demonstrate valuable techniques and insights for 3D generation research. - Links to projects and code repositories for further exploration. ### Tech Stack: HTML ### Why it Matters: This repository serves as a comprehensive resource for researchers and developers interested in 3D generation, providing a curated list of papers that can help in understanding and advancing the field of 3D content creation.

Overview: This repository contains curated, original high-craft prompts for AI video ads, designed for performance marketers and D2C founders. The prompts are suitable for multi-beat timing, real camera language, and ad-ready structure, covering 52 prompts across 10 formats. Key features: - Product Showcase - UGC & Authentic - Before / After - Unboxing & ASMR - Problem → Solution - Lifestyle & Brand - Feature & Explainer - Social & Trend - Luxury & Premium - Food & Beverage - How to use - Contributing - License Tech stack: JavaScript Why it matters: These prompts are crafted to enhance the performance of AI video ads, offering a high level of craft and detail, making them ideal for creating engaging and effective ads. Repository: LichAmnesia/awesome-ad-video-prompts GitHub stars: 82 GitHub description: Curated, original collection of high-craft prompts for AI video ads (Seedance 2.0 / Veo 3 / Kling / Runway). Companion to HeyDreaming. 🚀 Want these turned into finished, scored ad videos from just a product URL? Try [HeyDreaming](https://heydreaming.com) — paste a link, get four scored ad-video variations. This list is the open-source craft companion to HeyDreaming.

Overview: PDFChatBot is a Python-based chatbot designed to answer questions based on the content of uploaded PDF files, utilizing the Gradio library for a user-friendly interface and LangChain for natural language processing. Key features: - Process PDF files and extract information - Maintain chat history and provide detailed explanations - Generate responses using a Conversational Retrieval Chain - Display specific pages of PDF files according to the answer Tech stack: Python, Langchain, Llama2, ChromaDB, Hugging Face, Gradio Why it matters: This project offers a practical solution for anyone looking to create a custom RAG chatbot, making it accessible for developers and researchers alike.

### Overview: This repository is a curated list of AI agent frameworks, tools, platforms, research papers, and resources. It serves as a comprehensive guide for developers interested in AI agents, focusing on autonomous systems that use Large Language Models (LLMs) to reason, plan, and take actions. ### Key Features: - **Frameworks & Libraries**: - AutoGen - AG2 - Agent Swarm - AgentField - AgentScope - Agno - AutoGen - CAMEL - CopilotKit - CrewAI - Cumora - DeerFlow - dimos - firstmate - **Platforms & Low-Code**: - AG-UI Protocol - Claude Code - Codex - Gemini CLI - OpenCode - pi - Qwen Code - Rasa - **Agent Infrastructure**: - Multi-agent Orchestration - Role-based Multi-Agent Framework - Cross-platform Team Chat - Physical Space Operating System - **Evaluation & Testing**: - Evaluation Metrics - Testing Protocols - **Safety & Governance**: - Ethical Considerations - Safety Protocols - **Research Papers**: - AI Agent Research - **Tutorials & Courses**: - AI Agent Development - **Use Cases & Case Studies**: - Real-world Applications - **Community**: - Developer Forums -