Projects

### Overview: A tool for producing word-precise segmentation of recorded Qur'anic recitation, designed to work with EveryAyah style audio input. It assigns precise timestamps to each word within the recorded audio, allowing for features such as highlighting the current word being spoken, repeating a phrase, comparing against other audio, and analyzing a qari's speaking cadence. ### Key Features: - Word-accurate timestamps for Qur'anic audio - Matches words from the Qur'an to precise timestamps within the recorded audio - Highlights the current word being spoken during playback - Allows for repeating a certain word or phrase - Enables comparison against other audio recordings - Analyzes a qari's speaking cadence ### Tech Stack: C++ ### Why it Matters: This tool provides a valuable resource for researchers, educators, and enthusiasts interested in the recitation of the Qur'an. By accurately segmenting the recitation, it allows for detailed analysis and comparison of different recitations, aiding in the study of Arabic language and Qur'anic recitation techniques.

### Overview: NanoSage is a local language model-powered recursive search and knowledge explorer. It utilizes Python for its implementation, runs on your laptop, and is designed to refine user queries through a multi-source, relevance-driven, and recursive search pipeline. The system balances depth and breadth using Monte Carlo-based exploration, ensuring precision and avoiding unrelated tangents. It generates detailed, well-organized reports using retrieval-augmented generation (RAG), integrating the most valuable insights from local and web data. ### Key Features: - **Structured Search Pipeline**: Refines user queries through a multi-source, relevance-driven, and recursive search pipeline. - **Monte Carlo Exploration**: Balances depth and breadth, ranking each branch's relevance to ensure precision and avoid unrelated tangents. - **Evidence-Aware Deep Search**: Branches are scored by the evidence they add, coverage is broadened before recursing, and scholarly APIs bring in paper abstracts even when publisher pages are blocked. - **Grounded Reports**: The final report cites only sources that were actually read, and papers stay in the evidence. - **Search Engines**: Access to high-quality academic sources including PubMed, research journals, and scholarly databases via Tavily Search API. - **Hybrid Embedding System**: Intelligent model selection where SigLIP/CLIP handle vision tasks while all-MiniLM processes text content, optimizing performance. - **Web Crawler**: Comprehensive web content extraction with metadata generation, domain grouping, and fallback search engines for maximum reliability and coverage. - **LiteLLM Gateway Support**: OpenAI-compatible providers and run visualization

Overview: OramaCore is a comprehensive AI runtime that includes a full-text search engine, vector database, and various utilities for projects, answer engines, copilots, and search. Key features: - Fully-fledged full-text search engine - Vector database - LLM interface with action planning and reasoning - JavaScript runtime for custom agent development - NVIDIA GPU support for optimal performance Tech stack: Rust Why it matters: OramaCore simplifies AI development by providing a complete runtime for projects, answer engines, copilots, and search, making it easier to integrate AI into various applications efficiently. Repository: oramasearch/oramacore GitHub stars: 265 GitHub description: OramaCore is the complete runtime you need for your projects, answer engines, copilots, and search. It includes a fully-fledged full-text search engine, vector database, LLM interface, and many more utilities.

Overview: MVPaint is a novel generation-refinement framework designed for generating high-resolution, multi-view consistent textures on 3D meshes. The project addresses the challenges of generating consistent textures across multiple views, especially for arbitrary UV unwrapping, by introducing three key modules: Synchronized Multi-view Generation (SMG), Spatial-aware 3D Inpainting (S3I), and UV Refinement (UVR). Key features: - Synchronized Multi-view Generation (SMG) - Spatial-aware 3D Inpainting (S3I) - UV Refinement (UVR) Tech stack: Python Why it matters: MVPaint offers a significant improvement in the quality of generated textures by ensuring multi-view consistency and enhancing the generation versatility, making it a valuable tool for 3D asset production workflows.

Overview: This repository contains the official code for "Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation" presented at CVPR 2023. The project aims to address the limitations of generative adversarial networks (GANs) by proposing a novel diffusion-based framework called Diffusion Co-Speech Gesture (DiffGesture), which effectively captures audio-to-gesture associations and preserves temporal coherence for high-fidelity audio-driven co-speech gesture generation. Key features: - Diffusion Co-Speech Gesture (DiffGesture) framework - Diffusion Audio-Gesture Transformer (DAGT) for multimodal information handling - Diffusion Gesture Stabilizer (DGS) for temporal consistency - Implicit classifier-free guidance for diversity and quality trade-off Tech stack: Python Why it matters: This project demonstrates the potential of diffusion models in generating coherent and high-quality audio-driven co-speech gestures, showcasing advancements in multimodal audio-to-gesture association and temporal coherence. Repository: Advocate99/DiffGesture GitHub stars: 264 GitHub description: [CVPR'2023] Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

### Overview: ShareGPT4V is a state-of-the-art large-scale multi-modal model designed to improve the performance of large multi-modal models with better captions. This project showcases the implementation of ShareGPT4V at ECCV 2024, focusing on enhancing the descriptive quality of image-text datasets. ### Key Features: - **ShareGPT4V-7B**: A high-performance model that significantly improves the caption generation capabilities of large multi-modal models. - **ShareCaptioner**: A general image captioner that closely approaches the capabilities of GPT4-Vision. - **ShareGPT4V-7B Demo**: An open-source implementation for generating captions for images and videos. - **Share-Captioner Demo**: A web-based demo for generating captions using the ShareCaptioner model. ### Tech Stack: Python ### Why It Matters: ShareGPT4V addresses the limitations of existing large multi-modal models by providing high-quality captions that are more descriptive and closer to the capabilities of GPT4-Vision. This project showcases the effectiveness of the ShareCaptioner model in improving the performance of large multi-modal models, making it a valuable resource for researchers and practitioners in the field of computer vision and natural language processing. ### Repository: - **Repository**: [ShareGPT4Omni/ShareGPT4V](https://github.com/ShareGPT4V/ShareGPT4V) - **Stars**: 263 - **Description**: Official implementation of ShareGPT4V: Improving Large Multi-modal Models with Better

Overview: Backlot is a local emulator for enterprise SaaS APIs, providing the exact response shapes, pagination, authentication, and per-document ACLs as the official vendor SDKs. It uses a deterministic corpus controlled by the user, eliminating the need for vendor accounts, OAuth, secrets, or network access. Key features: - Emulates Slack, Gmail, Google Drive, GitHub, Jira, Notion, S3, and other enterprise APIs. - Supports real-time response shapes, pagination, authentication, and per-document access controls. - Uses a deterministic corpus controlled by the user, ensuring consistent behavior without vendor accounts, OAuth, secrets, or network access. - Provides a simple CLI interface for running and testing the APIs locally. - Supports both development and testing environments, allowing for immediate integration and testing of APIs. Tech stack: Python Why it matters: Backlot simplifies API testing by providing a local environment that closely mimics the official vendor SDKs, eliminating the need for complex setups, authentication, or network access, making it ideal for continuous integration and testing of enterprise SaaS APIs. Repository: brekkylab/backlot Primary language: Python GitHub stars: 260 GitHub description: A local emulator for enterprise SaaS APIs — Slack, Gmail, Google Drive, GitHub, Jira, Notion, S3 and more — with the real response shapes, pagination, auth and per-document ACLs, over a corpus you supply. README excerpt: # Backlot [![GitHub stars](https://img.shields.io/github/stars/brekkyl

Overview: traceAI is an open-source observability framework for AI applications, built on OpenTelemetry, designed to capture and trace every LLM call, prompt, token count, retrieval step, and agent decision as structured OpenTelemetry traces. Key features: - Standardized tracing for consistent OpenTelemetry spans and attributes. - Drop-in setup for 50+ AI frameworks across Python, TypeScript, Java, and C#. - Vendor-agnostic compatibility with any OpenTelemetry-compatible backend. - Rich context including prompts, completions, tokens, model parameters, tool calls, and more. Tech stack: Python. Why it matters: traceAI provides a seamless way to trace AI workflows, enabling developers to identify and debug issues in their AI applications by pinpointing the exact points where the model calls, prompts, and decisions occur, without the need for new vendors or dashboards. Repository: future-agi/traceAI GitHub stars: 262 GitHub description: Open Source AI Tracing Framework built on OpenTelemetry for AI Applications and Frameworks.

### Overview: Direct Retrieval-augmented Optimization (DRO) is a novel framework that leverages knowledge selection and language models to enhance the performance and factuality of knowledge-grounded Large Language Models (LLMs). This project introduces a synergistic approach where the knowledge selection and answer generation processes are optimized together, providing a more comprehensive and accurate output. ### Key Features: - **Synergistic Optimization**: DRO integrates the knowledge selection and answer generation processes, enabling mutual enhancement and improved performance. - **Collaborative Learning Loop**: The framework allows the Selector and Generator to learn from each other, leading to better factuality, relevance, and overall performance. - **Drop-in Upgrade**: DRO can be seamlessly integrated into existing knowledge-grounded LLMs, making it a valuable upgrade for researchers and practitioners. ### Tech Stack: - **Python**: The primary programming language used for implementing the framework. ### Why It Matters: - **Enhanced Performance**: DRO significantly improves the factuality, relevance, and overall performance of knowledge-grounded LLMs. - **Synergy Between Processes**: By optimizing the knowledge selection and answer generation processes together, DRO achieves better synergy and accuracy. - **Comprehensive Evaluation**: The project provides a comprehensive dataset and evaluation metrics, making it a valuable resource for researchers and practitioners. ### Repository: - **Repository**: [mangopy/direct-rag-learning](https://github.com/mangopy/direct-rag-learning) - **GitHub Stars**: 264 - **GitHub Description**: Official code for TOIS2026

### Overview: UmaDev is a coding agent that leverages existing AI coding CLIs like Claude Code, Codex, OpenCode, Grok Build, or Kimi Code, coordinating and managing their work to produce high-quality software artifacts. It integrates seamlessly with the selected base CLI, ensuring consistent and reproducible results. ### Key Features: - **Role-Based Team Orchestration**: UmaDev manages work assignments across multiple roles, such as product, architecture, UI/UX, frontend, backend, QA, security, and DevOps, ensuring depth-appropriate and shippable software. - **Bounded Sessions**: Work is divided into bounded sessions, with small edits being small edits, and larger changes requiring a visible plan and deliberate work schedule. - **Audit Trail**: The coordinator evaluates gates and leaves an audit trail, providing a transparent and honest report of failed or incomplete work. - **Cross-Role Collaboration**: Roles exchange bounded blackboard artifacts and structured verdicts, avoiding free-form cross-talk. - **Single Rust Binary**: UmaDev is a single Rust binary, delivered via npm or native installer, ensuring reproducibility and ease of use. ### Tech Stack: UmaDev is built using Rust, a language known for its safety and performance, and is delivered via npm or a native installer. ### Why It Matters: UmaDev simplifies the process of integrating AI coding agents into your development workflow, ensuring that the work is managed and delivered in a consistent and reproducible manner. It also provides a transparent and accountable way to manage and track the progress of projects, making it easier to collaborate

### Overview: SARA represents retrieved evidence as a mix of `k` natural-language passages and `n - k` compressed retrieval embeddings. This repository supports two evaluated methods: Standard RAG and SARA, both starting from the public `mistralai/Mistral-7B-Instruct-v0.2` base and trained with matched budgets. The methods differ in how they handle the compressed embeddings, making them comparable under identical conditions. ### Key features: - **Projector + `<COMPRESS>` tokens**: retriever embeddings are projected into the language model's hidden space at `<COMPRESS>` token positions. - **LoRA-first**: adapters + projector only — no merged or full-model checkpoints are ever saved. - **Fully batched generation and evaluation**: dynamic padding, one GPU call per batch. - **Leakage-safe**: document-level (per-paper) train/dev/test splits. - **Single CLI**: `python -m src.run_sara {finetune, evaluate, report}`. ### Tech stack: - Python 3.11 - PyTorch 3.11 - Transformers 4.57 - PEFT 4.57 - Mistral 7B Instruct-v0.2 ### Why it matters: - **Two-method, apples-to-apples**: Standard RAG vs SARA under identical data, retriever, prompts, decoding, and seeds. - **Projector + `<COMPRESS>` tokens**: retriever embeddings projected into the LM hidden space at `<COMPRESS>` token positions. -

Overview: Voice overlay helps users convert their voice into text, offering a polished user experience and handling permission requests. Key features: - Voice permission handling - Voice input dialog - Permission rationale explanation - Permission manual instructions - Customizable behavior Tech stack: Kotlin Why it matters: It simplifies integrating voice input into Android applications, providing a seamless user interface and permission management. Repository: algolia/voice-overlay-android GitHub stars: 263

### Overview: Create labeled datasets, enhance audio quality, identify speakers, and support diverse dataset types. This project is dedicated to advanced audio processing for training text-to-speech or speech-to-text models. ### Key features: 1. **Dataset Generation:** Creation of multilingual datasets with Mean Opinion Score (MOS). 2. **Silence Removal:** Feature to remove silences from audio files, enhancing the overall quality. 3. **Sound Quality Improvement:** Improves the quality of the audio when needed. 4. **Audio Segmentation:** Segments audio files within specified second ranges. 5. **Transcription:** Transcribes the segmented audio, providing a textual representation. 6. **Gender Identification:** Identifies the gender of each speaker in the audio. 7. **Pyannote Embeddings:** Utilizes pyannote embeddings for speaker detection across multiple audio files. 8. **Automatic Speaker Naming:** Automatically assigns names to speakers detected in multiple audios. 9. **Multiple Speaker Detection:** Capable of detecting multiple speakers within each audio file. 10. **Store speaker embeddings:** Speakers are detected and stored in a Chroma database, so you do not need to assign a speaker name. 11. **Syllabic and words-per-minute metrics** 12. **Multiple input sources:** You can either use your own files or download content by pasting URLs from sources such as YouTube, LibriVox, and TED Talks. ### Tech stack: Python ### Why it matters: This project is essential for anyone working on speech recognition, text-to-speech, or audio quality

### Overview: ReNoise: Real Image Inversion Through Iterative Noising ReNoise is an official implementation for real image inversion using the ReNoise paper. It enhances reconstruction accuracy without increasing the number of operations, making it particularly useful for recent models trained with fewer denoising steps. ### Key Features: - **ReNoise Method**: Iteratively refines the approximation of a predicted point along the forward diffusion trajectory. - **High Quality-to-Operation Ratio**: Enhances reconstruction accuracy without increasing the number of operations. - **Comprehensive Evaluations**: Evaluated using various sampling algorithms and models, including recent accelerated diffusion models. ### Tech Stack: Python ### Why It Matters: ReNoise offers a high-quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations, making it particularly useful for recent models trained with fewer denoising steps. Repository: garibida/ReNoise-Inversion GitHub Stars: 263

### Overview: This project involves finetuning a Falcon-7B language model using QLoRA (Quantized Low-Rank Adaptation) on a mental health conversational dataset. The model is trained to provide empathetic and non-judgmental responses to mental health inquiries, aiming to reduce stigma and promote understanding. ### Key Features: - **Mental Health Conversational AI**: Fine-tuned to respond to mental health-related questions and provide empathetic support. - **Pre-Processed Dataset**: Curated from online FAQs, healthcare blogs, and wiki articles to ensure conversational format. - **Custom Dataset**: Mental health dataset pre-processed to anonymize and remove unwanted characters. ### Tech Stack: - **Model**: Falcon-7B LLM - **Finetuning Technique**: QLoRA - **Training Environment**: Colab (Nvidia A100, Google Colab Pro) ### Why It Matters: - **Reducing Stigma**: Helps in reducing negative perceptions and misconceptions about mental health. - **Accessible Support**: Provides immediate assistance and emotional support, especially in moments of distress. - **Empathy and Understanding**: Employs an empathetic and non-judgmental approach to improve mental health awareness. ### Repository: - **GitHub Repository**: iamarunbrahma/finetuned-qlora-falcon7b-medical - **Stars**: 263 - **Description**: Finetuning of Falcon-7B LLM using QLoRA on Mental Health Conversational Dataset ### Primary Language: - **Jupyter

Overview: Vector Graph RAG is a pure vector search model designed for multi-hop reasoning scenarios, achieving state-of-the-art performance. Key features: - No graph database required - Configurable reranking with single-pass generative reranking or two-stage Jev relation and passage scoring - Optimized for dense factual domains like legal, finance, medical, and literature - Works out of the box with Milvus Lite - Supports multi-hop reasoning through subgraph expansion - Performs well on MuSiQue and 2Wiki datasets, reaching the quality–latency Pareto frontier Tech stack: Python Why it matters: This model simplifies complex reasoning tasks by leveraging pure vector search, making it highly efficient and scalable for large-scale applications.

Overview: This project is a simple implementation of a vision language model (VLM) created from scratch using PyTorch. The model is designed to be a simplified version of Grok 1.5 or GPT-4 Vision, focusing on the creation of a sparse mixture of experts based VLM. Key features: - Image Encoder: Extracts visual features from images using a custom implementation of the original vision transformer. - Vision-Language Projector: Projects image embeddings to match the text embedding space. - Decoder: A character-level language model that generates text, incorporating a projection module. Tech stack: PyTorch. Why it matters: This implementation serves as a basic demonstration of how a VLM can be constructed from scratch, providing insights into the components and architecture of modern vision language models.

Overview: This repository contains papers discussing the Small Language Models (SLMs) in the era of large language models, focusing on their technology, on-device applications, efficiency, enhancements for LLMs, and trustworthiness. Key features: - A Comprehensive Survey of Small Language Models: Technology, On-Device Applications, Efficiency, Enhancements for LLMs, and Trustworthiness - News: Includes updates on the extended survey on SLM-LLM collaboration, talks at NLP Reading Groups, and accepted papers in various conferences. Tech stack: The repository uses Markdown for documentation and includes images for visual representation. Why it matters: This survey is crucial for researchers and practitioners interested in understanding the advancements and challenges in Small Language Models, particularly in the context of Large Language Models (LLMs).

### Overview: go-agent is a Go framework designed for building AI agents with pluggable LLM providers, memory, file context, guardrails, UTCP tool orchestration, and multi-agent coordination. It is built for production and supports a variety of AI models and memory solutions. ### Key Features: - **Agent Core**: A small `agent.Agent` core with `Generate`, `GenerateWithFiles`, and `GenerateStream` methods. - **Provider Adapters**: Support for Gemini, OpenAI, Anthropic, Ollama, and a local dummy model. - **Memory**: Short-term memory plus vector-store backed long-term memory. - **ADK Modules**: Wiring models, memory, tools, sub-agents, CodeMode, and UTCP. - **Guardrails**: Input/output guardrails and checkpoint/restore support. - **Orchestration**: Composable retry, timeout, rate-limit, and token-budget model middleware. - **Local Skills**: Agents automatically load local instructions from `.skills` in the process working directory. ### Tech Stack: Go ### Why It Matters: go-agent provides a production-ready framework for building AI agents in Go, offering a small, idiomatic Go core, pluggable LLM providers, and comprehensive memory and orchestration features. This makes it suitable for both prototyping and production environments, ensuring your agents are efficient and reliable.

### Overview: This repository contains curated resources related to the paper "Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey," focusing on models, datasets, and other resources for biomedical language cross-modeling. ### Key features: - **Models:** - `BioText` (Bioinformatics 2019) - [Model](https://huggingface.co/dmis-lab) - `BioText` (EMNLP IJCNLP 2019) - [Model](https://github.com/allenai/scibert) - `BioText` (BioNLP@ACL 2019) - [Model](https://huggingface.co/bionlp) - `BioText` (EMNLP 2020) - [Model](https://github.com/NVIDIA/NeMo) - `BioText` (BioNLP@CHIL 2020) - [Model](https://github.com/kexinhuang12345/clinicalBERT) - `BioText` (BioNLP@ACL 2021) - [Model](https://aclanthology.org/2021.bionlp-1.24.pdf) - **Datasets & Benchmarks:** - [Datasets & Benchmarks](#datasets--benchmarks) - **Related Resources:** - **Related Surveys & Evaluations:** - [Related Surveys & Evaluations](#related-sur