Projects

### Overview: Alfie is an open-source PyTorch implementation of the paper "Alfie: Democratising RGBA Image Generation With No $$$" (AI for Visual Arts Workshop, ECCV24). This project aims to democratize RGBA image generation by providing a no-cost alternative to traditional image generation methods. ### Key features: - **Democratization of RGBA Image Generation**: Alfie offers a way to generate RGBA images without the need for expensive hardware or specialized software. - **No $$$ Requirement**: The project is designed to be completely free and accessible, making it ideal for educational and creative applications. - **AI4VA Workshop Contribution**: Alfie is a contribution to the AI4VA workshop at ECCV24, showcasing the potential of AI in visual arts. ### Tech stack: Python ### Why it matters: Alfie provides a practical solution for generating RGBA images without the financial burden, making it accessible to a wide range of users, including students, artists, and enthusiasts interested in AI-driven visual arts.

### Overview: Panacea is a framework designed to build collaborative, intelligent multi-agent AI systems, providing a robust infrastructure for creating and managing AI agents that can operate in dynamic environments. ### Key Features: - **Orchestrator**: Central hub for task assignment, execution, and monitoring. - **Agent**: Autonomous units programmed to perform tasks, make decisions, and communicate with other agents. - **Task**: Specific assignments completed by agents, providing all necessary details like description, tools, and responsibilities. - **Crew**: Collaborative groups of agents working together to achieve a set of tasks. - **Workflows**: Frameworks for agent collaboration, including sequential and hierarchical tasks. - **Models**: Backbone of intelligent agents, enabling capabilities like natural language understanding and reasoning. - **Tools**: Skill or function agents use to perform actions, including search, computer use, data extraction, file uploading, and advanced interactions. ### Tech Stack: Python ### Why It Matters: Panacea enables the creation and management of intelligent agents that can work together to solve complex problems in dynamic environments, ensuring efficient and seamless collaboration.

### Overview: The iOS Voice Activity Detection (VAD) library is designed to process audio in real-time and identify the presence of human speech in audio samples containing a mixture of speech and noise. The library supports three distinct models for voice activity detection: WebRTC VAD, Silero VAD, and Yamnet VAD. ### Key Features: - **WebRTC VAD**: Based on a Gaussian Mixture Model (GMM), it offers exceptional speed and effectiveness in distinguishing between noise and silence, but may exhibit lower accuracy for differentiating speech from background noise. - **Silero VAD**: Utilizes a Deep Neural Networks (DNN) model and executes on the ONNX Runtime Mobile, providing exceptional accuracy and processing time that is very close to WebRTC VAD. - **Yamnet VAD**: Based on a Deep Neural Networks (DNN) model and employs the Mobilenet_v1 depthwise-separable convolution architecture, it can predict 522 audio event classes (such as speech, music, animal sounds, etc.) and is trained on the AudioSet-YouTube corpus. ### Tech Stack: Swift ### Why It Matters: The iOS VAD library is lightweight (only 158 KB) and offers exceptional speed in audio processing. It is suitable for scenarios where a small and fast library is necessary, and where sacrificing accuracy is acceptable. For scenarios requiring high accuracy, models like Silero VAD and Yamnet VAD are more preferable. For a detailed comparison between DNN and GMM, refer to the Silero VAD vs Web

### Overview: BBQvec is an open-source, embedded vector database index designed for both Go and Rust, providing approximate K-nearest-neighbors (aKNN) functionality. It is optimized for scalability and efficiency, making it suitable for applications requiring fast and accurate vector similarity searches. ### Key Features: - **Scalable Embedded Vector Index**: Designed for both Go and Rust, BBQvec efficiently indexes and queries large datasets. - **Approximate K-NN (aKNN)**: Provides approximate nearest neighbors, which is crucial for applications where exact matches are not necessary. - **Embedded Backend**: Runs entirely within the language runtime, eliminating the need for external storage. ### Tech Stack: - **Go**: Implemented in Rust, BBQvec leverages the Go ecosystem for performance and simplicity. - **Rust**: Optimized for speed and efficiency, Rust is used to ensure the highest performance and reliability. ### Why It Matters: BBQvec is particularly valuable for applications requiring fast and accurate vector similarity searches, such as recommendation systems, image search, and bioinformatics. Its embedded nature ensures low latency and minimal overhead, making it ideal for real-time applications. ### Repository: Repository: barakmich/bbqvec GitHub Stars: 36 Primary Language: Rust GitHub Description: Scalable Embedded Vector Index for Go and Rust ### Getting Started: BBQvec is designed to be easy to integrate into projects. Here’s a quick guide to get started: #### Go: ```go package main import ( "fmt" bbq "github.com

### Overview: MAD is a unified cross-domain diffusion model designed for various makeup tasks, offering a comprehensive solution for makeup application and removal. ### Key features: - **Unified Makeup Tasks**: MAD supports beauty filtering, makeup removal, and makeup transfer across multiple domains. - **Cross-Domain Diffusion Model**: Leverages diffusion models to handle diverse makeup tasks effectively. - **GPU Optimization**: Pre-trained on high-performance GPUs for efficient processing. ### Tech stack: Python ### Why it matters: MAD provides a robust and versatile solution for makeup tasks, making it a valuable tool for makeup artists and developers alike.

### Overview: Visual text rendering is a challenging task, especially when precise font control is desired. This work demonstrates that diffusion models can achieve **font-controllable** multilingual text rendering using **just raw images without font label annotations**. ### Key features: - **Font controls require no font label annotations:** A text segmentation model can capture nuanced font information in pixel space without requiring font label annotations in the dataset, enabling zero-shot generation on unseen languages and fonts, as well as scalable training on web-scale image datasets as long as they contain text. - **Evaluating ambiguous fonts in the open world:** Fuzzy font accuracy can be measured in the embedding space of a pretrained font classification model, utilizing our proposed metrics `l2@k` and `cos@k`. - **Supporting user-driven design flexibility:** Random perturbations can be applied to segmented glyphs. While this won’t affect the rendered text quality, it accounts for users not precisely aligning text to best locations and prevents models from rigidly replicating the pixel locations in glyphs. ### Tech stack: PyTorch ### Why it matters: This work provides a novel approach to text rendering that can be applied to a wide range of languages and fonts without the need for font annotations, making it highly versatile and scalable.

### Overview: This project showcases how large language models (LLMs) and their smaller counterparts can be utilized to enable semantic product search on Databricks. Unlike traditional keyword-based product search, semantic search leverages the conceptual similarities between words to find products that align with a user's query. ### Key Features: - **Off-the-Shelf Model**: Demonstrates using an open-source pre-trained model for basic semantic search. - **Hybrid Approach**: Combines semantic and keyword search using a hybrid model. - **Advanced Reranking**: Enhances search results through advanced reranking techniques. - **Fine-Tuning**: Fine-tunes the model to specific product catalogs for more effective search results. ### Tech Stack: - Python ### Why It Matters: Semantic search enables more precise and relevant product recommendations by understanding the context and relationships between words, rather than relying solely on keyword matches. This approach can significantly improve user experience and search accuracy in e-commerce and retail applications. ### Repository: Repository: databricks-industry-solutions/product-search GitHub Stars: 36 Contributors: tim.lortz@databricks.com, saurabh.shukla@databricks.com, mustafaali.sezer@databricks.com, peyman@databricks.com, bryan.smith@databricks.com ### Project Structure: ``` product-search/ ├── 00_Setup.py ├── 01_Data_Prep.py ├── 02_Define_Basic_Search.py ├── 03_Define_Hybrid_Search.py ├──

### Overview: Saturate is a batch inference tool designed for processing datasets efficiently. It allows you to run any OpenAI-compatible endpoint on a dataset, ensuring that the model's output is a complete dataset rather than a series of responses. Saturate handles congestion-aware concurrency, retries, and resumable parquet output, making it ideal for batch processing tasks. ### Key Features: - **Batch Inference**: Processes entire datasets or files through a model. - **Concurrency Management**: Handles congestion-aware concurrency, ensuring sustainable throughput. - **Resumable Output**: Outputs are resumable parquet files, allowing for resuming of failed runs. - **Flexibility**: Supports various OpenAI-compatible endpoints and custom models. - **Ease of Use**: Supports both local and remote endpoints, including HF Jobs and datasets. ### Tech Stack: Python ### Why It Matters: Saturate simplifies the process of running models on datasets, ensuring efficient and reliable batch processing. It eliminates the need for manual tuning and retries, making it a powerful tool for developers and data scientists looking to automate their workflows. ### Repository: Repository: davanstrien/saturate ### GitHub Stars: GitHub Stars: 23 ### Primary Language: Primary Language: Python ### Status: The API and behavior can change in any 0.x release, often with no deprecation period. Pin an exact version (`saturate==0.3.0`) if you rely on how it behaves now, and read [CHANGELOG.md](https://github.com/davanstrien/saturate

Overview: A comprehensive audio processing module for PyTorch, providing Fourier Transform (FFT), Inverse Fourier Transform (IFFT), Short-Time Fourier Transform (STFT), and Inverse Short-Time Fourier Transform (ISTFT) functionalities. Key features: - FFT - IFFT - STFT - ISTFT Tech stack: PyTorch Why it matters: This module simplifies audio processing tasks by providing essential Fourier Transform and inverse Fourier Transform operations, making it a valuable tool for developers working with audio data in PyTorch projects.

### Overview: PeerCache is a lightweight, peer-to-peer L3 KV-cache storage backend built for SGLang HiCache, designed for PD-disaggregated inference. It leverages RDMA zero-copy for efficient data sharing across nodes without a centralized master. ### Key features: - **Embedded Service Discovery**: Nodes auto-host the discovery service in-process, registering endpoints, heartbeats, and pulling the live membership list. - **Consistent-Hash DHT**: Maps keys to nodes and remote addresses, using hashing to shard the mapping across all nodes. - **Slotmap Mode**: An alternative placement mode where keys map to fixed physical slots, reducing metadata lookup. - **Data Stays Local on Write**: Copies pages into a node-local published pool and pushes only a tiny location record to the directory. - **One-sided RDMA READ on Read**: Reads directly into SGLang's registered host buffer. - **Concurrent Multi-threaded I/O**: Per-peer channel pool for efficient I/O operations. - **Disk Persistence Tier**: Pages evicted from memory are promoted back into the pool on a later read. ### Tech stack: Python ### Why it matters: PeerCache provides a lightweight, efficient, and scalable KV-cache solution for distributed inference workloads, eliminating the need for a centralized master and metadata services.

### Overview: This repository contains scripts and documentation for a high-performance deep learning model serving system. The system is optimized for 4x RTX PRO 6000 GPUs and uses vLLM-Moet for speculative decoding, serving the model at 371-376 tok/s with a context size of 1M tokens. The model supports parallel tool calls, image input, and flexible start VMs, making it suitable for various applications. ### Key features: - **High Performance**: The system achieves up to 376 tok/s single-stream agent decode at 30k-120k context, with 1M context. - **Effortful Thinking**: The model supports thinking at maximum effort by default. - **Parallel Tool Calls**: The system allows parallel tool calls, enabling efficient use of computational resources. - **Image Input**: It supports image input, allowing for integration with image-based applications. - **SSE Streaming**: The model supports SSE streaming over the Anthropic Messages API, making it versatile for various use cases. - **OpenAI Chat Completions**: It also supports OpenAI chat completions, providing a comprehensive solution for natural language processing tasks. ### Tech stack: - **GPU**: NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, PCIe, no NVLink), tensor parallel 4. - **Compute Engine**: `g4-standard-192` VM with 4 RTX PRO 6000 cards, 192 vCPUs,

Overview: This repository contains a complete implementation of the ViT (Vision Transformer) model using Triton kernels. The weights are imported from HuggingFace, and a comparison with the official HF implementation is provided. The implementation is designed for educational purposes and can be used in pipelines. Key features: - Triton kernels for ViT implementation - Import from HuggingFace repository - Benchmarks comparing Triton implementation with HF implementation - Benchmarks for various operations (add, layernorm, matmul, matmul3, softmax, conv2d) - Comparison with HuggingFace implementation Tech stack: Python Why it matters: This implementation provides a practical example for learning GPU programming, using Triton kernels to optimize model inference and training. Repository: indri-voice/vit.triton GitHub stars: 36

Overview: This project implements a voice-based AI assistant using OpenAI's ChatGPT and Bing on a Raspberry Pi, demonstrating integration of multiple language models and APIs. Key features: - Wake word detection using Porcupine - Speech recognition with Google Speech Recognition - AI-based chat using ChatGPT and Bing APIs - Multilingual support for English, Mandarin, and Cantonese - Text-to-Speech using Google Text-to-Speech (gTTS) - Context-Aware interactions for both ChatGPT and Bing based on conversation history - Time-sensitive question filtering using custom NLP function Tech stack: Python Why it matters: This project showcases the integration of various AI models and APIs to create a more robust and context-aware voice-based assistant, suitable for educational and educational technology projects. Repository: ivan00105/Voice-Based-AI-Assistant-with-ChatGPT-on-Raspberry-Pi Primary language: Python GitHub stars: 36

### Overview: A Svelte library that converts speech from the microphone to text, making it available to Svelte components. This library is based on the React Speech Recognition library and aims to provide a functional and user-friendly speech recognition experience. ### Key Features: - **Speech Recognition Hook (`useSpeechRecognition`)**: A Svelte hook that allows components to access the transcript of speech picked up from the user's microphone. - **Speech Recognition State Management (`SpeechRecognition`)**: Manages the global state of the Web Speech API, enabling the turning on and off of the microphone. - **Polyfill Compatibility**: Utilizes a speech recognition polyfill to ensure compatibility across modern browsers, including Internet Explorer, and provides a consistent voice experience. ### Tech Stack: TypeScript ### Why It Matters: This library is crucial for developers looking to integrate speech recognition functionality into their Svelte applications, ensuring a seamless and user-friendly experience across various browsers and platforms.

Overview: This repository contains the audio-visual corruption modeling code for testing audio-visual speech recognition on the LRS2 and LRS3 datasets. The code is part of the paper "Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring" presented at CVPR 2023. Key features: - Audio corruption modeling using babble noise from NOISEX-92. - Visual corruption modeling using occlusion patches from a specific paper. - Separate audio files for LRS2 and LRS3 datasets. - Landmarks for LRS2 and LRS3 datasets from a repository. - Scripts for generating corrupted audio and visual data. Tech stack: Python 3.6 or higher. Why it matters: This repository is essential for researchers and developers interested in improving audio-visual speech recognition systems, particularly in the context of robustness against visual corruption.

### Overview: Local FAISS vector store as an MCP server – Agent Memory, drop-in local semantic search for Claude / Copilot / Agents. ### Key features: - **Local Vector Storage**: Uses FAISS for efficient similarity search without external dependencies. - **Document Ingestion**: Automatically chunks and embeds documents for storage. - **Semantic Search**: Query documents using natural language with sentence embeddings. - **Persistent Storage**: Indexes and metadata are saved to disk. - **MCP Compatible**: Works with any MCP-compatible AI agent or client. ### Tech stack: Python ### Why it matters: This project provides a local vector database solution for semantic search and retrieval, making it easier to integrate into existing AI workflows without the need for external dependencies.

Overview: A simple Python AI project that utilizes speech recognition and text-to-speech functionalities. Key features: - Speech recognition - Text-to-speech conversion - Offline text-to-speech with pyttsx3 - Voice commands for various tasks Tech stack: Python Why it matters: This project showcases basic AI capabilities using Python, demonstrating how to integrate speech recognition and text-to-speech functionalities for practical applications.

Overview: This is the Python SDK for the Rev AI platform, providing access to various functionalities for developers. Key features: - Submitting jobs for transcription, translation, and human transcription. - Handling file uploads and URLs for transcribing and translating files. - Configuring and managing custom vocabularies and language settings. - Providing options for skipping diarization, punctuation, and deleting after a certain time. - Supporting authentication headers for URL submissions. - Requesting transcript summaries and translations into multiple languages. Tech stack: Python Why it matters: This SDK simplifies the integration of the Rev AI platform into Python projects, making it easier for developers to leverage its advanced transcription, translation, and human transcription capabilities. Repository: revdotcom/revai-python-sdk Primary language: Python GitHub stars: 36

### Overview: Rostam is an open-source vector database and sub-microsecond key-value store built with Go. It offers high performance, including unmatched query speed and recall, and is designed to be embedded, replicated, or run as a standalone server. With features like HNSW/IVF/Vamana indexes, quantization, hybrid dense+sparse search, and efficient memory usage, Rostam stands out in the vector database space. ### Key Features: - **High Performance Vector Search Engine:** Rostam excels in vector search with features like HNSW, IVF, and Vamana indexes, SQ8/binary/PQ quantization, and hybrid dense+sparse fusion. - **Key-Value Store:** A robust key-value store with options for in-process, replicated, and TCP server stores. - **Standalone Server:** Runs as a standalone server with REST, gRPC, and binary TCP protocols. - **Replicated Cluster:** Supports per-shard Raft replication, online sharding, backups to S3, and RBAC/JWT/mTLS. - **Embedded Library:** Can be embedded directly into Go binaries without requiring a server or cgo. - **MCP Agent Memory:** Provides persistent agent memory and vector-DB tools for MCP clients, including Claude Code, Claude Desktop, and Cursor. - **LLM Response Caching:** An OpenAI-compatible caching reverse proxy for efficient LLM response caching. - **RAG in a Box:** Chunks and indexes documents for answering questions with cited sources, supporting both full-text search and dense

Overview: SaluteJS is a set of minimalistic utilities designed for the implementation of Salute Assistants scenarios. Key features: - Direct code autocomplete for intents and app state - Strongly typed API with SmartApp types - Common types between scenarios and Canvas Apps - Integration support for Node.js web frameworks (expressjs, hapi, koa) - Client frameworks integration support (NextJS, Gatsby) - Support for any types of recognizers (RegExp, String Similarity, SmartApp Brain) - Custom recognizer API - Intents and entities synchronization with SmartApp Brain - Session persisting adapters (memory, MongoDB, Redis) - Assistants based phrases dictionary declaration support Tech stack: TypeScript Why it matters: SaluteJS simplifies the development of Salute Assistants scenarios by providing a set of utilities and strong typing, making it easier to integrate with various frameworks and ensure consistent behavior across different platforms. Repository: sberdevices/salutejs GitHub stars: 36 GitHub description: SmartApp Framework for creating Salute Assistants' scenarios.