Projects

Overview: KitchenAI is an open-source tool designed to simplify the integration of AI implementations with AI teams, providing a modular, loosely coupled architecture that enhances production-grade reliability while allowing developers to focus on their core applications. Key features: - Modular Architecture: AI workflows encapsulated in "bento boxes" that can be updated, replaced, or scaled independently. - High-Performance Messaging: NATS-powered messaging layer for reliable and scalable communication between AI modules. - Framework Agnostic: Supports LangChain, LlamaIndex, and custom implementations, ensuring future-proof integration with any AI framework. Tech stack: Python Why it matters: KitchenAI simplifies the integration of AI with teams, making it easier for developers to focus on their core applications while ensuring reliability and flexibility. Repository: epuerta9/kitchenai Primary language: Python GitHub stars: 131 GitHub description: Open Source LLMOps tool for AI teams README excerpt: KitchenAI is a control plane for AI implementations — designed to bridge the gap between application developers and AI teams. Our platform simplifies AI integration with a loosely coupled, modular architecture that delivers production-grade reliability while letting your teams focus on what they do best.

Overview: LangStruct is a Python library designed to extract structured data from any text using Large Language Models (LLMs). It optimizes prompts for better performance and integrates seamlessly with various LLMs, making it a versatile tool for developers. Key features: - Automatically optimizes prompts using DSPy for better performance. - Supports type-safe Pydantic schemas for full validation and type hints. - Integrates with RAG systems to extract structured metadata from documents and parse user queries into filters. - Works with any LLM provider, including OpenAI, Claude, Gemini, and local models. - Supports multiple optimization strategies (MIPROv2 and GEPA). - Provides character-level mapping back to original text. - Supports multiple LLM providers without rewriting code. Tech stack: Python Why it matters: LangStruct simplifies the process of extracting structured data from text, making it easier and more efficient for developers. It optimizes prompts, integrates with RAG systems, and supports multiple LLM providers, making it a valuable tool for anyone working with text data.

Overview: VocalForge is an open-source toolkit designed to streamline the creation of voice datasets for various AI applications, including TTS models and hotword detection, by automating the process of audio and text formatting. Key features: - Automatically downloads audio from YouTube playlists or inputs raw audio files. - Removes non-speech and overlapping speech data. - Splits audio files into speakers and isolates the same speaker across multiple files. - Reduces background noise using DeepFilterNet. - Normalizes audio levels. - Generates a dataset in the format of LJSpeech. - Transcribes text using OpenAI's Whisper. - Runs text normalization and CTC segmentation for accurate alignment. - Verifies and edits timestamps using VCAuditor to ensure alignment with audio. Tech stack: Python, NVIDIA's NEMO, PyAnnote, CTC segmentation, and OpenAI's Whisper. Why it matters: VocalForge simplifies the process of creating voice datasets, allowing developers to focus on training models rather than manual data curation, thereby saving time and effort.

Overview: SoundStorm is a model designed for efficient parallel audio generation, leveraging Semantic Tokens from AudioLM and Bidirectional Attention with Confidence-Based Parallel Decoding. Key features: - Efficient parallel audio generation - Uses Semantic Tokens from AudioLM - Implements Bidirectional Attention and Confidence-Based Parallel Decoding Tech stack: Python Why it matters: SoundStorm offers a faster and more efficient way to generate neural audio codecs, making it suitable for real-time applications and parallel processing environments. Repository: rishikksh20/SoundStorm-pytorch Primary language: Python GitHub stars: 131

Overview: LumoKit is a Swift package designed for on-device Retrieval-Augmented Generation (RAG) workflows, combining PicoDocs for document ingestion with VecturaKit for vector storage and semantic search. Key features: - Document parsing from PDFs, Markdown, HTML, and text files using PicoDocs. - Multiple chunking strategies: sentence-based, paragraph-based, and semantic chunking with configurable overlap. - Content-aware processing for different types of content. - Semantic search powered by VecturaKit. - Async API implementation. - Database management features: reset, delete chunks, or re-index on demand. Tech stack: Swift. Why it matters: LumoKit provides a comprehensive solution for building searchable knowledge bases, making it essential for developers looking to integrate RAG into their Swift projects. Repository: rryam/LumoKit Primary language: Swift GitHub stars: 131 GitHub description: Swift package for on-device Retrieval-Augmented Generation (RAG).

Overview: HumanPrompt is a framework designed for researchers to easily design, manage, share, and use prompt-based methods with large language models. It aims to simplify the process of human-in-the-loop AI research and development. Key features: - Easy configuration and setup - Support for various datasets and prompting methods - Flexible and extensible architecture - Community-driven development and contributions Tech stack: Python Why it matters: HumanPrompt simplifies the process of human-in-the-loop AI research, making it accessible and efficient for researchers. It provides a standardized way to test and compare different prompting methods, facilitating the development of more effective AI models. Repository: HKUNLP/HumanPrompt GitHub stars: 130 Primary language: Python GitHub description: A framework for human-readable prompt-based method with large language models. Specially designed for researchers. (Deprecated, check out LangChain for better usage!)

### Overview: Affordance Diffusion synthesizes plausible ways of human interacting with objects based on a single RGB image. ### Key features: - **HOI Synthesis:** Generates multiple plausible hand-object interaction scenarios. - **Interpolation:** Smoothly interpolates HOI synthesis for horizontally flipped layouts. - **Heatmap Guidance:** Guides the inference process with keypoints for better results. ### Tech stack: Python ### Why it matters: This project offers a novel approach to generating realistic hand-object interactions from a single image, which has potential applications in virtual reality, augmented reality, and other interactive visual experiences.

### Overview: This repository hosts a collection of research papers focused on Embodied Multimodal Large Language Models (VLA models). It aims to provide the latest advancements in the field of vision-language-action models, with a particular emphasis on robotic manipulation and full-body humanoid control. The repository includes detailed projects, code, and datasets, making it a valuable resource for developers and researchers interested in AI and robotics. ### Key Features: - **ABot-Manipulation**: A large open-source robotic manipulation dataset with Action Manifold Learning (AML) for direct clean action prediction. - **RT-2**: An end-to-end vision-language-action model that integrates vision-language models trained on internet-scale data directly into robotic control pipelines. - **Helix**: The first VLA model achieving full upper-body humanoid control, including fingers, wrists, torso, and head. - **π0 (Pi-Zero)**: A generalist control model across various robot embodiments, utilizing large-scale pretraining and flow matching action generation. - **OpenVLA**: An open-source large-scale vision-language-action model pretrained on 970k+ robotic episodes. - **Gemini Robotics**: A multimodal generalization to physical action model built on Gemini 2.0, enabling complex real-world manipulation without task-specific training. ### Tech Stack: - **Python** - **PyTorch** - **Hugging Face** ### Why It Matters: This repository serves as a comprehensive resource for researchers and developers interested in the latest advancements in vision-language-action models, particularly in the context of robotic manipulation and full-body humanoid control

Overview: This project optimizes the llama.cpp library for IBM POWER8 processors, including vec_perm non-bijunctive collapse, PSE hardware entropy, and DCBT prefetch, making it more efficient and faster on POWER8 hardware compared to x86/ARM/CUDA. Key features: - POWER8 intrinsics compatibility layer (power8-compat.h) - Full L2/L3 cache-resident prefetch hints (ggml-dcbt-resident.h) - AltiVec/VSX performance benchmark (altivec_benchmark.c) Tech stack: C Why it matters: This optimization is crucial for efficient LLM inference on IBM POWER8 servers, providing significant speedup and resource efficiency, particularly beneficial for applications requiring high performance and low latency on POWER8 hardware. Repository: Scottcjn/llama-cpp-power8 GitHub stars: 130 GitHub description: llama.cpp optimizations for IBM POWER8: vec_perm non-bijunctive collapse, PSE hardware entropy, DCBT prefetch. Sovereign inference. Part of the Proof of Physical AI stack.

### Overview: LumaBrowser is a Chromium-based desktop app that integrates a GPU-aware LLM runtime, chat and agents, local image, video, music, and voice capabilities, as well as a Minecraft Multiplayer Server (MCP) for AI-driven browser automation. It simplifies the use of local AI tools by bundling them into a single application, ensuring all functionalities run locally and seamlessly. ### Key Features: - **GPU-Aware LLM Runtime**: Runs models on your hardware, optimizing performance. - **Local AI Stack**: Includes a chat interface, image and video generation, voice and music capabilities, and a Minecraft Multiplayer Server (MCP) for AI-driven browser automation. - **Built-in Chromium Browser**: Offers tabs, history, bookmarks, and a built-in ad blocker. - **Luma On Demand**: A floating voice or text panel that interacts with the current tab. - **Dashboard**: A grid of live widgets from chats and extensions. ### Tech Stack: - JavaScript ### Why It Matters: LumaBrowser simplifies the use of local AI tools by integrating them into a single, powerful desktop application. It optimizes performance, ensuring all functionalities run locally and seamlessly, making it ideal for developers who need a versatile and efficient local AI environment.

Overview: This project implements a text-to-video generation model that adapts to diverse audio samples, focusing on generating videos that are aligned both globally and temporally with the input audio. Key features: - Text-to-video generation model adaptation - Diverse audio-to-video generation - Temporal and global alignment of videos with audio Tech stack: Python Why it matters: This project is significant because it addresses the challenge of generating videos that are semantically aligned with natural audio samples, which is crucial for applications such as audio-visual summarization and interactive storytelling. The method's ability to generate videos that are both diverse and temporally aligned with the input audio is a novel contribution that could have wide-ranging applications in the field of audio-visual synthesis.

Overview: This project evaluates the effectiveness of large language models (LLMs) in email spam detection by comparing them with established machine learning techniques. The results demonstrate that LLMs significantly outperform traditional methods, particularly in few-shot scenarios. Key features: - Spam-T5: A modified Flan-T5 model specifically designed for email spam detection. - Baseline techniques: BERT-like, Sentence Transformers, and Seq2Seq models. - Public datasets: Four datasets used across different training sample sizes. Tech stack: Python. Why it matters: The project showcases the adaptability and superior performance of LLMs in detecting email spam, making them a promising tool for spam detection tasks with limited labeled samples and frequent updates.

Overview: This repository contains the code for the paper "KG-FIT: Knowledge Graph Fine-Tuning upon Open-World Knowledge" (NeurIPS 2024). The project focuses on fine-tuning knowledge graphs using large language models to enhance their performance on open-world datasets. Key features: - KG-FIT with KGE base models - Text and Hierarchical Constraints - Text Embedding within Entity Embedding Tech stack: Python Why it matters: KG-FIT is a significant contribution to the field of knowledge graph fine-tuning, providing a novel approach to improve the performance of knowledge graphs on open-world datasets by leveraging large language models. GitHub stars: 130 Repository: pat-jj/KG-FIT

### Overview: This project applies Connectionist Temporal Classification (CTC) to speech recognition using TensorFlow 1.0, compatible with TensorFlow 2.0. The application is based on the VCTK Corpus, a dataset of over 100,000 audio recordings of 100 speakers. The code includes a comprehensive setup for training and validation, as well as a detailed experiment to evaluate the performance of the CTC model on a small subset of the VCTK dataset. ### Key features: - **CTC Model**: Utilizes TensorFlow's CTC loss for speech recognition. - **VCTK Corpus**: Compatible with the VCTK Corpus, a large dataset of audio recordings. - **TensorFlow 1.0**: Supports TensorFlow 1.0 and is compatible with TensorFlow 2.0. - **Python**: Primarily written in Python with additional dependencies like dill, librosa, namedtupled, numpy, python_speech_features, and TensorFlow. ### Tech stack: - **Python**: The primary programming language used. - **TensorFlow**: The machine learning library used for implementing the CTC model. ### Why it matters: This project demonstrates the effectiveness of CTC in speech recognition, showcasing how it can be applied to a large dataset and evaluated on a small subset. The results highlight the potential of CTC in practical applications, especially in scenarios where a large amount of data is available but the training set is limited.

### Overview: Weightgain is a Python library that allows you to fine-tune any embedding model in under a minute, even with closed-source models from OpenAI, Cohere, and Voyage. It works by training an adapter that sits on top of the model, transforming the embeddings after they're generated, optimizing them for specific retrieval and RAG use cases. ### Key features: - **Fine-tune any embedding model in under a minute.** - **Supports OpenAI, Cohere, and Voyage models.** - **Easy installation with `pip install weightgain`** - **Generate synthetic or custom datasets with `Dataset.from_chunks`, `Dataset.from_synthetic_chunks`, or `Dataset.from_pairs`** - **Train an adapter with minimal code, even without a dataset** - **Apply the adapter to transform embeddings with `adapter.transform(old_embeddings)`** - **Generate a report with various plots for training and performance metrics** ### Tech stack: Python ### Why it matters: Weightgain simplifies the process of fine-tuning embeddings, making it accessible even for those without a dataset. It optimizes embeddings for specific retrieval and RAG use cases, enhancing the performance of your models.

Overview: MIA-Tuner is a method for adapting large language models (LLMs) as pre-training text detectors, presented at the AAAI'25 conference. Key features: - Adapting LLMs to detect text instances seen during pre-training. - Providing a unified script for running all existing baselines and the proposed MIA-Tuner. - Utilizing the WikiMIA-24 dataset for benchmarking. Tech stack: Python Why it matters: MIA-Tuner offers a novel approach to leveraging pre-trained LLMs for text detection tasks, potentially improving the efficiency and effectiveness of text detection models.

Overview: This project provides a comprehensive Skill suite for the MiniMax H3 directing system, including official prompt writing, creative director, mode routing, multi-shot planning, text video prompt generation, reference video prompt handling, keyframe video prompt processing, and prompt reviewer. It aims to streamline the OpenCode skill workflow for better efficiency and accuracy. Key features: - Official prompt writing - Creative director - Mode routing - Multi-shot planning - Text video prompt generation - Reference video prompt handling - Keyframe video prompt processing - Prompt reviewer Tech stack: PowerShell Why it matters: This Skill suite streamlines the OpenCode skill workflow, making it more efficient and accurate, thereby enhancing the overall quality of video productions. Repository: unknowlei/minimax-h3-opencode-skills GitHub stars: 129 GitHub description: OpenCode skill suite for MiniMax H3 directing, routing, multishot planning, prompt generation, and review.

### Overview: MosaicFusion is a novel diffusion-based data augmentation pipeline designed specifically for large vocabulary instance segmentation. It leverages off-the-shelf text-to-image diffusion models to generate image and mask pairs with multiple objects, enabling the training of various downstream detection and segmentation models. This approach is particularly beneficial for improving performance on rare and novel categories. ### Key Features: - **Training-free**: Generates multiple objects directly without requiring additional detectors or segmentors. - **Agnostic to Detection Architectures**: Works with any existing detection architecture. - **Off-the-Shelf Models**: Utilizes existing text-to-image diffusion models for image generation. ### Tech Stack: - Python ### Why It Matters: MosaicFusion provides a powerful tool for enhancing the performance of instance segmentation models, especially in scenarios where rare and novel categories are involved. By generating high-quality images and masks with multiple objects, it significantly improves the accuracy and diversity of training datasets, leading to better model generalization and performance improvements. ### Repository: - Repository: Jiahao000/MosaicFusion - GitHub Stars: 129 - GitHub Description: [IJCV 2024] MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation - Primary Language: Python - GitHub URL: [Jiahao000/MosaicFusion](https://github.com/Jiahao000/MosaicFusion)

### Overview: Mine StableDiffusion is a native, offline-first AI art generation studio built on Kotlin Multiplatform and powered by the stable-diffusion.cpp engine. It offers desktop-class inference capabilities across Android, iOS, and Desktop platforms, ensuring maximum hardware utilization and privacy. ### Key features: - **Native Performance**: Pure C++ backend combined with JNI bindings for maximum hardware utilization. - **100% Privacy**: Everything runs offline, no cloud, no subscriptions, no data harvesting. - **True Multiplatform**: A unified, beautiful Compose Multiplatform UX adapted for Mobile & Desktop. - **Pro-Level Controls**: Granular control over VRAM (Flash Attention, CPU Offloading, Direct Convolution). - **Endless Expansion**: Out-of-the-box support for external `.safetensors` LoRAs, advanced Samplers, and metadata injection. ### Tech stack: - Kotlin Multiplatform - Stable-diffusion.cpp engine - Compose Multiplatform - Vulkan - Metal ### Why it matters: - **Native Performance**: Ensures maximum hardware utilization. - **100% Privacy**: No cloud, no subscriptions, no data harvesting. - **True Multiplatform**: Unified, beautiful UX for Mobile & Desktop. - **Pro-Level Controls**: Granular control over VRAM, CPU offloading, and direct convolution. - **Endless Expansion**: Support for external LoRAs, advanced Samplers, and metadata injection. ### Tech stack: Kotlin Multiplatform Kotlin Multiplatform is a powerful framework that allows developers to write once, run anywhere

Overview: Circuitron is an agent-driven PCB design prototyper that converts natural language requirements into SKiDL scripts, KiCad schematics, and PCB layout files. It integrates a multi-agent pipeline with advanced reasoning and retrieval-augmented capabilities. Key features: - Reasoning-driven workflow - Retrieval-Augmented Generation (RAG) - Real part selection - Automatic schematic and PCB generation - Iterative correction loop - Containerized toolchain Tech stack: Python Why it matters: Circuitron is an innovative tool for agentic PCB design, providing a seamless workflow from natural language prompts to functional designs.