Projects

### Overview: Montycat is an AI-native NoSQL database with built-in semantic search, designed for RAG, AI agents, and LLM memory. It is a self-hosted alternative to Pinecone, Weaviate, Chroma, Qdrant, and Redis, offering a single engine for vector embedding and NoSQL storage, eliminating external dependencies and sidecar services. ### Key Features: - **Self-Hosted NoSQL + Vector Database**: One Rust-powered engine with semantic search built in. - **RAG, AI-Agent Memory, and Vector Search**: Live where your data already lives, eliminating the need for separate vector databases. - **Blazing Speed**: Powered by the Montycat Engine written in Rust, built for microsecond-level performance. - **Async-First Design**: Fully asynchronous, built on asyncio, perfect for APIs, pipelines, and real-time apps. - **Hybrid Storage**: In-memory for raw speed or persistent for durability, or mix both in one engine. - **AI Semantic & Vector Search**: Rank data by meaning with on-device embeddings, built-in kNN vector search for RAG, AI agents, and LLM apps. - **Schema-Aware**: Define data schemas in Python, enforce them at runtime with zero ceremony. - **True Data Mesh Architecture**: Each keyspace is a self-owned, domain-oriented data product. - **Reactive Subscriptions**: Subscribe to live updates in real-time per key or per keyspace. - **Memory-Safe & Secure**: Backed by Rust’s zero-cost abstractions and modern TLS communication

### Overview: This repository is a dataset preparation tool designed specifically for the Mochi-1 video model fine-tuning process. It includes features for clip extraction, captioning, and formatting videos into the required format. ### Key Features: - Video segmentation into 2.5-second segments - Resolution and frame rate standardization - Creation of caption placeholder files - Batched operation support for faster processing ### Tech Stack: Python ### Why it Matters: This tool streamlines the video processing required for fine-tuning the Mochi-1 model, ensuring that the videos are in the correct format and ready for training.

### Overview: Octolib is a Rust library designed for AI providers, offering a single `provider:model` string to access 31 different AI models. The crate includes built-in cost tracking, making it easy to manage pricing and token usage. It supports a variety of providers, including OpenAI, Anthropic, and others, with a unified interface and zero configuration for pricing and token tracking. ### Key Features: - **🔌 31 providers, one interface** — OpenAI, Anthropic, xAI, OpenRouter, Google Vertex & Studio, Amazon Bedrock, DeepSeek, Moonshot (Kimi), MiniMax, Z.ai, BytePlus, Alibaba Model Studio, Groq, Cerebras, NVIDIA NIM, Together, Featherless, Fireworks, Hetzner, Inception Labs, Meta, Tinker, OpenCode Zen/Go, OctoHub, Cloudflare Workers AI, Ollama, WebWright, Local, and CLI proxies (Codex, Claude, Gemini, Cursor). - **💰 Cost tracking with zero config** — per-model pricing tables ship in the crate; every response carries input, output, cache, and reasoning tokens with USD cost. - **🧰 Tool calling** — one `ToolCall` format across providers, with JSON Schema parameter validation and multi-turn conversations. - **📋 Structured output** — JSON and JSON Schema modes, validated locally even when the upstream doesn't enforce them. - **🧠 Thinking/reasoning** — reasoning content and token counts surfaced separately from the answer; the `ReasoningEff

### Overview: cuVS Lucene is a project that leverages NVIDIA's cuVS library to accelerate vector search and clustering on the GPU. This project provides a pluggable KnnVectorsFormat that uses cuVS for efficient vector index building and search, making it possible to integrate GPU acceleration into existing Lucene applications without significant code changes. ### Key Features: - **GPU Acceleration**: Offloads vector index build and search to NVIDIA GPUs, enabling faster processing times. - **Standard Codec Compatibility**: Plugs into standard Lucene codecs, allowing seamless integration with existing Lucene applications. - **Four Codecs**: Four codecs are provided for different use cases: - `Lucene101AcceleratedHNSWCodec`: GPU-accelerated HNSW build with CPU HNSW search. - `LuceneAcceleratedHNSWScalarQuantizedCodec`: Scalar-quantized vectors for smaller index footprints. - `LuceneAcceleratedHNSWBinaryQuantizedCodec`: Binary-quantized vectors for even smaller index footprints. - `CuVS2510GPUSearchCodec`: GPU-accelerated HNSW build and GPU search. ### Tech Stack: Java ### Why It Matters: This project is crucial for developers looking to enhance their vector search capabilities on the GPU, potentially improving the performance of their applications by leveraging NVIDIA's advanced hardware. The ability to integrate GPU acceleration into existing Lucene applications without significant changes makes it highly valuable for developers working with large datasets and complex search queries.

Overview: This project fine-tunes the Mask2Former model for semantic segmentation on the FoodSeg103 dataset, enhancing its performance in identifying and segmenting food items from images. The project includes deploying the model via Gradio for interactive inference. Key features: - Fine-tuning Mask2Former on the FoodSeg103 dataset - Deploying the fine-tuned model via Gradio for real-time inference Tech stack: Python Why it matters: This project showcases the potential of Mask2Former in food segmentation, offering a practical solution for developers looking to improve their image segmentation models.

### Overview: SOLVE-Med is a modular medical QA system designed to route questions to domain-specific Small Language Models (SLMs) and fuse their opinions using a lightweight orchestrator. The system is self-hostable, GPU-friendly, and supports multi-turn chat with automatic history reformulation. ### Key features: - **Ask-a-Specialist**: Directly ask a question to a specialist, bypassing the fusion stage. - **Plug-and-Play Specialists**: Drop new SLMs by editing a YAML file. - **Memory-Aware**: Low mem mode keeps only the orchestrator model resident, streaming SLMs on demand. - **Language-Agnostic Prompts**: Templates live in `backend/app/prompts/<lang>/`. ### Tech stack: - **Backend & REST API**: FastAPI + Uvicorn - **LLM Runtime**: Unsloth (4-bit quantization, LoRA) - **Orchestrator**: `gemma-2-9b-it` (quantized) - **Specialists**: 10× `Llama-3.2-1B-Instruct` fine-tuned SLMs - **Router Agent**: DistilBERT multilingual (multi-label) - **Frontend**: React + Vite - **DevOps**: Docker / Makefile ### Why it matters: - **Self-hostable**: Supports GPU-friendly and multi-turn chat. - **GPU-friendly**: 4-bit quantization for efficient computation. - **Cloud-free**: Eliminates the need for cloud-based infrastructure. ### GitHub stars

### Overview: Triton OpenAI Gateway is an OpenAI-compatible API gateway that integrates with NVIDIA Triton and vLLM, providing a consistent API for chat and non-chat models. It handles large multimodal inputs, backpressure, and observability without disrupting Triton's model lifecycle or vLLM scheduler. ### Key Features: - **OpenAI-style API for LiteLLM, LibreChat, SDKs, and internal clients.** - **Text/VL chat, dense or hybrid embeddings, and reranking.** - **Multimodal inputs: Images, video, audio, and PDF with bounded preprocessing.** - **Runtime: NVIDIA Triton `26.10`, vLLM `0.31.0`, and Python `3.12`.** - **Operations: Admission queues, cancellation, structured logs, Prometheus metrics, and OTLP traces.** - **Deployment: Digest-pinned Docker build and Kubernetes Helm charts.** ### Tech Stack: - Python ### Why It Matters: This project provides a practical and consistent API for clients like LiteLLM, LibreChat, OpenAI SDKs, and internal applications, addressing the gaps in raw Triton inference endpoints. It ensures that clients can seamlessly integrate with the existing infrastructure without disrupting the model lifecycle or scheduler.

Overview: This project is an intelligent web crawling and content extraction tool that converts web content into structured JSON data, specifically designed for fine-tuning Large Language Models (LLMs). It leverages web scraping capabilities and Google Gemini AI to generate question-answer pairs from crawled web pages. Key features: - Intelligent web crawling - Content extraction - AI-powered Q&A generation - Modern GUI interface - Command-line interface - Configurable settings - JSON output - Robots.txt compliance - Real-time progress tracking Tech stack: Python Why it matters: This tool simplifies the process of fine-tuning LLMs by providing structured JSON data directly from web content, enhancing the accuracy and relevance of the model's responses. Repository: Razee4315/jsonAgentWeb Primary language: Python GitHub stars: 10 GitHub description: An intelligent web crawling and content extraction tool that transforms web content into structured JSON data suitable for fine-tuning Large Language Models (LLMs).

Overview: caix is a local Core AI inference server designed for Apple silicon, enabling seamless integration with OpenAI and Anthropic APIs. It offers a web dashboard, chat interface, and streaming capabilities, all without requiring an internet connection. Key features: - A web dashboard for machine stats, model management, and live usage. - A chat interface with markdown, streaming, tools, skills, and MCP. - OpenAI-compatible and Anthropic-compatible APIs. Tech stack: Swift Why it matters: caix simplifies the integration of large language models with Apple silicon, making it accessible and efficient for developers and users alike. Repository: RedHillsMediaFL/caix GitHub stars: 9 GitHub description: caix — native Apple Core AI inference server for Apple silicon (beta): OpenAI/Anthropic API, dashboard, streaming chat with tools/skills/MCP, MTP speculative decoding.

Overview: This project provides a quick tutorial on fine-tuning Llama3.1 using only free tools and text data. The tutorial includes all the necessary code in an IPython Notebook, making it easy to follow along step by step. Key features: - Fine-tuning Llama3.1 - Using free tools and text data - All code included in an IPython Notebook Tech stack: Jupyter Notebook Why it matters: This tutorial offers a practical guide for developers looking to fine-tune Llama3.1 without requiring expensive hardware or specialized knowledge, making it a valuable resource for anyone interested in natural language processing.

### Overview: Slipstream is a macOS application that allows running large AI models on a Mac with limited RAM, by streaming MoE expert weights from an SSD instead of loading the entire model into memory. It is designed to be self-contained and lightweight, making it ideal for developers and coders who need to run complex AI models on their Macs without running out of memory. ### Key Features: - **MoE Expert Streaming**: Slips the MoE expert weights from an SSD into a bounded cache, allowing the model to run on a 16-36 GB Apple-Silicon Mac with a 35B-480B MoE model. - **Self-Contained App**: The app is bundled with the engine, making it easy to install and run without any dependencies. - **No Compile Required**: The app is ready to use with just a single click, no need to compile or install anything. - **User-Friendly Interface**: The app is designed to be accessible and usable, with clear and visible labels for all settings. - **Performance Optimization**: The app includes optimizations for performance and accessibility, such as fixed labels, proper tab strip functionality, and improved UI for screen readers. ### Tech Stack: - **Python**: The core engine is written in Python, leveraging the LLVM compiler infrastructure. - **Metal**: The app runs on macOS, utilizing Metal for rendering and performance optimization. - **Quic**: The app supports direct authenticated QUIC peers for remote inference, allowing for secure and encrypted communication. ### Why It Matters: Slipstream offers a powerful

### Overview: The Big Picture Large Language Models (LLMs) like Phi-3 are essentially **next-token prediction machines**. Given a sequence of tokens, they predict what token should come next by repeatedly processing the sequence through token-by-token generation. This approach allows models to generate entire responses by predicting the next token based on the previous ones. ### Key Features: - **Tokenization**: Converts text into numerical tokens, with each token represented by a unique ID. - **Architecture**: The model consists of three main components: token embeddings, transformer layers, and a language model head. - **Token Embeddings**: Converts token IDs into dense vectors that capture semantic meaning. - **Transformer Layers**: Composed of self-attention mechanisms and feed-forward networks, enabling efficient processing of sequential data. - **Self-Attention Mechanism**: Allows tokens to "look at" and learn from other tokens in the sequence, preventing "cheating" by only allowing tokens to attend to previous tokens. - **Feed-Forward Network (MLP)**: Processes each token's representation independently, enabling the model to learn complex patterns. - **Language Model Head**: Predicts the next token in the sequence, enabling the model to generate coherent responses. ### Tech Stack: Jupyter Notebook The project uses Jupyter Notebook for hands-on training workflows with Phi-3, providing a practical and interactive environment for exploring the Transformer architecture, token-by-token generation, and LLM training. ### Why It Matters: Practical Application This project offers a comprehensive guide to understanding how LLMs like Phi-3 work, focusing

### Overview: Krea 2 for Forge Run Krea 2 (Krea AI's 12B single-stream DiT) in Stable Diffusion WebUI Forge (Neo) — the first open-source Krea 2 extension. One-click model download, presets, fp8, full + piecewise loading. By Stable Yogi. ### Key Features: - **⭐ One-click Muse by Stable Yogi** (Krea 2 v1.5 Turbo, photoreal) — GGUF or fp8, straight from the "Krea 2" tab. - **Native Krea 2 architecture in Forge** — no ComfyUI needed. - **Both loading streams supported:** - **Full model** — one combined checkpoint with everything baked in. - **Pieces** — a bare DiT checkpoint; the TE + VAE are auto-loaded (or pick them yourself). - **fp8 supported** for the DiT and the text encoder (half the size/VRAM). - **One-click model downloader** — the "Krea 2" tab fetches every file into the right folder. - **`krea2` UI preset** — auto-sets sampler/steps/CFG and auto-selects the TE + VAE. - **Detail Boost** (opt-in) — rebalances the 12-layer text conditioning toward the deep, fine-detail taps with RMS-safe renormalisation: sharper identity/texture, no oversaturation. *Want more? The full **Enhancement Suite** (advanced Prompt

### Overview: UPARIS-DS (Uncertainty-Aware Paragraph-Level Iterative Reflective Deep Search Agents) 是一种基于压力驱动的段落级迭代反思搜索框架，通过五因子不确定性量化模型（5-Factor UQ Model）和动态反思压力机制（Dynamic Reflection Stress）来增强深度搜索代理的能力。该框架利用语义熵与方差惩罚对证据权威性、时效性和一致性进行多维建模，并引入多样性去噪排序和跨维度置信度评估，有效解决了RAG中的信息冲突与幻觉问题，实现了高事实保真度和逻辑鲁棒性的长文本深度合成。 ### Key features: - **段落级迭代反思**：通过迭代反思机制，提高搜索结果的可靠性。 - **反思压力模型**：动态分配检索预算与推理深度，适应不确定性。 - **智能体协作**：8个专业化代理协同工作，提高搜索效率。 - **不确定性量化**：通过5因子权重模型量化不确定性，确保结果的准确性和可靠性。 - **去噪重排序**：多样性去噪排序和跨维度置信度评估，提升搜索质量。 ### Tech stack: - **Python**：实现核心算法和框架。 - **asyncio**：并发处理和控制重试。 - **aiohttp**：异步HTTP请求处理。 - **pydantic**：数据验证和模型管理。 - **LLM编排**：智能体之间的通信和协作。 - **Tavily API**：多样本检索和去

### Overview Air.rs is a high-performance, zero-config, hardware-agnostic language model inference engine written in Rust. It auto-detects available hardware, selects the optimal attention and GEMM kernels, and routes through a pluggable six-family quantization engine — all without touching a config file. Key properties include quantized weights staying in their native format until the final multiply-accumulate cycle, attention kernel selection based on GPU compute capability and inference phase, and unbounded context with a paged KV cache enabling infinite context without pre-allocating peak VRAM. ### Key Features - **Quantized Weights**: Stay in their native format until the final multiply-accumulate cycle. - **Runtime Attention Kernel Selection**: Based on GPU compute capability and inference phase (prefill vs. decode). - **Paged KV Cache**: Enables unbounded context with a block size of 16, without pre-allocating peak VRAM. - **OpenAI-Compatible REST Server and SSE Streaming**: Built-in and ready for use. - **Native C ABI (`libair.so` / `libair.a`) and PyO3 Python Package (`air_rs`)**: Provides seamless integration. ### Tech Stack - **Rust**: The primary language for the project. - **Hardware**: Automatically detects and selects the optimal attention and GEMM kernels based on the hardware capabilities. - **Quantization Engine**: Six-family quantization engine for efficient inference. ### Why It Matters Air.rs simplifies the process of deploying language models by abstracting away the complexities of hardware selection and configuration.

Overview: TileMEM / TilePO is a research project aimed at demonstrating the potential benefits of tile-level memory placement and admission for high-throughput Multi-Headed Embedding (MoE) serving. It includes a method description, source code, evidence, and public manifests, with the primary language being Python. Key features: - Tile-level memory placement and admission optimization - BF16 profile-guided tile-level placement and admission - Open-source project for reproducibility and verification Tech stack: Python Why it matters: This project serves as a demonstration of potential benefits of tile-level MoE memory placement and admission, and it aims to provide reproducible and verifiable evidence for its claims. Repository: TerminusAkivili/TileMEM GitHub stars: 10 GitHub description: TileMEM / TilePO: profile-guided tile-level placement/admission for high-throughput MoE serving.

### Overview: This project provides a set of PHP classes for semantic search, utilizing Ropherta as the embedder and SQLite3 Vector Search as the vector database. It includes features for keyword search and advanced functionalities like using vector search as a reranker and applying HyDE for symmetric embeddings. ### Key Features: - **Semantic Search Classes**: Interfaces for semantic search. - **Ropherta Embedder**: Uses Ropherta for embedding text data. - **SQLite3 Vector Search**: Stores and searches vector data using SQLite3. - **Keyword Search**: Implements a keyword search using SQLite3 FTS4 and BM25. - **Ingestion**: Supports JSONL format for ingesting documents. - **Vector Index Dependencies**: Requires SQLite3 Vector Search and Ropherta. - **Multilingual Model**: Supports multiple languages with SentencePiece and XLM Tokenizer. ### Tech Stack: PHP ### Why It Matters: This project is useful for developers looking to implement semantic search functionalities in their PHP applications, enhancing the search capabilities of their projects.

Overview: Mila is a C++23 library designed for developers and researchers working with open LLMs, providing an explicit neural-network library that allows for understanding and inspecting every forward pass, backward pass, and gradient. It is crafted to be read, understood, extended, and challenged, making it suitable for researchers, engineers, and developers who want to know exactly what happens in every forward pass and trace every gradient. Key features: - Explicit neural-network components - Compile-time device and precision choice - Full control over gradients and kernels - Readable and understandable code Tech stack: C++23, CUDA, modern C++ features like modules, deducing-this, std::format, and concepts Why it matters: Mila is designed for developers and researchers who want to understand exactly what happens in every forward pass, backward pass, and gradient. It is crafted to be read, understood, extended, and challenged, making it suitable for researchers, engineers, and developers who want to know exactly what happens in every forward pass and trace every gradient.

Overview: AtomGen is a library designed for handling atomistic graph datasets, focusing on transformer-based implementations. It provides utilities for training various models, experimenting with pre-training tasks, and integrating pre-trained models with Hugging Face. Key features: - Aggregates and standardizes datasets from diverse sources. - Supports pre-processing of datasets for the S2EF pre-training task. - Provides utilities for training models, experimenting with pre-training tasks, and integrating pre-trained models with Hugging Face. Tech stack: Python Why it matters: AtomGen simplifies the process of handling and utilizing atomistic graph datasets, enabling large-scale pre-training and generative modeling on atomistic graphs. Repository: VectorInstitute/atomgen GitHub stars: 10 GitHub description: Library for handling atomistic graph datasets focusing on transformer-based implementations, with utilities for training various models, experimenting with different pre-training tasks, and a suite of pre-trained models with huggingface integrations

### Overview: A multi-modal product catalog platform that leverages CLIP and BGE embeddings for image and text embedding, hybrid search, and decision traceability. The system is designed for efficient image-based product cataloging with enhanced accuracy and explainability. ### Key features: - **Image and Text Embedding**: Automatically embeds product images and textual descriptions. - **Catalog Enrichment**: Enhances catalog entries with additional information. - **Duplicate Detection**: Identifies and removes duplicate product entries. - **Vector Indexing**: Indexes catalog entries for hybrid search. - **Hybrid Search**: Combines image and text-based search results for improved relevance. - **Recommendations**: Provides relevant product recommendations. - **Price Estimation**: Offers price estimates with explanations of comparables and exclusions. - **Decision Trace**: Provides structured decision traces for every intelligence feature. ### Tech stack: Python ### Why it matters: This platform offers a high level of explainability and reliability, making it valuable for businesses looking to enhance their product cataloging and search functionalities. The decision traces provide transparency and trustworthiness, ensuring that the system's recommendations and price estimates are backed by clear reasoning.