Projects

### Overview: The Grill is a tool designed to measure the performance improvements of your language model server. It provides a straightforward answer indicating whether your changes have made your server run faster, slower, or if the change is inconclusive. ### Key Features: - **Speed Benchmarks:** Measures the performance of your server by comparing two runs with the same setup but different changes. - **Quality Evaluation:** Grades the output of your model to ensure it correctly answers tasks. - **Token Usage Tracking:** Tracks the token usage while streaming to ensure accurate performance measurements. ### Tech Stack: Rust ### Why It Matters: The Grill is a simple yet powerful tool for developers to quickly evaluate the impact of their changes on the performance of their language model server. It helps identify whether your optimizations have indeed improved the server's speed or if the results are inconclusive. Repository: plotarmordev/thegrill GitHub Stars: 14 GitHub Description: Did your change make your LLM server faster? Measure it: faster, slower, or too close to call. Linux and Apple Silicon.

### Overview: Uni-DPO introduces a unified dynamic preference optimization paradigm for training large language models (LLMs) from preference data. Unlike prior DPO-based methods that treat all preference pairs equally, Uni-DPO jointly considers intrinsic data quality and model learning dynamics, enabling more effective and robust preference learning. ### Key features: - **Quality-aware**: Adaptive prioritization of high-quality preference pairs while down-weighting ambiguous ones. - **Dynamics-aware**: Training focus shift toward under-fitted samples to mitigate overfitting. - **Unified & lightweight**: Seamlessly integrates dual-perspective weighting and calibrated NLL into standard DPO with minimal overhead. ### Why it matters: Uni-DPO provides a more effective and robust approach to preference learning in large language models, addressing the limitations of prior methods that treat all preference pairs equally. By considering both intrinsic data quality and model learning dynamics, Uni-DPO enables more efficient and reliable preference optimization, leading to improved model performance and better overall model quality.

Overview: This is a toy implementation of FLUX diffusion transformers, focusing on understanding rectified flow matching, joint attention, and key design choices behind the FLUX models. Key features: - FLUX.1 and FLUX.2 DiT architectures - Rectified flow matching training with velocity prediction and logit-normal timestep sampling - Euler ODE inference with configurable timestep schedules - VAE architectures from the official Black Forest Labs repos Tech stack: Python Why it matters: This project serves as a learning tool to understand the FLUX diffusion transformer models and their training algorithms, specifically focusing on rectified flow matching and joint attention. Repository: purohit10saurabh/toyFLUX Primary language: Python GitHub stars: 15 GitHub description: A toy implementation of FLUX diffusion transformers README excerpt: A toy project to learn about the FLUX.1 and FLUX.2 diffusion transformers by Black Forest Labs, built for understanding rectified flow matching, joint attention, and key design choices.

Overview: LangGraphRAG is a terminal-based Retrieval-Augmented Generation (RAG) system designed for NLP researchers and developers, focusing on advanced conversational AI and information retrieval systems. Key features: - Message history caching - Query transformation - Vector database retrieval Tech stack: Python Why it matters: Ideal for NLP researchers and developers working on advanced conversational AI and information retrieval systems. Repository: ranguy9304/LangGraphRAG GitHub stars: 15

Overview: This project provides a reference architecture, guides, and examples using Amazon Bedrock and Redis as a knowledge base for Retrieval-Augmented Generation (RAG). It aims to streamline the use of LLMs with Redis as a vector database. Key features: - Amazon Bedrock integration - Redis Enterprise Cloud as a knowledge base for RAG - Vector database functionality for LLMs - LLM Cache optimization Tech stack: Amazon Bedrock, Redis Enterprise Cloud, AWS Secrets Manager, AWS Marketplace Why it matters: This collaboration offers a robust, scalable, and efficient solution for developers, streamlining the use of LLMs with Redis as a vector database, and improving the performance and cost-effectiveness of the system.

Overview: This project is a Model Context Protocol (MCP) server implementation for semantic vector search and memory management using TxtAI. The server provides a robust API for storing, retrieving, and managing text-based memories with semantic search capabilities. Key features: - Semantic search across stored memories - Persistent storage with file-based backend - Tag-based memory organization and retrieval - Memory statistics and health monitoring - Automatic data persistence - Comprehensive logging - Configurable CORS settings - Integration with Claude and Cline AI Tech stack: Python Why it matters: This project leverages the powerful semantic search capabilities of TxtAI to enhance the functionality of AI assistants like Claude and Cline, providing a robust and efficient memory management system for text-based data. Repository: rmtech1/txtai-assistant-mcp Primary language: Python GitHub stars: 15 GitHub description: Model Context Protocol (MCP) server implementation for semantic vector search and memory management using TxtAI. This server provides a robust API for storing, retrieving, and managing text-based memories with semantic search capabilities. You can use Claude and Cline AI as well.

Overview: This project provides a local, private, and cost-free persistent semantic memory solution for AI agents, using Qdrant and Model Context Protocol (MCP) integration. The solution includes a Raspberry Pi 5 hardware with Qdrant storage and MCP client integration, ensuring that AI agent memories persist across reboots and searches by meaning rather than keywords. Key features: - Local Qdrant vector database storage - MCP-compatible model context protocol integration - Persistent semantic memory for AI agents - No cloud dependencies, $0 cost - 3s per query performance Tech stack: JavaScript, Python, Node.js, SQLite, ONNX, ARM64 Why it matters: This project offers a practical, low-cost, and efficient solution for AI agents that need to remember information without the risk of forgetting it, ensuring consistent and accurate recall across sessions. Repository: rockywuest/qdrant-mcp-pi5 GitHub stars: 15 GitHub description: Persistent semantic memory for AI agents on Raspberry Pi 5 — local Qdrant + MCP, no cloud, ~3s per query

### Overview: FlashPrefillv2-MLX accelerates long-context causal prefill on Apple silicon by replacing dense attention with block-sparse attention while preserving a pooled correction for omitted blocks. This project implements FlashPrefill V2 for MLX, building upon the Hopper implementation by the paper authors. ### Key features: - Block-sparse attention - Pooled correction for omitted blocks - Acceleration on Apple silicon - Implemented on MLX 0.32.2 and MLX-LM 0.31.3 ### Tech stack: Python ### Why it matters: FlashPrefillv2-MLX provides significant performance improvements for long-context causal prefill on Apple silicon, enabling faster and more efficient processing of large language models.

Overview: This project is a hybrid application boilerplate using Python for the backend and TypeScript for the frontend, powered by Langchain for language model chaining. It includes Dockerization for deployment and benchmarks for inference performance. Key features: - Python backend with FastAPI and Langchain - TypeScript frontend with Vite, Radix UI Primitives, and Redux Toolkit - Dockerized for easy deployment Tech stack: Python, TypeScript, Langchain, FastAPI, Vite, Radix UI Primitives, Redux Toolkit, Docker, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Langchain, Lang

Overview: MindMeld is a simple, open-source GUI application for local AI chat powered by llama.cpp, designed for desktop and mobile platforms. Key features: - Supports multiple platforms: iOS, Android (un-accelerated), Windows, MacOS, Linux - Graphical user interface tailored for chatting with AI - Multiple chat logs with customizable parameters - Customizable character portraits - Automatic model downloading from HuggingFace - 'narrator' feature using slash-command '/narrator' - Lorebook support with customizable lore text - Prompt caching for fast AI chat reply regeneration - Real-time streaming responses - AI samplers for various temperature settings Tech stack: Dart Why it matters: MindMeld simplifies the integration of local AI chat with a user-friendly interface, making it accessible and compatible across various devices and platforms. Repository: tbogdala/mindmeld GitHub stars: 15 GitHub description: A simple-to-use, open source GUI for local AI chat.

Overview: This AI-powered expense affordability agent decides whether a user can afford an expense based on a 90-day balance forecast and real transaction history, with a deterministic decision core and a narrow multimodal evidence layer for receipts and messages. Key features: - Determines whether to pay in full, partially, use installments, or wait for an expense. - Provides a maximum amount safe to pay today, a dated payment plan, the earliest safe date for full payment, and any flexible expenses that must be stopped or reduced. - Personalizes decisions per user from real transaction history. - Uses a 90-day balance forecast against the user's minimum-balance floor for safety. Tech stack: Python. Why it matters: The model never decides money, making the ledger true. It corrected the human by disagreeing with a message that was later caught and produced no ledger amendment. The model is deterministic, unit-tested, and can be reproduced by construction.

Overview: VidGen is an open-source, production-ready AI video and image generation platform designed for creators, indie hackers, and entrepreneurs. Key features: - Multi-Modal AI Generation: Text-to-Image (Text2Img), Image-to-Image (Img2Img), Text-to-Video (Text2Video), and Image-to-Video (Img2Video). - Monetization & Credit Economics: Global Payments with PayPal and Stripe, and Granular Credit Pricing. - Bilingual Ad: Powerful Bilingual Ad. Tech stack: Vue, Nuxt 3, FastAPI, Celery, Tailwind CSS. Why it matters: VidGen provides a complete end-to-end solution for AI video and image generation, including a modern SSR Nuxt 3 frontend portal, bilingual admin dashboard, and Celery async generation engine. It supports Replicate, SiliconFlow, Gemini, and custom workflows, making it a valuable tool for creators and entrepreneurs looking to launch a full-featured AI platform in just 5 minutes.

### Overview: Seedance 1.0 Pro Prompts: Tested Image Prompt Recipes with Example Renders, Aspect Ratios, Model ID (`seedance-1.0-pro`), and Cost Per Image. ### Key Features: - **Model ID:** `seedance-1.0-pro` - **Aspect Ratios:** 16:9, 16:10, 1:1, 4:3, 3:2, 1:1.5, 1:2, 1:2.5, 1:3, 1:3.5, 1:4, 1:4.5, 1:5, 1:5.5, 1:6, 1:6.5, 1:7, 1:7.5, 1:8, 1:8.5, 1:9, 1:9.5, 1:10, 1:10.5, 1:11, 1:11.5, 1:12, 1:12.5, 1:13, 1:13.5, 1:14, 1:14.5, 1:15, 1:15.5, 1:16, 1:16.5, 1:17, 1:17.5, 1:18, 1:18

Overview: This repository demonstrates how to fine-tune the BERT model for text matching using a simplified approach based on the official code library. It includes simplifications of TPU-related logic and uses standard `tf.data` and `tf.estimator` APIs for model building. The model is deployed using TensorFlow Serving for production environments. Key features: - Simplified BERT fine-tuning - Use of standard `tf.data` and `tf.estimator` APIs - Deployment to production using TensorFlow Serving Tech stack: Python, TensorFlow, TensorFlow Serving Why it matters: This repository provides a practical example of how to apply BERT for text matching tasks, making it useful for developers looking to integrate BERT models into their applications. Repository: zhongbin1/bert_for_text_matching GitHub stars: 15

### Overview: MoExtend is an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. This work, presented in the paper "MoExtend: Tuning New Experts for Modality and Task Extension" at ACL 2024, aims to enable large language models (LLMs) to adapt and extend their capabilities to new modalities without the need for extensive model tuning. MoExtend seamlessly integrates new experts into pre-trained MoE models, addressing the challenge of accommodating new modalities within LLMs. ### Key features: - **Modality and Task Extension**: MoExtend enables the adaptation and extension of LLMs to new modalities or tasks, enhancing their versatility. - **Efficient Integration**: It seamlessly integrates new experts into pre-trained models, reducing the need for extensive model tuning. - **Reduced Risk of Catastrophic Forgetting**: By avoiding the need for model tuning, MoExtend mitigates the risk of catastrophic forgetting, ensuring a smoother learning process. ### Tech stack: Python The project is built using Python, leveraging the deep learning framework PyTorch for efficient training and inference. ### Why it matters: - **Versatility in LLMs**: MoExtend allows LLMs to be easily adapted to new modalities, expanding their application beyond text data. - **Reduced Model Tuning**: By avoiding the need for extensive model tuning, MoExtend mitigates the risk of catastrophic forgetting, ensuring a smoother learning process. - **Efficient Adaptation**: MoExtend streamlines the adaptation process, making

### Overview: Russian Vibe is a project that fine-tunes Stable Diffusion XL to generate Russian-themed images. It leverages the capabilities of the Stable Diffusion XL model to create visually appealing and culturally relevant images. ### Key features: - Fine-tuning Stable Diffusion XL to generate Russian corners. - Utilizes the Diffusers library for fine-tuning. - Generates high-quality Russian-themed images using the Stable Diffusion XL model. ### Tech stack: Python ### Why it matters: Russian Vibe showcases the potential of AI in generating culturally relevant and visually appealing images, making it a valuable project for developers interested in AI and image generation.

### Overview: AOI is a model-agnostic perception layer that provides adaptive visual and audio perception for computer-use agents, enabling them to perceive dynamic screen content such as video, animations, and transient UI events with zero retraining. It includes three gated components: inter-step keyframe capture, volume-gated audio observation, and visual narration. ### Key features: - **Inter-step keyframe capture**: Continuous 3 Hz screen sampling with pixel-change gating, optionally filtered by CLIP semantic. - **Volume-gated audio observation**: RMS-energy gate followed by Whisper large-v3 speech transcription. - **Visual narration**: CU model narrates new visual content as a side-output each step. ### Tech stack: HTML ### Why it matters: Across nine CU models (closed and open-source, ranging from 7B to frontier scale), AOI yields +17 to +48 pp on DynaCU-Bench with no retraining. The paper, website, and benchmark definitions are available, along with results, ablations, and a comparison with streaming baselines. Repository: 19PINE-AI/aoi Primary language: HTML GitHub stars: 14 GitHub description: AOI: a model-agnostic perception layer giving computer-use agents adaptive video + audio perception with zero retraining, plus DynaCU-Bench (100 dynamic browser tasks). Paper + benchmark + code.

Overview: Kraits is a plugin for llama.cpp that runs mixture-of-experts models on a 16 GB Mac, specifically targeting TypeScript coding with a 20.4 GB model. Key features: - Runs the 20.4 GB `Ling-3.0-flash-TS128-Q4_K_M.gguf` model on a 16 GB Mac. - Ships an expert pin list (`Ling-3.0-flash-TS128-Q4_K_M.kraits-pins.txt`). - Locks an 8.8 GB bank of the hottest experts in wired GPU memory at load. - Streams cold experts from an SSD. Tech stack: TypeScript. Why it matters: Kraits provides a streamlined solution for TypeScript developers by leveraging a larger model on a smaller device, improving performance and reducing memory usage compared to other models. GitHub stars: 14

### Overview: This project implements an Advanced Prompt Hacking Tester, allowing users to test the responses of an AI system by generating various types of prompts. It includes methods to generate random prompts, contextual adversarial prompts by modifying the original prompts semantically, and inappropriate prompts. The `execute_prompt_test` method processes a given prompt using the AI system and saves the prompt-response pair in a JSON file if specified. The `run_tests` method runs a specified number of tests, generating prompts of random types and executing them. ### Key Features: - **Generate Random Prompts**: Generates prompts with random lengths. - **Contextual Adversarial Prompts**: Modifies original prompts to create adversarial examples. - **Inappropriate Prompts**: Generates prompts that are not appropriate in context. ### Tech Stack: Python ### Why It Matters: This tool provides a comprehensive way to test AI systems' responses to different types of prompts, aiding in evaluating their robustness and performance.

### Overview: 基于Qwen2.5-VL-3B + QLoRA的智能驾驶场景结构化理解系统，通过微调训练模型，能够对驾驶场景进行车道线、车辆、交通标志、驾驶风险的结构化分析。 ### Key features: - Lane detection - Vehicle identification - Traffic sign recognition - Risk assessment ### Tech stack: - Qwen2.5-VL-3B-Instruct (7.1GB) - QLoRA (4-bit NF4 + LoRA r=8) - Transformers + peft + trl - BitsAndBytesConfig (bfloat16 + NF4) - NuScenes CAM_FRONT (500张训练 + 50张评估) - Gradio ### Why it matters: 该系统能够自动对驾驶场景进行结构化理解，提高自动驾驶场景的识别和分析能力，为智能驾驶提供有力支持。