Projects

### Overview: DocChat is an AI-powered RAG application designed to chat with any documentation. It scrapes, processes, and converts documentation into a searchable knowledge base, allowing users to interact with it through natural language queries. ### Key Features: - **Documentation Ingestion**: Accepts a documentation URL as input and recursively crawls internal pages up to config limits. - **Retrieval Strategies**: Supports both **Vector Mode** (vector embeddings) and **TreeIndex Mode** (structural tree retrieval). - **Knowledge Base Reuse**: Reuses existing Qdrant collections or generates a tree index for resource-constrained environments. ### Tech Stack: TypeScript ### Why it Matters: DocChat simplifies the process of integrating documentation into a searchable knowledge base, making it easier for developers to find and understand information quickly. The AI-powered RAG application ensures accurate and relevant answers, enhancing productivity and efficiency in the development process.

Overview: This repository provides a reproducible recipe and orchestration scripts to run large language models (LLMs) on the Qualcomm Hexagon NPU of Snapdragon 8-series phones, including the "unsupported" Snapdragon 8 Gen 1 (Hexagon v69). The project is verified to run Stable Diffusion 1.5 on an OnePlus 10 Pro with 31.3 tok/s token throughput and 107ms token-time-for-token (TTFT) time. Key features: - Run LLMs on Snapdragon 8 Gen 1 and 8 Gen 1 (Hexagon v69) phones - Verified on OnePlus 10 Pro with 31.3 tok/s token throughput and 107ms TTFT time - Includes an orchestration layer for device detection, asset download, ADB push, and prompt execution Tech stack: Shell Why it matters: This project challenges the conventional wisdom about the hardware requirements for running LLMs on Hexagon NPU, providing a solution that works on older devices and opens up the possibility of running LLMs on a wider range of Qualcomm devices, including the older 8 Gen 1 phones. Repository: avisre/snapdragon-npu-llm GitHub stars: 17 GitHub description: Run LLMs on Snapdragon NPU — including the 'unsupported' 8 Gen 1 (Hexagon v69). Verified at 31 tok/s on OnePlus 10 Pro.

Overview: This repository contains all the runnable code, data, and examples from the book "LLM Customization and Fine-Tuning: Adaptation, Distillation, and Alignment" by Manning Publications. It is a hands-on, cost-aware playbook for turning a general-purpose open-weights LLM into a focused, cheaper, privately hosted system that beats a generic API call on your domain. Key features: - Code for fine-tuning an LLM with various techniques like LoRA, full SFT, distillation, and DPO alignment. - Data and examples from the book, organized by chapter. - Reproducible results on a single GPU. - Companion code for the Manning book "LLM Customization and Fine-Tuning". Tech stack: Python. Why it matters: This repository provides a practical guide for adapting large language models to specific use cases, focusing on cost-efficiency, privacy, and reliability in production.

### Overview: BatchGen is a high-throughput batch inference engine designed for large sparse mixture-of-experts (MoE) models. It optimizes batch completion time by coordinating sequence-level scheduling, expert-level batching, and host/device KV-cache movement across GPU clusters. ### Key Features: - **Batch Completion Time (BCT) Optimization:** Minimizes the time to finish a batch of requests. - **Long-Context Prefill:** Keeps large prompt batches productive when model weights and KV state exceed a single GPU's capacity. - **MoE-Aware Scheduling:** Reorganizes sequence work to form larger expert-level batches and reduces long-tail stragglers. - **Host KV Cache:** Uses CPU memory as an extension of the KV-cache hierarchy for workloads that cannot fit entirely on device. ### Tech Stack: C++ ### Why It Matters: BatchGen is essential for both production users and researchers, providing a batch API that can drive offline generation, evaluation, synthetic-data pipelines, test-time scaling, and RL rollouts while making it straightforward to reproduce experiments and compare scheduling policies.

### Overview: The Pinecone Scala Client is a full-fledged Scala client designed for interacting with Pinecone's vector database. It supports a wide range of operations, including vector operations, collection operations, index operations, inference operations, and assistant operations. The client is designed for asynchronous processing and is compatible with both the PineconeVectorService and PineconeIndexService. ### Key Features: - **Vector Operations**: Support for operations such as describeIndexStats, query, delete, fetch, update, and upsert. - **Collection Operations**: Support for listCollections, createCollection, describeCollection, and deleteCollection. - **Index Operations**: Support for listIndexes, createIndex, describeIndex, deleteIndex, and configureIndex. - **Inference Operations**: Support for embedData, rerank, and evaluate. - **Assistant Operations**: Support for listAssistants, createAssistant, describeAssistant, deleteAssistant, listFiles, uploadFile, describeFile, deleteFile, and chatWithAssistant. ### Tech Stack: Scala ### Why It Matters: This library simplifies the interaction with Pinecone's vector database, making it easier for developers to integrate Pinecone into their projects. It provides a consistent and intuitive API, reducing the learning curve and making it easier to manage and scale vector database operations.

### Overview: This project presents a study on cross-modal projection in multimodal language models (MLLMs), specifically examining how visual attributes are projected into textual space. The research was conducted at the Georgia Institute of Technology and Harvard University, with a focus on improving the performance of multimodal language models by leveraging cross-modal projection techniques. ### Key Features: - **Zero-Shot Inference with CLIP**: The project includes a zero-shot inference method using CLIP, which is crucial for evaluating the model's ability to project visual attributes into textual space without any pre-training. - **Fine-Tuning Strategies**: The code supports fine-tuning strategies for the `llava-v1.5-7b` model, allowing for modifications to the projection layer while keeping the LLM frozen, or vice versa. - **Datasets**: The project utilizes four different datasets: Agriculture, Textures, Dermatology, and Humanitarian. Each dataset is preprocessed and formatted for fine-tuning the LLaVa-1.5-7b model. - **Model Zoo**: The project explores different variants of the `llava-v1.5-7b` model, providing a comprehensive model zoo for fine-tuning. ### Tech Stack: - **Python**: The project is built and executed using Python, leveraging the LLaVa's codebase. ### Why It Matters: This project is significant because it addresses the limitations of current multimodal language models in projecting visual attributes into textual space, which is crucial for improving their performance and effectiveness in various applications. By exploring different fine-tuning strategies and utilizing

Overview: This project presents a novel Multimodal Transformer with Variable-length Memory (MTVM) model designed for Vision-and-Language Navigation (VLN) tasks, achieving state-of-the-art performance on R2R datasets. Key features: - Explicitly models the history interactions between observations and the instruction. - Achieves a 65% Success Rate and 59% SPL on unseen test set. Tech stack: C++ Why it matters: The model is a significant advancement in the field of multimodal navigation, offering improved performance compared to existing methods. Repository: clin1223/MTVM GitHub stars: 19

### Overview: This toolkit provides a comprehensive solution for running large language models (LLMs) on AMD Vega 8 and 10 GPUs and APUs, utilizing ROCm 7 and Vulkan for inference. It includes Docker images and benchmarks for `llama.cpp` and LM Studio, focusing on performance optimization and stability. ### Key Features: - **ROCm 7 and Vulkan Backports**: Utilizes the latest ROCm 7 and Vulkan backports for enhanced performance. - **LLM Management**: Supports multi-GPU management, including both Vega and AMD/NVIDIA GPUs. - **Benchmarking**: Includes benchmarks for `llama.cpp` and LM Studio, providing detailed performance metrics. - **GPU Detection**: Automatically detects the Vega 8 iGPU using PCI ID, ensuring consistent results across different environments. ### Tech Stack: Shell-based scripts and Docker images. ### Why It Matters: - **Performance Optimization**: The toolkit leverages the latest ROCm and Vulkan backports to deliver superior performance, particularly in dense models. - **Stability and Consistency**: By auto-detecting the Vega 8 iGPU and managing multi-GPU setups, the toolkit ensures consistent results across different environments, including post-reinstall changes. - **Benchmark Accuracy**: The benchmarks provided are based on optimized memory tuning, ensuring accurate performance measurements for various micro-batches and token generation depths. ### Repository: Repository: daimonionnn/amd-vega-rocm-vulkan-llm-toolkit GitHub Stars: 19 GitHub Description: - Run L

Overview: This project is a RAG system that utilizes real-time news scraping and advanced AI models to generate summaries of related news articles. The system is built using ChromaDB, mixtral-8x7b-instruct-v0.1, and a fine-tuned Bart model. Key features: - Real-time news scraping - Vectorized news articles - Fine-tuned Bart model for text summarization - Fine-tuned mixtral-8x7b-instruct-v0.1 model for prompt categorization Tech stack: Python Why it matters: This project offers a comprehensive solution for generating summaries of related news articles, leveraging advanced AI models and vector databases for improved accuracy and efficiency. Repository: dhivyeshrk/Retrieval-Augmented-Generation-for-news Primary language: Python GitHub stars: 19 GitHub description: RAG system with real-time news scraping built using mixtral-8x7b, ChromaDB, bart summarizer

Overview: FlashAttention brought back to Tesla V100 — a deep llama.cpp fork: SM 7.0 D256 kernels, SplitKV3, q4_0 KV cache, DFlash2 speculative decoding and multimodal fixes. Key features: - Split-D D256 pre-fill kernel - Attention output fully fp32 - q4_0 kernel direct read - q8_0 kernel direct read (opt-in) - D256 tensor core decoding kernel - Adaptive KV split (kv_len/920, max 48) - IQ slimming rearrangement (opt-in) - 2×V100 tensor parallel (NVLink, Windows TCC) - Correctness: sm_70 dequant kernel, uncertainty root fixes, mtmd + DFlash2 crash fix, context checkpoint persistence Tech stack: C++ Why it matters: This fork of llama.cpp brings back FlashAttention to Tesla V100, improving performance and correctness compared to the official stock version.

### Overview: xb is an AI-first SQL/JSON builder designed for relational and vector databases. It simplifies the process of constructing SQL queries and JSON data for various database types, including Qdrant vector databases. The builder supports fluent API chaining, making it easy to build complex queries and pipelines. ### Key Features: - **Unified Vector Entry**: `JsonOfSelect()` now covers all Qdrant search/recommend/discover/scroll flows, replacing legacy `ToQdrant*JSON()` methods. - **Composable SQL**: `With/WithRecursive` and `UNION(kind, fn)` allow direct expression of ClickHouse-style analytics in Go. - **Smart Condition DSL**: auto-filter nil/zero, guard rails via `InRequired`, raw expressions via `X()`, reusable subqueries via `CondBuilderX.Sub()`, and inline conditional blocks. - **Adaptive JOIN Planner**: `FromX` + `JOIN(kind)` automatically skips meaningless joins, keeping SQL lean. - **Observability-first**: `Meta(func)` and interceptors carry TraceID/UserID across builder stages. - **AI-assisted maintenance**: code, tests, and docs co-authored by AI and reviewed by humans every release. ### Tech Stack: Go ### Why it Matters: xb simplifies the process of building SQL and JSON queries for various database types, including Qdrant vector databases. Its fluent API chaining and intelligent join planner make it easy to build complex queries and pipelines. The builder supports both SQL and JSON outputs, and it is designed with observability in mind, ensuring that the

### Overview: This repository contains a Streamlit application designed to demonstrate the latest Anthropic Claude 3 family of multimodal models, including Haiku, Sonnet, and Opus, on Amazon Bedrock. The application allows users to upload various media types such as JPG, PNG, GIF, PDF, CSV, MD, and TXT files to perform creative analysis tasks. The Streamlit application leverages the Anthropic Claude Messages API to generate responses based on the uploaded content. ### Key Features: - **Upload Media Types:** Supports JPG, PNG, GIF, PDF, CSV, MD, and TXT files. - **Model Integration:** Integrates with Amazon Bedrock's Anthropic Claude Messages API. - **Creative Analysis:** Enables users to generate analyses of advertisements and other creative content. ### Tech Stack: Python ### Why It Matters: This application showcases the capabilities of Anthropic Claude 3 in generating creative analyses of advertisements, providing a practical tool for advertising agencies and marketers to enhance their creative processes.

### Overview: This repository contains code from the ODSC Agentic Graph RAG workshop, focusing on combining vector, FTS, and graph retrieval for RAG. It includes observability and guardrails for evaluating the outputs. ### Key Features: - Code from the ODSC Agentic Graph RAG workshop - Combines vector, FTS, and graph retrieval for RAG - Includes observability and guardrails for evaluating outputs - Uses Kuzu for graph storage and LanceDB for embeddings - Provides tips for getting started with the workshop - Dataset of 2,726 FHIR records for training the system ### Tech Stack: Python ### Why it Matters: This workshop is designed to help developers understand and implement advanced AI techniques for graph retrieval and RAG, providing practical tools and guardrails for evaluating the system's performance effectively. ### Repository: graphgeeks-lab/odsc-agentic-ai-summit-2025 ### Primary Language: Python ### GitHub Stars: 19 ### GitHub Description: Code from the ODSC Agentic Graph RAG workshop combining vector, FTS & graph retrieval for RAG. Includes observability and guardrails for evaluating outputs.

### Overview: Squeak-SemanticText is a package designed for Squeak/Smalltalk that enhances the semantic understanding of text, making it accessible and interpretable within the programming environment. It includes features such as a chat completion client, SemanticConversation framework, SemanticCorpus for semantic search and generation, and experimental tools like an integration of semantic search and RAG into the Help Browser. ### Key Features: - **OpenAI API Client**: Supports chat completions, embeddings, and speech transcription/synthesis. - **SemanticConversation**: Framework for conversational agents. - **ChatGPT**: Conversational GUI for Squeak. - **SemanticCorpus**: Framework for semantic search, similarity search, and retrieval-augmented generation (RAG). ### Tech Stack: Smalltalk. ### Why It Matters: This package aims to make semantic context accessible in Squeak/Smalltalk by providing features that enhance the understanding and interpretation of text, making it easier to work with and analyze data within the programming environment.

Overview: llmq is a scheduler designed for distributed batched inference of large language models, providing researchers and production teams with control over their inference stack. Key features: - Supports distributed inference jobs - Uses vLLM for GPU acceleration - Utilizes RabbitMQ for reliable job distribution - Provides resumable downloads and multiple result consumers Tech stack: Python Why it matters: llmq offers a solution for researchers and teams needing to process large volumes of inference jobs while maintaining full control over their models, infrastructure, and costs. Repository: iPieter/llmq GitHub stars: 19 GitHub description: A Scheduler for Batched LLM Inference

### Overview: Facial Expression and Attribute Understanding through Instruction Tuning ### Key Features: - **Instruction Tuning for Facial Expression and Attribute Recognition**: This project leverages instruction tuning to enhance facial expression and attribute recognition, making it easier to understand and interpret facial expressions and attributes in various contexts. - **Model Weights and Dataset**: The project includes weights and a dataset for facial expression and attribute understanding, available on Hugging Face. ### Tech Stack: Python ### Why It Matters: This project is significant because it provides a more intuitive and user-friendly approach to facial expression and attribute recognition, making it easier for developers and researchers to integrate these capabilities into their applications.

Overview: Surogate Studio is an enterprise-grade LLMOps platform designed to accelerate the development and deployment of generative AI applications. Key features: - KV-aware routing, GPU sharding, replicas, and disaggregated serving for production-grade performance. - Git-like Data Hub for models & datasets. - Multi-GPU; Multi-node (Ray-based). - Import/Export from/to HuggingFace and ModelScope. - Pretraining; full fine-tuning; LoRA / QLoRA. - BF16, FP8, NVFP4, BnB; mixed-precision training. - Experiment metrics with charts (Loss, Eval loss, Learning rate, Grad norm., Tokens per second). - Logs and Test vLLM chat. - Integration with Axolotl and SOTA **[Surogate](https://github.com/invergent-ai/surogate)** training libraries. - Benchmark with MMLU, ARC, GSM8k, TruthfulQA, HellaSwag, and more. Red-team for toxicity, bias, misinformation, PII, and harms. - Custom benchmarks with judge and simulators. - Workload deployment on local or remote Kubernetes clusters and on public clouds. - Optional air-gapped. - Kubernetes-native scaling. - Workload/container isolation. - GPU & node monitoring. - Start from templates or deploy custom apps via an intuitive UI. Includes **[Surogate Agent](https://github.com/invergent-ai/surogate-agent)**. - Deterministic configs + predefined recipes. Tech stack: Java Why

### Overview: Crucible is a high-performance inference engine designed specifically for the AMD Strix Halo APU, optimized for long-context agent work with Qwen3.8-Flash-Next. It leverages ROCm/HIP kernels for enhanced performance and compatibility with OpenAI's standards. ### Key features: - **High-Performance Inference Engine**: Optimized for the AMD Strix Halo APU, Crucible excels in long-context inference tasks. - **Exclusive ROCm/HIP Kernels**: Utilizes custom kernels for maximum performance and compatibility with OpenAI standards. - **4 Sessions of 262K Context**: Crucible supports up to 4 sessions of 262K context, ensuring efficient handling of complex queries. - **Local OpenAI-Compatible Server**: Provides chat, streaming, function calls, and exact speculative decoding capabilities. - **Balanced Power Profile**: Runs on a Ryzen AI Max+ 395 with a Radeon 8060S (gfx1151) with 128 GB of unified memory. ### Tech stack: - **Language**: C++ - **GitHub Stars**: 18 ### Why it matters: Crucible is designed for researchers and developers looking to leverage the power of the AMD Strix Halo APU for high-performance inference tasks. Its exclusive ROCm/HIP kernels ensure optimal performance and compatibility with OpenAI standards, making it an ideal choice for AI research and development.

### Overview: Seedance 2.0 Prompts — 8 Tested Recipes With Outputs Dreamina — Seedance 2.0 prompts: tested image prompt recipes with example renders, aspect ratios, the model id (`seedance-2.0`) and the cost per image. ### Key Features: - **Model ID:** `seedance-2.0` - **Image Prompts:** 4 recipes - **Duration:** 5 seconds - **Cost:** $0.066 per second ### Tech Stack: Python ### Why It Matters: - **Access to 300+ Models:** The same base URL and headers provide access to Seedance 2.0 and over 300 image, video, and language models. - **Cost Efficiency:** No pre-paid subscription or monthly plan, starting at $1 per call. - **Real-Time Pricing:** Each task is billed upon completion, and failed tasks are refunded. - **Asynchronous Processing:** Sends tasks, retrieves `task_id`, and queries `GET /v1/tasks/{id}` for results, with background processing for efficiency. ### Also Known As: Dreamina ### Recipes: - **Product Ad:** A matte ceramic mug on a dark stone plinth, steam rising, soft key light from camera left, shallow depth of field, product commercial. - **Cinematic:** A rainy street at night, camera pushes in slowly past neon reflections, a lone figure with an umbrella walks away from camera, 35mm anamorphic, film grain. - **

Overview: FMRG (Flow Map Reward Guidance) is an efficient framework for few-step alignment in generative modeling, providing solutions that match or surpass existing methods across various tasks such as inverse problems, style transfer, and VLM rewards. Key features: - Flow Map Reward Guidance (FMRG) for few-step alignment - Training-free, single-trajectory approach - Integrates and guides the flow using the flow map - Matches or surpasses baselines with as few as 3 NFEs Tech stack: Python Why it matters: FMRG offers a novel and efficient solution for few-step alignment in generative modeling, providing significant speedup and accuracy improvements over existing methods. Repository: jrrhuang/fmrg GitHub stars: 19 GitHub description: Official codebase for the paper "How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance" (ICML 2026).