Projects

### Overview: Auto-tuning for vLLM aims to optimize performance across various LLM deployments, leveraging Optuna for efficient tuning and benchmarking with GuideLLM and other metrics. ### Key features: - 🎯 Flexible backends: Local execution by default; optional Ray for distributed runs. - 📊 GuideLLM benchmarking: Warmup/cooldown, ramp-up, trace replay, multimodal VLM, synthetic or custom datasets. - 🧮 Rich objectives: Multi-objective optimization with arithmetic metric expressions. - 🔀 Smart sampler selection: Automatic grid or random mode based on search-space size. - ⚡ Speculative decoding: Tune EAGLE3, ngram, Medusa, and MTP parameters from YAML. - 📈 Optuna integration: User attributes for extra metrics; dashboard launcher included. - 🗄 Flexible storage: SQLite for local use, PostgreSQL for production (optional). - ✅ Easy configuration: YAML-based study and parameter configuration. - ✅ Tested: Unit tests and CI on Python 3.10–3.12. ### Tech stack: Python ### Why it matters: Auto-tuning for vLLM helps in optimizing performance across various LLM deployments, ensuring efficient use of resources and improved user experience.

### Overview: This project fine tunes a GPT-2 model for code generation using the Hugging Face Transformers library. The script handles data preprocessing, model initialization, and training, making it a comprehensive solution for code generation tasks. ### Key features: - Fine-tuning of a GPT-2 model for code generation. - Use of Hugging Face Transformers library for model training. - Utilization of Datasets library for handling the dataset. - Training on a ByteLevelBPETokenizer and special tokens. - Data collator and masked language modeling enabled. - Training arguments set for multiple epochs and batch size. ### Tech stack: Python ### Why it matters: This script serves as a proof of concept for code generation using a GPT-2 model, demonstrating its effectiveness in generating code based on provided text data. The fine-tuning process on smaller models like GPT-2 can be used to prototype and improve the model's performance over time. Repository: Jayveersinh-Raj/code_generation_gpt2 Primary language: Python GitHub stars: 7 GitHub description: Fine tuning a gpt2 model for code generation/completion. This is the work under Advanced Engineering School (AES) Innopolis University. Data and model is not open source, the prototyping is for learning purposes.

Overview: This project is a V-lang API wrapper for the llama.cpp model, providing a convenient way to interact with the model without the need for C++ code conversion. The wrapper is based on the chatllm.cpp library, which is optimized for CPU-only execution and supports a limited set of model types. The project includes instructions for compiling and using the V-lang version, along with a script for downloading and using AI models. Key features: - V-lang API wrapper for llama.cpp model - Supports CPU-only execution - Limited support for model types and architectures - Includes instructions for compiling and using the V-lang version - Script for downloading and using AI models Tech stack: C Why it matters: This project simplifies the process of working with llama.cpp models by providing a V-lang API wrapper, making it easier for developers to integrate the model into their projects without the need for extensive C++ code conversion.

Overview: This project provides a Docker bundle that deploys the Qwen3.6-27B model on an AMD Instinct MI50 GPU, utilizing the vLLM framework with ROCm optimization. The bundle includes patches for the Qwen3.5/3.6 hybrid Gated-DeltaNet support and is designed to run on a single eGPU setup. Key features: - Deploy Qwen3.6-27B model on AMD Instinct MI50 GPU - Use OpenAI-compatible HTTP API on port 8000 - Support for Qwen3.5/3.6 hybrid Gated-DeltaNet - Patch files for vLLM to load and run Qwen3.5/3.6 models from GGUF checkpoints - Docker-compose configuration for eGPU setups Tech stack: Python Why it matters: This project simplifies the deployment of Qwen models on GPUs, making it easier to run large language models on eGPUs, which can significantly reduce training times and improve performance compared to traditional CPU setups. Repository: Kausik-A/qwen3.6-27b-mi50-vllm

Overview: This CLI tool is designed to generate high-quality synthetic datasets for Large Language Models (LLMs) through the use of seed examples or domain descriptions. It connects to any OpenAI-compatible LLM to ensure quality and controllability before training begins. Key features: - Seed expansion: Amplifies 20-50 hand-crafted seed examples into thousands of quality-filtered training examples. - Domain generation: Creates datasets from scratch using hierarchical topic trees. - Quality scoring: LLM-as-judge scoring with 1-10 scale and rule-based checks for toxicity, PII, duplicates, and placeholders. - Decontamination: Detects overlap with MMLU, GSM8K, HumanEval, ARC, and HellaSwag via exact match, n-gram, and embedding similarity. - Multiple formats: Export to JSONL, Alpaca, ShareGPT, ChatML, and HuggingFace. - HTML reports: Diversity metrics, topic coverage heatmaps, score distributions, and contamination evidence. - Provider-agnostic: Works with Ollama, OpenAI, Anthropic, vLLM, and any OpenAI-compatible API. Tech stack: Python Why it matters: Manual annotation costs $0.02-$5+ per example at scale. This toolkit addresses the issue of noise in naive synthetic pipelines and provides a way to measure and control dataset quality before training begins. Repository: KazKozDev/synth-dataset-kit Primary language: Python GitHub stars: 7 GitHub description: CLI tool for generating high-quality synthetic datasets

Overview: Throttle is a tool designed to help developers reduce their AI bill by identifying and moving cheaper AI models to handle the same tasks as your current models. It works by testing your app's traffic against different models and moving the ones that perform better. Key features: - Identifies and moves cheaper AI models to handle tasks currently performed by more expensive ones. - Runs in your code as a GitHub Action, requiring only one change to your API base URL. - Tests each AI call in your app against other models, either open or closed, on the traffic your app actually sends. - If a model performs as well as or better than the current one, your team approves, and the call moves. If a model fails, the request is retried on the original model. - Open source and designed for teams looking to cut their AI bill. Tech stack: Python Why it matters: Throttle helps developers reduce their AI costs by efficiently moving cheaper models to handle tasks currently performed by more expensive ones, ensuring that the team's approval is based on actual performance rather than just the model's name. Repository: KushagraKanaujia/throttle Primary language: Python GitHub stars: 7 GitHub description: Stop renting your AI. Throttle tests your AI calls against cheaper models on your own traffic and moves the ones that pass. Open source.

Overview: Plux simplifies AI workflows by providing a file-based interface, a plus button for quick AI interactions, and a built-in magic notepad for capturing insights. It supports multiple large language models and local inference engines, making it adaptable to different environments. Key features: - Filetree-driven interface - Plus button for AI content forwarding - Magic notepad for capturing insights - MCP integration for backend connectivity - Multi-LLM support - Local and remote inference - Cross-platform compatibility - Open and extensible design Tech stack: TypeScript Why it matters: Plux streamlines AI interactions, reduces copy-paste errors, and enhances productivity by keeping work in a single, intuitive workspace.

Overview: This project demonstrates a simple shoes classification model using ResNet50 with transfer learning for a deep hackathon challenge. It achieves ~77% accuracy on the validation dataset. Key features: - Fine-tuning of ResNet50 with a pre-trained model for improved accuracy - Unfreezing the last 2 residual blocks for fine-tuning - Utilizing a dataset of 14 classes in jpg image format Tech stack: Python, Keras, Scikit-learn Why it matters: This project showcases the effectiveness of transfer learning and fine-tuning techniques in improving model performance, making it valuable for developers looking to enhance their models with pre-trained knowledge.

Overview: This project is a modular, configuration-driven pipeline designed to transform raw financial market data into structured datasets suitable for Large Language Model training. The pipeline follows a medallion architecture with built-in auditing and quality checks, ensuring traceability and data reliability. Key features: - Medallion Architecture: Three-stage data processing (raw → silver → gold) for traceability and quality assurance. - Multiple Export Formats: Generates datasets in CPT, SFT, and TXT formats. - Configuration-Driven: All data sources, feature schemas, rate limits, and processing parameters are defined in YAML files. - Offline-First Design: Core logic developed and validated using a local bootstrap dataset before connecting to live data sources. - Built-in Auditing: Automated checks for data quality issues such as lookahead bias, missing values, and statistical anomalies. - Robust Online Transport: HTTP transport layer with proxy rotation, rate limiting, checkpointing, and resumable downloads. Tech stack: Python Why it matters: This project is crucial for developers and data scientists looking to integrate financial data into Large Language Models, ensuring the data is ready for training and maintaining high standards of quality and reliability.

Overview: PSAlign is an innovative framework designed to enhance the safety alignment in text-to-image diffusion models, specifically tailored for personalization and fidelity preservation. Key features: - Personalization: Adjusts safety thresholds based on user profiles (e.g., stricter for minors, culturally aware for diverse groups). - Fidelity Preservation: Maintains image quality and text alignment while suppressing harmful content. - Compatibility: Works with Stable Diffusion 1.5 and SDXL via lightweight adapters (no full model retraining). Tech stack: Python. Why it matters: PSAlign addresses the need for safer and more contextually sensitive text-to-image generation, ensuring that models can be used responsibly and respectfully across different demographics and contexts.

### Overview: This monorepo is a production-ready RAG system featuring local LLM inference, hybrid retrieval, semantic caching, and full observability. It supports multiple formats of document ingestion and is compatible with OpenAI APIs. ### Key features: - **🤖 Local LLM Inference**: Runs models locally with vLLM, avoiding API costs for development. - **🌐 Cloud LLM Fallback**: Routes complex queries to OpenRouter (Claude, GPT-4, etc.). - **🔍 Hybrid Retrieval**: Dense + Sparse vector search with Qdrant. - **📊 Full Observability**: Langfuse tracing with session & user tracking. - **💾 Semantic Caching**: Instant responses for similar queries. - **📄 Multi-Format Ingestion**: Supports PDF, DOCX, HTML, Markdown, and OCR for images. - **🎯 OpenAI-Compatible API**: Drop-in replacement for the OpenAI API. ### Tech stack: Python ### Why it matters: This monorepo provides a robust and efficient RAG system that can handle various document formats and supports both local and cloud-based LLM inference, making it highly versatile and scalable.

Overview: This project fine-tunes the Segment Anything Model (SAM) using Huggingface's PEFT and Trainer, leveraging techniques like QLoRA, DoRA, and SDPA for efficient training and inference. It supports the COCO dataset format and works with SAM models compatible with the Transformers library. Key features: - PEFT (Parameter-Efficient Fine-Tuning) - Bitsandbytes for quantization - COCO dataset format support - SDPA (Scaled Dot-Product Attention) for faster training and inference Tech stack: Python Why it matters: This project showcases the effectiveness of PEFT techniques for fine-tuning SAM, demonstrating improved performance with fewer parameters and faster training times. The use of SDPA for training and inference speeds up the process, making it suitable for larger models like SAM-H.

### Overview: This project is a hardware-focused fork of the llama.cpp repository, designed for local AI development on Windows with dual AMD RDNA4 GPUs. It includes a local web GUI (RDNA LLM Studio), reproducible benchmark tooling, long-context work, and AMD-specific optimizations. The primary workload focuses on Qwen3.8-27B, with a primary performance priority on prompt evaluation. The repository is maintained when necessary and may make smaller fixes or improvements, but larger optimization and research work will resume when resources allow. ### Key features: - **RDNA LLM Studio GUI**: A branded desktop application for local AI development. - **ROCm/HIP and Vulkan backends**: Optimized for AMD GPUs. - **MTP (Multi-Threaded Pre-fetching)**: Improves decode speed without sacrificing prompt evaluation. - **FP8 Attention**: Enhanced attention mechanism for better performance. - **Long-Context Work**: Supports long-prompt evaluations. - **GPU-Specific Optimizations**: Tailored for RDNA4 GPUs. ### Tech stack: - **Primary Language**: C++ ### Why it matters: This fork is designed to maximize the performance and efficiency of Qwen3.8-27B for agent workloads, ensuring prompt evaluation without sacrificing useful decode speed. The combination of ROCm/HIP and Vulkan backends, along with AMD-specific optimizations, makes it suitable for both local development and production use.

Overview: This project is an AI agent infrastructure landscape tool, offering a searchable catalog of 912 systems across 68 columns covering memory layers, agent frameworks, runtimes, vector stores, knowledge graphs, MCP servers, and benchmarks. It provides a comparative analysis of AI agent stacks with a citation behind every claim. Key features: - Build a stack: Describe the product; get one cited pick per layer (framework, model, gateway, memory, vector/graph store, tools and auth, sandbox, guardrails, observability), the runners-up, and why not, integrations, risk, OWASP coverage, cost, and a `stack.yml`. - Choose one layer: Memory, framework, or data store. - Check a stack you run: Archived, acquired, or stale dependencies, with replacements that fit the rest of your stack. Tech stack: HTML. Why it matters: This tool helps developers and system architects quickly identify and evaluate the best AI agent infrastructure components for their projects, ensuring a well-rounded and secure setup. Repository: MrPeppersDev/agent-infrastructure-landscape GitHub stars: 7 GitHub description: AI agent memory & infrastructure landscape — comparative catalog of 912 systems × 68 columns covering memory layers, agent frameworks, runtimes, vector stores, knowledge graphs, MCP servers, benchmarks. Searchable with typed edges, lineages, citations.

Overview: This repository contains a benchmark suite for evaluating the effectiveness of the Triad Engine in eliminating language model hallucinations across various models, including Claude 4.6, GPT-5.2, Mistral 7B, and Gemini 2.5 Pro. The benchmark includes 222 adversarial QA pairs from Ancient Rome, 110 CE, and demonstrates a 95-100% accuracy rate across all categories, with no regressions. Key features: - The Triad Engine eliminates LLM hallucinations at inference time without the need for fine-tuning. - It shows significant improvements in accuracy for multiple models across various categories. - The benchmark includes results from two independent judges, demonstrating consistent performance across different models and providers. Tech stack: The Triad Engine is a model-agnostic inference layer that grounds any base LLM in a domain guide, which is a structured system prompt that includes cultural grounding and domain-specific constraints. Why it matters: The benchmark is crucial for evaluating the effectiveness of the Triad Engine in eliminating language model hallucinations, which is essential for ensuring the reliability and consistency of language models across different domains and providers.

Overview: 这个项目旨在将中文场景描述转化为用于ANIMA模型的自然语言提示词，以提高模型的生成质量。 Key features: - 自然语言提示词生成 - 适用于多种LLM模型 - 包含正向7个槽位规则、负面三层结构、互斥表与输出协议 Tech stack: 使用Python和相关库实现 Why it matters: 通过这种方式，可以显著改善ANIMA模型的生成质量，提高最终图片的自然度和一致性。

Overview: This project is an LLM-Powered astrologer built by NishantkSingh0 under the task by the Vedaz organization. It predicts future plans and provides essential astrological details in a conversational style. Key features: - Predicts future plans - Provides essential astrological details - Sliding memory algorithm for natural conversation Tech stack: Python 3.10+, Flask, Flask-CORS, Groq API, python-dotenv, HTML + CSS + JS Why it matters: This project offers an interesting blend of language understanding and predictive-style responses, providing a fun and experimentally useful tool for exploring astrology.

Overview: This GitHub Action allows developers to request AI inference requests directly from their GitHub repositories, utilizing the latest models available. Key features: - Integrates with any GitHub repository - Enables seamless model selection and inference - Supports multiple AI models Tech stack: GitHub Actions Why it matters: This action simplifies the process of integrating AI capabilities into existing projects, making it easier for developers to leverage cutting-edge AI technologies without requiring extensive setup or knowledge of AI models.

### Overview: MatchAI is a job matching system designed to rank positions based on CV analysis, offering local development and cloud deployment options. It utilizes Groq's fast LLM API, semantic search, and deterministic filters for efficient and explainable job matching. ### Key Features: - **Fast Inference**: Uses Groq's optimized LLM API for quick CV parsing and explanations. - **Lightweight Embeddings**: Utilizes fastembed (ONNX Runtime) - 75% smaller than PyTorch. - **Smart Matching**: Combines semantic similarity with deterministic filters (skills, seniority, location). - **Explainable**: Generates human-readable explanations for each match. - **Interview Tips**: LLM-generated preparation suggestions based on skill gaps. - **Email Notifications**: Sends match results via email after each scheduled run. - **Cloud-Native**: Runs as scheduled Cloud Run Job with Supabase and Pinecone. - **Local Development**: SQLite + ChromaDB fallback for local testing. ### Tech Stack: MatchAI leverages Python, Groq API, fastembed (ONNX Runtime), ChromaDB, and Supabase for cloud deployment. It uses SQLite and fastembed for local development, and Pinecone and Supabase for cloud-based job matching. ### Why It Matters: MatchAI provides a robust solution for job seekers and recruiters, offering a fast, efficient, and explainable way to find relevant job positions based on a candidate's CV. Its cloud-native architecture allows for easy scaling and deployment, while its local development setup ensures quick feedback and testing.

Overview: This project involves training a model to predict characters from synthetic Captchas using a combination of ResNet and Transformer architectures, specifically designed for the Kaggle competition "Captcha Competition" in the Machine Learning II course. Key features: - Use of ResNet and Transformer for character prediction - Synthetic dataset for training - Kaggle competition for accuracy evaluation - Two phases: Experimentation and Competition Tech stack: PyTorch Why it matters: This project showcases advanced machine learning techniques for solving real-world problems, demonstrating the effectiveness of combining convolutional and transformer-based models for character recognition tasks.