Projects

Overview: Instruction tuning (IT) involves training large language models (LLMs) on datasets consisting of `(instruction, output)` pairs, bridging the gap between LLMs' next-word prediction objective and human instruction adherence. This systematic review covers the general methodology, construction of IT datasets, training of IT models, and applications across different modalities, domains, and applications. It also analyzes factors influencing IT outcomes and critiques existing strategies, suggesting avenues for future research. Key features: - Datasets - Models Tech stack: Python, Markdown, LaTeX Why it matters: This repository provides essential resources for understanding and evaluating instruction tuning, contributing to the field of large language model optimization and evaluation.

### Overview: This project evaluates the adversarial robustness of large vision-language (generative) models (VLMs), focusing on the threat model of black-box access and a targeted goal. The method aims to mislead VLMs into saying what you want, regardless of the input image query. The project uses Stable Diffusion for target image generation and is based on recent large VLMs such as MiniGPT-4, LLaVA, Unidiffuser, BLIP/2, and Img2Prompt. ### Key features: - Targeted Image Generation: Leverages a pretrained text-to-image model to generate targeted images for adversarial attacks. - Flexible Attack: Utilizes Stable Diffusion for demonstration and processing of targeted captions from the MS-COCO dataset. - Robustness Evaluation: Measures the adversarial robustness of VLMs under realistic and challenging threat models. ### Tech stack: Python ### Why it matters: This research is significant as it evaluates the robustness of large VLMs against targeted adversarial attacks, providing valuable insights for the development of safer and more robust models.

Overview: This project is a native compiled version of pgvector for PostgreSQL running on Microsoft Windows, providing a vectorized extension for PostgreSQL. Key features: - Native compiled PostgreSQL extension for vectorized operations - Compatible with PostgreSQL on Windows - Supports vectorized queries and operations Tech stack: Microsoft Windows x64 Why it matters: This project offers a faster and more efficient way to perform vectorized operations in PostgreSQL, making it ideal for applications requiring high performance in data analysis and machine learning tasks.

Overview: This is a lightweight Windows application that provides a local inference service for Qwen3.6-27B, running on a single RTX 3090 GPU. It offers a text-based user interface for managing and serving Qwen snapshots, with a focus on performance and privacy. Key features: - A text-based user interface for managing Qwen snapshots. - Configurable snapshots with performance metrics. - Native, no WSL, no Docker, no telemetry. - Supports various OpenAI-compatible clients and models. Tech stack: Python. Why it matters: This application simplifies the deployment of Qwen3.6-27B on Windows, ensuring privacy and performance without the need for additional tools or configurations. Repository: devnen/qwen3.6-windows-server Primary language: Python GitHub stars: 231 GitHub description: One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

### Overview: Memoket-kite is an open-source, vector-free long-term memory engine designed for AI agents. It achieves state-of-the-art results on LoCoMo and LongMemEval with significantly reduced context, specifically 93.51% on LoCoMo and 85.60% on LongMemEval-S using gpt-4.1-mini. ### Key features: - **Vector-Free Memory**: Unlike traditional memory engines that rely on vectors, Memoket-kite uses a vector-free approach, focusing on the content and context of the questions and answers. - **Topic-Based Memory**: Questions are mapped to specific topics, and the engine retrieves relevant facts from those topics. - **Reduced Context**: The engine's average reader context is only ~1.6K tokens per question, making it highly efficient and context-sensitive. ### Tech Stack: Python ### Why it Matters: Memoket-kite stands out by providing a memory engine that focuses on the actual meaning of questions and answers rather than just the similarity of words. This results in more accurate and contextually relevant responses, making it particularly useful for AI agents that need to handle complex, real-world scenarios. ### Repository: memoket/memoket-kite ### GitHub Stars: 229 ### GitHub Description: An open-source, vector-free long-term memory engine for AI agents, achieving SOTA on LoCoMo and LongMemEval with significantly less context.

Overview: This repository serves as a comprehensive resource for developers and machine learning practitioners, providing examples and use cases on how to effectively integrate LangChain and Ray libraries for building LLM-based applications. Key features: - Open Source LLM Search Engine - Fast and Scalable Embedding Generation - Retrival-Based Question Answering System Tech stack: Python Why it matters: This repository offers practical examples and tutorials on how to leverage LangChain and Ray to enhance the development and deployment of language models, making it easier and more efficient to create applications that utilize these models.

Overview: Traceloop Hub is a high-performance LLM gateway written in Rust, providing a unified OpenAI-compatible API for connecting to multiple LLM providers with built-in observability. Key features: - Multi-Provider Support: Supports OpenAI, Anthropic, Azure OpenAI, Google VertexAI, AWS Bedrock - OpenAI Compatible API: Drop-in replacement for OpenAI API calls - Two Deployment Modes: YAML Mode (static config) and Database Mode (dynamic config) - Built-in Observability: OpenTelemetry tracing and Prometheus metrics - High Performance: Written in Rust with async/await support - Hot Reload: Dynamic configuration updates (Database mode) - Pipeline System: Extensible request/response processing - Unified Architecture: Single crate structure with integrated Management API Tech stack: Rust Why it matters: Traceloop Hub simplifies the integration of large language models (LLMs) by providing a unified API and robust observability, making it a valuable tool for developers and organizations looking to leverage multiple LLM providers in a high-performance environment.

Overview: The Deep Search Toolkit is a Python SDK and CLI designed for interacting with the Deep Search platform, enabling users to explore and discover new knowledge. Key features: - Conversion of documents - Graph creation - Querying with semantic retrieval and RAG (Refine and Generate) capabilities Tech stack: Python Why it matters: The toolkit simplifies access to the Deep Search platform, facilitating efficient knowledge exploration and discovery through its comprehensive set of features.

### Overview: This repository is a curated list of resources related to machine-generated text, including large-scale pre-training, language analysis, and detection techniques. It aims to provide a comprehensive overview of the field, covering topics such as pre-training, analysis methods, detection algorithms, and practical applications. ### Key Features: - **Large Scale Pre-training for Language Generation:** Resources on how to train language models effectively. - **Analysis** - **Comprehensive:** Resources that cover various aspects of language generation. - **Hallucination \& Disinformation:** Techniques to identify and mitigate issues in generated text. - **Bias \& Toxicity:** Methods to ensure fairness and remove biases in language models. - **Security Risk:** Measures to protect against potential security threats. - **LM Attack:** Strategies to defend against language model attacks. - **Environment:** Guidelines for setting up and maintaining a safe environment for language models. - **Detection** - **Papers:** A collection of research papers on the topic. - **Survey:** A comprehensive survey of the field. - **Human Detection:** Techniques for detecting human-generated text. - **Automatic Detection:** Algorithms for detecting language models. - **Detector Attack:** Methods to defend against detector attacks. - **Benchmark:** A benchmark for evaluating language model detection systems. - **Watermarking:** Techniques to watermark language models for security. - **Other Related Work

### Overview: The **AI** framework is a comprehensive, open-source library designed to facilitate seamless integration with generative AI models. This framework is built on Swift and aims to provide developers with a robust and efficient way to interact with various AI providers, including OpenAI, Anthropic, Mistral, Groq, and ElevenLabs. ### Key Features: - **Swift Integration**: The framework is entirely written in Swift, ensuring compatibility and ease of integration with existing Swift projects. - **Open-Source**: The framework is released under an open-source license, allowing developers to freely use, modify, and distribute it. - **Cross-Platform Support**: The framework supports multiple platforms including macOS, iOS, iPadOS, watchOS, and tvOS, making it versatile for a wide range of applications. - **LLM Clients Abstraction**: The framework abstracts the client logic, allowing developers to easily switch between different LLM providers without changing the client code. - **Model Support**: The framework supports a variety of models, including LLM clients, Vision models (for image-to-text), and audio models. - **Completions**: Supports basic completions, vision models for image-to-text, and audio models for transcription and generation. ### Tech Stack: Swift ### Why It Matters: The **AI** framework is crucial for developers looking to integrate generative AI into their Swift projects. It simplifies the process of using multiple AI providers, making it easier to switch between different models and providers without the need for extensive code changes. This framework is essential for anyone working on applications that require

### Overview: STEVE-1 is a generative model designed for text-to-behavior tasks in Minecraft, utilizing a methodology inspired by unCLIP. This model, called STEVE-1, can follow short-horizon open-ended text and visual instructions in Minecraft. It is trained through two steps: adapting a pretrained Video Pretraining (VPT) model to follow commands in MineCLIP's latent space, and then training a prior to predict latent codes from text. The model is trained using only $60 of compute, making it highly accessible and cost-effective. ### Key Features: - **Text-to-Behavior Model**: STEVE-1 is specifically designed to respond to text instructions in Minecraft, focusing on sequential decision-making tasks. - **Pretrained Model Adaptation**: The model is adapted to follow commands in the MineCLIP latent space, enabling it to understand and execute text-based instructions. - **Latent Space Prediction**: A prior is trained to predict latent codes from text, allowing for fine-tuning through self-supervised behavioral cloning and hindsight relabeling. ### Tech Stack: - **Python**: The primary programming language used for developing and training the model. - **PyTorch**: Used for implementing the generative model and training scripts. - **MineDojo**: Provides the MineCLIP environment for training the model, including the necessary environment setup. - **MineRL**: The environment used for training the model, with specific requirements for MineRL setup. - **VPT**: A pretrained Video Pretraining model used as a starting point for adaptation

Overview: This repository contains a self-contained Text to Image Latent Diffusion model built using a Transformer core in PyTorch. The model is designed to be fast, small, and reasonably performant for training on a single GPU in a reasonable time frame. Key features: - Fast generation speed - Small model size (~100 million parameters) - Reasonably good performance - Trainable in a reasonable time on a single GPU (under 50 hours on an A100) - Simple, self-contained codebase - Uses 1 million images with high-quality data - Code provided for downloading and processing the data Tech stack: PyTorch Why it matters: This model offers a practical and accessible solution for generating images from text inputs, making it useful for applications requiring real-time or near-real-time image generation, such as in web applications or AI-driven content creation tools. Repository: apapiu/transformer_latent_diffusion GitHub stars: 230

### Overview: This repository provides a sample code for using Retrieval-Augmented Generation (RAG) with LangChain, leveraging Amazon Bedrock Titan text embeddings and Amazon OpenSearch vector engine. It demonstrates how to integrate Amazon Bedrock Titan LLMs with LangChain for more accurate responses. ### Key features: - **Amazon Bedrock Titan**: Utilizes Amazon Bedrock Titan text embeddings for text retrieval. - **Amazon OpenSearch**: Stores and retrieves text embeddings for prompt engineering tasks. - **LangChain**: Integrates with Amazon Bedrock Titan to enhance LLM responses. ### Tech stack: - Python ### Why it matters: This sample code showcases how to leverage Amazon Bedrock Titan text embeddings and Amazon OpenSearch vector engine to improve the accuracy of LLM responses, making it easier to create more contextually relevant prompts.

### Overview: Ellora is a collection of standardized, high-quality LoRA recipes designed to enhance Large Language Model capabilities without the need for new frameworks. It focuses on creating reproducible training methodologies that work with existing infrastructure, aiming for a quality-first approach. ### Key Features: - 📋 **Recipes, not frameworks**: Reproducible training methodologies. - 🎯 **Quality-first approach**: Rigorous evaluation and benchmarking. - 🔄 **Self-supervised data generation**: No dependency on external datasets. - 🏗️ **Infrastructure agnostic**: Works with existing tools (PEFT, LoRAX, etc.). - 🌍 **Community-driven**: Open recipes for the ecosystem. ### Tech Stack: Jupyter Notebook ### Why It Matters: Ellora simplifies the process of enhancing LLMs by providing standardized, high-quality LoRA recipes that can be easily integrated with existing infrastructure, ensuring a quality-first approach without the need for new frameworks.

Overview: This project introduces a novel architecture combining Mamba/SSMs with Transformers to enhance long context and high-quality sequence modeling. Key features: - Integrates Mamba/SSMs with Transformers for improved performance. - Enhances processing speed, context length, and perplexity. - Boosts reasoning ability without compromising size and compactness. Tech stack: Python Why it matters: This architecture offers significant advancements in sequence modeling, surpassing previous limitations, making it a valuable addition to the field of natural language processing. Repository: kyegomez/MambaTransformer GitHub stars: 230 GitHub description: Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling

Overview: Kaleido-BERT is a vision-language pre-training model designed for fashion cross-modality representations, introducing a novel kaleido strategy to enhance semantic relations between images and text. Key features: - Novel kaleido strategy for fashion cross-modality representations - Alignment guided masking for self-supervised pre-training - Five novel tasks: rotation, jigsaw, camouflage, grey-to-color, and blank-to-color - State-of-the-art results on four downstream tasks: text retrieval, image retrieval, category recognition, and fashion captioning Tech stack: Python Why it matters: Kaleido-BERT achieves significant improvements in performance on various downstream tasks, demonstrating its effectiveness in real-world applications, particularly in e-commerce websites.

Overview: NPi is an open-source platform providing tool-use APIs to empower AI agents with the ability to operate and interact with a diverse array of software tools and applications. Key features: - Tool-use APIs - Command-line tool for quick experimentation - High documentation and examples Tech stack: Python Why it matters: NPi is an open-source platform that aims to enhance the capabilities of AI agents by providing them with the ability to interact with various software tools and applications through a natural language interface. This can be particularly useful in fields such as automation, where integrating AI with existing systems can lead to more efficient workflows. Repository: sheet0/npi Primary language: Python GitHub stars: 230 GitHub description: Action library for AI Agent README excerpt: NPi is an open-source platform providing tool-use APIs to empower AI agents with the ability to operate and interact with a diverse array of software tools and applications!

### Overview: GraphRag.Net 是一个基于微软论文的.NET版本实现，旨在使用Semantic Kernel来快速构建图谱并生成社区摘要。该项目通过C#实现，可以轻松集成到现有的.NET项目中。 ### Key features: - **图谱构建**：将源文档分割成文本块，从每个文本块中提取图节点和边的实例。 - **社区检测**：通过社区检测算法将图划分为社区。 - **摘要生成**：为每个社区生成摘要，最终生成全局答案。 - **社区算法查询**：通过社区摘要和全局摘要综合分析生成答案。 ### Tech stack: C# ### Why it matters: GraphRag.Net 通过使用Semantic Kernel和社区检测算法，能够快速构建图谱并生成社区摘要，适用于需要快速信息检索和分析的场景。其简化的流程和易用的.NET实现方式使其成为快速学习和应用的绝佳选择。

Overview: This project provides a complete Python script to rebuild the Stable Diffusion Model, including training on classic datasets like MNIST and CelebA. Key features: - Rebuild the Stable Diffusion Model in a single Python script - Train a toy version of Stable Diffusion on classic datasets - Build your own Stable Diffusion UNet model from scratch Tech stack: Python Why it matters: This project offers a self-contained solution for those interested in learning about Stable Diffusion models and understanding their architecture, making it a valuable resource for both academic and practical applications in the field of machine learning.

### Overview: This repository is a collection of RAG systems powered by large language models (LLMs). It includes various frameworks and components designed to enhance the performance and utility of LLM-based applications. ### Key features: - **LLM-Powered-RAG-System**: A comprehensive framework for building RAG-based LLM applications. - **Chat with Documents**: Facilitates interaction with documents using LLMs. - **Database**: Enables querying and retrieving information from databases. - **Optimize Method**: Provides methods for optimizing RAG systems. - **Others**: Includes additional features and utilities not detailed in the README. ### Tech stack: The repository utilizes LangChain, LlamaIndex, GraphRAG, Embedchain, FlagEmbedding, FastRAG, LLMware, LLM-Applications, DB-GPT, pandas-ai, and canopy. ### Why it matters: This collection of tools and frameworks is essential for developers looking to build robust and efficient LLM-based applications, particularly those requiring powerful retrieval and generation capabilities.