Projects

Overview: This repository is a curated collection of research on AI and foundation models across the game lifecycle, focusing on how AI is used in various roles within games. Key features: - Organizes survey literature into six roles - Includes a visual survey map - Provides a literature search and AI-crafted games section Tech stack: None specified Why it matters: This repository helps developers understand how AI is integrated into game development, from design to testing, and provides valuable insights into the application of AI in games.

### Overview: RISE is a self-improving robot policy framework designed to enhance the robustness and performance of robot manipulation tasks. It leverages compositional world models and scalable RL methods to improve real-world robot manipulation capabilities. ### Key Features: - **Compositional World Model**: A principled design that combines a controllable multi-view dynamics model with a progress value model, providing informative advantages for policy improvement. - **RL in Imagination**: A scalable self-improving framework that leverages imaginary rollouts to improve robot policies without the need for hardware or real-world interaction resets. - **Real-World Manipulation Gains**: Non-trivial performance improvements on challenging dexterous tasks, including +35% on dynamic brick sorting, +45% on backpack packing, and +35% on box closing. ### Tech Stack: Python ### Why It Matters: RISE offers a practical solution for improving robot manipulation tasks by combining advanced RL techniques with a robust world model, leading to significant performance enhancements in real-world applications.

Overview: Predikit is a Python library that bridges the gap between machine learning models and large language models (LLMs), enabling the integration of trained scikit-learn or XGBoost models into LLM-callable tools. It automatically generates JSON schemas, typed I/O, and zero boilerplate code, making it easy to create LLM-friendly models. Key features: - Auto-generated JSON schemas for any scikit-learn or XGBoost model - Typed I/O for consistent data handling - Zero boilerplate code for LLM integration - Integration with OpenAI and LangChain frameworks - Support for ensemble routing with 5 built-in strategies - Confidence handling with threshold checks and on-low-confidence logic - Model registries for easy model loading from MLflow and Snowflake - Async functionality for improved performance Tech stack: Python Why it matters: Predikit simplifies the process of integrating machine learning models into LLMs, making it easier to deploy models that can reason about complex inputs and outputs, and ensuring that models are validated and ready for use in an LLM environment.

### Overview: TrueMemory is an AI memory system designed to automatically capture and recall your conversations, providing a seamless and personalized experience. It eliminates the need for manual tagging or prompt engineering, ensuring your AI agent remembers everything you've discussed without any effort. ### Key Features: - **Automatic Memory Capture**: TrueMemory automatically captures memories from your conversations and injects the right ones into your next session. - **Personalized Learning**: The system grows with you, resolving contradictions and updating stale facts, continuously improving over time. - **100% Local**: One SQLite file on your machine, ensuring your memories never leave your device. No cloud storage or API keys required. - **Zero Cloud Dependency**: Works seamlessly without any external dependencies, ensuring privacy and security. ### Tech Stack: - Python ### Why It Matters: - **Signal in Noise**: Finds and filters out the most relevant information from your conversations. - **Continuous Learning**: Improves over time as you use it, resolving contradictions and updating facts. - **Seamless Integration**: Works without any manual intervention, making it perfect for real-world applications. - **Local Experience**: Ensures your memories are always accessible and never leave your device. ### GitHub Stars: 383 ### GitHub Description: The memory your AI should have had from the start. Automatic capture, automatic recall, 100% local. Works with Claude Code, Claude CLI, Cursor, Codex CLI, Gemini CLI. ### Tech Stack: - Python ### Why TrueMemory: - **Signal in Noise**: Finds and filters out the most relevant information

Overview: This repository contains a project that explores and demonstrates the pre-training of Chinese language models using PyTorch, with a focus on improving their performance on specific domains and tasks. Key features: - Pre-training of language models on target domains (DAPT) - Pre-training of language models on specific tasks (TAPT) - Combination of DAPT and TAPT for further improvement - Use of curated training data for additional pre-training (Curated-TAPT) - Minimal code cost for pre-training using PyTorch Tech stack: PyTorch Why it matters: This project provides a detailed analysis of the pre-training process for Chinese language models, including how to improve their performance on specific tasks and domains. It offers practical insights into using PyTorch for language model pre-training, making it valuable for developers looking to enhance the performance of their models.

Overview: This project is a Model Context Protocol (MCP) server designed to query Erick Wendel's contributions across various platforms, including talks, blog posts, and videos. It leverages natural language processing through tools like Claude, Cursor, or similar services. Key features: - Retrieves a paginated list of talks with filtering capabilities. - Fetches posts with filtering and pagination. - Retrieves videos with filtering and pagination. - Verifies if the API is alive and responding. Tech stack: TypeScript. Why it matters: This MCP server provides a convenient way to access Erick Wendel's contributions, making it easier for developers to find and utilize his content across different platforms. GitHub stars: 383. Repository: https://github.com/ErickWendel/erickwendel-contributions-mcp

### Overview: This repository is a curated list of awesome AI leaderboards, focusing on various development tools and evaluation organizations related to the AI domain. It serves as a valuable resource for researchers, developers, and enthusiasts interested in AI leaderboard operations and related technologies. ### Key Features: - Comprehensive model ranking: Includes model rankings across different categories such as text, code, image, video, math, agent, research, business, safety, medical, audio, embodied, 3D, time series. - Database ranking: Covers database-related leaderboard operations. - Dataset ranking: Lists datasets used in AI leaderboard evaluations. - Metric ranking: Ranks metrics commonly used in AI leaderboard evaluations. - Infrastructure ranking: Highlights leaderboard infrastructure and tools. - Paper ranking: Lists papers related to AI leaderboard operations. - Usage ranking: Tracks the usage of AI leaderboards and their impact on the field. - Company ranking: Profiles companies involved in AI leaderboard operations. ### Tech Stack: The repository is built using Markdown and GitHub Pages. ### Why It Matters: This curated list provides a comprehensive overview of AI leaderboard operations, helping researchers, developers, and enthusiasts stay updated on the latest practices and technologies in the field.

Overview: This repository provides a step-by-step guide to building a ChatGPT-like language model (LLM) using PyTorch. The project breaks down the architecture into key components and explains each one in detail. Key features: - Tokenization - Token Embeddings - Positional Embeddings - Self Attention Mechanism - Masked Multi-Head Attention - Feedforward Neural Networks - Residual Connections - Layer Normalization - Transformer Block - MiniGPT Tech stack: Jupyter Notebook Why it matters: This project is useful for developers looking to understand and implement advanced natural language processing (NLP) techniques, specifically those related to language models and neural networks.

### Overview: Bodo DataFrames is a high-performance, scalable data processing library that offers exceptional performance and ease of use, replacing Pandas with a single line of code change. It leverages MPI technology for true parallel execution and integrates with modern data infrastructure, including Apache Iceberg and Snowflake. ### Key Features: - **Drop-in Pandas Replacement**: Replace `import pandas as pd` with `import bodo.pandas as pd` for seamless integration. - **Intuitive APIs**: Simplify and scale AI workloads with Pandas-compatible APIs. - **Advanced Query Optimization**: Optimized query execution for faster data processing. - **Just-in-Time (JIT) Compilation**: Accelerate custom transformations and performance-critical functions with native support for Pandas, Numpy, and Scikit-learn. - **Streaming Execution**: Process larger-than-memory datasets with streaming execution. - **High Performance SQL Engine**: Integrate a native SQL engine for extra flexibility. - **Interoperable with Modern Data Infrastructure**: Support for Apache Iceberg and Snowflake, enabling smooth integration with existing ecosystems. ### Tech Stack: MPI-based high-performance computing (HPC) technology. ### Why It Matters: Bodo DataFrames offers exceptional performance and ease of use, making Python run much faster than it normally does. It scales and accelerates Pandas workloads with a single line of code change, eliminating runtime overheads and leveraging MPI technology for true parallel execution. This library seamlessly integrates with the Python ecosystem, making it a powerful tool for data processing and AI/ML workloads.

### Overview: VectorHub is a fast and easy-to-use platform for building AI websites or applications using embeddings. ### Key features: - Quickly and easily create AI-powered web applications. - Utilizes embeddings for generating conversational AI. - Supports LangchainJS and Chatbot-ui for advanced chatbot functionalities. - Deployed with Vercel for seamless hosting. ### Tech stack: TypeScript ### Why it matters: VectorHub simplifies the process of integrating AI into web applications, making it accessible and efficient for developers.

### Overview: Sophia Optimizer Sophia Optimizer is an all-new simple plug-in and play optimizer designed to significantly reduce model training costs by 50%. This optimization algorithm is 2x faster than Adam on large language models, making it an essential tool for accelerating the training process without compromising performance. ### Key Features: - **Efficient Optimization**: Sophia uses an inexpensive stochastic estimate of the diagonal of the Hessian as a pre-conditioner and a clipping mechanism to control the worst-case update size. - **50% Cost Reduction**: By cutting model training costs in half, Sophia can save millions of dollars in computational resources. - **Seamless Integration**: Sophia can be seamlessly integrated into existing training pipelines, making it easy to adopt without requiring special requirements on model architecture or computing infrastructure. - **Versatile Estimators**: Supports both Hutchinson and Gauss-Newton-Bartlett Hessian estimators for flexibility. ### Tech Stack: - **Python**: The core implementation is written in Python, leveraging the power of the language for efficient computation and readability. ### Why It Matters: - **Cost Reduction**: Sophia achieves the same validation pre-training loss with 50% fewer number of steps than Adam, making it a significant improvement in terms of both validation loss and computational efficiency. - **Performance**: By reducing the number of steps required for training, Sophia can significantly decrease the total compute time and wall-clock time, making it an ideal choice for large language models. - **Ease of Use**: Sophia is designed to be simple and easy to use, requiring no special requirements on the model architecture

Overview: This project is a local implementation of CLIP Guided Diffusion, allowing developers to experiment with the process without needing a Colab environment. Key features: - Uses OpenAI's unconditional ImageNet diffusion models for 256x256 and 512x512 resolutions. - Fine-tunes the 512x512 model from the original 512x512 class-conditional model. - Connects CLIP text prompts with images. - Supports both 256 and 512 model outputs. - Provides examples of generated images with enhanced quality using Real-ESRGAN. - Includes an environment setup guide for Ubuntu 20.04 with Anaconda and Nvidia RTX 3090 requirements. Tech stack: Python Why it matters: This project offers a practical way for developers to explore and experiment with CLIP Guided Diffusion, which combines CLIP for text-to-image prompts with diffusion models for image generation.

Overview: This repository contains the code used to build the training dataset, preprocess existing open-source music datasets, train the model, and run inference for the paper "LLark: A Multimodal Instruction-Following Language Model for Music" by Josh Gardner, Simon Durand, Daniel Stoller, and Rachel Bittner. Key features: - Training dataset creation - Preprocessing of music datasets - Model training - Inference functionality Tech stack: Jupyter Notebook Why it matters: This repository provides the necessary code for the research paper, allowing others to replicate the work and potentially build upon it.

Overview: Mini NanoGPT is a tool designed to simplify the process of training GPT models, making it accessible for beginners, researchers, and developers alike. Whether you're starting with deep learning, experimenting with large language models, or building AI-powered applications, Mini NanoGPT provides an intuitive interface for training models through a visual interface, flexible tokenization, efficient training, and real-time feedback. Key features: - Visual Interface: Simplifies command-line operations. - Bilingual UI: Supports English and Chinese. - One-click Operations: Data processing, training, and generation. - Flexible Tokenization: Supports character-level or custom tokenizers. - Efficient Training: Multi-process acceleration and distributed training. - Real-time Feedback: Live training progress and loss curves. - Parameter Visualization: Adjusts all training parameters in the UI. - Model Database: Manages and reuses training settings. - Checkpoint Support: Resumes training from any saved state. - SFT Training: Supports supervised fine-tuning for conversational AI models. - Alpaca Format: Standard dataset format support. - Chat Inference: Interactive conversation with fine-tuned models. Tech stack: Python, PyTorch, Gradio, Docker, and OpenAI. Why it matters: Mini NanoGPT offers a user-friendly approach to training GPT models, making it accessible for beginners, researchers, and developers. It simplifies the process of training models, supports multiple languages, and provides real-time feedback, making it an essential tool for anyone interested in large language models and AI development. Repository: ystemsrx/

Overview: Sonobarr is a music discovery tool designed for Lidarr power users, integrating with Last.fm and ListenBrainz to provide personalized recommendations. It leverages Lidarr's library and Last.fm's insights to surface artists you'll enjoy. Key features: - Deep Lidarr integration - Personal discovery hub - AI assistant - Artist requests workflow - Real-time UX - Role-based access - OIDC Single Sign-On - Per-user API keys - Hardened configuration - Update & schema self-healing - Docker-first deployment Tech stack: Python Why it matters: Sonobarr simplifies the music discovery process by combining Lidarr's library with Last.fm's insights, making it easier for users to find and enjoy music they'll actually like. Repository: Dodelidoo-Labs/sonobarr GitHub stars: 383 GitHub description: Music discovery tool that integrates with Lidarr and provides recommendations using Last.fm and ListenBrainz README excerpt: # Sonobarr > Music discovery for Lidarr power users, blending Last.fm insights, ListenBrainz playlists, and a modern web UI.

### Overview: Inst-Inpaint is a novel inpainting framework that enables the removal of objects from images using natural language instructions. This project addresses the challenge of removing unwanted objects from images by utilizing diffusion models and GANs, providing a more efficient and accurate method compared to traditional binary mask-based approaches. ### Key Features: - **Real Image Inpainting**: Inst-Inpaint can remove objects from real images, making it suitable for various applications. - **Textual Instructions**: Users can provide text prompts to specify which objects should be removed, making the process more intuitive and less error-prone. - **Dataset**: The GQA-Inpaint dataset is used to train and evaluate models, providing a comprehensive training ground for the inpainting task. ### Tech Stack: - Python ### Why It Matters: Inst-Inpaint offers a more efficient and accurate solution for image inpainting by using natural language instructions, making it particularly useful in applications where precise control over the removal process is required. This approach can significantly reduce the time and effort needed for manual masking, leading to improved user experience and better results in various image editing tasks.

Overview: HAL-9100 is an edge-based full-stack LLM platform built with Rust, designed for creating AI assistants that don't require internet access. It offers 75% cheaper and 23x faster assistants compared to traditional methods. Key features: - Code Interpreter: Runs Python code in a sandboxed environment autonomously. - Knowledge Retrieval: Retrieves external knowledge autonomously. - Function Calling: Defines and executes custom functions autonomously. - Actions: Executes requests to external APIs autonomously. - Files: Supports a range of file formats. - OpenAI compatible: Works with OpenAI (Assistants) SDK. Tech stack: Rust. Why it matters: HAL-9100 is a comprehensive solution for creating edge-based AI assistants that can perform tasks without internet access, offering significant cost and speed improvements. Repository: llm-edge/hal-9100 GitHub stars: 383 GitHub description: Edge full-stack LLM platform. Written in Rust. [Undergoing large refactor](https://github.com/llm-edge/hal-9100/tree/0.1). Primary language: Rust. GitHub description: Edge full-stack LLM platform. Written in Rust. [Undergoing large refactor](https://github.com/llm-edge/hal-9100/tree/0.1).

Overview: A browser-based tool that instantly converts PDFs to Markdown, ensuring secure and fast processing without any file uploads. Key features: - Browser-based conversion - Privacy-first approach - Supports headings, lists, tables, basic formatting - Instant results - Offline capable - Open source Tech stack: TypeScript Why it matters: This project is perfect for developers, writers, and anyone working with Markdown, offering a fast, accurate, and secure way to convert PDFs to Markdown.

Overview: AI LocalBase 是一个本地优先的AI知识库系统，旨在将本地文档接入向量检索与大模型对话流程。它支持多种文件格式，包括MD、TXT、PDF、XLSX、CSV，提供Web UI和模型接入功能。 Key features: - 支持多种本地文件格式上传与解析 - 实现文本自动切分与批量嵌入 - 动态召回候选结果 - 关键词覆盖增强重排 - 嵌入缓存与可选语义缓存 - 可选 Hybrid Search、Semantic Reranker、Query Rewrite、Context Compression - 低置信度场景二次扩召回 - 基于Qdrant的向量检索 - 检索增强问答功能 - 聊天记录持久化到本地SQLite数据库 - 配置持久化到本地JSON文件 - Docker Compose部署，支持本地单进程调试 Tech stack: Go + Gin, React + Vite + TypeScript, Qdrant, Ollama / OpenAI Compatible API, Docker Compose Why it matters: AI LocalBase 为个人或小团队提供了一个快速搭建本地知识库问答系统的解决方案，适用于自托管环境，支持多种文档格式和模型接入，简化了知识库的管理和扩展能力。

### Overview: LLaVA-Interactive is an all-in-one demo designed for image chat, segmentation, generation, and editing. It is an open-source project that aims to provide a comprehensive solution for various AI applications. ### Key Features: - **Image Chat**: Real-time text-based chat using images. - **Segmentation**: Interactive segmentation of images. - **Generation/Editing**: Editable image generation and manipulation. ### Tech Stack: Python ### Why it Matters: LLaVA-Interactive offers a versatile platform for developers and researchers to explore AI-driven image manipulation and interaction, making it a valuable tool for applications in the field of computer vision and AI. ### Repository: LLaVA-VL/LLaVA-Interactive-Demo - **Primary Language**: Python - **GitHub Stars**: 382 - **GitHub Description**: LLaVA-Interactive - **README Excerpt**: An All-in-One Demo for Image Chat, Segmentation and Generation/Editing. ### Install - **Requirements**: CUDA 11.7 or above, Python 3.10.8, PyTorch 2.0.1 - **Steps**: Clone the repository, create a conda environment, install dependencies, and run the demo. ### Run the Demo - **Command**: `./run_demo.sh` - **Output**: A workflow image showing the demo's functionality. ### Citation - **BibTeX**: ```bash @article{chen2023llava_interactive, author = {Chen,