Projects

Overview: 4DGen is a project that focuses on 4D content generation, specifically designed for video-to-4D and image-to-4D tasks. It leverages spatial-temporal consistency to generate 4D content from videos or images, addressing the challenges of text-to-4D and image-to-4D generation. Key features: - Supports text-to-4D and image-to-4D tasks. - Utilizes stable diffusion-XL and stable video diffusion for text-to-image-to-video-to-4D generation. - Implements image-to-video-to-4D and video-to-4D generation using video diffusion models. - Provides a complete pipeline script for reference. Tech stack: Python Why it matters: 4DGen offers a novel approach to 4D content generation, particularly useful for applications requiring spatial-temporal consistency in video and image-based 4D content creation. Repository: VITA-Group/4DGen Primary language: Python GitHub stars: 251 GitHub description: "4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency", Yuyang Yin*, Dejia Xu*, Zhangyang Wang, Yao Zhao, Yunchao Wei README excerpt: "4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency"

### Overview: This benchmark evaluates large language models (LLMs) based on how frequently they produce non-existent answers (confabulations or hallucinations) in response to misleading questions that are based on provided text documents. These documents are recent articles not yet included in the LLM training data. Minimizing these types of confabulation is crucial when using Retrieval-Augmented Generation (RAG). ### Key features: - Confabulation rate - Non-response rate ### Tech stack: HTML ### Why it matters: Minimizing confabulation is crucial when using Retrieval-Augmented Generation (RAG) models.

Overview: LLMTuner is a Python-based library designed to fine-tune large language models like Whisper and Llama using best practices such as LoRA and QLoRA. It simplifies the process by providing a scikit-learn-inspired interface, making it easy to fine-tune models with minimal code. Key features: - Effortless Fine-Tuning: Fine-tune state-of-the-art LLMs like Whisper, Llama with minimal code. - Built-in utilities for techniques like LoRA and QLoRA. - Interactive UI for launching webapp demos with one click. - Simplified Inference: Fast inference without separate code. - Deployment Readiness: (Coming Soon) Deploy models to AWS, GCP, etc., ready to share. Tech stack: Python Why it matters: LLMTuner streamlines the process of fine-tuning large language models, making it accessible and efficient for developers and researchers. Repository: promptslab/LLMtuner GitHub stars: 251 GitHub description: FineTune LLMs in few lines of code (Text2Text, Text2Speech, Speech2Text) License: Apache 2.0 Community: Join the PromptsLab Discord for discussions on fine-tuning LLMs, building scalable models, and prompt-engineering.

### Overview: This project is a web-based interface designed to facilitate the learning and exploration of large language models. It leverages Langchain and Fastchat frameworks, integrating Streamlit for the frontend, aiming for offline deployment and testing of various open-source models from Hugging Face. ### Key features: - **Chat Interface**: Enables real-time conversations with the model. - **Model Quantization**: Simplifies the model for easier deployment. - **Fine-tuning**: Allows for customizing the model for specific tasks. - **Role Play**: Facilitates role-based interactions for practice. - **Multimodal Support**: Combines text, images, and more for enhanced functionality. - **Code Interpreter**: Integrates with programming languages for code execution. - **Speech Recognition**: Converts spoken commands into text. - **Image Recognition**: Identifies objects and scenes in images. - **Network Search Engine**: Facilitates web searches. - **Function Calling**: Enables calling external functions. - **ToolBoxes**: Provides a collection of tools for various tasks. ### Tech stack: Python The project is built using Python, leveraging open-source libraries such as Langchain, Fastchat, and Streamlit. ### Why it matters: This project serves as an educational tool for developers and enthusiasts to understand and experiment with large language models, making it easier to integrate these models into various applications and projects.

Overview: GALIP is a fast, efficient, and high-quality text-to-image model designed for large-scale training. It achieves impressive results on COCO and CC12M datasets, demonstrating superior performance compared to the original paper version. Key features: - Fast generation on CPU - High-quality images - Efficient training and sampling Tech stack: Python 3.9, Pytorch 1.9, CUDA 11.7 Why it matters: GALIP provides a practical solution for generating high-quality images from text inputs, making it highly relevant for applications in AI art, generative design, and other fields where image synthesis is crucial.

Overview: This project aims to create a library for Godot that utilizes Large Language Models (LLM) to enhance game development, specifically for NPC models, game mechanics, and design assistance. Key features: - GDLlama node for text generation - GDEmbedding node for text embedding - GDLlava node for multimodal text generation - LlmDB node for vector database - LlmDBMetaData node for metadata management Tech stack: C++ Why it matters: This project provides a practical solution for integrating LLMs into Godot games, addressing legal concerns related to integrating LLM-generated content into games. GitHub stars: 250 Repository: Adriankhl/godot-llm

Overview: This repository contains curated Seedance 2.5 prompts with real video previews and a Seedance 2.5 Skill that optimizes prompts, creates storyboards, and generates videos via Seedance models. The Skill is an installable Agent Skill that enhances the overall experience. Key features: - Curated Seedance 2.5 prompts with real video previews - Seedance 2.5 Skill for prompts, storyboards, and video generation - Optimized prompts, storyboard creation, and video generation using Seedance 2.5 models - Supports Seedream 5.0 Pro for storyboard images and Seedance 2.0 as the current executable video default, switching to Seedance 2.5 when available Tech stack: TypeScript Why it matters: This library provides a comprehensive solution for creating engaging video content using Seedance models, making it easier and more efficient for developers to create and optimize prompts, storyboards, and videos.

Overview: This project is a library designed for real-time voice processing in web browsers, utilizing the Web Audio API and Web Workers to ensure compatibility across modern browsers, including mobile devices. Key features: - Uses the Web Audio API to access microphone audio. - Leverages Web Workers to offload compute-intensive tasks. - Converts microphone sampling rate to 16kHz, the de facto standard for voice processing engines. - Provides a flexible interface for passing in arbitrary voice processing workers. Tech stack: TypeScript Why it matters: This library simplifies the process of integrating voice processing capabilities into web applications, making it easier for developers to implement real-time voice recognition and processing without the need for complex browser-specific code. Repository: Picovoice/web-voice-processor GitHub stars: 250 GitHub description: A library for real-time voice processing in web browsers

### Overview: Trex is a powerful tool designed to enforce structured output from Large Language Models (LLMs) 100% of the time. It intelligently restructures unstructured data into predefined schemas, ensuring compliance and consistency. ### Key Features: - **Structured JSON Generation**: Converts unstructured data into structured JSON format. - **Custom Context-Free Grammar (CFG) Generation**: Allows for generating structured data based on custom context-free grammars. - **Custom Regular Expressions (Regex) Generation**: Enables the creation of structured data from custom regular expressions. ### Tech Stack: - Python ### Why It Matters: Trex simplifies the process of ensuring structured data is generated from LLMs, making it easier to integrate structured data into applications and systems. This tool is particularly useful in scenarios where data consistency and compliance are critical. ### Repository: Repository: [automorphic-ai/trex](https://github.com/automorphic-ai/trex) GitHub Stars: 250 GitHub Description: Enforce structured output from LLMs 100% of the time

### Overview LLM7.io is a single API gateway that connects developers to a wide array of leading AI models from various providers. It offers a public model catalogue, comparison tools, and sitemaps to help developers easily integrate these models into their projects. ### Key Features - **Public Model Catalogue Refresh**: Model, comparison, and sitemap assets are generated during `npm run build` from the public API snapshot. - **Automatic Cloudflare Deployment**: The Cloudflare Pages workflow builds and publishes `llm7-landing` on every push to `main`. - **Live Model Pricing**: Model pages, catalogue, comparisons, alternatives, and calculators read current prices from `https://api.llm7.io/v1/models` in the browser. ### Tech Stack - TypeScript ### Why It Matters LLM7.io simplifies the process of integrating leading AI models into projects, making it easier for developers to leverage the capabilities of various providers without needing to set up their own infrastructure. The automatic deployment and live pricing features ensure that developers can quickly access the latest models and pricing information, enhancing their development workflows.

### Overview: LLM Wiki v2 is a powerful tool that turns your documents into a Markdown wiki, answering your questions with citations. It supports any LLM provider, including OpenAI, and is optimized for fast and efficient operation. ### Key features: - **Document Integration**: Automatically files documents into the wiki, merging them with existing notes. - **Question Answering**: Answers questions directly from the notes, citing the sources. - **CROW Mode**: Uses a small classifier to decide where things go, making the wiki fast and cost-effective. - **Web Interface**: Available as a desktop app (Crow) or through the command line (uv), with Python 3.11 or later. - **Easy Installation**: Simple installation via pip or uv for Python 3.11 or later. - **Demo**: An example wiki with a fictional coffee roastery, ready to be used or modified. ### Tech stack: Python ### Why it matters: - **Efficiency**: The system is optimized for fast and efficient operation, making it suitable for large-scale use. - **Scalability**: It can handle any LLM provider, ensuring compatibility with various models. - **Cost-Effectiveness**: The classifier-based approach keeps the wiki fast and cost-effective as it grows. - **User-Friendly**: The web interface makes it easy to use, and the command line allows for flexibility. - **Community Support**: The project has a dedicated community, offering resources and support. ### GitHub stars: 247 ### Repository: crow-hq/llm-wiki-v

### Overview: This project explores the idea of instruction-tuning Stable Diffusion to enhance its capabilities in following specific image editing instructions. By creating an instruction-promoted dataset and conducting InstructPix2Pix style training, the model is made better at processing images according to user instructions. ### Key features: - Instruction-tuned Stable Diffusion - Dataset preparation for instruction-promoted training - InstructPix2Pix style training - Memory-efficient training with xformers ### Tech stack: Python ### Why it matters: This project aims to improve the ability of Stable Diffusion to perform image transformations based on specific instructions, making it more versatile and useful for various image editing tasks.

Overview: This project implements an end-to-end speech recognition model using RNN Transducers in TensorFlow 2.0. Key features: - End-to-end speech recognition - RNN Transducers model - Tensorflow 2.0 framework Tech stack: Python Why it matters: This project showcases the implementation of advanced speech recognition technology using modern machine learning techniques, making it valuable for developers interested in natural language processing and speech recognition.

Overview: This repository contains comprehensive interview questions and study materials for AI and LLM engineers, covering 33 companies and 10 engineering roles. Key features: - 1,172 questions with worked answers - 10 system design case studies - 19 runnable coding challenges - Company-specific question sets - Role-based study maps Tech stack: Python Why it matters: This resource is designed for software engineers transitioning into AI and ML practitioners, providing a structured approach to preparing for various roles and companies. GitHub stars: 235

### Overview: This repository compiles a list of must-read papers on Large Language Models (LLM) as optimizers and automatic optimization for prompting LLMs. The papers cover various approaches to enhancing the performance and efficiency of LLMs through optimization and instruction-based methods. ### Key features: - **RLPrompt**: Optimizing Discrete Text Prompts with Reinforcement Learning - **Large Language Models Are Human-Level Prompt Engineers**: Demonstrating that large language models can effectively engineer prompts. - **GrIPS**: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models - **Automatic Prompt Optimization with "Gradient Descent" and Beam Search** - **InstructZero**: Efficient Instruction Optimization for Black-Box Large Language Models - **Retroformer**: Retrospective Large Language Agents with Policy Gradient Optimization - **Forget Demonstrations, Focus on Learning from Textual Instructions** - **Large Language Models as Optimizers** - **Promptbreeder**: Self-Referential Self-Improvement Via Prompt Evolution - **Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers** - **Self-Taught Optimizer (STOP)**: Recursively Self-Improving Code Generation - **DSPy**: Compiling Declarative Language Model Calls into Self-Improving Pipelines ### Tech stack: Python, JavaScript, and various optimization techniques. ### Why it matters: These papers highlight the potential of large language models as powerful optimization tools and provide practical strategies for improving prompt engineering and instruction-based optimization. The insights from these papers can significantly enhance the performance and

Overview: everything-ai is a fully proficient, AI-powered and local chatbot assistant designed to provide comprehensive support and assistance across various tasks. Key features: - Retrieval-text-generation: Use `qdrant` backend to build a retrieval-friendly knowledge base, allowing you to query and refine the output of your model. - Agnostic-text-generation: Offers a ChatGPT-like text generation capability, supporting every text-generation model on the Hugging Face Hub. - Text-summarization: Summarizes text and PDFs, supporting every text-summarization model on the Hugging Face Hub. - Image-generation: Utilizes stable diffusion to generate images, supporting every text-to-image model on the Hugging Face Hub. - Image-generation-pollinations: Uses Pollinations AI API for image generation, requiring no additional configuration. - Image-classification: Classifies images, supporting every image-classification model on the Hugging Face Hub. - Image-to-text: Describes an image, supporting every image-to-text model on the Hugging Face Hub. - Audio-classification: Classifies audio files or microphone inputs. Tech stack: everything-ai leverages Python for its development and deployment. Why it matters: everything-ai is designed to be a versatile, AI-powered chatbot assistant that can handle a wide range of tasks, from text generation to image and audio classification, making it a valuable tool for developers and professionals alike.

### Overview: Lore is a lightweight, local AI-powered memory assistant that allows you to capture and recall thoughts instantly. It uses a local LLM (Ollama) and vector database (LanceDB) to store and retrieve information, ensuring complete privacy and no reliance on cloud services. ### Key features: - **Quick Capture**: Press a global shortcut (Ctrl+Shift+Space on Windows, Cmd+Shift+Space on macOS) to pop up a chat bar and type a thought. - **Smart Recall**: Ask questions in natural language and get answers sourced from your stored thoughts. - **AI Classification**: Input is automatically classified as a thought, question, command, or instruction. - **Todo Management**: Add, list, complete, and organize todos with priority and categories. - **RAG Pipeline**: Retrieval-Augmented Generation finds relevant context from your notes before answering. - **Fully Local**: All data and AI processing stays on your machine. ### Installation: 1. **Download Installer**: Download the latest release from the [Releases](https://github.com/ErezShahaf/lore/releases) page. - **Windows**: `Lore-x.x.x-Setup.exe` - **macOS**: `Lore-x.x.x.dmg` - **Linux**: `Lore-x.x.x.AppImage` 2. **Install the Software**: Open the installer and follow the prompts to choose directories for your LLM models and Ollama. 3. **Choose Your Models**: After installation, open the settings by clicking the Lore icon in the system tray

Overview: This repository contains the code for "LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs". We are still in the process of polishing the repo. Key features: LongRAG consists of a "long retriever" and a "long reader". It uses a 4K-token retrieval unit, which is 30x longer than before, offering insights into the future roadmap for combining RAG with long-context LLMs. Tech stack: Python Why it matters: This framework aims to improve the performance of retrieval-augmented generation by using a more efficient retrieval and reader design. Repository: TIGER-AI-Lab/LongRAG GitHub stars: 249

### Overview: ABQ-LLM is a novel arbitrary bit quantization scheme designed to achieve excellent performance across various quantization settings while enabling efficient computation at the inference level. This library supports precise weight-only quantization (W8A16, W4A16, W3A16, W2A16) and weight-activation quantization (W8A8, W6A6, W4A4, W3A8, W3A6, W2A8, W2A6) for large language models (LLMs) such as LLaMA and LLaMA-2. ### Key Features: - Support for precise weight-only quantization (W8A16, W4A16, W3A16, W2A16) and weight-activation quantization (W8A8, W6A6, W4A4, W3A8, W3A6, W2A8, W2A6). - Pre-trained ABQ-LLM model weights for LLaMA and LLaMA-2. - A set of out-of-the-box arbitrary bit quantization operators that support arbitrary bit model inference in Turing and above architectures. ### Tech Stack: - C++ ### Why It Matters: ABQ-LLM provides a robust and efficient quantization solution for large language models, ensuring high performance and low computational requirements across different quantization settings. This makes it particularly valuable for applications where model size and inference speed are critical factors.

### Overview: Ahnlich is a suite of tools that includes an in-memory vector datastore and an AI proxy. It provides powerful capabilities for storing and searching similar vectors using linear and non-linear similarity algorithms, as well as filtering entries based on metadata values. The system is designed to be highly scalable and efficient, making it suitable for a wide range of applications. ### Key features: - **`ahnlich-db`**: An in-memory vector key-value store for storing embeddings/vectors with corresponding metadata. - **`ahnlich-ai`**: An AI proxy that communicates with `ahnlich-db`, allowing developers to issue queries using raw input. - **`ahnlich-client-rs`**: Rust client for interacting with `ahnlich-db` and `ahnlich-ai`. - **`ahnlich-client-py`**: Python client for `ahnlich-db` and `ahnlich-ai`. - **`ahnlich-client-go`**: Go client for `ahnlich-db` and `ahnlich-ai`. - **`ahnlich-client-node`**: Node.js/TypeScript client for `ahnlich-db` and `ahnlich-ai`. - **`ahnlich-cli`**: A CLI for querying `ahnlich-db` and `ahnlich-ai`. ### Tech stack: Rust ### Why it matters: Ahnlich is designed to be highly scalable and efficient, making it suitable for a wide range of applications. The system is built with a focus on performance and ease of use, ensuring that developers can quickly integrate it into their projects.