Projects

### Overview: This repository contains the official implementation of the LLM-grounded Video Diffusion Models (LVD) paper presented at ICLR 2024. The codebase is based on the LMD repo and includes features for generating dynamic scene layouts from text prompts, which are then used to generate videos. The project aims to enhance the prompt understanding capabilities for text-to-video generation. ### Key Features: - **Dynamic Scene Layouts (DSLs):** The LLM takes a text prompt and generates dynamic scene layouts consisting of reasoning statements, captioned boxes, and background prompts. - **Layout Caches:** Pre-generated layout caches are available for benchmarking, allowing users to skip the layout generation step if they only need the layouts for benchmarking purposes. - **Prompt Understanding:** The LLM can perform chain-of-thought reasoning to generate more detailed and accurate layouts. - **OpenAI API Integration:** The project includes an example of using OpenAI's API for batch text-to-layout generation, demonstrating how to integrate the code with the LLM. ### Tech Stack: - Python ### Why It Matters: This project offers a significant improvement in the ability to generate text-to-video content by leveraging large language models to understand and generate dynamic scene layouts, leading to better prompt understanding capabilities.

### Overview: This project is a self-evolving multi-agent newsroom that turns your documents into a cross-linked markdown wiki. The agent writes pages, but the reviewer is not the same as the author. Every new document enriches the existing pages, creating a structured and persistent alternative to RAG. ### Key features: - Cross-referenced wiki - Local-first approach - Structured and persistent knowledge - Domain-agnostic framework ### Tech stack: Python ### Why it matters: This project challenges the traditional roles of writer and reviewer, providing a structured alternative to RAG. It leverages an AI agent to read and understand your collected documents first, then organizes them into a wiki with cross-references, automatic flagging of conflicting claims, and per-topic synthesis built in from the start. ### Repository: alfadur7/llm-wiki-newsroom - GitHub stars: 173 - GitHub description: Harness engineering applied to knowledge production: a self-evolving multi-agent newsroom that turns your documents into a cross-linked markdown wiki. A "reground" loop pulls published pages back in before they go stale — writer ≠ reviewer, local-first, a structured alternative to RAG. ### Primary language: Python ### GitHub description: This project is a multi-agent newsroom that turns your documents into a cross-linked markdown wiki. The agent writes pages, but the reviewer is not the same as the author. Every new document enriches the existing pages, creating a structured and persistent alternative to RAG.

### Installation To install `conversant`, you can use pip: ```bash pip install conversant ``` ### Streamlit Demo You can run the Streamlit demo by cloning the repository and running: ```bash streamlit run app.py ``` ### Running Your Own Streamlit Demo If you want to run your own Streamlit demo, you can create a `requirements.txt` file with the following content: ```txt streamlit==2.0.0 ``` Then, clone the repository and run: ```bash streamlit run app.py ``` ### Creating a Custom Persona To create a custom persona, you can follow these steps: 1. Clone the repository: ```bash git clone https://github.com/cohere-ai/sandbox-conversant-lib.git cd sandbox-conversant-lib ``` 2. Edit the `persona.json` file to define your persona's description and example conversations. 3. Run the Streamlit demo: ```bash streamlit run app.py ``` ### Editing a Persona on the Demo You can edit the persona on the demo by clicking on the persona card in the Streamlit app. The changes will be saved automatically. ### Usage `conversant` is a framework that allows you to quickly build conversational AI agents using Cohere's large language models. You can customize personas, leave chat history management to `conversant`, and ground conversations in your provided facts. ### How Conversant Works `conversant` is designed

### Overview: CustomDiffusion360 is a project that allows users to control the viewpoint of generated images using text-to-image diffusion models. It leverages a 360-degree multiview dataset to fine-tune FeatureNeRF blocks in the intermediate feature space of the diffusion model, enabling the customization of new objects based on a target camera pose. ### Key features: - Customizing text-to-image diffusion models to control the viewpoint of generated images. - Fine-tuning FeatureNeRF blocks in the intermediate feature space of the diffusion model. - Target camera pose control for generating images with specific viewpoints. ### Tech stack: Python ### Why it matters: CustomDiffusion360 provides a powerful tool for generating images with controlled viewpoints, making it useful in various applications such as art, design, and simulation. The ability to customize the viewpoint based on a target camera pose allows for more realistic and contextually relevant images.

Overview: Deploy a self-hosted AI stack with Docker Compose, featuring local LLMs, chat, document parsing, embeddings, transcription, and text-to-speech tools powered by Ollama, LiteLLM, AnythingLLM, and more. The stack is secure by default, supports HTTPS, and is lightweight and multi-arch-compatible. Key features: - Zero-config auto-configuration - Secure password protection - Optional HTTPS support - Local-first deployment - AI agent integration with Goose - Lightweight stacks for lower memory usage - GPU acceleration via NVIDIA CUDA - Multi-architecture support Tech stack: Shell Why it matters: The Self-Hosted AI Stack provides a complete self-hosted AI environment with Docker Compose, making it ideal for developers looking to deploy AI tools locally without the need for external dependencies. Repository: hwdsl2/self-hosted-ai-stack GitHub stars: 173 GitHub description: Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollama, LiteLLM, AnythingLLM and more. Local-first with auto-configured authentication, lightweight stacks, optional HTTPS and NVIDIA GPU support. MIT licensed. Multi-arch: amd64, arm64.

Overview: GraphRAG is an implementation of a Graph RAG approach for query-focused summarization, built on top of LangChain, designed to be extensible and modular. Key features: - Supports LangChain integration - Provides a modular design for easy component swapping - Implements the GraphRAG algorithm from "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" Tech stack: Python Why it matters: This implementation aims to provide a versatile and extensible solution for query-focused summarization, leveraging the LangChain framework for improved integration and extensibility.

### Overview: FlowFlow is a voice notes app that captures what you say, transcribes it, and lets you chat with your notes later. It's designed to be used on both iOS and macOS, featuring a dark voice capsule that records and transcribes your speech, and a chat interface that allows you to ask your notes questions. ### Key Features: - **Voice Recording and Transcription**: One composer for notes and chat that records, transcribes, and hands back the text where you were typing. - **Live Waveform and Playback**: A live 60fps waveform that pauses and resumes, with a dynamic island timer and a pill button for dictation. - **Cloud and Offline Transcription**: Cloud or fully offline transcription with local Whisper models. - **Long Recording Support**: Local Whisper works in one-minute chunks, saves after each one, and resumes after a lock or crash. - **Personal Dictionary**: Personalized dictionary with your names and brands spelled right. - **Read a Long Meeting**: A recorded note reads like a document, with paragraphs split at pauses, highlighted words, and a reading bar. - **AI Chat with Claude or OpenAI**: Hybrid search with BM25, vector, and rerank, and the ability to ask your notes questions through a Claude or OpenAI API key. - **Smart Filters**: AI titles, tags, and themes, searchable chats, and smart filters in the search bar. - **Organized for You**: AI titles, tags, and themes as you write, and searchable chats. - **Encrypted LAN P2P Sync**:

Overview: This repository is designed for beginners learning PyTorch. It provides foundational knowledge and practical examples to help you understand the basics of computer vision. Key features: - Basic CV knowledge - Practical examples in Jupyter Notebook Tech stack: Jupyter Notebook Why it matters: This repository serves as an excellent resource for beginners looking to learn PyTorch and computer vision concepts. Repository: sagizty/Insight Primary language: Jupyter Notebook GitHub stars: 173

### Overview: Selene is a desktop app that integrates AI agents with popular communication tools, allowing users to write code, generate images, and control browsers directly from their devices. It prioritizes user data privacy by running AI locally and offers a seamless integration experience across multiple platforms. ### Key Features: - **AI Integration:** Connects to WhatsApp, Telegram, Slack, and Discord using AI agents. - **Code Writing:** Write and debug code directly within the app. - **Image Generation:** Generate high-quality images using AI. - **Browser Control:** Control web browsers with AI-powered actions. - **Offline Support:** Runs fully offline or uses local API keys for cloud-based services. - **Customization:** Mix and match different AI tools and services. ### Tech Stack: TypeScript ### Why It Matters: Selene simplifies the integration of AI with communication tools, making it easier for developers to create personal assistants and automate workflows without the need for complex setups or external dependencies. The app's agent-first approach ensures that all actions are contextually relevant and efficient, reducing the cost of AI usage per turn.

### Overview: This repository contains a comprehensive benchmark list for evaluating large language models (LLMs) and agent-powered systems, aiming to provide a streamlined resource for researchers and developers seeking to advance artificial general intelligence (AGI). ### Key Features: - **Survey of Evaluation Methods**: Includes reviews of various evaluation methods and benchmarks for LLMs and agents. - **ToolUse**: A tool for evaluating the tool manipulation capability of open-source large language models. - **API-Bank**: A benchmark for tool-augmented LLMs. - **T-Eval**: Evaluates the tool utilization capability of large language models step by step. - **NPHardEval**: A dynamic benchmark on the reasoning ability of large language models via complexity classes. - **Reasoning with Foundation Models**: A survey of reasoning with foundation models. ### Tech Stack: - **GitHub**: Repository management and collaboration platform. ### Why it Matters: This repository serves as a valuable resource for researchers and developers looking to evaluate the performance and capabilities of large language models and agent-powered systems, contributing to the advancement of artificial intelligence and AGI.

Overview: Vinaya Journal is a cross-platform journaling desktop app that integrates local AI via Ollama and a retrieval-augmented generation (RAG) pipeline using ChromaDB. It is designed for offline-first, privacy-conscious AI journaling, ensuring that your thoughts stay on your device. Key features: - Semantic Search of Your Entries - Mood Insight and Visualization - Personalized Chat with Vinaya’s Contextual Ollama AI Tech stack: TypeScript Why it matters: Vinaya Journal offers a secure, local journaling experience, keeping all data on your device, ensuring privacy and control over your journal. The Semantic Search and Mood Insights features help enhance the journaling experience by providing deeper insights into your thoughts and emotions. The personalized chat feature allows for engaging conversations with an AI that understands your entries, aiding in reflection and insight. Repository: BarsatKhadka/Vinaya-Journal Primary language: TypeScript GitHub stars: 171 GitHub description: A secure, local RAG journal that understands you better the more you write.

Overview: FlexPrefill is a dynamic and context-aware sparse attention mechanism designed to optimize computational efficiency during long-sequence inference for large language models (LLMs). It achieves this by dynamically adjusting sparse attention patterns and computational budgets in real-time based on input demands and attention head requirements. Key features: - Dynamic adjustment of sparse attention patterns - Real-time optimization of computational budgets - Context-aware sparse attention mechanism Tech stack: Python Why it matters: FlexPrefill is a significant contribution to the field of large language models, offering a method to enhance the efficiency of inference processes without sacrificing the quality of the output. This makes it particularly relevant for applications requiring high throughput and low latency, such as real-time language translation, chatbots, and other AI-driven services.

### Overview: This repository serves as a survey paper exploring the self-evaluation and self-updating mechanisms in large language models, focusing on Internal Consistency and Self-Feedback. It includes resources for measuring consistency and results from GPT-4o responses, aiming to provide a unified perspective on these mechanisms. ### Key Features: - **Code and Results for Consistency Measurement:** `expt-consistency-types` - **Results from GPT-4o Responses:** `expt-gpt4o-responses` - **Comprehensive List of References:** `Paper List` ### Tech Stack: Jupyter Notebook ### Why It Matters: This survey paper aims to provide a comprehensive overview of self-evaluation and self-updating mechanisms in large language models, offering a unified perspective on Internal Consistency and Self-Feedback. It includes practical resources and references, making it valuable for researchers and developers interested in the field. ### Repository: IAAR-Shanghai/ICSFSurvey ### Primary Language: Jupyter Notebook ### GitHub Stars: 172 ### GitHub Description: Explore concepts like Self-Correct, Self-Refine, Self-Improve, Self-Contradict, Self-Play, and Self-Knowledge, alongside o1-like reasoning elevation🍓 and hallucination alleviation🍄.

### Overview: StyleStudio is an advanced text-driven style transfer model designed for fine-tuning and integration into existing style transfer frameworks. It addresses the challenges of overfitting, stylistic control, and misalignment with textual content by introducing adaptive instance normalization, selective control over stylistic elements, and a teacher model to stabilize spatial layouts. ### Key Features: - **Cross-Modal Adaptive Instance Normalization (AdaIN)**: Enhances the integration of style and text features. - **Style-based Classifier-Free Guidance (SCFG)**: Enables selective control over stylistic elements. - **Teacher Model Integration**: Reduces irrelevant influences and mitigates artifacts during early generation stages. ### Tech Stack: Python ### Why It Matters: StyleStudio offers significant improvements in style transfer quality and better alignment with textual prompts. Its fine-tuning-free approach allows seamless integration into existing style transfer frameworks, making it a valuable tool for enhancing the aesthetic and textual coherence of generated images.

Overview: CEP is a software platform designed for users who want to learn or rapidly prototype using standard AI components. Key features: - Visual programming interface (based on Microsoft's MakeCode project) - Low-code Python API - Middleware system (CURT) for distributing AI computations to different computing devices on the local network - Built-in modules for computer vision, voice, NLP, control, and smart home Tech stack: JavaScript Why it matters: CEP provides a user-friendly interface for AI development, allowing users with no prior programming experience to create their own AI projects and deploy them on the OAK-D camera, while experienced users can quickly create AI programs using Python. The decentralized mesh architecture of CURT ensures device failover and load balancing, making it suitable for distributed AI computations. The built-in modules offer a variety of AI functionalities, including computer vision, voice recognition, natural language processing, and control, making it a versatile platform for developers. Repository: cortictechnology/cep GitHub stars: 172 GitHub description: CEP is a software platform designed for users that want to learn or rapidly prototype using standard AI components.

Overview: Rageval is a comprehensive evaluation tool designed for Retrieval-augmented Generation (RAG) methods. It evaluates six sub-tasks including query rewriting, document ranking, information compression, evidence verification, answer generation, and result validation. Key features: - Supports evaluation of RAG systems - Measures answer correctness and groundedness - Provides metrics such as F1, NLI, EM, Bleu, TER, chrF, Disambig-F1, Rouge, and LCS Ratio - Integrates with Python workflows and CI/CD pipelines Tech stack: Python Why it matters: Rageval offers a robust framework for evaluating RAG systems, ensuring that the generated answers are both correct and factually consistent, thereby enhancing the overall performance and reliability of RAG models.

Overview: This Nix flake provides a curated ComfyUI environment with Python 3.12 and custom nodes, supporting macOS (Apple Silicon) and Linux. It includes CUDA and ROCm GPU support with pre-built wheels for fast builds. Key features: - A slightly opinionated ComfyUI flake with curated custom nodes - Supports macOS (Apple Silicon) and Linux - CUDA builds available for Linux with NVIDIA GPUs - ROCm builds available for Linux with AMD GPUs - Pre-built PyTorch wheels for fast builds - Supports Turing (RTX 20 series) through Blackwell (RTX 50 series) in one package - Supports `gfx1100` (7900XTX) with ROCm 7.1 libraries Tech stack: Nix Why it matters: This flake simplifies the setup of ComfyUI, providing a stable and efficient environment for developers on both macOS and Linux. The pre-built wheels for CUDA and ROCm ensure fast builds and support for a wide range of GPUs, making it suitable for both modern and older hardware. Repository: utensils/comfyui-nix GitHub stars: 171 GitHub description: A slightly opinionated Nix flake for ComfyUI with curated custom nodes. Supports macOS (Apple Silicon) and Linux with CUDA.

Overview: Libre Chat is a free and open-source chatbot web service that allows users to easily configure and deploy a fully self-hosted, offline-capable chatbot based on open-source Large Language Models (LLMs). It provides a web UI and API, and supports various types of agents, including generic conversation and documents-based question answering. Key features: - 🌐 Free and open-source chatbot web service with UI and API. - 🏡 Fully self-hosted, not tied to any service, and offline capable. - 🔌 Web API described using OpenAPI specs: GET/POST operations, websocket for streaming response. - 🪶 Chat web UI working well on desktop and mobile, with streaming response, and markdown rendering. Alternative Gradio-based UI also available. - 🚀 Easy to setup, no need to program, just configure the service with a YAML file, and start it with 1 command. - 📦 Available as a `pip` package 🐍, or `docker` image 🐳. - 🦜 Powered by `LangChain` and `llama.cpp` to perform inference locally. - 🤖 Various types of agents can be deployed: generic conversation, documents-based question answering (experimental). Tech stack: Python Why it matters: Libre Chat is designed for developers and enthusiasts who want to create a chatbot without the need for deep knowledge in machine learning. It provides a user-friendly interface and is highly customizable, making it accessible to a wide range of users. The self-hosted nature of the service ensures security

### Overview: This repository is a curated collection of papers, models, and resources focused on the field of video generation. It aims to provide the most comprehensive and cutting-edge map of the latest research in video generation. ### Key Features: - **Survey Papers**: Comprehensive surveys on various aspects of video generation. - **Text-to-Video (T2V) Generation**: Papers and models for generating videos from text descriptions. - **Image-to-Video (I2V) Generation**: Techniques for converting images into videos. - **Video-to-Video (V2V) Editing**: Methods for editing and manipulating videos. - **Controllable Video Generation**: Papers on generating videos with specific control over elements. - **Audio-Driven Video Generation**: Research on generating videos based on audio inputs. - **Human Image Animation**: Techniques for animating human images. - **Fast Video Generation (Acceleration)**: Papers on accelerating video generation processes. - **Reinforcement Learning for Video Generation**: Research on using reinforcement learning for video generation. ### Tech Stack: Python, Markdown, Git, and various academic papers and models. ### Why It Matters: This repository serves as a valuable resource for researchers, developers, and enthusiasts interested in the latest advancements in video generation. It provides a one-stop shop for the most up-to-date and cutting-edge research in the field, making it an essential tool for staying informed and contributing to the ongoing developments in video generation.

Overview: The repository contains the Docker Compose files for running the Spellbook AI Assistant stack, including the UI and middleware components. This stack is designed to interact with various AI models such as LLMs, Stable Diffusion, Whisper, and xTTS, requiring an ExLlama GPU for real-time results. Key features: - Spellbook AI Assistant stack - Docker Compose files for running the stack - Real-time results with ExLlama GPU support - UI and middleware components - Stack architecture diagram - Ubuntu 22 installation instructions Tech stack: Shell Why it matters: The Spellbook AI Assistant stack is essential for developers and designers looking to interact with a wide range of AI models, providing a unified interface and seamless integration with real-time AI capabilities. GitHub stars: 171