Projects

### Overview: Noobot is a self-hosted AI agent application that provides isolated user workspaces, durable sessions, extensible tools, semantic workflows, and model-family routing across OpenAI-compatible providers. ### Key features: - Multi-user workspace - Persistent attachments and execution history - File operations, native/script execution, browser automation, LibreOffice, FFmpeg, multimodal parsing/generation, services, and reusable skills - OpenAI-compatible provider interfaces, provider/model-family routing, tool calling, streaming, and multimodal capability configuration - Task delegation, semantic workflows, workflow plugins, and harness-based planning/guidance/review - MCP servers plus database, terminal, email, and custom service connectors - Import 3D characters and use natural language to create, preview, and refine AI character animations ### Tech stack: JavaScript, Vue 3, Electron, Node.js ### Why it matters: Noobot offers a cost-effective, self-hosted AI agent solution for tool calling, multi-model routing, and multi-agent workflows, providing isolated user workspaces and seamless integration with various OpenAI-compatible providers.

Overview: This project is a paper titled "Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering" that was accepted by the ACL 2024 Findings conference. It introduces a new pipeline called KnowPAT, which aims to align large language models (LLMs) with human preference for better user experience in domain-specific applications. Key features: - KnowPAT is a pipeline that aligns LLMs with human preference. - It incorporates domain knowledge graphs to construct preference sets. - It designs a new alignment objective to fine-tune the LLMs. Tech stack: The project is developed using Python. Why it matters: This paper addresses the need for LLMs to better understand and align with human preferences, particularly in domain-specific applications, leading to improved user experience and better decision-making capabilities. Repository: zjukg/KnowPAT Primary language: Python GitHub stars: 192 GitHub description: [Paper][ACL 2024 Findings] Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering

### Overview: TIFA is a tool designed to evaluate the fine-grained alignment between text inputs and generated images from text-to-image models like Stable Diffusion. This repository includes the code and models for a paper titled "TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering." ### Key Features: - **Text-to-Image Alignment Evaluation:** Measures the accuracy of how well the text matches the generated image. - **Interpretable Results:** Provides clear and understandable results that can be easily explained. - **Question Answering Model:** Uses a VQA (Visual Question Answering) model to generate questions that help evaluate the text-to-image faithfulness. ### Tech Stack: - Python ### Why It Matters: TIFA is more accurate than existing tools like CLIP while being fine-grained and interpretable, making it an ideal choice for evaluating image generation models. The repository also includes tools to generate your own questions without the need for OpenAI APIs. ### Repository: - **Repository:** Yushi-Hu/tifa - **GitHub Stars:** 191 - **GitHub Description:** TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering - **Primary Language:** Python ### Quick Links: - **Installation:** `git clone https://github.com/Yushi-Hu/tifa.git && cd tifa && pip install -r requirements.txt && pip install -e .` - **Quick Start:** Generate images from `sample/sample_text_inputs.json`, create a JSON file for image locations

### Overview: DSH Crew is a plugin designed to integrate Claude Code, Codex, Antigravity, and Grok with DeepSeek Harness (DSH), allowing users to dispatch work to DSH agents without losing the host's native subagent interface. It provides native progress tracking, tier-specific policies, and a multimodal bridge that leverages the text-only capabilities of Claude Code while maintaining image generation capabilities. ### Key Features: - **Native Progress UI**: Workers appear as regular subagents in Claude Code / Codex / Antigravity / Grok, with detailed progress showing in the host's task panel. - **Tier Policy and Escalation**: Dispatch tiers (`flash` / `pro`) manage agent capabilities, with tier policy clamping and escalation for failed flash runs. - **In-Host DSH Sessions**: Enables DSH agents to run within the host's native subagent interface, improving usability in CI/CD environments. - **Vision and Image Generation**: Utilizes DeepSeek's VL model for image generation, falling back to user-provided models or OpenAI-compatible APIs. ### Tech Stack: JavaScript ### Why It Matters: DSH Crew simplifies the integration of Claude Code, Codex, Antigravity, and Grok with DSH, providing a seamless workflow that retains the host's native subagent interface while leveraging the text-only capabilities of Claude Code. This plugin enhances the flexibility and usability of DSH agents in various deployment scenarios, ensuring that users can maintain their preferred workflows without compromising on the performance and reliability of their DSH harness.

Overview: This repository contains the implementation code for a novel method called Genie, which uses equivariantly diffusing oriented residue clouds for de novo protein design. It is based on an ICML paper and provides a sample illustration of the protein design process. Key features: - Equivariantly diffusing oriented residue clouds for de novo protein design - Implementation of a protein design method using equivariant diffusion models - Scripts for downloading and cleaning the SCOPe dataset - Configuration and training scripts for the Genie model - Sample and evaluation scripts for generating and evaluating protein designs Tech stack: Python Why it matters: This method offers a new approach to de novo protein design by leveraging equivariant diffusion models, potentially leading to more efficient and accurate protein design compared to existing methods. Repository: aqlaboratory/genie Primary language: Python GitHub stars: 191

Overview: This repository contains the code and results for the paper "Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models". The paper is available on arXiv and ACL Anthology. Key features: - Detailed reports and summary table in the `results/` directory - `run_verification.sh` script for running new models - `magikarp/unused_tokens.py` for updating unused token ids - `generate_results.py` for generating plots and markdown reports Tech stack: Python Why it matters: This repository provides valuable insights into the challenges and solutions for detecting under-trained tokens in large language models, contributing to the field of language model evaluation and improvement.

Overview: Cogitator is a Python toolkit designed for experimenting with chain-of-thought (CoT) prompting methods in large language models (LLMs). It aims to simplify the integration and usage of popular CoT strategies and frameworks for research or application development. Key features: - Unified API for CoT strategies - Support for OpenAI, Ollama, and OpenRouter as LLM providers - Pydantic validation for structured model outputs - Customizable benchmarking framework - Implementations of Self-Consistency CoT, Automatic CoT, Least-to-Most Prompting, Tree of Thoughts, Graph of Thoughts, and Clustered Distance-Weighted CoT Tech stack: Python Why it matters: Cogitator provides a convenient and comprehensive solution for researchers and developers looking to explore and integrate CoT strategies into their AI applications, thereby enhancing the performance and interpretability of LLMs on complex tasks. Repository: habedi/cogitator GitHub stars: 191 GitHub description: A Python toolkit for chain-of-thought prompting 🐍

Overview: This repository contains the dataset and code for the paper "Few-Shot Keyword Spotting in Any Language" presented at Interspeech 2021 and the Multilingual Spoken Words Corpus, which was presented at NeurIPS 2021 Datasets and Benchmarks Track. Key features: - Multilingual Spoken Words Corpus dataset - Few-shot Keyword Spotting model Tech stack: Jupyter Notebook Why it matters: This repository provides a valuable resource for researchers and developers interested in few-shot keyword spotting across multiple languages, making it an essential tool for advancing natural language processing and speech recognition technologies.

### Overview This project is an end-to-end Speech Recognition neural network implemented using Keras, designed as part of an Artificial Intelligence Nanodegree program at Udacity. The goal is to build a deep neural network that functions as part of an end-to-end automatic speech recognition (ASR) pipeline. The project involves investigating the LibriSpeech dataset, which is used to train and evaluate the models. The algorithm converts raw audio to feature representations and then maps these features to transcribed text using neural networks. The project also includes creating and testing state-of-the-art models. ### Key Features - **LibriSpeech Dataset**: The dataset is used for training and evaluating the models. - **Deep Neural Network**: The project involves building a deep neural network using Keras. - **ASR Pipeline**: The project includes an ASR pipeline that converts audio to feature representations and maps them to transcribed text. ### Tech Stack - **Programming Languages**: Python, Keras - **Frameworks**: TensorFlow, Keras - **Libraries**: NumPy, Matplotlib, TensorFlow ### Why It Matters This project is significant because it demonstrates the application of deep learning techniques in speech recognition, showcasing the capabilities of neural networks in processing and understanding speech. The project also provides a comprehensive guide on how to set up and implement an ASR pipeline, making it valuable for both students and professionals in the field of AI and speech technology.

### Overview: Emeltal is a local machine learning voice chat application that utilizes high-performance models to provide a user-friendly experience. The project aims to create a self-contained, out-of-the-box solution with minimal dependencies. ### Key Features: - **Voice Chat Integration**: Enables real-time voice communication within a local environment. - **Model Selection**: Offers a curated list of open-source models, including Qwen, Dolphin, Samantha, Gemma, Llama, and Devstral, for optimal performance. - **Model Performance**: Supports models ranging from 3.5B to 70B parameters, with varying context sizes to ensure the best fit for each application. ### Tech Stack: - **Programming Languages**: C++ - **Model Platforms**: Hugging Face, Meta, and Cognitive Computation ### Why It Matters: Emeltal aims to provide a seamless and efficient voice chat experience, leveraging the latest machine learning models to enhance the quality and reliability of the communication. The project is designed to be as self-contained as possible, minimizing the need for external dependencies, making it ideal for users who may not have access to powerful hardware or extensive internet connectivity.

Overview: This repository contains a practical guide to building production-quality Retrieval Augmented Generation (RAG) systems using knowledge graphs for enhanced accuracy, traceability, and performance. Key features: - Practical Examples: Vector similarity search tools, Agentic RAG applications, and performance evaluation - Production-Ready: Learn to extract structured knowledge from text and deploy real-world systems - Hybrid Approach: Combine vector-based and graph-based retrieval methods - Comprehensive: From basic concepts to advanced implementation techniques Tech stack: Jupyter Notebook Why it matters: This book demonstrates how to harness knowledge graphs to supercharge RAG systems, providing more relevant and comprehensive context to LLMs. Repository: tomasonjo/kg-rag GitHub stars: 191

Overview: Machtiani is a powerful email client that allows users to email back and forth with their computer using their existing email address and agents they already know and use. It leverages git for its memory model and provides features like secure group conversations, local non-cloud-based memory, and the ability to use any agent installed. Key features: - Securely add others to a group email conversation and use as many simultaneous threads as desired. - Remembers what you're working on with local, non-cloud-based memory. - Use your favorite agents already installed and with whatever sandboxing you want. - Watch the work on your computer by running `machtiani run --attach` on the terminal. Tech stack: Python. Why it matters: Machtiani empowers users by providing a seamless and secure way to communicate with their computer, leveraging existing email infrastructure and agents, and managing sessions effectively without compaction. Repository: tursomari/machtiani Primary language: Python GitHub stars: 191 GitHub description: Empowering users.

Overview: X-CLIP is an official implementation of the paper "X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval," which was accepted by ACMMM22. It uses cross-grained contrastive learning and attention over similarity matrix modules to enhance video-text retrieval. Key features: - Multi-grained Contrastive Learning - Attention over Similarity Matrix Module Tech stack: Python Why it matters: X-CLIP achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet, demonstrating its effectiveness in improving retrieval accuracy.

### Overview: Imagine360: Immersive 360 Video Generation from Perspective Anchor ### Key features: - Perspective-to-360 video generation - Curated YouTube 360 video dataset - Inference code and checkpoints available - Data and video demo links provided - VR mode guideline for enhanced experience ### Tech stack: Python ### Why it matters: Imagine360 offers a powerful tool for generating immersive 360 videos from perspective anchors, making it a valuable resource for creators and researchers in the field of 360 video technology. The curated dataset and associated tools provide a comprehensive solution for creating high-quality 360 videos, enhancing the user experience with a mobile-friendly interface and detailed VR mode guidance.

Overview: AI SDK CPP is a modern C++ toolkit designed to simplify the integration of AI services like OpenAI and Anthropic into C++ applications. Key features: - Unified API for interacting with multiple AI providers - Built with C++20 features for clean, expressive code - Minimal external dependencies for easy integration Tech stack: C++ Why it matters: AI SDK CPP provides a convenient, unified interface for C++ developers to work with popular AI services, making it easier to build AI-powered applications.

Overview: This submodule provides the source code for the embedded version of AnythingLLM, allowing developers to expose their workspace as a chat interface within a website or HTML. Key features: - Embeds workspace as a chat widget via `<script>` or `<iframe>` elements - Supports random session ID for chat persistence - Limits chat model abuse through configuration - Includes `<script>` and `<iframe>` HTML embed options Tech stack: JavaScript Why it matters: This submodule is essential for integrating the chat functionality of AnythingLLM into custom applications, ensuring secure and controlled access to the workspace's knowledge base. Repository: Mintplex-Labs/anythingllm-embed Primary language: JavaScript GitHub stars: 189

### Overview: Project Lakechain is a cloud-native, AI-powered document processing pipeline built on AWS, designed to be scalable, cost-efficient, and ready to use with 60+ built-in middlewares. ### Key Features: - 🤖 **Composable** — Express document processing pipelines using middlewares. - ☁️ **Scalable** — Scales out-of-the-box, automatically scaling to zero when done. - ⚡ **Cost Efficient** — Uses cost-optimized architectures to reduce costs. - 🚀 **Ready to use** — 60+ built-in middlewares for common document processing tasks. - 🦎 **GPU and CPU Support** — Balances performance and cost with GPU and CPU support. - 📦 **Bring Your Own** — Create custom transform middlewares to extend Lakechain. - 📙 **Ready Made Examples** — 50+ examples for quickstart projects. ### Tech Stack: TypeScript ### Why it Matters: Project Lakechain simplifies the process of building document processing pipelines, making it easier and more efficient to integrate AI and machine learning into your applications.

Overview: This is a Python library designed to facilitate the creation of prompts for various applications, making it easier to generate and format text-based instructions and examples. Key features: - The `template` decorator allows users to create Jinja templates for their prompts, which can be passed to the `Template` object to render them. - The `few_shots` function demonstrates how to use the `template` decorator to provide examples alongside a set of instructions. Tech stack: Python Why it matters: This library simplifies the process of creating prompts by abstracting the creation of templates and examples, making it more efficient and easier to manage for developers and project managers.

### Overview: 本项目是关于大模型LLM的学习教程与笔记，旨在帮助程序员从零开始学习LLM的基础知识。教程涵盖了从PyTorch到Transformer架构、LangChain基础、大模型训练与微调、以及RAG和Agent等多方面的内容。 ### Key features: - PyTorch基础 - Transformer基础 - LangChain基础 - 大模型之基础 (CS336) - 大模型之训练与微调 - 大模型之 RAG (Retrieval-Augmented Generation) - 大模型之 Agent ### Tech stack: Python ### Why it matters: 本教程为程序员提供了一个全面的路径，从基础的机器学习框架到复杂的自然语言处理任务，帮助他们理解和应用这些技术。通过学习，读者可以掌握从零开始构建和训练大模型所需的技能，从而在实际项目中发挥更大的作用。

Overview: This project is an open-source Next.js frontend platform designed to automate the generation of movies and videos from script generation to text-to-video conversion. The platform aims to democratize the creation of high-quality videos, enabling anyone to become a director by quickly translating their ideas into engaging videos, whether for films, marketing videos, or social media videos. Key features: - Video script generation using GPT models from OpenAI, Baidu Kuaikan, and Qiming ChatGLM. - Text-to-video conversion using fliki API (API temporarily unavailable). - Frontend development based on Next.js, with design references to Notus Next.js and Next.js Template. Tech stack: Next.js Why it matters: This project aims to provide a comprehensive solution for generating videos using AI, especially for movie and video creation. By democratizing the video creation process, it aims to make high-quality video creation accessible to everyone, including those who may not have the necessary skills or resources to do so traditionally. GitHub stars: 189 Repository: JiaqiLi404/IAmDirector-Text2Video-NextJS-Client