Projects

### Overview: ToK is a fully local, high-quality, and extensible Retrieval-Augmented Generation (RAG) application that leverages AI models and indices to query documents and generate contextually accurate responses. It is designed for users to upload documents and receive more informed answers tailored to their queries. Additionally, the app automatically stores chats, allowing users to revisit and continue conversations at any time. ### Key Features: - **Fully Local, Secure, and Privacy-Focused:** Ensures all operations are performed locally, keeping data private and secure. - **Versatile Document Uploading:** Supports uploading of various types of text documents and folders for creating an index. - **Optional Metadata Assignment:** Enhances document uploads with custom metadata for better organization and accessibility. - **Comprehensive UI Settings:** Access and modify important settings like chunking options, temperature, context window size, and chat mode through the user interface. - **Custom Prompts:** Allows adding and using personalized prompts for both the Large Language Model (LLM) and the chat engine. - **Persistent Chats with Titles:** Chats are saved with customizable titles, enabling revisit and continuation of conversations later. - **Persistent Chat History:** Access and review past conversations at any time. - **Model Selection from Ollama:** Allows choosing and pulling AI models directly from Ollama for specific tasks and preferences. - **Streaming Responses with Smart Snippets:** Provides real-time responses with smart output formatting, especially for code snippets. - **Extensible:** Easily extend ToK's functionality with additional features or integrations. ### Tech Stack:

Overview: This repository contains code for fine-tuning the Whisper speech-to-text model, supporting features like multi-dataset validation, WER, CER, NLL, log-probability, entropy, and calibration metrics, and training with single-GPU and PyTorch DDP, LoRA, and more. Key features: - Multi-dataset validation with macro averages - WER, CER, NLL, log-probability, entropy, and calibration metrics - Single-GPU and PyTorch DDP training - LoRA training and LoRA checkpoint merging - Timestamp training - Prompt training - Stochastic depth, SpecAugment, gradient checkpointing, and mixed precision - W&B logging and local checkpointing Tech stack: Jupyter Notebook Why it matters: This repository is essential for developers looking to fine-tune Whisper models for various applications, ensuring accurate and efficient speech-to-text processing. Repository: i4Ds/whisper-finetune GitHub stars: 27 GitHub description: This repository contains code for fine-tuning the Whisper speech-to-text model.

### Overview: This repository contains the dataset and code for the EMNLP 2022 paper "Visual Named Entity Linking: A New Dataset and A Baseline," which introduces the first large-scale visual input-oriented multimodal named entity linking dataset. The dataset consists of over 48,000 annotated images based on 120,000 Wikipedia knowledge base, focusing on Visual Named Entity Linking. ### Key Features: - **Dataset**: The repository includes the Visual Named Entity Linking (VNEL) dataset, which is a high-quality human-annotated visual person linking dataset focusing on Visual Named Entity Linking. - **Data Format**: The dataset is labeled on the News-related dataset with diverse agencies such as USA TODAY, BBC, the Washington Post, etc., making it of higher quality than images directly obtained from search engines. - **Example**: An example of the dataset is shown, featuring each image with the entity's bounding box and corresponding Wikidata ID. - **Legal Notices**: The contributors grant a license to the WIKIPerson Dataset and other content under the MIT License, and reserve all other rights. ### Tech Stack: - **Python**: The primary programming language used for the project. ### Why It Matters: This dataset and code provide a comprehensive solution for the Visual Named Entity Linking task, offering a new perspective on linking visual entities to their corresponding knowledge bases. The inclusion of large-scale Wikipedia knowledge base ensures a rich and diverse dataset, making it suitable for evaluating and improving various models in the field. ### Repository: - **Repository

### Overview: VECTR8 is a web application that allows users to upload CSV/JSON files, create vector embeddings, and query them. It is currently under development, with features like converting unstructured data to JSON/CSV using an integrated LLM. ### Key features: - Upload CSV/JSON files - Create vector embeddings - Query vector database - Convert unstructured data to JSON/CSV ### Tech stack: JavaScript, Python 3.7+, Flask, Flask-CORS, transformers, torch, numpy, pandas ### Why it matters: VECTR8 is designed to help users easily convert and analyze unstructured data into structured formats, making it easier to search and retrieve information. The project is still under development, so features like embeddings and query pages are currently in progress.

Overview: AnnoTheia is a data annotation toolkit designed to identify when a person speaks in a scene and transcribe their speech, offering flexibility to adapt to different languages. Key features: - Detects when a person speaks in a scene - Transcribes spoken speech - Flexible to replace modules for different languages Tech stack: Python Why it matters: This toolkit aims to cover a wide range of languages in audio-visual speech technologies, promoting inclusivity and accessibility. Repository: joactr/AnnoTheia GitHub stars: 27

Overview: 译声实时翻译（白色新版）是一款免费的Windows实时日语字幕与翻译工具，支持系统声音、本地GGUF/Ollama/OpenAI兼容API翻译，提供历史回看、自定义背景等功能。 Key features: - 支持系统声音和麦克风 - 日语识别和本地或API翻译 - 悬浮字幕和历史回看 - 自定义背景设置 - 优化透明度和重绘问题 - 整句翻译优先，失败重试机制 Tech stack: Python Why it matters: 译声实时翻译（白色新版）为开发者提供了一个高效、实用的实时翻译工具，特别适合需要实时字幕和翻译的场景，如游戏、会议或远程协作。其简洁的界面和丰富的功能使其成为提高工作效率的得力助手。

Overview: This project provides a simple and up-to-date wrapper for the Prodia API, including all available features. Key features: - Text to Image - Transform Image - SDXL Generation - Face Swap Tech stack: TypeScript Why it matters: This library simplifies the process of generating images and applying transformations using the Prodia API, making it easier for developers to integrate Prodia's features into their projects.

Overview: Exa is a developer-friendly tool designed to maximize the performance of large language models (LLMs) on consumer-class GPUs, achieving a 300% boost in performance without requiring long-term adjustments or additional hardware costs. Key features: - World-Class Quantization: Enhances model performance with top-tier accuracy. - Automated PEFT: Simplifies fine-tuning workflows. - LoRA Configuration: Explores flexible LoRA configurations for improved performance. - Seamless Integration: Works seamlessly with popular models like LLAMA, Falcon, and more. Tech stack: Python Why it matters: Exa leverages cutting-edge LLMs with minimal code, ensuring unparalleled performance on consumer-class GPUs, making it an essential tool for developers looking to maximize their LLMs' potential without the need for complex setups or expensive hardware upgrades. Repository: kyegomez/Exa Primary language: Python GitHub stars: 27 GitHub description: Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and minimal learning curve. README excerpt: [](https://discord.gg/qUtxnK2NMf) # Exa Boost your GPU's LLM performance by 300% on everyday GPU hardware, as validated by renowned developers, in just 5 minutes of setup and with no additional hardware costs. ----- ## Principles - Radical Simplicity (Utilizing super-powerful LLMs with as minimal lines of code

### Overview: This repository provides a comprehensive guide on how to integrate multimodal and RAG functionalities using LangChain, specifically focusing on applications utilizing Anthropic Claude3, Amazon Bedrock, and LangChain ChatBedrock. ### Key Features: - **Multimodal Integration**: Supports the analysis of text and images. - **Multi-Region LLMs**: Utilizes multiple AWS Regions to alleviate request and token limits. - **RAG (Retrieval-Augmented Generation)**: Implemented using OpenSearch for fast and high-performance results. - **Internet Search**: Utilizes Google Search API when RAG-related documents are not available. - **Priority Search**: Filters out less relevant documents to improve accuracy. - **Chatbot Infrastructure**: Configured with LangChain ChatBedrock for efficient handling of chatbot requests. - **DynamoDB Integration**: Retrieves chat logs from DynamoDB for continuous conversation support. - **WebSocket Integration**: Facilitates bidirectional communication using API Gateway. - **AWS CDK Deployment**: Simplifies infrastructure deployment and management using AWS CDK. ### Tech Stack: Python ### Why It Matters: This guide showcases the practical application of LangChain and its integration with AWS Bedrock and Anthropic Claude3, providing a robust framework for developers to build multimodal and RAG-based applications efficiently.

### Overview: LinkML-Store is an AI-ready data management and integration platform that provides a common abstraction layer over multiple backends, including DuckDB, MongoDB, Neo4j, and local filesystems. It allows for common query, index, and storage operations, making it a versatile tool for data management and integration. ### Key Features: - Supports multiple backends: DuckDB, MongoDB, Neo4j, and local filesystems. - Provides a common abstraction layer for data management and integration. - Supports indexing and search operations using semantic search using LLM embeddings. - Composable indexes: You can use different indexing schemes with different backends. - Supports analytic faceted queries across all backends. ### Tech Stack: Python ### Why it Matters: LinkML-Store is designed to be flexible and extensible, allowing for common query, index, and storage operations across multiple backends. It provides a union of features from each backend, making it a versatile tool for data management and integration.

Overview: TokenRouter simplifies the integration of multiple AI providers by providing a single, OpenAI-compatible endpoint for all requests. This allows for easier setup, better security, and more efficient management of access and quotas. Key features: - One `/v1` endpoint for all OpenAI-compatible clients. - Support for multiple providers and model discovery. - Built-in presets and provider connection tests. - Access token hierarchy (`admin`, `keymaster`, `inferrer`). - Token expiry and quota enforcement. - Persistent usage analytics. - Built-in admin UI for easy access. Tech stack: Go Why it matters: TokenRouter simplifies the integration of multiple AI providers by providing a single, OpenAI-compatible endpoint, making it easier to manage and secure the use of AI services across different tools and applications.

Overview: This project is an offline streaming speech-to-text transcription system that processes audio entirely offline on the CPU, utilizing a 1B param streaming speech recognition model. The demo is built using Rust and WebAssembly (WASM) and is designed for real-time transcription in the browser. Key features: - Offline streaming speech-to-text transcription - Uses a 4-bit quantized version of a 1B param streaming speech recognition model - CPU parallelization with Rayon for efficient processing Tech stack: Rust, WebAssembly (WASM), Kyutai STT Model, Candle, Rayon, wasm-bindgen-rayon Why it matters: This project offers a practical solution for real-time speech transcription in the browser, making it ideal for applications requiring offline processing and low-latency performance.

### Overview: This repository contains the code for an automatic speech recognition (ASR) model designed for the Nepali language. The model utilizes CNN, bidirectional LSTM, and ResNet architectures to improve the accuracy of speech recognition. ### Key Features: - **Model Architecture**: The model combines CNN, bidirectional LSTM, and ResNet to enhance speech recognition accuracy. - **Training Pipeline**: The repository includes a training pipeline implemented in `trainer.py` that can be used to replicate the results. - **Evaluation**: The model is evaluated using CTC loss and beam search decoding for text generation. ### Tech Stack: - **Python**: The primary programming language used for the implementation. ### Why It Matters: This project showcases advancements in ASR for the Nepali language, providing a valuable resource for researchers and developers interested in improving speech recognition technologies for minority languages.

Overview: mlserve is a tool that turns Python models into RESTful APIs and serves web pages with form inputs that match the input data of the training dataset. Key features: - Model predictions via RESTful API endpoint - Model predictions via generated UI - Web page to simplify model usage - Automatic UI generation to match input data - Simple dashboard for monitoring Tech stack: Python Why it matters: mlserve simplifies the process of integrating machine learning models into existing applications by providing a RESTful API and a user-friendly web interface. It's useful for quickly demonstrating models and integrating them into existing applications without requiring extensive backend development.

Overview: This project provides a blueprint for transcribing audio files using the Speaches API, which is compatible with Mozilla's AI initiative. It includes instructions for setting up both a local demo and a Docker container for running Speaches. Key features: - Using the Speaches API for transcribing audio files - Setting up a local demo on HuggingFace Spaces - Using Docker to run Speaches on a server - Accessing the server via HTTP Tech stack: Python, Docker, OpenAI API Why it matters: This project is useful for developers and researchers looking to integrate speech transcription capabilities into their applications or projects, providing a streamlined and efficient way to transcribe audio files.

Overview: DataGenFlow is a minimal tool designed to generate and validate datasets from seed documents with full visibility. Key features: - Easy to Extend: Add custom blocks with auto-discovery - Faster Development: Visual pipeline builder eliminates boilerplate code - Simple to Use: Intuitive drag-and-drop interface, no training required - Full Transparency: Complete execution traces for debugging Tech stack: Python Why it matters: DataGenFlow simplifies the process of generating and validating data from seed documents, making it accessible and efficient for developers.

Overview: This project is a Java client library for the Ollama API, designed to simplify interaction with the Ollama web service. Key features: - OllamaService: Interface for interacting with the Ollama web service. - OllamaServiceFactory: Factory class for creating instances of the OllamaService. - StreamResponseProcessor: Interface for processing streaming completion responses. Tech stack: Java. Why it matters: This client library simplifies the integration of Ollama into Java applications, making it easier for developers to access the Ollama API without needing to understand the underlying API documentation.

### Overview: OpenCrawling is a reference implementation of the Open Ingestion Standard (OIS) that provides a secure, decoupled, and vendor-neutral enterprise data integration platform. It leverages modern Java 25 features, Spring Boot, Spring AI, and vector search infrastructure to orchestrate data flows. This project is designed to be used as a building block for integrating data across different systems. ### Key Features: - Secure data integration platform - Decoupled architecture for easy scalability - Vendor-neutral integration capabilities - Modern Java 25 features - Spring Boot integration - Spring AI for AI-driven data processing - Vector search infrastructure for efficient data retrieval ### Tech Stack: Java 25 ### Why It Matters: OpenCrawling is crucial for enterprises looking to integrate data from multiple sources securely and efficiently. Its modern Java 25 features and Spring Boot integration make it a versatile tool for integrating data across different systems. The project's focus on vendor-neutral integration and modern technologies ensures that it can be used in a wide range of environments, making it a valuable resource for developers and data integration professionals.

Overview: FastAssert is a Dockerized LLM inference server designed to provide faster, cheaper, and rate-limited access to large language models (LLMs). It leverages vLLM and outlines to achieve high accuracy while significantly reducing validation errors and costs. Key features: - Faster inference speed compared to GPT-3.5 - Reduced validation errors using outlines - Lower cost compared to function calling - No rate limits on API calls Tech stack: FastAPI, vLLM, outlines Why it matters: FastAssert offers a more efficient and cost-effective alternative to traditional LLM APIs, making it ideal for developers looking to integrate advanced language processing capabilities without the limitations of current providers. Repository: phospho-app/fastassert Primary language: Jupyter Notebook GitHub stars: 27 GitHub description: Dockerized LLM inference server with constrained output (JSON models), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API provider.

Overview: PostgreSQL-V is a fast vector search system designed for PostgreSQL, offering high performance comparable to specialized vector databases. Unlike previous systems that inherit overhead, PostgreSQL-V decouples vector indexes from PostgreSQL's core engine, leveraging native vector index libraries for improved performance. However, it introduces the challenge of index inconsistency, which is addressed with a lightweight mechanism. Key features: - Fast vector search performance - Compatible with pgvector APIs - Decoupled architectural design from PostgreSQL's core engine - Lightweight consistency mechanism for index inconsistency - Compatible with PostgreSQL Tech stack: PostgreSQL, pgvector Why it matters: PostgreSQL-V provides a viable solution for vector search in PostgreSQL, offering high performance and compatibility with existing tools, making it a valuable addition to the PostgreSQL ecosystem. Repository: purduedb/PostgreSQL-V Primary language: C GitHub stars: 27 GitHub description: Fast vector search in PostgreSQL