Projects

### Overview: Docling Java is a Java API designed to simplify the processing and integration of diverse document formats, including advanced PDF understanding and seamless integration with the Generative AI ecosystem. ### Key Features: - 🗂️ Parsing of multiple document formats, including PDF, DOCX, PPTX, XLSX, HTML, WAV, MP3, VTT, images (PNG, TIFF, JPEG, ...), and more. - 📑 Advanced PDF understanding, including page layout, reading order, table structure, code, formulas, image classification, and more. - 🧬 Unified, expressive DoclingDocument representation format. - 🗂️ Various export formats and options, including Markdown, HTML, DocTags, and lossless JSON. - 🔒 Local execution capabilities for sensitive data and air-gapped environments. - 🤖 Plug-and-play integrations, including LangChain4j. - 🔍 Extensive OCR support for scanned PDFs and images. - 👓 Support for several Visual Language Models (GraniteDocling). - 🎙️ Audio support with Automatic Speech Recognition (ASR) models. ### Tech Stack: Java ### Why It Matters: Docling simplifies document processing, parsing diverse formats, and provides seamless integrations with the Generative AI ecosystem. This project offers a Java API for working with the data types used by Docling for document representation, making it easier to integrate with advanced AI tools.

Overview: EpiDiff is a localized interactive multiview diffusion model designed to enhance multi-view synthesis. It introduces a lightweight epipolar attention block to improve cross-view interaction among feature maps. Key features: - Localized epipolar attention block - Cross-view interaction among feature maps - Compatible with various base diffusion models Tech stack: Python Why it matters: EpiDiff offers a novel approach to enhancing multi-view synthesis, making it compatible with a wide range of diffusion models and improving cross-view interaction.

### Overview: SimGen is a simulator-conditioned generative model designed to create realistic driving scenes. It addresses the gap between simulation and reality by leveraging the appearance and layout diversity of web videos and synthesized data. ### Key Features: - **Simulator-Conditioned Generative Models**: SimGen uses generative models to create realistic driving scenes, conditioned on simulated data. - **Appearance and Layout Diversity**: It generates diverse driving scenes by incorporating appearance and layout diversity from web videos and synthesized data. - **Cascade Diffusion Paradigm**: SimGen follows a cascade diffusion paradigm to generate realistic driving scenes, guided by layout cues from simulators and text prompts. ### Tech Stack: Python ### Why It Matters: SimGen addresses the Sim2Real gap by using simulator-conditioned generative models to create realistic driving scenes, making it valuable for research and practical applications in the field of driving simulation and generative modeling. ### Repository: metadriverse/SimGen ### Primary Language: Python ### GitHub Stars: 142 ### GitHub Description: Simulator-conditioned Driving Scene Generation

### Overview: SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning In this project, we propose SpAtten, an innovative algorithm-architecture co-design that leverages token sparsity, head sparsity, and quantization opportunities to reduce the attention computation and memory access. By pruning unimportant tokens and heads, SpAtten significantly improves the efficiency of large language models like language understanding and generation tasks. ### Key Features: - **KV Token Pruning**: Prunes away unimportant tokens in the sentence to reduce attention computation. - **Head Pruning**: Removes unessential heads to further optimize model efficiency. - **Cascade Pruning**: Different from weight pruning, cascade pruning selects tokens and heads on the fly for efficient hardware support. - **KV Progressive Quantization**: Fetches MSBs only and performs computation; if confidence is low, fetches LSBs and recomputes attention outputs for memory reduction. ### Tech Stack: Scala The project is implemented using Scala, a powerful and efficient programming language that is well-suited for scientific computing and large-scale data processing. ### Why It Matters: SpAtten is a significant advancement in the field of large language models, offering substantial improvements in efficiency and resource utilization. By reducing the computational and memory requirements of attention mechanisms, SpAtten enables more effective and faster processing of natural language tasks, making it a valuable tool for researchers and developers working on advanced AI systems.

Overview: This project evaluates the performance of OpenAI's GPT-3.5 model on the Multistate Bar Examination (MBE) section. Key features: - Fine-tuning over GPT-3.5's zero-shot performance - Hyperparameter optimization and prompt engineering positively impact zero-shot performance - Achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam Tech stack: HTML Why it matters: This research suggests that an LLM will likely pass the MBE component of the Bar Exam in the near future, indicating significant potential for AI in legal licensing tasks.

### Overview: This project evaluates the Information Extraction (IE) capabilities of ChatGPT across various fine-grained tasks, including entity typing, named entity recognition, relation classification, relation extraction, event detection, and event argument extraction. The findings reveal that while ChatGPT struggles in standard IE settings, it excels in open IE tasks, particularly in providing high-quality and trustworthy explanations for its decisions. The project also highlights the issue of overconfidence leading to low calibration and demonstrates high faithfulness to the original text in most cases. ### Key Features: - **Entity Typing (ET):** Uses BBN and OntoNotes datasets. - **Named Entity Recognition (NER):** Uses CoNLL2003 and OntoNotes datasets. - **Relation Classification (RC):** Uses TACRED and SemEval2010 datasets. - **Relation Extraction (RE):** Uses ACE05-R and SciERC datasets. - **Event Detection (ED):** Uses ACE05-E and ACE05-E+ datasets. ### Tech Stack: Python ### Why it Matters: This project provides a comprehensive assessment of ChatGPT's performance, explainability, calibration, and faithfulness, contributing to the field of information extraction and AI research.

### Overview: This repository contains concise implementations of various generative models, focusing on pedagogical purposes. It includes bare-bones versions of β-VAEs, diffusion models, normalizing flows, consistency models, flow matching, and more. ### Key Features: - β-VAEs - Diffusion models - Normalizing flows - Continuous normalizing flows - Consistency models - Flow matching - Diffusion distillation - Discrete walk-jump sampling ### Tech Stack: Jupyter Notebook ### Why it Matters: These models serve as a practical introduction to various generative techniques, making complex concepts accessible through simple, minified implementations.

Overview: Spokestack is a Python library designed to easily integrate voice interfaces into Python applications, particularly for embedded systems, including Raspberry Pi and Django web applications. Key features: - Integrates voice interfaces into Python applications - Supports Raspberry Pi and Django web applications - Offers pre-configured profiles for various use cases - Provides fine-grained control over speech pipelines Tech stack: Python Why it matters: Spokestack simplifies the process of adding voice functionality to Python projects, making it accessible for developers working on embedded systems and web applications alike.

### Overview: This repository includes code and demos for the session "Unlock insights with AWS GenAI services" (re:Invent BOA303) at the [re:Invent 2023](https://reinvent.awsevents.com/). The demos showcase how RAG (Retrieval-Augmented Generation) can enhance AI models by leveraging external data sources to provide context-aware answers and unlock insights. ### Key features: - Deploying Llama2 with Amazon SageMaker, CodeWhisperer, and Kendra for data retrieval. - Using Claude 2 with Amazon Bedrock and Vector DB for data retrieval. - Demonstrating the integration of GenAI models with external data sources to enhance their capabilities. ### Tech stack: Jupyter Notebook ### Why it matters: These demos provide practical insights into how AWS GenAI services can be effectively integrated with existing data infrastructure, enhancing the capabilities of AI models to provide more accurate and contextually relevant answers.

Overview: Spatial-SSRL, a self-supervised reinforcement learning framework, has been released for enhancing spatial understanding in large language models (LLMs). This innovative approach optimizes Qwen2.5-VL-7B, significantly improving its performance across seven spatial benchmarks in both image and video settings. Key features: - Spatial-SSRL leverages lightweight, easily integrable tools to enhance large language models' spatial understanding. - It can be seamlessly extended to various pretext tasks, including image and video understanding. - The framework requires only ordinary RGB and RGB-D images for five formulated tasks. Tech stack: Python Why it matters: Spatial-SSRL provides a novel and effective method for improving the spatial capabilities of large language models, making them better equipped to understand and interpret spatial information in real-world applications.

Overview: Markify is a tool that converts files into Markdown, making it easier for RAG and LLM to understand your documents. It supports various formats like PDF, Word, PPT, Excel, and more, with a focus on high-quality PDF parsing. Key features: - Converts documents, multimedia, web pages, and compressed files into Markdown. - Supports multiple PDF parsing modes: fast, advanced, and cloud. - Integrates with APIs and Streamlit for easy conversion and integration. Tech stack: Python Why it matters: Markify simplifies the process of making documents accessible to AI models, enhancing their ability to understand and process information effectively. Repository: KylinMountain/markify Primary language: Python GitHub stars: 141 GitHub description: Convert files into markdown to help RAG or LLM understand, based on markitdown and MinerU, which could provide high quality pdf parser.

### Overview: DKG V10 is a decentralized knowledge graph for multi-agent AI systems, providing a shared, verifiable memory layer that agents can collaborate on. This system ensures that knowledge is owned and verified by the publisher, and it supports provenance tracking on-chain. ### Key Features: - **Decentralized Knowledge Graph (DKG)**: A shared, verifiable memory layer for multi-agent AI systems. - **Provenance Tracking**: Knowledge is committed to the blockchain, ensuring integrity and history. - **Context Graphs**: Agents can keep knowledge private, open it to specific peers, or back it with on-chain signatures for group governance. - **Knowledge Assets**: Units of published knowledge committed to the blockchain, ensuring integrity and history. - **Working Memory**: Writes knowledge in the cheapest, most private layer first and promotes it outward as it matures. ### Tech Stack: TypeScript ### Why It Matters: DKG V10 provides a robust, decentralized memory layer that supports collaboration and collaboration governance, ensuring that knowledge is owned and verified by the publisher. This system is essential for multi-agent AI systems that require shared, verifiable, and controllable knowledge. ### Repository: OriginTrail/dkg ### GitHub Stars: 141 ### Why Knowledge Assets Enable Trust: DKG V10 packages knowledge into Knowledge Assets, which are committed to the blockchain, ensuring that knowledge is owned and verified by the publisher, and that later updates cannot silently rewrite its history. ### Why Context Graphs Enable Collaboration: DKG V10 supports Context Graph

### Overview: Video-Bench is a comprehensive benchmark and toolkit designed to evaluate Video-based Large Language Models (Video-LLMs). It features a three-level assessment system that evaluates models in video-exclusive understanding, prior knowledge incorporation, and video-based decision-making abilities. ### Key Features: - **Comprehensive Evaluation**: Measures models across various aspects of video understanding, knowledge integration, and decision-making. - **User-Friendly Toolkit**: Provides a toolkit that streamlines the performance assessment of Video-LLMs using datasets and QA pairs. - **Extensive Experiments**: Conducts experiments to evaluate prominent Video-LLMs, analyzes limitations, and proposes improvements. ### Tech Stack: Python ### Why It Matters: Video-Bench is a crucial tool for evaluating the capabilities of Video-LLMs, providing a standardized benchmark that helps researchers and developers understand and improve the performance of these models in handling video-related tasks.

Overview: This repository provides a comprehensive example of developing a GenAI application on SAP Business Technology Platform (SAP BTP) using Retrieval Augmented Generation (RAG) and LangChain integration. It includes single and multitenant versions, showcasing the integration of LLMs via SAP AI Core, LangChain in CAP, and advanced techniques like RAG. Key features: - Single-tenant application - Multitenant SaaS solution - Integration of LLMs via SAP AI Core - Use of LangChain in SAP CAP - Advanced techniques like RAG and embeddings for enhanced insights Tech stack: TypeScript Why it matters: This repository offers a practical guide for developers aiming to integrate advanced GenAI solutions on SAP BTP, focusing on enhancing customer support and automation through email insights and automated responses.

Overview: DALL-Eval is a project that evaluates the reasoning skills and social biases of text-to-image generation models, specifically focusing on DALL-E, DALLE, and Stable Diffusion. It includes visual reasoning, social bias, image quality, and image-text alignment evaluations. Key features: - Visual Reasoning: Evaluates the ability of models to reason about visual concepts. - Social Bias: Measures the presence of gender and skin tone biases in image generation. - Image Quality: Assesses the quality of generated images using the FID score. - Image-Text Alignment: Evaluates the alignment between images and text descriptions using CLIP. Tech stack: Jupyter Notebook Why it matters: This project is crucial for understanding the limitations and biases inherent in text-to-image generation models, providing valuable insights for improving the ethical and effective use of these technologies. Repository: j-min/DallEval Primary language: Jupyter Notebook GitHub stars: 141 GitHub description: DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (ICCV 2023)

### Overview: OpenCinema is an open-source video studio designed for AI agents. It allows developers to plan, generate, and edit short drama videos on a local-first canvas, using their own models and short drama video. The platform connects storyboards, assets, 3D previs, generation, and editing, ensuring every step taken by hand is preserved in the agent's workflow. ### Key Features: - **Previs First, Then Generate, Then Fix Only What Needs Fixing:** Block out movement and cameras in 3D, review each shot, and replace a single shot without redoing the film. - **One Canvas for the Whole Film:** Scripts, references, previs, generation, review, and the edit plan are nodes on one canvas, and edges are reference relationships. - **Your Agent Works on the Same Canvas:** Describe what you want in Codex, Claude Code, or any MCP client. The agent reads and writes nodes through public MCP tools, and you can open an editor and change things by hand at any time. Both paths share one command path, one revision, and one undo history. - **Swap the Generation Backend:** Cloud models, local services, and your existing ComfyUI workflows all plug into the same generation node. - **Everything Stays Local:** The Go server listens on loopback only, a project is a plain folder on your disk, and credentials stay in local settings. Agents only ever see “configured” or “not configured.” ### Tech Stack: TypeScript ### Why It Matters: OpenCinema simplifies the process of

Overview: MLX Flash is a lightweight, high-performance solution for running large models on Apple Silicon Macs without additional quantization. It allows models up to 400B in size to be streamed directly from SSDs, enabling seamless integration with the MLX ecosystem. Key features: - Runs models larger than RAM on base-spec Macs - Uses native precision for models - Supports any model size (30B, 70B, 400B+) - Provides high-quality, production-grade integration layer Tech stack: Python Why it matters: MLX Flash enables efficient execution of large models on Apple Silicon hardware, addressing performance bottlenecks and providing a seamless experience for developers working with MLX. Repository: matt-k-wong/mlx-flash Primary language: Python GitHub stars: 140 GitHub description: Flash weight streaming for MLX: run massive models larger than your RAM on Apple Silicon.

### Overview: OmniHuman is an AI model designed to understand both text and humanoids, providing a comprehensive solution for various applications. ### Key Features: - **Text Understanding**: The model can interpret and analyze written text. - **Humanoid Recognition**: It can recognize and understand humanoids in images and videos. - **Video Generation**: The model can generate videos based on provided text or images. ### Tech Stack: - Python ### Why It Matters: OmniHuman is a powerful tool for applications that require text and image processing, such as virtual assistants, security systems, and more. ### Repository: - **Repository**: [mdsrqbl/omnihuman](https://github.com/mdsrqbl/omnihuman) - **Stars**: 141 - **Description**: AI model that understands text & humanoids. ### Installation: ```bash pip install omnihuman ``` ### Usage: ```python import omnihuman import PIL.Image text = "Raise both hands and clap overhead." frames = omnihuman.read_frames("path/to/image.jpg") # (1, channels, height, width) # frames = omnihuman.read_frames("path/to/video.mp4") # (n_frames, channels, height, width) # model = omnihuman.OmniHuman() # frames = model.generate_video(text, frames) PIL.Image.fromarray(frames[-1].permute(1,2,0).numpy()).show() ``` ### Citation & License: ```bibtex @misc{md

Overview: VivaDicta is an iOS and watchOS speech-to-text app powered by Apple Foundation Models, WhisperKit, NVIDIA Parakeet, and 20+ AI providers. It offers on-device transcription, chat with your notes, and a flexible AI voice keyboard. Key features: - On-device transcription using WhisperKit and Parakeet - Cloud-based transcription with OpenAI, Groq, Cohere, Deepgram, ElevenLabs, Gemini, Mistral, Soniox, and more - 100+ languages with automatic detection - Speaker-separated diarization - Filler word removal, paragraph formatting, and custom word replacements - 40+ AI presets across categories - AI assistant for asking questions, fact-checking, and summarizing - Auto-translation for speaking in one language and getting output in another - Create custom presets and mark favorites for quick access Tech stack: Swift, Xcode 26+, iOS 18.0+, watchOS 10.0+, Swift 6.2, Apple Foundation Models, WhisperKit, NVIDIA Parakeet, 20+ AI providers, CloudKit, OAuth sign-in, CLI agent bridge, and on-device RAG with chat. Why it matters: VivaDicta provides a flexible and powerful solution for dictating text on iOS and watchOS devices, with on-device transcription and AI processing capabilities. Its AI voice keyboard and chat feature make it a versatile tool for note-taking and communication.

Overview: HackGPT is a powerful and customizable AI chatbot interface designed specifically for developers, offering a chat-like experience with advanced features like fine-tuning, set-up prompts, knowledge cut-off, and on-demand model switching. Key features: - ChatGPT-like Interface - Fine-tuning - Set-up Prompt Selection - Knowledge Cut-off - On-demand Switching Tech stack: Python Why it matters: HackGPT provides a streamlined and customizable way for developers to interact with AI, enhancing their hacking and development processes with advanced features tailored to their needs.