Projects

Overview: This project explores multimodal document understanding by combining vision-language models with text retrieval to answer questions over PDFs containing figures, tables, and complex layouts. Key features: Baseline retrieval, VLM evaluation pipeline, fused reranker, LLaVA answer generation. Tech stack: LLaVA-1.6 (Mistral-7B variant), LangChain, LlamaIndex, pdf2image, Tesseract, accelerate, PyTorch, transformers (HF). Why it matters: This project aims to improve the handling of technical PDFs by maintaining visual context and querying both modalities, aiming to provide better answers to questions over complex layouts. Repository: 0xzall/multimodal-doc-qa Primary language: Python GitHub stars: 13 GitHub description: Multimodal document QA: vision + retrieval over PDFs (LLaVA + LlamaIndex) Read this for the actual story: The README is mostly here so GitHub has something pretty. The honest version of what's working, what isn't, and why I made each decision lives in [RESEARCH-LOG.md].

Overview: code-llm-completion is a focused training & serving stack for fine-tuning open code LLMs into fast, IDE-grade completion models, with support for low-resource languages. Key features: - Fine-tuning of StarCoder2-7B and DeepSeek-Coder-6.7B models - Support for low-resource languages like Indonesian PHP/Laravel codebases, Lua game scripts, and in-house DSLs - Focus on low latency, high acceptance rate, and easy deployability Tech stack: Python Why it matters: This project aims to provide developer tools for the on-prem inference market, focusing on low-resource languages and fast, easy-to-deploy completion models. Repository: 0xzuky/code-llm-completion GitHub stars: 13 GitHub description: AI/ML research project on AMD ROCm

Overview: Longbow is a high-performance vector engine built on Apache Arrow, designed for modern AI/Agentic workloads. It leverages zero-copy data paths, SIMD optimizations, and advanced storage backends to deliver sub-millisecond latency. Key features: - Built on Apache Arrow for zero-copy data transfer. - Consistent hashing and gossip-based membership (SWIM protocol). - Optional `io_uring` WAL backend for high-throughput ingestion. - NUMA-aware memory allocation and SIMD vector distance calculations. - Resilient Go SDK for transparent request routing. Tech stack: Go Why it matters: Longbow is designed for high throughput and low latency, making it ideal for modern AI/Agentic workloads where performance is critical. Repository: 23skdu/longbow GitHub stars: 13 GitHub description: Apache Arrow Flight clustered vector engine for high throughput Agent memory sharing

### Overview: BDM (Bridge Diffusion Model) is a novel method designed to develop non-English language-native TTI (text-to-image) models that are compatible with the English Stable Diffusion community. This approach addresses the inherent language-related biases in existing English TTI models, ensuring a more balanced representation of diverse linguistic communities. ### Key Features: - **Language-Neutral Representation**: BDM generates images that are consistent with Chinese language semantics, making it suitable for Chinese-native users. - **Compatibility with SD Community**: The model is designed to be seamlessly integrated with the English Stable Diffusion ecosystem, ensuring compatibility and ease of use. - **Incorporation of External Translation**: By fine-tuning with Chinese-native data, BDM can assimilate Chinese-native language semantics without compromising compatibility with the English community. ### Tech Stack: Python ### Why it Matters: Developing non-English language-native TTI models is crucial because existing English models often exhibit language-related biases. By bridging these biases, BDM ensures a more balanced and diverse representation of linguistic communities, particularly those in non-English environments. This approach not only enhances the inclusivity of AI-generated content but also fosters a more collaborative and innovative ecosystem for AI development. ### Repository: 360CVGroup/Bridge_Diffusion_Model ### GitHub Stars: 13 ### Primary Language: Python ### GitHub Description: Chinese-native image generation while compatible with SD eco-system, 1st-gen, AAAI2025

### Overview: RWKV-Server 是一款高性能的大型语言模型推理服务，专为高并发场景设计。它支持动态扩缩容，兼容OpenAI API，采用CUDA优化，适用于低显存部署的大模型推理。RWKV-Server 提供了自动入队、动态容量管理和无头模式的压力测试脚本，适用于多种使用场景。 ### Key features: - **高性能推理**：支持2.9B和7.2B模型，峰值吞吐量分别为11,081 tok/s和6,373 tok/s。 - **动态扩缩容**：支持从256个任务开始，逐步减少到1个任务，以适应不同规模的部署需求。 - **兼容OpenAI API**：支持与OpenAI API的无缝集成，提供流畅的对话体验。 - **低显存部署**：适用于低显存环境，有效降低显存使用。 - **无头模式**：支持Web UI和无头模式，方便管理和监控。 ### Tech stack: Python ### Why it matters: RWKV-Server 是一个高性能、低显存、易扩展的推理服务，适用于各种需要大规模模型推理的场景，如自然语言处理、对话系统等。它的兼容性、动态扩缩容和低显存部署特性使其成为此类应用的理想选择。

### Overview: LLM Energy Lab is a web application and API designed to benchmark and compare the energy consumption, emissions, and behavior of Large Language Models (LLMs) across different inference configurations, middleware scaffolds, and post-training implementations. It provides insights into the impacts of various factors on LLM's raw behaviors and energy footprint, offering a comprehensive view of LLM's performance and environmental impact. ### Key features: - **Energy Testing Lab (primary)**: - **Per-input/output-token energy metrics**: Metrics are normalized by input or output, with Wh/1000 e-tokens. - **Live Hardware Tests (RAPL)**: Snapshot-based cumulative energy counters with start/split/end snapshots for Prefill and Decode energy. - **Estimated Benchmark Tests**: Ability to switch and create benchmarks, recalculate session with new benchmarks. - **Middleware Injections and Configuration**: Custom and Frontier System Prompts, Conversation Context, free-form injections, and temperature toggles. - **RAPL Batch Runner**: Multiple live runs using current UI query + injections with CLI logs and stop support; results table appears in Test Results. - **Real-time streaming**: Token streaming over WebSocket, supports reasoning "thinking" streams when present. - **LLM Behaviour Lab (WIP)**: - **Multi-model comparison panes with per-pane controls**. - **Streaming outputs, token counts (prompt/completion), and TPS**. ### Tech stack: Python ### Why it matters: LLM Energy Lab provides a valuable tool for developers and researchers to understand

### Overview: The Daily Paper Club is a collaborative learning community dedicated to understanding multi-modality models and their applications. Hosted at Agora every 10 PM NYC time, the club aims to foster a deep understanding of AI and mathematics through discussions on topics such as multi-modality reasoning, generation, and swarm collective intelligence. ### Key Features: - **Weekly Discussions**: Topics are covered every Thursday at 10 AM Eastern time. - **Moderation by AI**: AI-driven moderation ensures a safe and engaging environment. - **Community Engagement**: Members contribute to the community through discussions, sharing insights, and offering solutions. ### Tech Stack: The project uses Python for organizing and managing the discussions. ### Why It Matters: Understanding the intersection of various modalities in AI and mathematics will lead to breakthroughs in AI applications, making this community essential for the future of AI. ### Repository: The repository is hosted on GitHub and is open for contributions. It has received 13 GitHub stars, indicating a growing interest in the project. ### Primary Language: The primary language is not specified in the README excerpt. ### GitHub Description: The repository for the exclusive Daily Paper Club hosted at Agora every 10 PM NYC time at this discord: https://discord.gg/Gnzh6dnzyz

### Overview: ThingiPano is a comprehensive dataset containing metadata, images, and panoramic renders of 3D printing designs, designed to facilitate large-scale analysis and support design reuse in 3D printing. ### Key Features: - **Metadata**: Publicly available metadata from Thingiverse (up to 2018), including 283,873 users and 1,017,687 designs. - **Images**: 1,816,295 images associated with designs. - **Panoramic Renders**: 1,052,017 3-view panoramic 3D depth-map renders of 3D files, available for both Thingiverse and Shapenet. - **Validation Data**: Panoramic renders for Shapenet, along with Matlab source code for generating these renders. - **Compressed Sizes**: - [Thingiverse Images]: 97.5 GB - [Rendered Panoramas]: 27.6 GB - [Design Metadata]: 717 MB - [User Metadata]: 28.1 MB - [Shapenet Panoramas]: 1.34 GB ### Tech Stack: MATLAB ### Why It Matters: ThingiPano is a valuable resource for researchers and designers interested in exploring design reuse, 3D printing metadata, and large-scale analysis of 3D files. The dataset supports qualitative analysis and machine-learning applications, making it a cornerstone for advancing the field

Overview: This project fine-tunes T5, a large language model, on a dataset of varied text to improve its performance on NLP tasks such as Text-to-SQL. Key features: - Fine-tuning T5 on a large dataset of text - Pre-trained on diverse text datasets including Wikipedia, news articles, and books - Evaluated on the SPIDER dataset for Text-to-SQL tasks - Improved performance by 16% from 80% to 93% - Implemented rule-based post-processing to further improve scores Tech stack: Jupyter Notebook Why it matters: This project showcases the effectiveness of fine-tuning a large language model on a diverse dataset, achieving significant improvements in performance for Text-to-SQL tasks.

Overview: This repository contains the source code and results for the paper "Unified-modal Salient Object Detection via Adaptive Prompt Learning" presented at TCSVT 2025. Key features: - Unified-modal Salient Object Detection framework - Baseline SOD framework comparison - Performance metrics for RGB, RGB-D, and RGB-T modalities Tech stack: Python Why it matters: This project offers a comprehensive solution for salient object detection across different modalities, demonstrating the effectiveness of adaptive prompt learning in unified modal detection. Repository: Angknpng/UniSOD GitHub stars: 13 GitHub description: Unified-modal Salient Object Detection via Adaptive Prompt Learning

### Overview: This repository contains the code for the paper "Generating Diverse and Meaningful Captions: Unsupervised Specificity Optimization for Image Captioning" by Annika Lindh et al., presented at ICANN 2018. The project focuses on improving image captioning models to generate more diverse and meaningful captions without human supervision. ### Key Features: - **Unsupervised Specificity-Guided Optimization**: The code optimizes image captioning models to encourage diversity in the generated captions. - **Image Captioning Model**: The repository includes a specific image captioning model that is trained unsupervised to generate captions that are both diverse and meaningful. - **Unsupervised Learning**: The project leverages unsupervised learning techniques to enhance the diversity of captions. ### Tech Stack: - Python 2.7 - PyTorch 0.3.1 - torchvision - h5py 2.7.1 - sklearn 0.19.1 - scipy 1.0.1 - scikit-image (skimage) 0.13.1 - ijson ### Why It Matters: The project aims to address the challenge of generating captions that are both diverse and meaningful, which is crucial for applications such as image retrieval and image captioning. By optimizing the model to focus on diversity, the project seeks to improve the quality of generated captions, making them more useful and relevant for various applications. ### Repository: The repository is hosted on GitHub and is described as "Generating Diverse and Meaningful Captions

### Overview: This guide provides a comprehensive setup and configuration for a 2-node NVIDIA DGX Spark cluster, optimized for distributed training and inference tasks. The cluster is equipped with 200 Gb/s direct fabric networking and NVIDIA's latest NVMe-TCP shared storage, ensuring high performance and efficiency. ### Key features: - Distributed training with DDP scaling efficiency of 93.5% - CUDA inference with EXO model serving - NCCL tuning for optimal performance - 200 Gb/s direct fabric networking - Grace Blackwell GB10 DGX Spark systems connected point-to-point - NCCL configuration and GID index setup - End-to-end distributed training pipeline - GPU Direct RDMA explanation and solution ### Tech stack: Python ### Why it matters: This guide is essential for developers and researchers looking to deploy and optimize distributed training and inference tasks on the DGX Spark cluster, ensuring high performance and efficient resource utilization.

Overview: This project utilizes the Stable Diffusion Pipeline to generate images from text prompts, offering a simple GUI application for users to input prompts and view generated images. Key features: - Generates images based on user prompts - Uses the Stable Diffusion Pipeline from the diffusers module - Provides a Python GUI application - Supports text-based image generation Tech stack: Python, Jupyter Notebook Why it matters: This project simplifies the process of generating images from text prompts, making it accessible for a wide range of users, including artists, designers, and anyone interested in exploring the intersection of text and image generation.

Overview: Engram is a decentralized vector database on Bittensor, designed to store and retrieve text, images, and documents securely and permanently. It leverages content-addressed semantic memory to ensure each piece of knowledge has a unique identifier, eliminating the need for a central authority. Key features: - Content-addressed: Each piece of knowledge is uniquely identified by its CID, ensuring consistency across miners. - Decentralized: Replicated across competing miners on Bittensor subnet 450, providing redundancy and fault tolerance. - Permanent: Stores binary files (images, PDFs) on Arweave, ensuring permanence and durability. - Private: X25519 hybrid encryption with differential privacy noise, and namespace isolation at every endpoint. - Incentivized: Miners earn TAO for storing and serving vectors, incentivizing the network. - Verifiable: Merkle commitment over the full memory corpus with HMAC challenge-response proofs for transparency. - Mobile: Mine from your phone via managed Akash Network nodes, with pay-as-you-go usage on Base with USDC. Why it matters: Engram replaces a single-vendor vector database with a decentralized, incentivized network, ensuring the security and reliability of AI memory without relying on a single point of failure. Tech stack: Python Why it matters: Engram is a Bittensor subnet that turns text, images, and documents into permanently stored, content-addressed memories, providing a decentralized and private solution for AI memory. GitHub stars: 13 Why it matters: The repository has a moderate number of

### Overview: Qwen3.8-Flash-Next is a high-performance AI model optimized for running on a single RTX 3090 with 64 GB of RAM. This setup allows for up to 2,100 tokens per second prefill and 47 tokens per second decode, making it ideal for tasks that require quick response times. ### Key Features: - **GPU:** One NVIDIA RTX 3090 with 24 GB of memory. - **RAM:** 64 GB of system RAM. - **Runtime:** vLLM with hot MoE experts on the GPU and cold experts computed on the CPU. - **Int8 KV Cache:** Utilizes INT8 KV cache to optimize memory usage. - **Docker:** Runs in a Docker container for easy deployment. - **API Compatibility:** Compatible with OpenAI's API. ### Tech Stack: - Python-based runtime. ### Why It Matters: This setup provides a powerful and efficient way to run Qwen3.8-Flash-Next, leveraging the full potential of the RTX 3090 GPU and 64 GB of RAM to achieve high performance and fast response times. The use of INT8 KV cache and hot MoE experts ensures optimal performance and resource efficiency. ### Repository: Repository: DominikBucko/qwen38-flash-next-3090 GitHub Stars: 13 GitHub Description: Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) +

### Overview: This repository is a comprehensive guide to Large Language Models (LLMs), covering both theoretical and practical aspects of these models. It provides insights into the development, evaluation, and applications of LLMs, focusing on their capabilities and limitations. ### Key Features: - **Theoretical and Practical Aspects**: Covers both the theoretical foundations and practical implementation of LLMs. - **Statistical Language Modeling**: Explains N-gram models and their applications. - **Neural Modeling**: Discusses neural language modeling techniques (NLM) and their importance. - **Evaluation of LMs**: Details methods for evaluating the performance of language models. - **Transformer-Based Language Models**: Focuses on the advancements in transformer-based models. - **Generative Configuration Parameters**: Provides guidance on configuring parameters for inference. - **Computational Challenges and Quantization**: Addresses the computational challenges and techniques for efficient model deployment. - **Prompt Engineering**: Covers techniques for in-context learning and auto prompt engineering. - **Fine-Tuning**: Discusses the process of fine-tuning models for specific tasks. - **RAG (Retrieval-Augmented Generation)**: Explains how to incorporate retrieval in language models. - **LangChain**: Demonstrates how to build applications using LangChain. - **LLM Model Evaluation**: Provides evaluation metrics and techniques for LLMs. - **AI Agents**: Introduces the concept of AI agents and their integration with LLMs. - **References**: Offers additional resources for further reading. ### Tech Stack: The primary language used in this repository is Jupyter

### Overview: This repository is a comprehensive guide to system design for the AI era, focusing on modern patterns and infrastructure for large language models (LLMs), vector databases, and agent architectures. ### Key Features: - **Modern Patterns**: Covers AI infrastructure patterns that have been deployed in production after 2022. - **Original Synthesis**: Provides original explanations and diagrams instead of aggregating links. - **Reference Manual**: A free, open-source reference manual designed for software engineers preparing for system design interviews or building production systems using LLMs, agents, and vector databases. ### Tech Stack: The repository uses Markdown for documentation and Mermaid for flowcharts. ### Why It Matters: - **Beyond the Primer**: Adds AI infrastructure patterns that have reached production after 2022. - **Original Synthesis**: Provides original explanations and diagrams instead of aggregating links. - **Reference Manual**: A free, open-source reference manual designed to be maintained indefinitely. ### Who This Is For: - **Software Engineers**: With 2+ years of experience, preparing for system design interviews at L4–Staff level or building production systems using LLMs, agents, and vector databases. - **Not For**: Total beginners, ML researchers, or those seeking code tutorials. ### Structure Overview: ```mermaid flowchart LR F[Foundations] --> A[AI Era] A --> M[Modern Patterns] M --> C[Case Studies] C --> I[Interview Guide] ``` ### Table of Contents: - **Foundations** - [

Overview: This project implements four different multi-modal neural machine translation models based on the PyTorch port of OpenNMT, an open-source neural machine translation system. Key features: - Four different multi-modal neural machine translation models - Uses PyTorch and torchtext for implementation - Based on the research papers by Iacer Calixto - Supports PyTorch version 1.3.1 and torchtext version 0.2.3 - Includes pre-processing scripts for text and image data Tech stack: PyTorch, torchtext Why it matters: This project provides a practical demonstration of multi-modal neural machine translation models, showcasing their effectiveness in handling multimodal data for translation tasks.

### Overview: Kedro Serving is a plugin for the Kedro framework that allows developers to serve Kedro Pipelines as an API. This plugin simplifies the deployment and accessibility of complex data pipelines, making it easier to share and integrate with other applications. ### Key Features: - **Serves Kedro Pipelines as APIs**: Easily expose Kedro pipelines as RESTful APIs, enabling seamless integration with other systems. - **Integrated with Kedro**: Seamlessly integrates with the Kedro ecosystem, ensuring compatibility and ease of use. - **Flexible Configuration**: Configurable settings for API endpoints, authentication, and more. ### Tech Stack: Python ### Why It Matters: Kedro Serving simplifies the deployment and accessibility of Kedro Pipelines, making it easier to share and integrate with other applications. This plugin is particularly useful for developers working on complex data pipelines who need to expose their workflows as APIs for external use.

### Overview: This repository contains the evaluation code for a paper published in *npj Digital Medicine* titled "*An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing for Information Extraction*". The datasets used in the paper include the NCBI-Disease, 2018 n2c2, and GAD datasets for named entity recognition (NER) and relation extraction (RE). ### Key features: - Federated Learning Implementation - Comparison of LLMs with Blue BERT and GPT 2 trained with federated learning - Results show improvements in F1 scores for NER and RE tasks compared to Blue BERT and GPT 2 ### Tech stack: Jupyter Notebook ### Why it matters: This repository provides a comprehensive evaluation of LLMs in the context of biomedical NLP, demonstrating the effectiveness of federated learning in improving information extraction tasks.