Projects

### Overview: RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark (ECCV 2024) introduces a novel dataset and benchmarking framework for radar-based human pose estimation and localization. This project aims to address the limitations of traditional RGB-based methods in real-world applications by leveraging the unique attributes of radar technology. ### Key features: - **4D Radar Tensor Dataset**: Contains 4D radar tensors, LiDAR point clouds, and RGB images for diverse actions across 240 sequences. - **Annotation Process**: Accurately labels 3D human skeletons using RGB images and LiDAR point clouds. - **HRRadarPose Architecture**: A single-stage architecture that extracts high-resolution representations of 4D radar tensors for improved human keypoint estimation. ### Tech stack: Python ### Why it matters: RT-Pose addresses the limitations of traditional RGB-based methods by providing a more privacy-preserving and through-wall recognition alternative. The dataset and benchmarking framework enable researchers to evaluate the performance of radar-based HPE methods in real-world scenarios, making it a valuable resource for advancing the field of human pose estimation and localization.

Overview: This library provides state-of-the-art text representations for natural language processing tasks, focusing on the Polish language. Key features: - Text classification - Sequence labeling Tech stack: Python Why it matters: It is an initial version of the library focusing on the Polish language, aiming to facilitate the development of NLP tasks in the fastest possible manner. Repository: CLARIN-PL/embeddings Primary language: Python GitHub stars: 37

Overview: DrPlanner is a framework designed to diagnose and repair motion planners for automated vehicles using large language models. It has been accepted by RA-L 2024 and is available on Arxiv. Key features: - Diagnosis and repair of motion planners - Utilizes LLMs to improve as they scale - Demonstrated with an initial and final planner comparison Tech stack: Python Why it matters: DrPlanner offers a novel approach to enhancing motion planner performance through the integration of large language models, showcasing the potential of AI in autonomous vehicle technology.

### Overview: V.I.S.O.R. is a Go-based voice assistant in development, designed to be used on both desktop and Android devices. ### Key Features: - **Desktop Client**: Runs 24/7 on Raspberry Pi, managing tasks like RSS feed notifications and email sending. - **Server**: Runs on a Raspberry Pi, handling all operations and tasks. - **Communication**: Uses the GPT Communicator module for natural language processing. - **Email Sender**: Sends emails based on notifications. - **RSS Feed Checker**: Checks for news updates. - **System Information**: Provides system and hardware checks. - **Speech Recognition**: Uses the Festival speech recognition engine on Linux. - **Smart Check**: Checks system and hardware health. - **Webhook Integration**: Integrates with webhooks for additional functionality. ### Tech Stack: Go ### Why it Matters: V.I.S.O.R. is a versatile and robust voice assistant designed to be accessible and reliable across various operating systems, making it suitable for both desktop and mobile use.

### Overview: A fast parallel implementation of the "CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition" by George0828Zhang, featuring a PyTorch-based solution for end-to-end speech recognition. This implementation is optimized for performance and parallel processing. ### Key Features: - **Efficient Parallel Processing**: Utilizes PyTorch for parallel computation, significantly speeding up the process. - **Continuous Integrate-and-Fire (CIF) Architecture**: Implements the CIF algorithm for speech recognition, providing a continuous firing mechanism. - **Integration and Scattering**: Combines weighted source features using `cumsum` and `scatter` operations, ensuring accurate integration and delayed firing. - **Beta Parameter Handling**: Allows for flexible handling of boundary conditions in the CIF algorithm. ### Tech Stack: - Python ### Why It Matters: This implementation offers a fast and efficient solution for speech recognition tasks, making it suitable for real-time applications and large-scale deployments. The use of PyTorch ensures compatibility with existing speech recognition frameworks and accelerates performance through parallel processing. ### GitHub Stars: 37 ### Repository: - Repository: [George0828Zhang/torch_cif](https://github.com/George0828Zhang/torch_cif) - Primary Language: Python - GitHub Stars: 37

Overview: DataSpace is a benchmarking project designed to evaluate data agents for verifiable analytics over heterogeneous workspaces. It includes 410 tasks with 6 modalities, supports cross-language workspaces, and uses exact and semantic evaluation. Key features: - 410 tasks - 6 modalities (CSV, JSON, SQLite, Markdown, PDF, video) - Cross-language workspaces - Header-invariant column alignment with type-aware value and row comparison - 60 public references with gold results and frozen configs for local end-to-end evaluation - 15.0 GB of context spanning structured data, long documents, and multimedia Tech stack: Python Why it matters: DataSpace provides a comprehensive benchmark for evaluating data agents, ensuring that they can handle diverse data formats, languages, and analytical tasks in a verifiable manner. This is crucial for developing robust and reliable data-driven applications.

### Overview: This project tracks the latest papers from the Arxiv, providing developers with real-time updates on the latest research in their field. ### Key features: - Automatic tracking of new papers - Detailed summaries and links to full articles - Integration with popular development tools - Customizable alerts for specific topics ### Tech stack: Node.js, Express, MongoDB ### Why it matters: Keeping up with the latest research is crucial for developers, and this project helps them stay informed quickly and easily.

Overview: ProVision is an instruction data generation system designed for large multimodal language models, synthesizing data through Python programs and state-of-the-art scene graph models. Key features: - Generates single-image and multi-image instruction data - Supports extendability by adding new data generators - Uses Python programs and scene graph models for data synthesis Tech stack: Python, Jupyter Notebook Why it matters: ProVision provides researchers and developers with a flexible and programmable method to generate instruction data for multimodal language models, enhancing the training and evaluation of these models.

Overview: This repository contains various implementations of generative models, including Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based models, and diffusion models. Key features: - Vanilla GANs - Conditional GANs (cGANs) - CycleGAN - Self-Attention GANs (SAGAN) - Vanilla VAE - Glow: Generative Flow with Invertible 1x1 Convolutions - DDPM: Denoising Diffusion Probabilistic Models - SR3: Image Super-Resolution via Iterative Refinement, Diffusion model applied to Super Resolution Task Tech stack: Jupyter Notebook Why it matters: This repository provides a comprehensive collection of generative models, making it an excellent resource for developers looking to implement and experiment with various generative techniques in their projects.

### Overview: ReVolta is a high-performance inference server optimized for multi-GPU NVIDIA V100 servers, designed to run Qwen3.8 Flash-Next with advanced features like multimodal input, CPU/disk KV offload, and speculative decoding. It leverages the 1Cat-vLLM fork to achieve stable and fast inference, validated on four V100 GPUs with NVLink or P2P interconnects, ensuring reliable and scalable performance. ### Key Features: - **Stable and Fast Inference**: Optimized for multi-GPU NVIDIA V100 servers, ReVolta runs Qwen3.8 Flash-Next with high throughput. - **Multimodal Input**: Supports text, images, and video inputs for comprehensive processing. - **CPU/Disk KV Offload**: Offloads KV cache to CPU memory or disk for improved performance. - **MTP or DFlash2 Speculative Decoding**: Supports both MTP and DFlash2 for faster decoding. - **Experimental Whole-Table PLE Disk Option**: Recommended for better performance with 80 GiB of system memory. ### Tech Stack: - Python ### Why It Matters: ReVolta provides a reliable and efficient solution for running AI models on multi-GPU NVIDIA V100 servers, ensuring high throughput and stable performance. It is built on 1Cat-vLLM, a V100-focused fork of vLLM, making it well-suited for tasks requiring high parallelism and fast inference. ### Repository: - Repository: Leonccaa

Overview: This project is an AI chatbot with voice integration that can perform various tasks such as fetching events from Google Calendar, making notes, searching queries on Google, sending emails, opening applications, and providing weather updates. Key features: - Voice-based conversation - Google Calendar integration - Note-taking functionality - Google search and browser navigation - Email sending capability - Music and VS Code application opening - Direct Wikipedia search - Real-time weather report Tech stack: Python Why it matters: This project showcases the integration of AI with voice technology and various web applications, making it a versatile tool for automating tasks and enhancing user experience. Repository: Madhur215/Chatbot-cum-voice-Assistant Primary language: Python GitHub stars: 37

### Overview: WhisperX on NVIDIA Blackwell (DGX Spark / GB10 / GB200) Running legacy AI audio workloads (WhisperX, Pyannote) on next-generation NVIDIA Blackwell GPUs (SM_121) currently fails with: ``` nvrtc: error: invalid value for --gpu-architecture (-arch) ``` The NVRTC compiler doesn't recognize `sm_121` (Blackwell) yet, even though: - PyTorch can see the GPU - CUDA toolkit supports it - The hardware is ready Standard Python monkeypatching fails because Jiterator queries hardware architecture directly from C++, bypassing Python-level patches. ### Key Features: - **Architecture Spoofing:** Forces PyTorch's `get_device_capability()` to return `(9, 0)` (Hopper) instead of `(12, 1)` (Blackwell). - **JIT Bypass:** Patches `torchaudio` source code to avoid `.abs()` on complex tensors, which triggers the broken Jiterator path. ### Tech Stack: Python ### Why It Matters: This repository contains the "Blackwell Bridge Patch" - a surgical Dockerfile fix that allows SM_90 (Hopper) code to run natively on SM_121 (Blackwell) due to binary compatibility. The result is a speedup of 115x for 24-minute audio tasks, including transcription, alignment, and diarization. The health check and transcribe audio commands are provided for quick

Overview: Sophia AI Assistant is a Python-based desktop AI that can perform a variety of tasks such as answering questions, opening applications, browsing websites, and making calls via phone or WhatsApp. It uses the Hugging Face API for responses and offers activation via voice, text input, or a keyboard shortcut. Key features: - Voice Activation - Text Input - App Control - Website Navigation - Multimedia Control - Phone and WhatsApp Communication Tech stack: Python, HTML, CSS, JavaScript, Bootstrap, SQLite, Hugging Face API Why it matters: Sophia AI Assistant is designed to be versatile and extensible, making it a valuable tool for automating tasks and enhancing productivity.

### Overview: LinguaRelay is a real-time Windows system-audio translation captions tool that supports Chinese, Japanese, English, and Korean. It offers local-first, low-latency functionality and optional real-time or offline language model revision. ### Key features: - Real-time translation and subtitles for Windows system-audio - Capture of system output, selected processes, or microphone audio - Editable subtitles for recorded or imported audio/video - Supports Chinese, Japanese, English, and Korean in all directions - Manual source language design - Local-first, low-latency processing - Optional real-time or offline language model revision ### Tech stack: Python ### Why it matters: LinguaRelay provides a fast, local-first solution for real-time audio translation and subtitles, ideal for live and offline use. Its user-friendly interface and support for multiple languages make it a valuable tool for translators and language enthusiasts alike. ### Repository: MuzeAnisichael/LinguaRelay ### GitHub stars: 37 ### GitHub description: Real-time Windows system-audio translation captions for Chinese, Japanese, English, and Korean, with optional LLM revision.

### Overview: ProteinGPT is a comprehensive protein analysis system that leverages multimodal input, advanced encoding, and large language model integration to provide precise and contextually relevant responses. ### Key Features: - **Multimodal Input**: Accepts protein sequences (FASTA) and structures (PDB). - **Advanced Encoding**: Utilizes pre-trained sequence and structure encoders for feature extraction. - **LLM Integration**: Aligns protein representations with large language models for insightful responses. - **Extensive Training Data**: Trained on a large-scale dataset of 132,092 proteins with annotations and optimized using instruction tuning with GPT-4o. - **Modality Alignment & Instruction Tuning**: Two-stage training for enhanced understanding and response accuracy. ### Tech Stack: Python ### Why it Matters: ProteinGPT excels in protein-related tasks by providing higher semantic similarity scores and improved factual accuracy in closed-ended QA experiments, outperforming standard LLMs (GPT-4, LLaMA, Mistral) in protein-related tasks.

Overview: VITRIOL is an experimental local LLM inference project designed for old hardware, aiming to optimize modern AI models for systems that are typically too constrained to run them. It leverages various optimizations to make the most of limited resources, including direct memory access, sparse KV, MTP, and a VRAM governor. Key features: - RAM Shot streaming for MoE experts in page-locked host RAM. - Optimal inference on constrained hardware. - Custom copy engine for direct memory access. - Sparse KV eviction for context optimization. - MTP speculative decoding for depth per byte stretching. Tech stack: C++ and Rust. Why it matters: VITRIOL is a cutting-edge project that pushes the boundaries of AI inference on old hardware, potentially introducing optimizations for newer systems. Its experimental nature and the use of alchemical terminology make it a fascinating and potentially valuable resource for developers looking to optimize their hardware for AI performance.

Overview: HexGen is a framework designed for serving Large Language Models (LLMs) on heterogeneous decentralized clusters, specifically for the ICML 2024 conference. It supports hybrid model parallelism and request dispatch automation. Key features: - Comprehensive support for hybrid pipeline parallelism and tensor parallelism. - An automated mechanism for request dispatch. - Seamless integration with Ocf for efficient request coordination. Tech stack: Python Why it matters: HexGen simplifies the deployment and inference of LLMs on complex, heterogeneous environments, making it a valuable tool for researchers and developers working with decentralized computing clusters. Repository: Relaxed-System-Lab/HexGen GitHub stars: 38 GitHub description: [ICML 2024] Serving LLMs on heterogeneous decentralized clusters.

### Overview: DiffAffinity is a model designed to predict the effects of mutations on protein-protein binding affinity. This innovative approach leverages a side-chain diffusion probabilistic model to analyze the impact of mutations on protein structures, providing valuable insights into the molecular mechanisms of protein-protein interactions. ### Key Features: - **Model Type**: Predicts mutational effects on protein-protein binding via a side-chain diffusion probabilistic model. - **NeurIPS 2023 Poster**: Presented at NeurIPS 2023. - **GitHub Repository**: Repository hosted on GitHub at [TianZhuAI4S/DiffAffinity](https://github.com/oxcsml/geomstats). - **Core Tools**: Uses Jupyter Notebook and Python libraries such as JAX and Geomstats. - **Star Count**: 37 stars on GitHub. ### Tech Stack: - **Jupyter Notebook**: The primary development and presentation tool. - **JAX**: For efficient computation and numerical operations. - **Geomstats**: A library for geometric deep learning on manifolds. ### Why It Matters: - **Molecular Biology Insight**: Provides a deep understanding of how mutations affect protein-protein interactions. - **Practical Applications**: Useful for drug design, protein engineering, and understanding the mechanisms of protein-protein binding. ### Installation: - **Environment Setup**: Requires Anaconda and JAX installed. - **Dependencies**: `requirements.txt` lists the essential Python packages. - **Data Download**: Available datasets include PDB-REDO and SKEMPI

### Overview: AgentRunKit is a Swift 6 agent SDK designed for type-safe tools, streaming, and cloud/on-device inference via MLX on Apple Silicon. It provides a zero-dependency core, full sendable tools, asynchronous/await support, cloud + local capabilities, and is compatible with SPM. ### Key Features: - **Agent Loop with Iteration Limits and Token Budgets**: Configurable to manage run duration and token usage. - **Streaming with `AsyncThrowingStream` and `@Observable` SwiftUI Wrapper**: Stream events and tools in a type-safe manner. - **Type-Safe Tools with Compile-Time JSON Schema Validation**: Ensures type safety and error handling. - **Sub-Agent Composition with Depth Control and Streaming Propagation**: Allows for complex tool chains. - **Context Management**: Automatic context management for tools and sub-agents. ### Tech Stack: Swift 6.1 ### Why It Matters: - **Type-Safe Tools**: Ensures type safety and reduces runtime errors. - **Streaming**: Supports streaming events and tools for real-time applications. - **Cloud + Local**: Provides both cloud-based and local inference capabilities. - **On-Device MLX + Foundation Models**: Optimized for Apple Silicon devices. - **SPM Compatibility**: Compatible with SPM for easy integration and dependency management. ### Repository: Repository: Tom-Ryder/AgentRunKit ### GitHub Stars: 37 ### GitHub Description: Swift 6 agent SDK: type-safe tools, streaming, cloud + on-device inference via MLX on Apple Silicon

Overview: This project introduces TLB-VFI, a method for video frame interpolation that leverages temporal information extracted from both the pixel space and latent space. It aims to enhance the temporal consistency of the model and achieve state-of-the-art performance in various metrics compared to other recent methods. Key features: - Temporal-aware latent Brownian Bridge Diffusion - Improved temporal consistency through extraction of temporal information - Achieves state-of-the-art performance in LPIPS/FloLPIPS/FID metrics Tech stack: Python Why it matters: TLB-VFI is designed to address the challenges in video frame interpolation by incorporating temporal information from multiple perspectives, leading to better visual quality and temporal consistency, making it a significant advancement in the field of video processing and computer vision. Repository: ZonglinL/TLBVFI Primary language: Python GitHub stars: 37 GitHub description: ICCV 2025