Projects

### Overview: strixite is an LLM inference engine I developed from scratch using HIP for an AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next model. It achieves a decode speed of ~50 tokens/s with a context window of 512k tokens, making it highly efficient for real-time applications. ### Key Features: - Real-time performance with a context window of 512k tokens. - Decode speed of ~52-56 tokens/s, flat from 0 to 490k context. - Prefill speed of ~1,460 tokens/s. - 19 out of 19 tasks completed on terminal-bench-mini's core suite. ### Tech Stack: - C++ implementation using HIP. ### Why it Matters: - Demonstrates the potential for high-performance inference with minimal hardware resources. - Shows the feasibility of running a large model on a single chip using specialized hardware and optimized algorithms. ### Repository: - Repository: shawnshekari/strixite - GitHub Stars: 89 - GitHub Description: LLM inference engine written from scratch in HIP for AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next - ~50 tokens/s out to 492k context ### Release Speed: - Releases measured the same way: one request at a time through the server, a fresh prompt at each length. - Release points: - v0.1.0: 6f4

Overview: This repository provides a framework for building voice-based applications, simplifying the integration of custom speech services into websites and standalone devices. Key features: - Audio capture and playback services for local hardware - Automated speech recognition (ASR) using streaming for fast transcriptions - Hotword-optimized audio to text using Picovoice - Text to speech (TTS) - Natural Language Understanding (NLU) for intent and slot determination - Machine learning-based routing for RASA Tech stack: Python Why it matters: This software simplifies the integration of voice-based applications, making it easier to build custom speech services and standalone devices without requiring a network. Repository: syntithenai/hermod GitHub stars: 94 GitHub description: voice services stack from audio hardware through hotword, ASR, NLU, AI routing and TTS bound by messaging protocol over MQTT README excerpt: This repository provides a framework for building voice based applications. It was created to simplify integrating custom speech services into a website. It can also be used to build standalone alexa like devices that do not need a network. Inspired by Snips, the software is provided as a suite of microservices that collaborate using a shared MQTT server. Services include audio capture and playback, ASR, hotword optimization, NLU, RASA routing, and TTS. A sequence of messages passes between the services as the dialog progresses from hotword triggering through speech to text, NLU, routing, and finally TTS in reply to the user. The software also provides a vanilla JavaScript library

### Overview: TAGAN is an official PyTorch implementation of the paper "Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language" presented at NeurIPS 2018. This code is designed to manipulate images using natural language. ### Key features: - **Text-Adaptive Generative Adversarial Network (TAGAN)**: A deep learning model that leverages text inputs to generate images. - **PyTorch**: The code is written in PyTorch, a popular framework for building and training neural networks. - **Pretrained Word Vectors**: Utilizes pre-trained fastText word vectors for improved performance. ### Tech stack: - Python ### Why it matters: This implementation provides a practical tool for researchers and developers interested in image manipulation using natural language, offering a powerful method for generating images based on textual descriptions.

### Overview: UCDIR is a unified conditional framework based on diffusion models for image restoration. It leverages a lightweight UNet to predict initial guidance and a diffusion model to learn the residual of the guidance. The framework integrates conditional information into every block of the diffusion model to achieve spatially-adaptive generation conditioning, addressing the issue of overlooking conditional information in existing works. The proposed method demonstrates significant improvements in perceptual quality and generalization to restoration tasks. ### Key Features: - **Unified Conditional Framework**: A novel approach to integrate conditional information into diffusion models for image restoration. - **Spatially-Adaptive Conditioning**: Guidance and auxiliary conditions integrated into every diffusion block. - **High-Resolution Support**: Inter-step patch-splitting strategy for arbitrary-resolution images without grid artifacts. - **State-of-the-Art Performance**: Superior results on extreme low-light denoising, deblurring, and JPEG restoration tasks. ### Tech Stack: - Python 3.8+ - PyTorch ### Why It Matters: UCDIR addresses the challenge of integrating conditional information into diffusion models for image restoration, leading to superior performance in challenging tasks such as extreme low-light denoising, deblurring, and JPEG restoration.

Overview: This project focuses on Indian accent speech recognition using traditional ASR methods and deep neural networks (DNNs). It includes both custom models and a pre-trained model based on the Baidu DeepSpeech framework. Key features: - Custom DNN models for speech recognition - Traditional ASR techniques (Signal Analysis, MFCC, DTW, HMM) - Pre-trained DeepSpeech model for Indian accent speech recognition Tech stack: Jupyter Notebook Why it matters: This project provides a comprehensive solution for Indian accent speech recognition, combining traditional ASR methods with modern deep learning techniques to achieve high accuracy and efficiency in speech recognition tasks. GitHub stars: 93

Overview: DiffPack is a novel torsional diffusion model designed for predicting the conformation of protein side-chains based on their backbones. It significantly improves angle accuracy across various benchmarks for protein side-chain packing. Key features: - DiffPack (Vanilla) - DiffPack (with Confidence Prediction) Tech stack: Python Why it matters: DiffPack offers a significant improvement in angle accuracy for protein side-chain packing, making it a valuable tool in structural biology and drug design.

Overview: Franken Whisper is an agent-first Rust ASR stack built on a real in-process pure-Rust Whisper engine, featuring adaptive Bayesian backend routing, real-time NDJSON streaming, DTW word timestamps, and SQLite-backed run history. It is optimized for performance and compatibility with the latest Rust 2024 edition. Key features: - Real in-process pure-Rust Whisper engine - Adaptive Bayesian backend routing - Real-time NDJSON streaming - DTW word timestamps - SQLite-backed run history - Performance improvements over whisper.cpp by 2.99x on a 124.5-second whole job - Tiny.en is 1.52x faster on a 124.5-second transcribe-only workload - Tiny.en is 1.51x faster on a 300-second transcribe-only workload - Installer prebuilt binaries for Linux, macOS, and WSL, with SHA-256 verification Tech stack: Rust 2024 edition Why it matters: Franken Whisper offers a fast, reliable, and secure agent-first ASR stack that performs well on both live and batch tasks, making it ideal for real-time applications and batch processing. Repository: Dicklesworthstone/franken_whisper GitHub stars: 93

Overview: This plugin allows developers to integrate speech recognition functionality into their NativeScript applications, leveraging the available speech engines on the device. Key features: - Speech recognition using multiple engines - Cross-platform support across iOS and Android - Real-time speech recognition and transcription - Permissions management for microphone and speech recognition Tech stack: TypeScript Why it matters: This plugin simplifies the process of incorporating speech recognition into NativeScript applications, making it easier for developers to create voice-based applications without the need for external libraries or additional dependencies. GitHub stars: 93 Repository: EddyVerbruggen/nativescript-speech-recognition

Overview: This repository is a curated collection of papers and resources focused on Audio-Visual Intelligence (AVI) within the context of large foundation models. It provides a comprehensive survey of AVI methods, datasets, and benchmarks, organized around three interconnected pillars: perception, generation, and interaction. Key features: - Representation-Centric Methods - Generation-Centric Methods - LLM-Centric Methods Tech stack: Python, Markdown, and various open-source tools. Why it matters: This survey offers a systematic and in-depth treatment of AVI within the foundation-model paradigm, making it a valuable resource for researchers and practitioners interested in the latest advancements in AVI. Repository: JavisVerse/Awesome-AVI GitHub stars: 93 GitHub description: Awesome Audio-Visual Intelligence, Survey of Audio-Visual Intelligence README excerpt: <h1 align="center">Audio-Visual Intelligence in Large Foundation Models</h1> <h1 align="center">A Comprehensive Survey</h1> <div align="center"> [](https://arxiv.org/abs/2605.04045) [](https://huggingface.co/papers/26

Overview: DiffStyle is a project that leverages the Diffusion Prior to perform One-for-All Style Transfer, built on Prompt-to-Prompt and Null-Text. This project is a course project of CS3310 Computer Graphics. Key features: - Leverages Diffusion Prior for style transfer - Built on Prompt-to-Prompt and Null-Text - Compatible with Python 3.8 and Pytorch 1.11 - Supports both stable diffusion and latent diffusion models - Provides quick start commands for generating results Tech stack: Python Why it matters: DiffStyle is a project that combines the principles of Prompt-to-Prompt and Null-Text with the Diffusion Prior, offering a unique approach to style transfer that can be applied to various types of images, including landscapes and portraits, demonstrating its versatility and wide applicability. GitHub stars: 93

### Overview: LLaMA-MoE v2 is a series of open-sourced Mixture-of-Experts (MoE) models based on LLaMA3. It explores the sparsity of LLaMA from the perspective of Mixture-of-Experts and uses post-training supervised fine-tuning with open-source data. The project aims to provide a smaller, more affordable MoE model for everyone. ### Key Features: 1. **Support building Attention MoE and MLP MoE**: - Build Attention MoE models with attention layers. - Build MLP MoE models with MLP layers. 2. **Multiple Expert Construction Methods**: - Random MLP MoE construction (vanilla). - Residual MLP MoE construction (residual). 3. **Packed Padding Training**: - Supports training with megablocks. 4. **Two-stage & Open-source data for SFT**: - First-stage: OpenHermes-2.5, SlimOrca, sharegpt_gpt4, lima, Infinity-Instruct, Llama-3-Magpie-Air-3M-v0.1. - Second-stage: Infinity-Instruct, MetaMathQA. 5. **Support building MoE for different Models**: - Llama3-8B. ### Tech Stack: Python ### Why it Matters: LLaMA-MoE v2 offers a more efficient and affordable alternative to traditional MoE models, making it accessible for a wider range of users. The project's focus on post

### Overview: Auto Subtitle Generator is an Android app designed to automatically generate subtitles for videos locally, using whisper or vosk for speech recognition and ML Kit for translation. It supports offline generation, multilingual transcription, and on-device processing, ensuring no cloud service is required. ### Key Features: - **Local Subtitle Generation:** Extracts audio from videos, runs speech recognition locally, and generates timed subtitle cues. - **Recognition Engines:** Supports whisper (whisper.cpp) and vosk, allowing you to choose a model from a single catalog. - **Model Support:** Offers English-only and multilingual variants, including a bundled English Base model. - **Translation:** On-device translation with automatic language detection. - **Voice Activity Detection (VAD):** Skips silence and speeds up long videos. - **Multiple Video Support:** Generates subtitles for multiple videos in the background. - **Export Options:** Subtitle files in SRT or VTT formats, and videos with soft or hard burned-in subtitles. - **Shorts Management:** Supports exporting subtitles from long-video transcripts using local Gemma 4 E2B. - **Batch Export:** Export subtitle files or captioned videos. - **Export Library:** Tracks completed exports. ### Tech Stack: C ### Why It Matters: Auto Subtitle Generator simplifies subtitle creation, translation, and management, making it ideal for users who prefer local processing and don’t want to rely on cloud services. The bundled English Whisper model ensures quick setup, and the ability to handle multiple videos and languages makes it versatile for various use cases. ### GitHub Stars

Overview: MATLABER is an official implementation of the paper "MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR," which aims to enable material-aware text-to-3D conversion. Key features: - Material-aware text-to-3D conversion - LAtent BRDF auto-EncodeR for improved performance Tech stack: Python Why it matters: This project addresses the challenge of converting text descriptions into 3D models, making it valuable for applications in computer graphics, virtual reality, and more. Repository: SheldonTsui/Matlaber Primary language: Python GitHub stars: 93

Overview: VidForge is an open-source AI pipeline designed for generating short-form commerce videos, integrating multimodal analysis, RAG scripts, multi-agent orchestration, FFmpeg composition, and cost tracing. Key features: - Multi-Agent Orchestration - Knowledge-Grounded Generation - Context Engineering - Executable Video Pipeline Tech stack: TypeScript Why it matters: VidForge provides a robust infrastructure for video generation, enabling the integration of various AI and media technologies, making it a valuable tool for developers and content creators in the field of commerce and product storytelling. Repository: WANGLEVY9/VidForge GitHub stars: 92 GitHub description: Open-source AI pipeline for short-form commerce videos: multimodal analysis, RAG scripts, multi-agent orchestration, FFmpeg composition, and cost tracing. README excerpt: ... <p align="center"> <img src="./.github/assets/social-preview.jpg" alt="VidForge — Multi-Agent video generation workspace" width="920" /> </p> <h1 align="center">VidForge</h1> <p align="center"> <strong>Open-source Multi-Agent infrastructure for knowledge-grounded video generation.</strong><br /> 将素材理解、品牌知识、上下文工程、剧本规划、视频生成、质量反馈和媒体合成连接成一条可观测的生产流水线。 </p> <p align="center"> <a href="https://github.com/WANGLEVY9/VidForge/actions/workflows/ci.yml"><img src="https://github

### Overview: Taiwan Tongues ASR CE 是一個專門為台灣多元語言環境設計的開源語音辨識模型專案。它支援國語、台語、客語和英語，提供本地多語混合語音辨識功能，幫助開發者和資訊服務業者低成本、高效地進行ASR模型訓練、微調和在地化應用開發。 ### Key features: - 支援國語、台語、客語和英語語種 - 提供本地多語混合語音辨識功能 - 低成本、高效率進行ASR模型訓練、微調和在地化應用開發 - 由數位發展部數位產業署「114年數位產業跨域軟體基盤系統建置案」實證成果之一，推動台灣語音技術開源生態 ### Tech stack: Python ### Why it matters: 本專案旨在推動台灣語音技術開源生態，協助資訊服務業者強化智慧應用能量，落實在地AI技術自主發展，由台灣大哥大執行與維護。

Overview: CoMPaSS enhances the spatial understanding of text-to-image diffusion models, enabling them to generate images that faithfully reflect the spatial configurations specified in the text prompt. Key features: - Enhances spatial understanding - Provides reference weights and SCOP dataset for training and inference - Supports both SCOP and TENOR modules Tech stack: Python Why it matters: CoMPaSS offers a significant improvement in the spatial understanding of diffusion models, making them more effective in generating images that accurately represent the spatial configurations described in the text prompts.

### Overview: Toyyib Olaitan OLANREWAJU is a researcher and software engineer with a focus on developing Trustworthy AI systems. He is pursuing a PhD at the Technical University of Munich (TUM) and is interested in areas such as Explainable & Trustworthy AI, Computer Vision, and addressing socio-technical challenges. ### Key Features: - **Explainable & Trustworthy AI:** Developing formal justification mechanisms for hallucinations in Generative AI. - **Computer Vision & Automated Systems:** AI-driven proctoring systems and real-time activity detection. - **Algorithmic Bias & Inclusivity:** Quantifying algorithmic bias and integrating African narratives into AI models. - **Distributed Computing & Optimization:** Task scheduling in Fog-Cloud computing using Machine Learning and Metaheuristic approaches. ### Tech Stack: - Python - TensorFlow - PyTorch - Scikit-learn - OpenCV - Metaheuristic algorithms ### Why it Matters: Toyyib's research aims to enhance the reliability and fairness of AI systems, making them more trustworthy and inclusive. His work addresses real-world socio-technical challenges and contributes to the development of AI that can effectively assist and protect society.

### Overview: This repository provides a simple PyTorch implementation of conditional denoising diffusion probabilistic models (DDPM) for training and sampling on MNIST, Fashion-MNIST, and Sprite datasets. ### Key features: - MNIST dataset training and sampling - Fashion-MNIST dataset training and sampling - Sprite dataset training and sampling ### Tech stack: PyTorch ### Why it matters: This implementation serves as a practical example of how to apply DDPM models to real-world datasets, showcasing the model's effectiveness in generating images from noise.

Overview: This repository provides a reference blueprint for building modern, scalable web applications using TypeScript, Angular, Node.js, and Google Gemini API. It focuses on maintainability, developer experience, and deployment readiness. Key features: - Angular frontend - Node.js/Express backend - TypeScript monorepo Tech stack: TypeScript Why it matters: This blueprint serves as a solid foundation for developers to build robust, maintainable, and scalable applications using cutting-edge tech stacks. Repository: damiansire/genai-fullstack-blueprint GitHub stars: 93

Overview: The Deepgram Go SDK is an official Go library designed to simplify integration with Deepgram's powerful speech and language AI services. Key features: - Comprehensive support for Deepgram's API - Synchronous and asynchronous pre-recorded (synchronous) and pre-recorded (asynchronous/callbacks) functionality - Support for streaming audio and voice agents - Text to speech and text intelligence capabilities - Detailed documentation and migration guides for upgrading from previous versions Tech stack: Go Why it matters: This SDK streamlines the integration process, making it easier to incorporate Deepgram's AI into various applications and projects, enhancing the quality of speech and language processing. Repository: deepgram/deepgram-go-sdk GitHub stars: 93 GitHub description: Official Go SDK for Deepgram.