Projects

Overview: GibRAM is an in-memory knowledge graph server optimized for retrieval augmented generation workflows, designed to efficiently retrieve related information from text-based data. Key features: - Graph in-Buffer: Stores graph structure in RAM for quick access. - Retrieval: Facilitates context retrieval in RAG workflows. - Associative Memory: Traverses relationships to access related information. Tech stack: Go Why it matters: GibRAM combines a lightweight graph store with vector search, enabling efficient retrieval of related text data in memory, making it suitable for short-lived analysis and exploration rather than persistent storage.

### Overview: jargo is a conversational AI framework for Go, built on top of the Pipecat framework. It provides a native WebRTC audio-first conversational AI solution, including speech-to-speech capabilities, turn-taking, and barge-in features. The framework is designed for real-time voice agents and is optimized for Go's performance and concurrency. ### Key Features: - **WebRTC Transport**: Supports WebRTC, WebSockets, LiveKit, and local audio. - **Pure Go Audio**: Uses Opus for audio encoding and decoding, and resampling via `go-resample`. - **Streaming Voice Pipeline**: Transcribes speech to text, processes with a language model, and generates speech. - **Speech-to-Speech**: Supports models from OpenAI, NVIDIA, and others. - **Turn-Taking & Barge-In**: Utilizes `Silero VAD` and `Smart Turn v3` for turn-taking and barge-in. - **Telephony**: Optional inbound/outbound phone calls via Twilio Media Streams. - **User-Idle Watchdog**: Automatically re-engages or hangs up when the caller is silent. - **RTVI Data Channel**: Works with existing RTVI clients. - **Pluggable Services**: Allows swapping STT, LLM, TTS, and speech-to-speech services. - **Concurrent by Design**: Independent processors; interruptions are frames. ### Providers: - **Stt**: Deepgram, AssemblyAI, Gladia, Speechmatics, Soniox, Whisper (OpenAI/Groq/local), Azure,

### Overview: SimpleSTT is a speech-to-text and keyboard input captions plugin for OBS (Open Broadcaster Software). It offers a web-based and standalone version, supports various STT and TTS services, and integrates with popular platforms like Discord, Twitch, and VRChat. ### Key Features: - **Web Version**: Accessible via a web interface. - **Standalone Version**: Runs independently without internet connection. - **Text to Speech**: Supports speech-to-text-to-speech functionality. - **Keyboard Only Mode**: Ideal for text-heavy environments. - **Deep Visual Customization**: Allows for extensive customization of captions. - **Twitch/BTTV/FFZ Emotes Integration**: Seamless integration with popular emote systems. - **Google Fonts Integration**: Utilizes over 1000 free fonts. - **Simple OBS Integration**: Easy integration with OBS. - **Auto Saves Styles and Configuration**: Saves styles and configurations automatically. - **Style Changes Applied on the Fly**: Allows for real-time style adjustments. - **Share Captions with Multiple Browser Sources Over the Network**: Facilitates sharing captions across multiple browsers. ### Tech Stack: The project uses TypeScript for development and utilizes the OBS API for integration. ### Why It Matters: SimpleSTT provides a comprehensive solution for captions in OBS, offering both STT and TTS capabilities, and supports various platforms and emote systems. The web-based version is user-friendly, while the standalone version ensures offline functionality. The integration with Google Fonts and OBS makes it versatile and powerful for streamers and content creators. ### GitHub Stars: The

### Overview: Morphable Diffusion is an official implementation of the CVPR 2024 paper that introduces Morphable Diffusion, a novel approach for creating 3D-consistent avatar images from single images. This project leverages the FLAME topology for training and evaluation, and supports both virtual and real camera trajectories for seamless inference. ### Key Features: - **Pretrained Face Model**: Available with FLAME topology and inference script. - **Bilinear Topology Model**: Available with evaluation code. - **Realistic Trained Model**: Retrained with flame meshes obtained from more accurate FLAME-tracking. - **Exportable Data**: Preprocessed data in NeuS2 format for mesh reconstruction. ### Tech Stack: - Python ### Why It Matters: Morphable Diffusion offers a powerful method for generating realistic avatar images from single images, making it a valuable tool in the field of computer vision and image synthesis. The project's contributions to the FLAME topology and real-time inference capabilities make it a significant advancement in the field. ### GitHub Stars: 107 ### Repository: xiyichen/morphablediffusion

Overview: Visual News Captioner is an entity-aware model designed for news image captioning, introducing Visual News, a large-scale benchmark of more than one million news images. Key features: - Entity-aware model for news image captioning - Large-scale benchmark of over one million news images - Novel multi-modal feature fusion techniques and attention mechanisms Tech stack: PyTorch Why it matters: Our method effectively combines visual and textual features to generate captions with rich information, such as events and entities, outperforming competing methods in terms of fewer parameters and slightly better prediction results. Repository: FuxiaoLiu/VisualNews-Repository Primary language: Jupyter Notebook GitHub stars: 106 GitHub description: [EMNLP'21] Visual News: Benchmark and Challenges in News Image Captioning

### Overview: DiffMM: Multi-Modal Diffusion Model for Recommendation is a PyTorch implementation of a novel recommendation model proposed in the paper "DiffMM: Multi-Modal Diffusion Model for Recommendation" at ACM MM 2024 Oral. This model enriches the probabilistic diffusion paradigm with modality awareness, enhancing recommendation performance through multi-modal graph diffusion and cross-modal data augmentation. ### Key Features: - **Multi-Modal Graph Diffusion**: Reconstructs a comprehensive user-item graph using a multi-modal graph diffusion model. - **Cross-Modal Data Augmentation**: Provides valuable self-supervision signals through a cross-modal data augmentation module. - **Performance Comparison**: Conducted extensive experiments to compare with competitive baselines, establishing the superiority of DiffMM in terms of Recall@20, NDCG@20, and Precision@20. ### Tech Stack: - Python - PyTorch - NumPy - SciPy ### Why It Matters: DiffMM significantly improves recommendation performance by leveraging multi-modal data and enhancing the probabilistic diffusion paradigm, making it a valuable tool in the field of recommendation systems. ### Repository: - **Repository**: HKUDS/DiffMM - **Stars**: 106 - **Description**: PyTorch implementation of DiffMM, a multi-modal recommendation model for ACM MM 2024 Oral. ### Environment: - **Python**: 3.9.13 - **Packages**: numpy==1.23.1, torch==1.11.0, scipy

### Overview: This project is an official PyTorch implementation of the paper "Getting it Right: Improving Spatial Consistency in Text-to-Image Models" by Agneet Chatterjee et al. The repository includes the dataset, models, and a demo for evaluating the performance of text-to-image diffusion models. ### Key features: - **Dataset**: SPRIGHT (Spatially Focused Large Scale Dataset) - A large-scale dataset designed to improve spatial consistency in text-to-image models. - **Models**: Official PyTorch implementation of the model introduced in the paper. - **Training**: Detailed instructions and scripts for training the model. - **Inference**: A simple script for generating images using the model. - **Demo**: A local demo of the model using the provided script. ### Tech stack: Python ### Why it matters: This project is significant because it addresses a key limitation in current text-to-image models, improving their ability to generate images that faithfully follow the spatial relationships specified in the text prompt. The dataset and methods presented in the paper are designed to achieve state-of-the-art performance, making it a valuable resource for researchers and practitioners in the field of text-to-image models.

Overview: This repository contains the official implementation of the AAAI 2023 paper "Parameter-efficient Model Adaptation for Vision Transformers". The project aims to study parameter-efficient model adaptation strategies for vision transformers on the image classification task. Key features: - Parameter-efficient Model Adaptation for Vision Transformers - Benchmark and comparison with state-of-the-art methods - Measured using the vision transformer (ViT-B-224/32) via CLIP pretraining across 20 image classification datasets Tech stack: Python Why it matters: Our method places in the topleft corner and achieves the best tradeoff between accuracy and parameter efficiency, making it a significant contribution to the field of parameter-efficient model adaptation for vision transformers. Repository: UCSB-AI/PEViT GitHub stars: 106

Overview: CLIPAway is an official implementation of NeurIPS 2024's CLIPAway, a method for improving the performance of computer vision models. Key features: - Efficient model optimization - Improved accuracy and speed - Customizable for various applications Tech stack: Python Why it matters: CLIPAway is crucial for researchers and developers looking to enhance their computer vision models, offering significant improvements in both accuracy and speed.

Overview: EZIGen is an official implementation of the paper "Enhancing zero-shot personalized image generation with precise subject encoding and decoupled guidance." It aims to improve zero-shot personalized image generation by focusing on precise subject encoding and decoupled guidance, achieving state-of-the-art results on various benchmarks. Key features: - Subject Image Encoder - Separation of Text and Subject Guidance - Balanced Subject Identity Preservation and Text Following Tech stack: Python Why it matters: EZIGen introduces a new approach that effectively balances subject identity preservation and text-following, leading to superior results in zero-shot personalized image generation. Repository: ZichengDuan/EZIGen Primary language: Python GitHub stars: 106 GitHub description: [BMVC 2025] Official implementation for paper EZIGen: Enhancing zero-shot personalized image generation with precise subject encoding and decoupled guidance

### Overview: Variational Diffusion Models (VDM) This is a PyTorch implementation of Variational Diffusion Models, focusing on optimizing likelihood rather than sample quality. The implementation aims to match the official JAX implementation of VDM, with the primary goal being educational and simplicity. The model is currently only trained on CIFAR10 and does not include the variance minimization with the $\gamma_{\eta}$ network, which is only used with augmentations. ### Key Features: - **Diffusion Process**: The forward diffusion process is defined using sigmoid functions to control the noise level. - **Reverse Generative Process**: The discrete generative process is defined in terms of denoising steps, with the one-step denoising distribution being the true denoising distribution. - **Optimization in Continuous Time**: The loss function is the variational lower bound, and the diffusion loss is the sum of KL divergences between the predicted and true denoising distributions. ### Tech Stack: Python ### Why It Matters: The implementation aims to provide a clear and educational view of VDM, focusing on likelihood optimization rather than sample quality, which is a key aspect of probabilistic generative modeling. ### Repository: addtt/variational-diffusion-models ### Primary Language: Python ### GitHub Stars: 106 ### GitHub Description: PyTorch implementation of Variational Diffusion Models.

Overview: A large-scale dataset and baseline for music-driven group dance generation, presented at CVPR 2023. Key features: - AIOZ-GDANCE dataset with in-the-wild videos, music audio, and 3D group dance motion. - Semi-autonomous labeling method for obtaining 3D ground truth. - 16.7 hours of paired music and 3D motion sequences from 7 dance styles and 16 music genres. - New evaluation metrics for group dance quality measurement. - Proposed method for generating multiple group-coherent choreographies from music sequences and 3D positions. Tech stack: Python. Why it matters: This dataset and method provide a comprehensive resource for researchers interested in music-driven group dance generation, enabling the study of group choreography and improving the accuracy of dance motion synthesis from music.

### Overview: This repository, `download_audioset`, provides a straightforward method to download the raw audio files from AudioSet, a comprehensive library for speech object detection research. The dataset consists of 632 audio event classes and 2,084,320 labeled sound clips, totaling 32.45 GB of data. The script, `as_download.py`, efficiently downloads and converts the audio files from YouTube, making it accessible for various data science and modeling purposes. ### Key Features: - Downloads and converts all AudioSet audio files from YouTube. - Provides a hierarchical ontology of 632 audio event classes. - Supports both balanced and unbalanced datasets. - Includes a progress bar for the download process. - Supports both macOS and Linux environments. ### Tech Stack: Python ### Why It Matters: The ability to easily access and work with AudioSet's extensive dataset is crucial for researchers and developers in speech recognition, object detection, and related fields. The script simplifies the process of obtaining the necessary raw audio files, enabling more efficient data analysis and model training. ### Repository: - **Repository**: `jim-schwoebel/download_audioset` - **GitHub Stars**: 106 - **Primary Language**: Python - **GitHub Description**: This repo makes it easy to download the raw audio files from AudioSet (32.45 GB, 632 classes). - **GitHub URL**: [https://github.com/jim-schwoebel/download_audioset](https://github.com/j

### Overview: This project aims to detect diffusion model deepfakes using various deepfake detectors. It includes a detailed evaluation of different models and their performance metrics. ### Key Features: - **Detector Evaluation**: Reconstructs results from Table 2 and Figures 1a and 1b for the diffusion model deepfakes dataset. - **Model Evaluation**: Evaluates the performance of different models using various metrics such as AUROC and PD@5% PD@1%. - **Model Versions**: Provides checkpoint files for different models, including ProGAN, StyleGAN, ProjectedGAN, Diff-StyleGAN2, Diff-ProjectedGAN, DDPM, IDDPM, ADM, PNDM, LDM, and GAN. ### Tech Stack: - Python 3.8 ### Why It Matters: This project is significant for the field of deepfake detection, providing a comprehensive evaluation of different models and their performance metrics. The detailed evaluation helps researchers and practitioners to choose the most suitable model for their specific use cases. ### Repository: - **GitHub Repository**: [jonasricker/diffusion-model-deepfake-detection](https://github.com/jonasricker/diffusion-model-deepfake-detection) - **GitHub Stars**: 106 - **GitHub Description**: [VISAPP2024] Towards the Detection of Diffusion Model Deepfakes ### Code: - **Source Code**: The project provides the source code and instructions on how to recreate the results in the paper. - **Installation**: Requires Python 3.8.

Overview: FastDeploy simplifies the deployment of deep learning inference pipelines by reducing the need for extra code and providing auto-generated dockerfiles and Kubernetes health checks. Key features: - Auto batching of concurrent inputs - No changes to inference code - Promethues metrics exposed for monitoring - Clean dockerfiles and Kubernetes health check - Sequentially chained inference pipelines - REST APIs for querying from any language - Easy to understand architecture Tech stack: Python Why it matters: FastDeploy streamlines the deployment process for deep learning inference pipelines, making it easier to integrate models into existing systems without requiring significant modifications to the inference code. This tool is particularly useful for applications where individual inputs are processed sequentially, such as in microservices architectures, where the model, preprocessing, and post-processing steps are separated and can be scaled independently.

Overview: Specifai is an AI-powered platform that transforms project requirements into structured documents and actionable tasks, streamlining the software development lifecycle. Key features: - 🚀 AI-Powered Document Generation - 💬 Intelligent Chat Interface - 🪄 AI-Powered Inline Editing - 📊 Business Process Visualization - 🔗 Linking BRDs and PRDs -📋 User Story Generation Tech stack: TypeScript Why it matters: Specifai accelerates the SDLC process by automating documentation and generating detailed requirements, making it easier for teams to manage and refine software projects efficiently. Repository: presidio-oss/specif-ai Primary language: TypeScript GitHub stars: 106 GitHub description: Transform your software development lifecycle with Specifai - an intelligent platform that turns project ideas into structured requirements and actionable tasks in minutes.

### Overview: Cog implementation of Meta's MusicGen models is a cog project that uses the Meta MusicGen model to generate melodies. This project is based on the paper "MusicGen: A Simple and Controllable Model for Music Generation" by [the authors](https://arxiv.org/abs/2306.05284). It is designed to be used with Replicate's API, allowing users to interact with the model through a web interface or API. ### Key Features: - **Model Implementation**: Uses Meta's MusicGen model for generating melodies. - **EnCodec Tokenizer**: Trained on a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. - **Parallel Codebook Prediction**: Predicts all 4 codebooks in parallel, reducing the number of auto-regressive steps to 50 per second. - **Training Data**: Relys on 20K hours of licensed music to train the model, with an internal dataset of 10K high-quality music tracks and ShutterStock and Pond5 music data. ### Tech Stack: - Python ### Why It Matters: This project is significant for its ability to generate melodies efficiently and effectively, making it a valuable tool for music generation and composition. It bridges the gap between high-quality music generation and the practicality of real-time applications, making it suitable for various creative and production uses. ### Repository: - **GitHub Repository**: [replicate/cog-musicgen](https://github.com/replicate/cog-musicgen) -

### Overview: A real-time speech-to-text WebSocket server with pluggable ASR backends, energy-based VAD, streaming partial results, and Prometheus observability. ### Key features: - WebSocket/HTTP interface - OpenAI-compatible - Real-time processing - Pluggable ASR backends (Mock, Sherpa-Onnx, FunASR, Qwen3-ASR, Nemotron) - Energy-based VAD - Streaming partial results - Prometheus observability ### Tech stack: Python ### Why it matters: This system is designed for real-time speech-to-text processing, making it suitable for applications requiring immediate transcription, such as live streaming or real-time communication platforms. The pluggable ASR backends allow for flexibility in choosing the best model for the task, while the energy-based VAD ensures accurate speech detection. The streaming partial results feature allows for real-time updates, and Prometheus observability provides insights into system performance and usage.

Overview: M3Exam is a multilingual, multimodal, and multilevel benchmark designed to evaluate Large Language Models (LLMs) across various domains and levels. Key features: - Multilingual questions sourced from real and official human exam questions. - Multimodal questions requiring images. - Multilevel evaluation across different language levels (low, mid, high). Tech stack: Python Why it matters: M3Exam provides a comprehensive and realistic benchmark for evaluating LLMs in a challenging and diverse context, ensuring that models can perform well in a wide range of applications.

### Overview: This project is a Python utility designed to convert Compiled HTML Help (`.chm`) files into clean Markdown, supporting multiple document types through a profile system. ### Key features: - **Profile-based conversion** — `generic` profile for any CHM, `revit` profile for Autodesk Revit API docs - **Auto encoding detection** — handles UTF-8, GB18030, GBK, GB2312, and more via `chardet` - **Cross-platform 7-Zip detection** — finds `7z`/`7za`/`7zz` on PATH (Windows, Linux, macOS) - **Flexible CHM structure support** — handles `html/` subdirectory, flat, and deeply nested layouts (e.g. DirectX SDK) - **`--preserve-structure`** — optionally mirrors the CHM's internal folder hierarchy in the output instead of flattening everything into `data/` - **Code block preservation** — detects language from class names and named divs; supports C#, VB, C++, F#, Python, Java, JS/TS, Bash, SQL, XML, JSON - **Table normalization** — cleans and re-formats Markdown tables - **Index generation** — produces `file_index.json`, `id_lookup.json`, and `index.md` for search and AI integration - **Async + batched processing** — bounded concurrency and periodic GC prevent memory overflow on large CHM files (6 000+ pages) ### Tech stack: Python