Projects

Overview: Fucina is a CPU-first, eager-execution tensor/autograd runtime and language model inference engine written in pure Zig. It ensures compile-time axis checks and eager computation, enabling real-time performance without the need for C/C++ build systems or Python dependencies. Key features: - CPU-first tensor/autograd library - Eager execution - Comptime-checked axis tags and rank - Comptime facts for axis names, contraction, and result tag sets - GPU offload for matrix multiplication via Metal or CUDA - Applications validated against reference implementations and benchmarks Tech stack: Zig 0.16.0 Why it matters: Fucina provides a high-performance, low-overhead solution for training and inference of language models, enabling real-time applications without the need for complex build systems or Python dependencies. The library's direct translation into a book-length course and its real-world applications in various domains demonstrate its practical value.

Overview: This repository contains code for evaluating the role of pre-training dataset size and diversity on the performance of single-cell foundation models, specifically focusing on scVI, SSL, and Geneformer. Key features: - Code for evaluating the performance of scVI, SSL, and Geneformer on single-cell RNA-seq data. - Three different downsampling schemes for pre-training datasets. - Two main tasks: cell type classification and batch integration. - Pre-training on subsets of the scTab corpus using uniform random downsampling, cell type re-weighting, and geometric sketching. - Results presented in figures and a preprint. Tech stack: Python, Jupyter Notebook. Why it matters: This project assesses how pre-training dataset size and diversity impact the performance of single-cell foundation models, providing insights into improving model performance in the context of single-cell RNA-seq analysis.

### Overview: SuperPilot is a multi-model framework designed to build and execute various AI applications using language models like LLMs (Large Language Models). It leverages machine learning models from providers such as OpenAI to perform tasks such as text summarization, web searching, and more. ### Key Features: - **OpenAI Provider**: Manages the interaction with OpenAI models for language completion. - **SimplePrompt**: Generates prompts for OpenAI models. - **AbilityRegistry**: Registers and manages the abilities of the framework. - **SuperTaskPilot**: Executes tasks by invoking the registered abilities and model providers. ### Tech Stack: Python ### Why it Matters: SuperPilot simplifies the process of building and executing AI applications by providing a robust architecture that integrates various LLMs and model providers. This framework is essential for developers looking to leverage the capabilities of large language models to create intelligent applications.

### Overview: mlx-serve is a local inference server designed for Apple Silicon Macs, providing hot-swappable MLX models (LLM, vision, embeddings, TTS, STT, and decision) via the OpenAI API. It ensures efficient memory management and streamlined performance on the Apple Silicon architecture. ### Key Features: - **Hot-Swap by Model Name**: Send a request to any configured model; the server automatically loads it and unloads the previous one. - **OpenAI-Compatible API**: Compatible with various OpenAI SDKs and direct HTTP requests. - **All Six MLX Model Types**: Supports text (`mlx-lm`), vision (`mlx-vlm`), embeddings (`mlx-embeddings`), TTS (`mlx-audio`), STT (`mlx-whisper`), and decision (`laya_mlx`) models. - **Subprocess Isolation**: Text and vision models run in isolated subprocesses, while embeddings, TTS, and STT models run in-process. - **Auto-Unload on Inactivity**: Configurable timeout (default 10 minutes) to free memory when idle. - **Per-Request `keep_alive`**: Override the idle timeout per request, with `keep_alive` options for permanent or immediate unload. - **Prompt Caching**: `max_kv_cache_size` per model caps KV cache token capacity for efficient prefix reuse. - **Model Management API**: Preload, force-unload, delete from disk, show detail, and pull from HuggingFace. - **Observability**: Request metrics, memory monitoring, lifecycle

Overview: The LLM Semantic Router is an intelligent Mixture-of-Models (MoM) system designed to direct OpenAI API requests to the most suitable backend models based on semantic understanding of the request content. This system uses BERT classification to intelligently select models, improving overall inference accuracy and reducing prompt tokens. Key features: - Auto-Selection of Models: Directs requests to the most suitable models based on semantic understanding. - Auto-Selection of Tools: Selects the appropriate tools based on the prompt to reduce prompt tokens. - PII Detection: Detects and prevents the transmission of Personally Identifiable Information (PII). - Prompt Guard: Detects and prevents jailbreak prompts from being sent to the LLM. - Semantic Caching: Caches the semantic representation of the prompt to reduce prompt tokens and improve inference latency. Tech stack: The system is implemented in two ways: Golang (with Rust FFI based on Candle) and Python. Benchmarking will be conducted to determine the best implementation. Why it matters: This system is crucial for improving the performance and security of language models, ensuring privacy and preventing misuse, while also optimizing the efficiency of the system by reducing prompt tokens and improving inference accuracy.

Overview: This project is a software designed for drawing with the help of stable diffusion, utilizing a C++ interface with an embedded Python interpreter. Key features: - Fast interface for drawing with stable diffusion support - Input image, result, outpaint example, and inpainting, image2image, and controlnet examples Tech stack: C++ with embedded Python interpreter Why it matters: This project aims to provide a user-friendly interface for drawing with stable diffusion, making it accessible to a wider audience. Repository: rodjjo/diffusion-expert GitHub stars: 22 GitHub description: A software for drawing with stable diffusion support Installation video tutorial: Click on the video below: [Installation Tutorial](http://img.youtube.com/vi/cv_RmXgOrjQ/0.jpg)

Overview: Agent Notch is a macOS computer-use agent that sits in the notch, allowing users to interact with their computer via voice commands, screenshots, and mouse clicks. It leverages Claude's capabilities to drive the mouse and provides a seamless integration with OpenAI's Whisper and Piper. Key features: - Long-press the cursor to trigger interactions. - Voice commands for various tasks. - Screenshot functionality. - Mouse and keyboard inputs. - Live tool calls in the notch. Tech stack: Swift Why it matters: Agent Notch simplifies computer use by providing a more intuitive and efficient interface, making it particularly useful for those who prefer voice commands or need to interact with their computer in a more hands-free manner. Repository: sam-siavoshian/agent-notch Primary language: Swift GitHub stars: 22 GitHub description: macOS computer-use agent in the notch. Long-press, talk, Claude drives the mouse. Landing + downloads: [agent-notch.vercel.app](https://agent-notch.vercel.app) [](LICENSE) [](#) [](#) [![CI](https://github.com/sam-siavoshian/agent-notch/actions/workflows/ci.yml/badge.svg

Overview: ToriLinux is a Linux LiveCD distribution designed specifically for offline AI training and inference, featuring pre-installed projects and dependencies. Key features: - Automatic - ComfyUI - InvokeAI - Llama.cpp - SillyTavern - vllm Tech stack: ArchLinux Why it matters: Ideal for researchers and developers needing offline access to AI tools for training and inference, ensuring consistent performance and reduced dependency on internet connectivity. Repository: sasha0552/ToriLinux GitHub stars: 22 GitHub description: Linux LiveCD for offline AI training and inference.

### Overview: OmniJev is a multimodal research project designed to enhance robot decision-making capabilities. It provides a finite-choice decision SDK, an HTTP service, and a MuJoCo robotic arm workbench. The system maps unstructured states to structured decisions over predefined candidates, supporting routing, classification, and finite action selection. It integrates visual inputs from cameras and wrist views, allowing for trajectory replay, decision probes, and benchmark panels. ### Key features: - **Decision SDK**: Enables finite-choice decision-making with predefined candidates. - **HTTP Service**: Provides an interface for querying and receiving decisions. - **MuJoCo Workbench**: Offers a robotic arm workbench for MuJoCo simulations, including trajectory replay and reproducible evaluation. ### Tech stack: - **Python**: The primary programming language used for development and experimentation. ### Why it matters: OmniJev is significant because it bridges the gap between unstructured visual inputs and structured decision-making, making it easier for robots to understand and respond to complex environments. The system's ability to simulate and replay trajectories adds a layer of realism to the decision-making process, providing valuable insights and benchmarking tools for researchers and developers alike.

### Overview: Ferrum is a Rust-native language model serving engine designed to maximize output throughput while meeting strict latency SLOs. It is a single-binary solution that runs on Metal, CUDA, and CPU, eliminating the need for Python dependencies. ### Key Features: - **Single Binary**: A Rust-native engine that runs as a single binary, eliminating the need for Python runtime. - **Metal, CUDA, and CPU Support**: Supports Metal, CUDA, and CPU execution. - **Latency Optimization**: Optimized execution policies to meet time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) SLOs. - **Performance Metrics**: Tracks and optimizes output throughput, latency, and other performance metrics. ### Tech Stack: - Rust ### Why It Matters: Ferrum is designed to maximize output throughput while ensuring compliance with strict latency SLOs for time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL). It eliminates the need for Python dependencies, making it ideal for applications requiring high performance and low latency. ### GitHub Stars: 22 ### Repository: - Repository: sizzlecar/ferrum-infer-rs - Stars: 22 ### Installation: - **macOS Apple Silicon / Linux x86_64**: ```sh curl -fsSL https://ferrum.pandaailabs.com/install.sh | sh ``` - **Windows x64**: ```powershell

### Overview: ContextCache is an open-source middleware that caches the KV states of tool definitions, ensuring that tool schemas are precompiled and reused across all users, sessions, and requests. This reduces the number of tool schema reprocessing from 6,000 tokens per request to just 651 tokens, resulting in a significant speedup and reduced token usage. ### Key Features: - **Cached Tool Schemas**: Compiles tool schemas into a KV cache once, reducing the number of tokens processed per request. - **TTFT Reduction**: TTFT (Token Time for First Token) is reduced from 5.6 seconds to 200 milliseconds, with 50 tools instead of 6,000. - **Speedup**: Achieves a speedup of 29.2x with 50 tools compared to 200 milliseconds with 5 tools. ### Tech Stack: - Python ### Why It Matters: - **Reduced Token Usage**: ContextCache significantly reduces the number of tokens processed per request, leading to faster response times and lower costs. - **Performance Improvement**: The system achieves a speedup of 29.2x with 50 tools, making it highly efficient for large-scale applications. - **Scalability**: The system can handle up to 50 tools without a noticeable increase in response time, making it suitable for high-traffic environments. ### GitHub Repository: - **Repository**: spranab/contextcache - **Stars**: 22 - **Description**: Tool schemas get re-p

Overview: `sml` is a CLI application designed for launching Large Language Models (LLMs) and Vision Language Models (VLMs) on High-Performance Computing (HPC) clusters, specifically using SLURM and FirecREST for direct SLURM launches. Key features: - Direct SLURM launches - Interactive CLI for quick model launches - Public serving endpoint: <https://serving.swissai.svc.cscs.ch/> - Initialization script for setting up credentials - Documentation for various usage scenarios Tech stack: Python Why it matters: Simplifies the process of launching complex LLMs and VLMs on HPC clusters, making it accessible to developers and researchers without deep expertise in HPC or SLURM. Repository: swiss-ai/model-launch GitHub stars: 22 GitHub description: A CLI app for launching LLM/VLM models on Alps.

Overview: This project focuses on the development of a video summarization technique that utilizes multimodal features extracted from user-generated videos captured by wearable cameras. The aim is to create a tool that can automatically summarize videos based on audio and visual information, providing a concise representation of the content. Key features: - Multimodal summarization using audio and visual features extracted with specific libraries. - A dataset containing audio and visual features along with ground truth annotations for video summaries. - Source code available in Jupyter Notebook format. - Primarily used for academic purposes in a Master's degree thesis. Tech stack: Python, PyAudioAnalysis, multimodal_movie_analysis. Why it matters: This project offers a valuable solution for automatically summarizing user-generated video content, which can be particularly useful in applications like video surveillance, education, or entertainment. The ability to create concise summaries from raw video data can enhance user experience and provide quick access to key information. Repository: theopsall/Video-Summarization GitHub stars: 22 GitHub description: Multimodal summarization of user-generated videos from wearable cameras.

Overview: This project is a simple HTML-based profile for developers, focusing on readability and ease of use. Key features: - Readable and user-friendly interface - Clean and minimalistic design - Easy to navigate and access Tech stack: HTML Why it matters: This project aims to provide a straightforward and accessible way for developers to showcase their projects and skills, making it easier for potential employers or collaborators to understand their capabilities.

Overview: detLLM is a tool designed to verify and ensure the deterministic and verifiable inference of Large Language Models (LLMs), providing minimal reproducibility packs when outputs diverge. It measures run-to-run variance and batch-size variance to generate explicit and capability-gated guarantees. Key features: - Verifies reproducibility for LLM inference - Produces minimal reproducibility packs when outputs diverge - Measures run-to-run variance and batch-size variance - Reports results with explicit guarantees (only claimed when backend can enforce them) Tech stack: Python Why it matters: detLLM is crucial for ensuring the reliability and consistency of LLM outputs, especially in applications where reproducibility is critical. Repository: tommasocerruti/detllm Primary language: Python GitHub stars: 22 GitHub description: Deterministic-mode checks for LLM inference: measure run/batch variance, generate repro packs, and explain why outputs differ. README excerpt: detLLM verifies reproducibility for LLM inference and produces a minimal repro pack when outputs diverge. It measures run-to-run variance and batch-size variance, and reports results with explicit, capability-gated guarantees (only claimed when the backend can actually enforce them).

Overview: MarkAI is an open-source OpenAI wrapper designed to guide employees towards efficiency, integrating seamlessly with documents. It offers fast access to data, security, file compatibility, and is open-source and free to use. Key features: - Fast and efficient access to data - Secure data management - File compatibility with text, Markdown, and HTML - Open-source and free to use Tech stack: TypeScript Why it matters: MarkAI simplifies the integration of AI with documents, making it accessible and efficient for both personal and professional use. Repository: umuthopeyildirim/markai GitHub stars: 22

### Overview: This project is a work in progress focused on fine-tuning large language models using LoRA adapters with 4-bit/8-bit quantization optimized for NVIDIA DGX Spark and Blackwell GPUs. ### Key features: - **NVFP4 (4-bit)**: Native Blackwell FP4 training via Transformer Engine - **MXFP8 (8-bit)**: High-precision training with Transformer Engine - **bitsandbytes FP4**: Works on any CUDA GPU - **DGX Spark optimized**: Tested on GB10 (~41GB VRAM for 3B model) - **LoRA adapters**: Memory-efficient fine-tuning (~240MB output) - **TensorBoard logging**: Real-time training metrics - **Extended thinking**: `/think` and `/no_think` modes for reasoning models ### Tech stack: Python, PyTorch, and NVIDIA DGX Spark ### Why it matters: This project allows fine-tuning large language models with optimized quantization and GPU support, making it suitable for production environments.

### Overview: Janus-Pro-R1 is an advanced visual generation model designed to unlock Aha Moments through reinforcement learning. It introduces a two-stage training paradigm: supervised fine-tuning (SFT) and reinforcement learning (RL), enabling seamless collaboration between text-to-image generation and image editing. ### Key Features: 1. **Two-Stage Training Paradigm:** - **Stage 1 – Supervised Fine-Tuning (SFT):** - Learns structured visual reasoning through text-to-image generation, image-text consistency self-evaluation, and image regeneration through reflection. - **Stage 2 – Reinforcement Learning (RL):** - Trained using a token-level Markov decision process with bi-level QA-based rewards to encourage spontaneous reasoning and correction, optimizing via Generalized Regret Proportion Optimization (GRPO). 2. **Self-Reflective Capabilities:** - Bridges the gap between text-to-image generation and image editing, enabling a unified and coherent visual reasoning process. ### Tech Stack: - Python 3.11 ### Why It Matters: Janus-Pro-R1 is a significant advancement in visual generation, offering a novel approach to unlocking Aha Moments through reinforcement learning. It addresses the limitations of traditional text-to-image generation by integrating introspective reasoning, leading to more coherent and insightful visual outputs. ### Repository: - **Repository:** wendell0218/Janus-Pro-R1 - **GitHub Stars:** 22 - **Primary Language:** Python - **GitHub Description:** Official repository of the paper "Unlock

Overview: Chorus is a high-performance local LLM runtime for native applications, simulations, and games. It embeds one model with shared inference, supports concurrent agents, and manages continuous batching, priority scheduling, and persistent conversation state. Key features: - Local inference - Concurrent agents - Priority scheduling - Continuous batching - Conversation control - Structured output - Per-request generation settings - Explicit request outcomes - Asynchronous model loading - Text embeddings - Embeddable by default Tech stack: C++/C, Vulkan, Metal, Godot Why it matters: Chorus simplifies LLM integration by providing a unified, embeddable runtime that supports multiple engines and hardware, making it ideal for interactive applications and simulations.

### Overview This repository provides visualizations for the process of ANNS (Approximate Nearest Neighbor Search). Currently, it supports IVFFlat and HNSW indexing methods. ### Key features - **Backend Server**: Manages vector data storage and mapping, indexes building and search, and provides projection utilities. - **Web for Visualizations**: Designed for IVFFlat and HNSW, featuring Voronoi diagrams, D3-force-links/radius/collision, DR project methods (PCA, MDS, t-SNE, UMAP) for visualization. ### Tech stack - **Backend**: Python 3.6 - **Web**: TypeScript ### Why it matters This tool offers a visual interface for understanding and optimizing the process of vector search index, making it easier to analyze and improve the performance of ANN algorithms. ### Repository - **Repository**: zilliztech/vector-index-visualization-tool - **GitHub Stars**: 22 - **GitHub Description**: Visualization tool for vector search index - **Primary Language**: TypeScript - **GitHub Description**: Visualization tool for vector search index ### Quick start 1. **Prepare**: - Create a new conda environment (Python 3.6) - Install required packages - Download and install `hnswlib4vis` and compile it to Python 2. **Build and Run**: - Clone the repository - Install dependencies - Run the backend server and web application ### IVFFlat Visualization - ![image](./fig/ivfflat