Projects

Overview: This project provides a pure Visual Basic .NET implementation of LLaMA 3.2 with 1B parameters and FP16 precision, optimized for CPU inference. Key features: - Pure Visual Basic .NET implementation - Supports LLaMA 3.2 with 1B parameters and FP16 precision - Optimized for CPU inference Tech stack: Visual Basic .NET Why it matters: This project offers a lightweight and efficient solution for CPU inference, leveraging the capabilities of the original C++ repository and Gemini 2.5 Pro Preview for porting.

### Overview: This benchmark aims to measure the cost of moving evicted KV blocks to CPU RAM, CPU pinned memory, NVMe SSD, and discarding them. It evaluates the optimal KV cache tiering policy for large language models like Qwen, focusing on the trade-offs between different storage tiers and their respective reusability probabilities. ### Key features: - **CPU fp16 pinned pool**: Preallocated pinned CPU buffers. - **CPU fp16 unpinned**: Pageable CPU memory. - **CPU fp16 pinned**: Dynamically pinned CPU memory. - **CPU int8**: CPU with int8 quantization. - **Disk fp16**: NVMe SSD via safetensors with fsync. - **Disk int8**: NVMe SSD with int8 quantization. ### Tech stack: Python, PyTorch, HuggingFace, CUDA, and PyTorch. ### Why it matters: This benchmark helps determine the most efficient KV cache tiering strategy for large language models, optimizing storage costs and reusability probabilities.

### Overview: HealthSure InsurTech is an end-to-end machine learning application designed to predict insurance premium categories based on customer demographics, lifestyle factors, health indicators, and financial information. The platform integrates machine learning actuarial underwriting, dynamic batch file ingestion, Google Gemini LLM executive risk intelligence, and longitudinal Electronic Health Records (EHR) analytics into a unified system. ### Key Features: - **Machine Learning Actuarial Underwriting:** Predict risk-adjusted insurance premium tiers with real-time biometric readouts and automated Google Gemini actuarial briefs. - **Batch Underwriting:** Upload custom `.csv` or `.xlsx` policyholder files, trigger high-throughput asynchronous machine learning scoring with live progress bars, and dynamically render interactive Plotly analytics. - **Clinical Comorbidity Burden:** Compute Charlson Comorbidity Index (CCI) scores, calculate associative disease lift, and analyze length of stay vs. inpatient cost. - **Patient Cohort Analysis:** Define dual clinical cohorts with multi-variable filters and run automated statistical hypothesis tests. - **Audit Longitudinal Records:** Deep-dive into high-risk applicants with multi-year diagnoses, lab trends, and 1-click clinical PDF export. - **REST API & Model Diagnostics:** Live FastAPI endpoint tester, feature importance ranking, and ready-to-run code snippets in Python, cURL, and JavaScript. ### Tech Stack: - **Python:** Primary programming language. - **Streamlit:** Frontend framework for building interactive dashboards. - **FastAPI:** Backend API framework for real-time scoring. - **Sc

### Overview: Derate is a project designed to simplify the deployment and management of multiple models on your own machine, ensuring optimal performance and resource utilization without the need for reprogramming. ### Key Features: - **Unified Endpoint:** Derate provides a single endpoint for every model you run, ensuring efficient resource allocation. - **Hardware Optimization:** It measures the interconnect speed and model shapes to plan parallelism, preventing launches that exceed memory limits. - **Real-Time Metrics:** Derate dynamically adjusts the model split based on measured performance, providing accurate and actionable insights. ### Tech Stack: Python ### Why It Matters: Derate simplifies the deployment and management of models, ensuring optimal performance and resource utilization without the need for reprogramming. It measures the interconnect speed and model shapes to plan parallelism, preventing launches that exceed memory limits, and provides real-time metrics for accurate and actionable insights. Repository: Pizzaman213/derate Primary Language: Python GitHub Stars: 2 GitHub Description: One endpoint for every model you run and call, planned from what your hardware can actually do.

### Overview: Ternair is a standalone Python/PyTorch implementation of ternary language-model inference inspired by Microsoft's BitNet b1.58. It provides efficient inference with packed storage, straight-through training, and a packed storage model size of 1.1 GB, making it 4x smaller than the original model. Ternair supports OpenAI-compatible HTTP server, KV-cache inference, and a variety of features including incremental KV-cache generation and cached dequantisation. ### Key Features: - **Ternary Weights**: `{-1, 0, +1}` with straight-through training - **Packed Storage**: `packed` (1.6 bits/value) or `fastpacked` (2 bits/value) - **Decoder-Only Causal LM**: with GQA/RoPE attention, SwiGLU MLP, optional hybrid SSM blocks - **Incremental KV-cache**: generation (prefill once, decode one token per step) - **Greedy and Sampled Generation**: streaming, repetition penalty, chat templates - **Cached Dequantisation**: frozen weights unpacked once (~900× faster) - **Vectorised NumPy Matmul**: + optional Triton GPU / native C++ CPU backends - **SafeTensors Export**: and model-size/compression reports - **OpenAI-Compatible HTTP Server**: with batching, Prometheus metrics, health checks - **CLI**: for inspection, size estimation, demos, training, inference, serving, benchmarking ### Why Ternair? - **Same Tr

### Overview: MLsuite is an open-source platform designed to manage, run, and collect feedback on machine learning models. It provides a seamless workflow from training models to collecting and reviewing user feedback. ### Key Features: - **Model Management**: Store and manage trained model artifacts. - **Schema Analysis**: Analyze model inputs and generate versioned schemas. - **Form Rendering**: Render schemas as validated forms with MLForm. - **Prediction Execution**: Run individual and bulk predictions. - **History Preservation**: Preserve prediction history with exact model and schema identity. - **Feedback Collection**: Collect reviews, corrections, questionnaires, and explanation feedback. - **Export Capabilities**: Export inputs, outputs, and feedback for downstream use. - **Public Feeds**: Publish schema bookmarks to public pages, which can be filled by anyone, including signed-in users. ### Tech Stack: - **Frontend**: React - **Backend**: Spring Boot API - **Database**: PostgreSQL - **Artifact Storage**: RustFS - **Runners**: Docker Compose ### Why It Matters: MLsuite simplifies the process of testing and reviewing machine learning models. It automates the steps that typically require custom infrastructure, ensuring that each model and its interactions are reproducible and reviewable. This makes it easier to iterate on models and improve their performance based on user feedback, without the need for additional setup. ### Repository: The repository is located at [https://github.com/UlloaSP/mlsuite](https://github.com/UlloaSP/mlsuite). ### GitHub Stars: The repository has been

Overview: tabctx serves tabular foundation models by encoding and caching tables, enabling fast repeat predictions on multiple tenants. Key features: - Fits a table once and keeps the encoded context on the GPU for fast predictions. - Supports multiple models, including `tabicl-v2`, `tabpfn-3.5`, and `tabdpt-1.3`. - Offers prediction intervals for regression predictions. Tech stack: Python Why it matters: tabctx optimizes tabular model performance by reducing the expensive encoding step, making it ideal for real-world applications where training tables are often large.

Overview: This project demonstrates a complete ML infrastructure lifecycle, including distributed multi-node training on a Slurm cluster, scaled model deployment via Ray Serve, and performance monitoring/analysis using Prometheus/Grafana and DCGM. Key features: - Distributed multi-node training on a Slurm cluster - Scaled model deployment via Ray Serve - Performance monitoring/analysis using Prometheus/Grafana and DCGM Tech stack: Python Why it matters: This project provides hands-on experience with production-grade model deployment, performance monitoring, and scaling using Kubernetes, Ray, and Slurm to manage multi-node cluster configurations.

### Overview: This project, `triton-hf-builder`, automates the process of generating Triton model repositories from Hugging Face checkpoints. It supports Transformers, vLLM, and TensorRT-LLM, offering comprehensive configuration, runtime scripts, and dependency files for efficient inference. ### Key Features: - **Triton Configuration**: Generates Triton model configuration files. - **Runtime Scripts**: Provides runtime scripts for easy deployment. - **Dependency Files**: Includes necessary dependency files for the model. - **Versatile Support**: Compatible with Transformers, vLLM, and TensorRT-LLM. ### Tech Stack: - **Python 3.10+**: The project is built using Python 3.10 and above. ### Why It Matters: This tool streamlines the process of creating Triton model repositories, making it easier for developers to integrate Hugging Face checkpoints into their models without manual configuration. It ensures compatibility across different models and provides a consistent workflow for deployment and inference.

### Overview: TensorTorrent is a PyTorch compiler and Rust runtime designed for efficient multi-CPU and multi-GPU inference. It optimizes model execution across heterogeneous hardware, leveraging native Rust for placement and simulation, and compiling the best solution. ### Key Features: - **Profile → Plan → Simulate → Compile → Run**: TensorTorrent profiles the host, searches placements in a native Rust planner, simulates finalists, then compiles and runs the winner. - **Multi-CPU and Multi-GPU Support**: Trades PCIe/host bandwidth for capacity when it makes sense, optimizing resource usage. - **Heterogeneous Execution Planning**: Automatically searches placements in a native Rust planner for optimal execution across CPUs, GPUs, memory, and storage. ### Tech Stack: - Python 3.10–3.13 - PyTorch 2.4+ - Rust ### Why It Matters: TensorTorrent addresses the fragility of hand-written `.to(device)` maps and ad-hoc offload, especially when models approach or exceed accelerator memory. It leverages native Rust for placement and simulation, ensuring efficient execution across heterogeneous hardware, making it a valuable tool for developers working with PyTorch models on multi-CPU and multi-GPU setups. ### Repository: alhussein-jamil/TensorTorrent - GitHub Stars: 2 - GitHub Description: PyTorch compiler and Rust runtime for multi-CPU and multi-GPU inference. - Primary Language: Python - GitHub Actions CI: [ci.yml](https://github.com/alhussein-jamil/TensorTorrent/actions/work

Overview: This project is a complete machine learning pipeline for predicting diabetes using XGBoost and SHAP, serving predictions via FastAPI. Key features: - Data cleaning for biologically invalid zeros (Glucose, BloodPressure, SkinThickness, Insulin, BMI) - Baseline model (Logistic Regression) + decision-threshold tuning to prioritize **recall** - Stronger model: **XGBoost** with class imbalance handling (`scale_pos_weight`) - Model metrics (on test split): **ROC-AUC ≈ 0.817**, **Recall ≈ 0.80** (using tuned threshold) - Explainability: **SHAP** global and per-prediction explanations - Serving: **FastAPI** endpoint with Pydantic validation and Swagger docs Tech stack: Python Why it matters: This project demonstrates a practical application of XGBoost and SHAP for diabetes prediction, showcasing model performance and interpretability through SHAP explanations. The use of FastAPI for real-time predictions highlights the importance of efficient and scalable APIs in healthcare applications.

### Overview: This project is a two-node DGX Spark cluster serving GLM-5.3-Flash-NVFP4 using vLLM TP=2. The system is composed of a single entrypoint, one compose file, and one orchestrator, sharing a single configuration file for all stacks. The stacks are pure configurations with a `stacks/<name>.env` file that sets the image, weights path, and serving flags. The default stack is `nvfp4-dflash2`, which is 900K context with NVFP4 and DFlash2, pinning 9 GiB KV, and 1024/4 batched/seqs. The lab-vision stack is a one-command alternative, and the standing config is the `nvfp4-dflash2` stack with 95/100 hardmode performance. ### Key features: - **Single-Stack Policy:** Only one profile runs at a time. - **Configuration Files:** All stacks share a single `stacks/<name>.env` file. - **Orchestrator:** The `cluster.sh` script manages the stacks, stopping and starting them. - **Site-Specific Values:** All site-specific values are stored in `.env` files, which are copied and shared across nodes. ### Tech stack: - Shell ### Why it matters: This project is designed to be highly flexible and easy to manage, allowing for quick switching between different stacks without requiring any code changes. The single-stack policy ensures that the system is always running the same configuration,

Overview: This project is a low-latency streaming inference server designed for real-time AI applications. It supports token streaming, request batching, dynamic model hot-swapping, and backpressure management. Key features: - Token streaming via server-sent events - Async request batching with priority queues - Model registry for versioned hot-swapping - Middleware for rate limiting, logging, and metrics - Clean typed configuration for runtime behavior Tech stack: asyncio and aiohttp Why it matters: This server is essential for real-time AI applications that require high reliability and low latency, ensuring that the serving path does not block or stall, and that stream state is maintained gracefully under load. Repository: anmoldhingra1/realtime-ai-serve Primary language: Python GitHub stars: 2 GitHub description: Low-latency streaming inference server for real-time AI applications: token streaming, batching, hot swaps, and backpressure.

Overview: The Arbitrary Number project explores a groundbreaking mathematical system that transcends the limitations of the traditional Rational Number system. This system, known as the Arbitrary Number, is defined by its unique representation of numbers using ordinals and operations that can be easily converted into a form compatible with Rational Numbers through the Quantition operation. Key features: - The Arbitrary Number system is a universal representation of numbers that can be easily converted into a form compatible with Rational Numbers. - It introduces the concepts of Simple Arbitrary Numbers, Standard Arbitrary Numbers, and Complex Arbitrary Numbers. - The Arbitrary Number system is based on ordinals and operations that can represent any concept in the Universe of concepts. - It eliminates the loss of precision and loss of meaning that occurs in the Rational Number system. - The Arbitrary Number system can represent any concept in the Universe of concepts through increasing the magnitute of one of the ordinals and decreasing the magnitute of another by 1. Tech stack: Python Why it matters: The Arbitrary Number system offers a more precise and universal representation of numbers that can be easily converted into a form compatible with Rational Numbers, making it a significant advancement in mathematical representation and computation.

Overview: This project fine-tunes a large language model (LLM) specifically for mental health assistance, aiming to provide empathetic and informative responses to users seeking support. Key features: - Fine-tuned LLM for mental health support - Integration with mental health resources - Personalized advice and guidance Tech stack: Python Why it matters: This project addresses a critical need in mental health support by leveraging advanced AI technology to offer personalized and empathetic assistance, making it easier for individuals to seek and understand mental health resources.

### Overview: tamserve is a live multi-camera object tracking system that leverages SAM 2 and EdgeTAM models to serve every camera with a single model, achieving faster performance and reduced GPU memory usage compared to the original code. ### Key Features: - **Single Model Serving Multiple Cameras**: One model tracks every object on every camera, providing identical masks across all cameras. - **Significantly Faster Performance**: About 4x less GPU memory usage, with speeds up to 24x faster than the original code. - **Reduced Memory Usage**: 2.2 GB of GPU memory for 8 cameras, compared to 9.8 GB for 64 cameras. - **Lighter Model**: 321 frames/s for 8 cameras, 0.28 GB for 64 cameras, with no memory issues. - **Accuracy**: Maintains the same accuracy as SAM 2.1 models on DAVIS 2017. ### Tech Stack: - **Python 3.10+** - **PyTorch 2.6+** - **Models**: EdgeTAM, SAM 2.1 ### Why It Matters: tamserve simplifies the process of tracking multiple cameras by sharing the model across all cameras, making it faster, lighter, and more efficient than the original code. This approach is particularly useful in real-time applications where multiple cameras are streaming video feeds simultaneously.

Overview: This repository contains the code for the paper "MDInference: Balancing Inference Accuracy and Latency for Mobile Applications" published at IC2E'20. Key features: - Increased aggregate accuracy by over 7% and up to 40%. - Demonstrated impact of varied network conditions. - Introduced a holistic approach to mobile-aware framework design. Tech stack: Python Why it matters: This paper addresses the need for making mobile deep inference frameworks truly mobile-aware by balancing inference accuracy and latency, showcasing the effectiveness of dynamic model selection and on-device inference execution. Repository: cake-lab/MDInference GitHub stars: 2 GitHub description: Repository for the MDInference paper published at IC2E'20

Overview: This repository showcases a production-ready MLOps deployment that serves a trained machine learning model via a RESTful FastAPI microservice, containerized with Docker for seamless cloud deployment. It includes FastAPI endpoints, input validation, and health monitoring features. Key features: - FastAPI Endpoint Service: High-performance prediction endpoints with automated OpenAPI documentation. - Input Validation: Strict payload typing and schema verification using Pydantic models. - Containerized Deployment: Multi-stage Dockerfile ensuring dependency isolation and platform reproducibility. - Health Monitoring: Dedicated status routes to verify model loading states and API stability. Tech stack: Python, FastAPI, Docker. Why it matters: This repository demonstrates best practices in MLOps, model serving, and Docker containerization, making it a valuable resource for developers looking to deploy machine learning models efficiently and reliably.

### Overview: DevProof.AI is an open-source managed AI platform that allows developers to run open models and autonomous agents on their own Kubernetes infrastructure. It provides a self-hosted alternative to hosted managed-agent platforms, offering secure, scalable, and customizable AI solutions. ### Key features: - **Managed Agents**: Define an agent once, and the platform runs thousands of sessions of it, with isolated per-session Kubernetes pods that idle to zero between messages. - **Traceability**: Every step is recorded, including live session transcripts, per-request gateway traces, and token metering per session, agent, and workspace. - **Easy Development**: Create an agent in the console in a minute, with skills (progressive disclosure), memory stores, file attachments, and a Python client. - **Secured Environments**: Sessions get no network egress unless their environment allows specific hosts, and secrets live in credential vaults, injected as pod-level placeholders. ### Tech stack: TypeScript ### Why it matters: Own your scalable AI. DevProof.AI empowers developers to run AI models and agents on their own infrastructure, providing secure, customizable, and scalable solutions without the need for a managed platform.

### Overview: Onpar is a tool designed to help developers prove which open-source models can replace their closed-source APIs. It evaluates models based on traffic and hardware performance, ensuring a quality gate and rollback process. The tool is designed to be used without risk, making it a valuable resource for anyone looking to migrate their services. ### Key Features: - **Proving Models**: Determines which open-source models can replace closed-source APIs. - **Traffic and Hardware Analysis**: Evaluates models based on real-world traffic and hardware performance. - **Quality Gate and Rollback**: Ensures a quality gate and rollback process to protect against risks. - **Versatile Use Cases**: Suitable for new developers, curious engineers, and experienced agents. ### Tech Stack: Python ### Why it Matters: Onpar simplifies the process of migrating from closed-source to open-source models, ensuring that the transition is smooth and risk-free. It provides a clear, step-by-step guide to prove which models can replace existing APIs, making it a valuable resource for developers looking to modernize their services.