Projects

### Overview: Clef is a self-hosted calibrated classification system that uses a 9B multimodal model from OpenAI. It provides an OpenAI-compatible API, evaluation tools, async jobs, and a web console. Clef runs on your own GPU and supports various GPU configurations, including 24 GB, 16 GB, 12 GB, and 8 GB. ### Key features: - **One Pass Classification:** Returns probabilities for every option in a single forward pass. - **Multimodal Input:** Accepts text, JSON, images, and video in the same request. - **OpenAI SDK Compatible:** Runs against your own machine using the official OpenAI SDK. - **Quality Metrics:** Measures accuracy, F1, confusion matrix, and calibration metrics. - **Async Jobs and Batches:** Supports up to 100,000 rows per job, with persistence in SQLite or Postgres. - **Label Set Growth:** Keeps texts that cannot be placed by the classifier and asks for new labels from an LLM. - **Hardware Independence:** Runs on NVIDIA, AMD (ROCm), Apple Silicon, and CPU. - **Local by Default:** Binds to loopback only, with hashed API keys and rate limits. ### Tech stack: - Python ### Why it matters: - Clef simplifies the process of using a generative LLM as a classifier, providing calibrated probabilities for every option in a single forward pass. - It supports various GPU configurations, making it versatile and efficient. - It measures quality metrics and allows for growth

### Overview: A Python-powered ML-powered smart file tagging system for Nextcloud. ### Key Features: - **Smart File Tagging**: Automatically tags uploaded files into predefined categories or allows users to define custom categories. - **Personalized Document Classification**: ML model trained to classify documents into 5 categories: Lecture Notes, Problem Set, Exam, Reading, and Other. - **User Feedback Loop**: User corrections are logged and trigger retraining when enough feedback accumulates. ### Tech Stack: Python, FastAPI, MLflow, PostgreSQL, Docker Compose v1. ### Why It Matters: This system streamlines the tagging process for Nextcloud users, improving efficiency and organization. The use of ML and Docker ensures scalability and ease of deployment.

### Overview: This project demonstrates the Iris classification model using Flask and a saved scikit-learn Pipeline. The model accepts four flower measurements and returns the predicted species. The Flask form validates input and returns HTTP 400 for missing, nonfinite, or nonpositive values. The model is validated through regression tests and can be run locally with Python scripts. ### Key features: - Iris classification model using Flask and scikit-learn Pipeline - Form validation for input fields - Regression tests for model accuracy - Local run with Python scripts ### Tech stack: - Python - Flask - scikit-learn - Random Forest ### Why it matters: Training and inference share a saved scikit-learn Pipeline, ensuring consistency and ease of use. Form values are validated by feature name, preventing silent field order changes. Missing, nonfinite, and nonpositive values return HTTP 400, ensuring robust input validation. The interface accepts decimals, maintains submitted measurements, and works across desktop and mobile devices. The locally running app is accessible through `http://127.0.0.1:5000`, and tests check for decimal values, reordered form fields, invalid requests, and agreement between HTTP response and saved pipeline. GitHub Actions runs regression tests to ensure model accuracy. The bundled Iris data uses a fixed stratified 70/30 split, and the model is not an independent benchmark. The app binds to localhost with debug disabled, making it a learning demonstration. The original CodePen reference is retained in the HTML for context

Overview: This project is a benchmarking service that exports PyTorch models to ONNX format, applies INT8 quantization, and serves them behind a FastAPI endpoint. The benchmark includes latency, throughput, model size, and cost per 1,000 inferences across three backends. Key features: - ONNX export from PyTorch model - INT8 quantization - FastAPI endpoint for serving the model - Benchmarking of three backends: PyTorch, ONNX, and ONNX-INT8 Tech stack: Python Why it matters: This project provides a clear, measured comparison of the performance, size, and cost of ONNX models versus PyTorch models, specifically focusing on INT8 quantization. The benchmark results are crucial for developers looking to optimize their models for production environments, especially in terms of latency and cost. Repository: Mojtaba-Alehosseini/onnx-inference-service Primary language: Python GitHub stars: 0 GitHub description: FastAPI + ONNX + INT8 quantisation benchmark. ONNX 2.6x faster than PyTorch; INT8 1.71x smaller, accuracy delta -0.0005. 24 tests.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It includes semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by intelligently managing resources and failover mechanisms, ensuring optimal performance in local AI labs.

Overview: A revolutionary GPU virtualization technology that leverages CPU bypass to achieve 85% NVLink performance at a cost of 15% of traditional GPU costs. This project aims to unify multiple GPUs into a single accelerator, enhancing performance without the need for expensive hardware upgrades. Key features: - GPU virtualization - CPU bypass - 85% NVLink performance - 15% cost savings Tech stack: Pure Rust Why it matters: This project represents a significant advancement in GPU virtualization, offering a cost-effective solution for scaling GPU performance without the need for expensive hardware upgrades. The use of Rust ensures high performance and reliability, making it an ideal choice for developers looking to optimize their systems.

Overview: A high-performance memory system for large-scale machine learning models, designed to run models larger than your GPU's VRAM, utilizing tiered memory structures: VRAM, DRAM, and SSD. Key features: - Supports running models larger than your GPU's VRAM. - Utilizes tiered memory structures for efficient model execution. - Optimized for high-performance computing environments. Tech stack: C++ Why it matters: This system is crucial for researchers and developers working on large-scale machine learning models, enabling them to run more complex models without the need for additional hardware, thus reducing costs and improving efficiency.

Overview: SmarterRouter is an intelligent LLM gateway and VRAM-aware router designed for Ollama, llama.cpp, and OpenAI. It includes semantic caching, model profiling, and automatic failover for local AI labs. Key features: - Semantic caching - Model profiling - Automatic failover Tech stack: Python Why it matters: SmarterRouter enhances the performance and reliability of AI models by optimizing VRAM usage and providing a more efficient gateway for Ollama, llama.cpp, and OpenAI.

Overview: This project demonstrates a complete MLOps pipeline using Git, DVC, MLflow, FastAPI, and GitHub for version control, data and model versioning, experiment tracking, reproducible pipelines, model serving, and an interactive prediction UI. Key features: - Version control with Git and GitHub - Data and model versioning using DVC - Experiment tracking and metrics with MLflow - Reproducible pipelines via dvc.yaml - Model serving with FastAPI - Interactive prediction UI Tech stack: Python Why it matters: This project provides a comprehensive example of how to implement an MLOps pipeline, showcasing best practices in version control, data management, experiment tracking, and model deployment using modern tools like Git, DVC, MLflow, and FastAPI. Repository: NDDimension/iris-mlops Primary language: Python GitHub stars: 0 GitHub description: Complete ML workflow with versioning, experiment tracking, and FastAPI prediction UI.

Overview: Distributed AI Compute Engine is a unified framework designed for scaling Python applications and Machine Learning workloads from a single developer laptop to multi-node, multi-GPU clusters. Key features: - Tasks: Asynchronous, stateless functions scheduled across cluster nodes. - Actors: Stateful remote worker processes holding GPU model weights in VRAM. - Plasma Object Store: Zero-copy shared-memory object store for high-speed tensor sharing. Tech stack: Python Why it matters: It abstracts distributed systems complexity into three core primitives, making it an enterprise-grade distributed execution framework for scaling Python applications and Machine Learning workloads. Repository: NadeemAhmad3/distributed-ai-compute-engine Primary language: Python GitHub stars: 0 GitHub description: Unified Framework for Distributed AI Scaling, Ray Serve Model Orchestration, and Cluster Computing.

Overview: This repository is a high-performance Large Language Model (LLM) inference engine designed for scale and low latency. It leverages PagedAttention, Continuous Iteration Batching, and CUDA kernels to optimize memory usage and throughput. Key features: 1. PagedAttention Memory Management: Eliminates up to 96% of GPU memory waste by managing KV caches using virtual memory paging. 2. Continuous Iteration-Level Batching: Dynamically batches requests at the token iteration level to reduce latency. 3. Prefix Caching & Chunked Prefill: Caches and reuses Key-Value attention states for recurring prompt prefixes, reducing prefill latency by up to 80%. 4. Quantization & Hardware Acceleration: Supports native quantization (FP8, INT8, INT4, AWQ) and custom CUDA kernels for optimized attention and GEMM/MoE operations. 5. OpenAI-Compatible API Server: Includes a production HTTP server for OpenAI-compatible endpoints, supporting API calls and structured JSON output. Tech stack: Python Why it matters: This engine is crucial for applications requiring high-throughput and low-latency LLM inference, such as chatbots, language translation, and more, by efficiently managing memory and processing large volumes of text data. Repository: NadeemAhmad3/vllm GitHub stars: 0 GitHub description: High-Performance LLM Inference & Serving Engine powered by PagedAttention, Continuous Batching, and CUDA Kernels.

Overview: This project showcases a production-ready model canary platform that employs deterministic canary rollout decisions, side-by-side baseline and canary prediction comparisons, and multi-window rollout reviews with fresh approval and rollback evidence. Key features include FastAPI, Prometheus metrics, Kubernetes, and Terraform deployment assets. Key features: - Deterministic canary rollout decisions based on a stable hash of `ticket_id` - Side-by-side baseline and canary prediction comparison for rollout review - CLI and API rollout evaluation gates with promote, hold, and rollback decisions - Multi-window rollout reviews with stage-staleness and rollback-completion evidence gates - Staged traffic control reviews with fresh independent approval and ordered completion evidence - Prometheus metrics for selected model traffic, latency, and shadow mismatches - Docker, Kubernetes, and Terraform assets that make the repo readable Tech stack: Python Why it matters: This project demonstrates best practices in model deployment, ensuring safe and efficient rollouts with minimal risk and maximum visibility into the operational state.

### Overview: A provisioned fast EBS StorageClass tailored for Namazu Cloud tenants, providing up to 16,000 IOPS and 1,000 MiB/s for AI model weights and LoRA checkpoints, ensuring quick cold starts and namespace-guarded storage. ### Key Features: - **Namespace Guarded StorageClass**: Ensures that only the tenant's namespace can access the storage class, preventing unauthorized access. - **Provisioned IOPS/Throughput**: Up to 16,000 IOPS and 1,000 MiB/s for high-performance storage. - **Fast Load Times**: Models can be loaded within minutes instead of half an hour, improving cold start times. - **Flexible Storage Types**: Supports gp3 and io2 storage classes, with gp3 offering up to 16,000 IOPS and 1,000 MiB/s. ### Tech Stack: Kubernetes and AWS EBS CSI. ### Why It Matters: This addon is crucial for Namazu Cloud tenants, providing a fast and secure storage solution for their AI models, accelerating cold starts, and ensuring efficient data access.

### Overview: This is a shared ECR repository addon for Namazu Cloud, designed to securely store and manage AI inference and worker images for multiple tenants. The repository is immutable, pin-based, and ensures per-namespace pull control, with lifecycle management and audit features. ### Key features: - **Immutable ECR Repository**: The repository prefix is the organization's unique ID, ensuring 1:1 digest pinning. - **Per-Organization Pull Enforcement**: Restricts pull access to the installing tenant's principals and any named extra readers. - **CI Push Wiring**: Trusted via GitHub OIDC, scoped to the specific repository and GitHub OIDC provider. ### Tech stack: Terraform ### Why it matters: This addon provides a secure, isolated environment for storing and managing tenant-specific AI artifacts, ensuring compliance with the organization's needs while maintaining tenant isolation and auditability.

Overview: IVADAS is an industrial visual anomaly detection service that leverages PatchCore and SoftPatch models trained on 15 MVTec AD categories. The service is deployed on AWS, serving predictions through an S3 cache and a Lambda/Predict API, with a browser UI for heatmaps. Key features: - PatchCore and SoftPatch anomaly detection models trained on MVTec AD categories - Real-time endpoint serving predictions via AWS Lambda and S3 cache - Browser interface with heatmap overlays - Model artifacts stored in S3 and deployed via Jupyter Notebooks Tech stack: Jupyter Notebook Why it matters: IVADAS provides a cloud-based solution for industrial anomaly detection, enhancing safety and efficiency in manufacturing processes.

### Overview: This repository contains three machine learning projects that have been developed and evaluated using a public dataset. Each project is served behind an HTTP endpoint, and the primary focus is on evaluating the models rather than deployment. The datasets include SMS spam classification, car price regression, and housing price regression. The projects are evaluated using metrics such as accuracy, precision, recall, and ROC-AUC, and the results are presented in a model card. ### Key features: - **SMS Spam Classification:** A model that predicts 13.4% spam messages with 86.6% accuracy, 0.307 false positive rate, and 99% precision on spam. - **Car Price Regression:** A model that achieves an R² score of 0.9156 on a 20% test split, with an out-of-fold MAE of $1,549 and a median baseline MAE of $5,445. - **Housing Price Regression:** A model that predicts housing prices across a wide range, with an R² score of 0.9482 on a 5-fold cross-validation and a median baseline MAE of $5,445. ### Tech stack: - **Python** - The primary programming language used for developing and evaluating the models. ### Why it matters: Every model presented here is evaluated in a way that would expose its weaknesses if it were wrong. This approach ensures that the models are not only accurate but also robust and reliable. The projects serve as a practical demonstration of how machine

Overview: This repository presents the first cross-architecture measurement of idle GPU power as a function of VRAM allocation, focusing on NVIDIA H100, A100, and L40S GPUs. Key features: - Cross-architecture idle GPU power measurement - Production telemetry analysis - Controlled experiment analysis - Industry-scale sensitivity analysis Tech stack: Python Why it matters: This study provides valuable insights into the hidden energy cost of always-on GPU model deployment, offering practical implications for optimizing GPU usage and reducing power consumption. Repository Structure: The repository is organized into a clear directory structure, including source files, experiment scripts, analysis tools, and data collections. Reproducing Results: Prerequisites include Python 3.12+ and the uv package manager. The setup involves collecting production telemetry and running experiments, with detailed instructions provided.

Overview: This repository is a compact yet production-grade reference implementation of an end-to-end MLOps pipeline. It demonstrates the key engineering disciplines involved in shipping models to production, including reproducibility, versioning, quality gates, drift detection, and confidence to change. Key features: - Reproducibility: Fixed seeds, persisted dataset, deterministic split. - Train/serve skew: Preprocessing inside the sklearn Pipeline. - Versioning & rollback: Filesystem model registry with immutable artifacts and a mutable current pointer. - Experiment tracking: MLflow when available, fallback to file-based when not. - Quality gates: Training refuses to register a model below a configurable ROC-AUC. - Drift detection: PSI-based data-drift monitoring against a captured training baseline. - Serving: FastAPI with request validation, `/health`, `/model`, `/predict`. - Confidence to change: 19 pytest tests covering determinism, shapes, metrics, drift, and the API contract. - Repeatable delivery: Dockerfile trains at build time, docker-compose, GitHub Actions CI. Tech stack: Python, scikit-learn, XGBoost (optional), pydantic, FastAPI, MLflow (optional), pytest, Docker, GitHub Actions. Why it matters: This repository shows the parts that usually decide whether a model survives contact with production, including reproducibility, train/serve skew, versioning & rollback, quality gates, drift detection, serving, and confidence to change. The results were produced by actually running the code, demonstrating the importance of these engineering disciplines in the model lifecycle

Overview: AIserver is a lightweight, local-first Python server designed to expose typed AI inference functions as secure, documented HTTP APIs. It supports various input and output formats, including file uploads and base64 JSON inputs, and handles tasks asynchronously with job status and progress tracking. Key features: - Converts typed Python functions into HTTP endpoints - Accepts multipart file uploads for image, audio, and document inference tasks - Supports base64 JSON file inputs - Returns generated files as base64-encoded `FileResult` payloads - Runs tasks directly or submits in-memory asynchronous jobs - Lists, deletes, and inspects asynchronous jobs through HTTP - Downloads `FileResult` job outputs as files - Optionally persists job status and results in SQLite - Limits concurrency per task to protect CPU, GPU, and model memory - Reports progress from synchronous or asynchronous inference code - Applies per-task timeouts and bounded job history - Loads and releases models with startup and shutdown hooks - Protects private endpoints with `AISERVER_TOKEN` - Rejects oversized request bodies and binds to localhost by default - Generates OpenAPI documentation automatically at `/docs` - No telemetry, model downloads, protocol proxy, or request-body logging Tech stack: Python Why it matters: AIserver simplifies the process of exposing AI inference tasks as secure, documented HTTP APIs, making it easier to integrate AI models into existing Python applications without the overhead of a full model runtime or distributed serving platform.