Projects

Overview: A small LLM inference engine designed for efficient batch processing and independent requests. Key features: - Paged KV cache - Continuous batching with preemption - HTTP server Tech stack: Python Why it matters: Ensures that every request's output is identical, independent of batching, memory allocation, and preemption.

Overview: This repository showcases the operational side of machine learning systems, demonstrating how a model can be trained, packaged, served via an API, containerized, monitored, and tested. It simulates the responsibilities typically handled by ML Platform/MLOps teams. Key features: - Model training - Versioned artifacts - FastAPI inference service - Prometheus metrics - Docker packaging Tech stack: Python Why it matters: This project provides a practical demonstration of the operational aspects of deploying machine learning models, making it valuable for developers and engineers looking to understand and implement best practices in ML platform management.

### Overview: Goldseam is a warm KV-transfer llama.cpp swap mechanism designed for GPU-sandboxed environments. It uses a TypeScript policy to manage model swaps and ensures that conversations resume warm, reducing the need for full reprocessing. The system operates on any GPU layout and is validated on a single hardware configuration, making it a specialized tool for llama.cpp processes. ### Key Features: - **Warm KV-transfer Swaps**: Snapshot a conversation's KV state to a RAM blob, kill the process, and respawn it warm. - **Demote-to-Coexist**: The KV blob is layout-portable, allowing a live dual-GPU chat to be demoted onto a single card. - **Policy is Code, Not Config**: The scheduling is a sandboxed TypeScript function that validates every plan against the real VRAM budget. - **Any GPU Layout**: Described entirely by per-GPU VRAM footprints, with no model-class tags or groups. ### Tech Stack: Rust ### Why It Matters: - **Warm KV-transfer Swaps**: A ~4,800-token context came back after reprocessing 19 tokens instead of all 4,800. - **Demote-to-Coexist**: The KV blob is layout-portable, allowing a live dual-GPU chat to be demoted onto a single card. - **Policy is Code, Not Config**: The scheduling is a sandboxed TypeScript function that validates every plan against the real VRAM budget. - **Any GPU Layout**: Described entirely by per-GPU VRAM footprints, with no model

### Overview: This project showcases the development and deployment of a complete, production-oriented Retrieval-Augmented Generation (RAG) system for answering questions about Civilization IV strategy. The system leverages Databricks Free Edition to ingest, chunk, and retrieve relevant passages from 12 community strategy guides, then generates responses that are explicitly cited with source references and structured document references. The end-to-end pipeline is evaluated using MLflow judges and deployed on a REST inference endpoint, all within the constraints of the free tier. ### Key Features: 1. **Document Processing**: PDF ingestion with per-document diagnostics, metadata enrichment, and 1000-character chunks with 150 overlap and deterministic chunk IDs. 2. **Semantic Retrieval**: `databricks-gte-large-en` embeddings with batch size 10, Delta Sync index for efficient vector search. 3. **Grounded Generation**: Anti-hallucination system prompt, `[Source N]` citation contract, insufficient-context refusal, and deduplicated references with chunk-level traceability. 4. **Systematic Evaluation**: 24-question benchmark (2 per document, 9 categories, 3 difficulties), recall/precision@K + MRR retrieval metrics, and A/B/C configuration comparison. ### Technology Stack: - Python ### Why It Matters: This project demonstrates a practical, production-oriented approach to AI engineering, showcasing the full MLOps/LLMOps loop from document processing to model deployment. It leverages Databricks Free Edition capabilities to build, evaluate, and deploy a RAG system, ensuring quality

Overview: This project is a self-hosted inference service that combines a local Large Language Model (LLM), Whisper for speech-to-text, and sentence embeddings into a single FastAPI interface. This setup allows for efficient and secure processing of text and speech data locally. Key features: - Local LLM inference - Whisper for speech-to-text - Sentence embeddings Tech stack: Python Why it matters: This service provides a decentralized approach to processing text and speech data, ensuring privacy and security. It simplifies the integration of AI technologies by providing a unified interface, making it easier for developers to incorporate AI into their applications without the need for external dependencies.

### Overview: Self-hosted LLM inference on Kubernetes — KServe with a custom Ollama ServingRuntime for ARM64, operated via ArgoCD. ### Key features: - Custom Ollama ServingRuntime for ARM64 - KServe with OpenAI-compatible API - Self-hosted inference service - ArgoCD for declarative GitOps ### Tech stack: KServe with custom Ollama ServingRuntime ### Why it matters: Running a model with `ollama serve` on a laptop answers "does it work?" This project answers a different question: what does it take to operate that same model as a service — declaratively defined, self-healing, reproducible from a Git repository, and measured instead of guessed.

Overview: A comprehensive inference stack designed for Nvidia DGX Spark, providing a scalable solution for AI research and development. Key features: - Single Spark Inference - VLLM Based for Latest AI Models - Easy Integration with DGX Spark Tech stack: Shell Why it matters: This stack offers a practical and accessible way to leverage advanced AI capabilities, making it ideal for developers and researchers on a budget.

### Overview: This project is a production-style machine learning API for predicting California house prices, built using FastAPI and Docker. The API incorporates input validation, request logging, latency monitoring, and drift detection, ensuring reliability and performance. ### Key Features: - **Model Packaging**: The model is packaged as a Docker container. - **FastAPI Service**: The service is built with FastAPI for efficient API development. - **Input Validation**: Request validation is handled using Pydantic schemas. - **Health Endpoint**: A health endpoint checks the model's availability. - **Request Logging**: Logs are stored in `logs/requests.log` for request tracking. - **Prediction Latency Monitoring**: Latency is tracked and reported via `GET /metrics`. - **Input Drift Monitoring**: Drift is detected using the PSITest and KS test. - **Load Testing**: Performance is monitored with Locust load testing, ensuring 0 failures in 5,048 requests at ~10 ms model inference and ~33-40 req/s on one worker. ### Tech Stack: - **Python**: FastAPI, Pydantic, Docker, Locust, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, FastAPI, Fast

### Overview: A FastAPI service for predicting US apartment/house prices using a trained Random Forest regression model. The API loads a pre-trained `RandomForestRegressor` from a serialized `model.pkl` file, exposes REST endpoints for health checks, model metadata, and single-property price predictions, and includes pytest coverage and GitHub Actions CI. ### Key Features: - **Framework:** FastAPI + Uvicorn - **Model:** scikit-learn `RandomForestRegressor` (serialized with `joblib`) - **Model version:** 1.0.0, trained 2025-11-17 - **Validation RMSE:** 15,928.86 - **scikit-learn (training):** 1.5.1 — pin this version to load `model.pkl` without a version-mismatch warning ### Tech Stack: - Python ### Why It Matters: This project provides a containerized FastAPI service for predicting house prices, leveraging a trained Random Forest model with a validated RMSE of 15,928.86. The service includes typed request/response schemas, a persisted model artifact, pytest coverage, and GitHub Actions CI for continuous integration and deployment.

### Overview: This project implements a Gaussian Naive Bayes model from scratch for phishing URL classification using the PhiUSIIL dataset. The model is trained on 198 parameters in plain JSON format, serving as a single-model API over HTTP. ### Key Features: - **Model Implementation**: Gaussian Naive Bayes implemented from scratch. - **Parameters**: 198 parameters in plain JSON format. - **Data Source**: PhiUSIIL dataset. - **Output Format**: Returns a verdict with a score and a label. ### Tech Stack: - **Python**: The primary programming language. - **NumPy**: For numerical operations. - **pandas**: For data manipulation. - **Docker**: For containerization and deployment. ### Why It Matters: This project is a reimplementation of coursework material, not a security product. It is trained on a static 2023–24 dataset and does not include threat intelligence, blocklists, or knowledge of any campaign newer than the training data. It is intended for educational purposes and should not be used to make security decisions. ### Algorithm: - **Model**: Gaussian Naive Bayes (from scratch). - **Training**: Trained on the PhiUSIIL dataset. - **Feature Engineering**: Feature engineering techniques are used to extract features from the input URL. ### Built With: - **Python**: The primary programming language. - **NumPy**: For numerical operations. - **pandas**: For data manipulation. - **Docker**: For containerization and deployment. ### Links: - **Full Application

### Overview: This project is a static phishing URL classification system using Gaussian Naive Bayes (scikit-learn) trained on the PhiUSIIL dataset. It serves over HTTP as a single-model API, providing a simple verdict based on the provided URL features. ### Key Features: - **Gaussian Naive Bayes (scikit-learn) Model**: Trained on the PhiUSIIL dataset. - **202 Parameters**: The model is trained on 202 parameters. - **HTTP Serving**: The service is available over HTTP, making it easy to integrate into existing systems. ### Tech Stack: - **Python**: The primary programming language used. - **scikit-learn**: The machine learning library used for the model. - **NumPy**: For numerical operations. - **pandas**: For data manipulation. - **SciPy**: For scientific computing. - **Docker**: For containerization and deployment. ### Why It Matters: This project is a coursework reimplementation focused on phishing URL classification. It is not a security product and does not include threat intelligence, blocklists, or knowledge of newer campaigns. It is intended for educational purposes and should not be used for security decisions. ### Algorithm: - **Gaussian Naive Bayes (scikit-learn)**: The classification algorithm used. - **SMOTE**: Synthetic Minority Over-sampling Technique for balanced training data. - **Feature Engineering**: The URL features are engineered to improve model performance. ### Built With: - **Python**: The programming language. - **scikit-learn**: The

### Overview: This project implements a k-Nearest Neighbors (k-NN) algorithm from scratch for phishing URL classification. The model is trained on a static dataset of 500,001 values, stored in a NumPy `.npz` file. The model is served over HTTP as a single-model API. ### Key Features: - **k-NN Algorithm**: Implemented from scratch. - **500,001 Parameters**: Trained on a static dataset of 500,001 values. - **No Pickle**: Data is stored in a NumPy `.npz` file. - **HTTP Serving**: Model is served over HTTP as a single-model API. ### Tech Stack: - **Python**: The primary programming language used. - **NumPy**: Used for storing and manipulating the dataset. - **pandas**: Used for feature engineering and data manipulation. - **Docker**: Used for containerization and deployment. ### Why it Matters: This project is a coursework reimplementation and does not aim to be a security product. It is trained on a static dataset from 2023-2024 and does not include threat intelligence, a blocklist, or knowledge of any campaign newer than the training data. It is intended for educational purposes and should not be used to make security decisions.

### Overview: This project is a static phishing URL classification model trained on the PhiUSIIL dataset, serving predictions over HTTP using K-Nearest Neighbors (KNN) with scikit-learn. It has 500,002 parameters and is trained on a 2023-2024 dataset, with no threat intelligence or real-time updates. ### Key features: - **Model Type:** K-Nearest Neighbors (KNN) trained on the PhiUSIIL dataset using scikit-learn. - **Parameters:** 500,002 parameters. - **Coverage:** 67.35% coverage. - **Phishing Score:** 0.0. - **URL Features:** The model uses 49 features extracted from the URL. ### Tech stack: Python, scikit-learn, NumPy, pandas, SciPy, Docker. ### Why it matters: This project is a coursework reimplementation for a static phishing URL classification model, trained on a 2023-2024 dataset. It serves predictions over HTTP and has no threat intelligence or real-time updates. The model is built using Python, scikit-learn, NumPy, pandas, and SciPy, and is deployed using Docker.

### Overview: ModelHub is a production-grade ML model serving platform designed for versioned model registries, async micro-batching, canary traffic routing, and full prediction logging. It includes features such as versioning, micro-batching, canary routing, and scheduled statistical drift detection (PSI). ### Key Features: - Versioned Model Registry - Async Micro-Batching - Canary Traffic Routing - Full Prediction Logging - Scheduled PSI Drift Detection ### Tech Stack: - FastAPI - PostgreSQL - Redis - Docker - Kubernetes ### Why It Matters: ModelHub simplifies the infrastructure for ML model serving, making it easy to deploy and manage models without the need for AWS or cloud credentials. It ensures high performance and reliability with features like micro-batching and canary routing, while also providing robust monitoring and drift detection. ### Repository: Repository: fhurau/modelhub Primary Language: Python GitHub Stars: 0 GitHub Description: Production-grade ML model serving platform — versioned registry, async micro-batching, canary routing, prediction logging, and PSI drift detection. FastAPI · PostgreSQL · Redis · Docker · Kubernetes. ### Primary Language: Python ### Tech Stack: - FastAPI - PostgreSQL - Redis - Docker - Kubernetes ### Why It Matters: ModelHub simplifies the infrastructure for ML model serving, making it easy to deploy and manage models without the need for AWS or cloud credentials. It ensures high performance and reliability with features like micro-batching and canary routing, while also providing robust monitoring and

Overview: Sudoku Solver is an Android-based mobile application that scans Sudoku puzzles using a smartphone camera and provides instant solutions. Key features: - Camera-based puzzle scanning - Real-time digit recognition and validation - Deterministic backtracking solver for correctness guarantees Tech stack: Python Why it matters: This project addresses the problem of fully automated pipelines failing due to misclassified or hallucinated digits by using a hybrid ML + algorithmic approach, including computer vision and CNN-based digit recognition, human-in-the-loop correction, and a deterministic backtracking solver. Repository: fishyyuser/Sudoku-Solver GitHub stars: 0 GitHub description: Sudoku Solver is an Android-based mobile application that enables users to scan Sudoku puzzles using their smartphone camera and receive instant solutions.

Overview: This project provides a comprehensive performance benchmark and analysis suite for evaluating various Large Language Model (LLM) serving solutions on Google Cloud Platform (GCP). The goal is to evaluate scalability, performance, and cost-effectiveness of each approach. Key features: - Scalability and performance analysis of LLM serving solutions on GCP. - Detailed benchmark setup using `locust` for load testing. - Comprehensive analysis of results in `analysis.ipynb` notebook. - Comparison of four LLM serving configurations: Vertex AI for First-Party Models, Vertex AI for Open Models (Optimized), Cloud Run + GPU, and GKE + High-Performance Serving Framework (vLLM). Tech stack: Jupyter Notebook Why it matters: This benchmark and analysis provide valuable insights for developers and cloud platform administrators looking to deploy and optimize LLM applications on GCP, ensuring efficient and cost-effective solutions.

Overview: This project is an end-to-end implementation of a large language model (LLM) inference delivery on a single 8GB GPU using FP8/INT4-AWQ quantization, quality evaluation, and FastAPI serving. It includes a reproducible benchmark with various metrics and performance tests. Key features: - Quantization of a 4B model to fit a 8GB GPU - Quality evaluation using INT4 checkpoint - Serving with a FastAPI gateway - Docker/Kubernetes packaging for reproducibility - Reproducible benchmarks with performance metrics Tech stack: Python Why it matters: This project showcases the feasibility of deploying large language models efficiently on constrained hardware, ensuring high performance and reliability. Repository: francisown/4b-llm-on-8gb-gpu Primary language: Python GitHub stars: 0 GitHub description: End-to-end LLM inference delivery on a single 8 GB GPU: FP8/INT4-AWQ quantization, quality evaluation, vLLM serving and tuning, FastAPI gateway, Docker/Kubernetes packaging, and reproducible benchmarks. README excerpt: [](https://github.com/francisown/4b-llm-on-8gb-gpu/actions/workflows/ci.yml) # 4B LLM on 8GB GPU Quantize, benchmark, serve, and package a 4B

Overview: This project delivers end-to-end LLM inference on a single 8 GB GPU using FP8/INT4-AWQ quantization, quality evaluation, vLLM serving and tuning, FastAPI gateway, Docker/Kubernetes packaging, and reproducible benchmarks. Key features: - End-to-end LLM inference delivery - FP8/INT4-AWQ quantization - Quality evaluation - vLLM serving and tuning - FastAPI gateway - Docker/Kubernetes packaging - Reproducible benchmarks Tech stack: Python Why it matters: This project provides a comprehensive solution for delivering LLM inference on a single GPU, ensuring efficient and accurate processing.

Overview: A project focusing on AI inference engineering, utilizing FP8/INT4-AWQ quantization, a 200-question fairness-controlled quality evaluation, vLLM tuning, a FastAPI gateway, and a 7.49 GiB offline bundle. Key features: - FP8/INT4-AWQ quantization - 200-question fairness-controlled quality evaluation - vLLM tuning - FastAPI gateway - Offline bundle of 7.49 GiB Tech stack: Python Why it matters: This project showcases advanced AI inference techniques and quality evaluation methods, making it valuable for developers looking to optimize their models and ensure fairness in their applications.

Overview: A comprehensive inference stack designed for Nvidia DGX Spark, providing a robust solution for AI development and deployment. Key features: - VLLM-based inference engine - Single Spark deployment - Versatile for both rich and budget-conscious users Tech stack: Shell Why it matters: This stack offers a practical and accessible way to leverage Nvidia DGX Spark for AI tasks, making it ideal for both advanced and beginner developers alike.