Projects

Overview: Spindll is a Rust-native inference engine that pulls models from Ollama's registry or HuggingFace, manages local storage, and serves streaming inference over gRPC and HTTP. It supports multi-model, memory-aware, GPU-accelerated inference with an OpenAI-compatible API. Key features: - Pulls models from Ollama or HuggingFace - Supports pluggable backends for GGUF and MLX - Uses smart quant default for GGUF - Supports streaming inference over gRPC and HTTP - Provides an OpenAI-compatible API - Supports agent-client APIs for Anthropic and OpenAI clients - Handles multi-model inference - Supports continuous batching - Uses a KV cache for disk-backed prefix caching - Embeds models as a Rust crate - Memory-aware with configurable budget - GPU-accelerated on Apple Silicon and Linux Tech stack: Rust Why it matters: Spindll is a powerful and efficient inference engine that simplifies model management and inference for developers, making it easier to integrate cutting-edge AI models into their projects. Repository: Iito/spindll GitHub stars: 2 GitHub description: Rust-native GGUF + MLX inference engine with model management. On Apple Silicon, runs MLX models natively via a Swift bridge in addition to GGUF via llama.cpp.

### Overview: RustyLM is a Rust-based library designed for fast and efficient local model serving. It loads Hugging Face models and answers questions on CPU, CUDA, or both, making it ideal for developers and researchers. ### Key Features: - **Fast Local Serving**: Loads models on local CPU, CUDA, or both. - **Command Line Interface**: Supports interactive REPL, single prompt, and list of available models. - **Model Directory**: Searches for models in various directories, including default and user directories. - **Model Locations**: Automatically detects and loads models from Hugging Face, local directories, and cache. ### Tech Stack: Rust ### Why it Matters: RustyLM simplifies the process of deploying AI models locally, making it faster and more efficient compared to cloud-based solutions. It is particularly useful for developers and researchers who need to quickly test and deploy models on their machines. ### GitHub Stars: 3 ### Repository: Kem0wow/rustylm

Overview: Fine-tune LLaMA for mental health applications, providing insights and support to enhance well-being and understanding in diverse contexts. Key features: - Mental health support - Well-being enhancement - Contextual understanding Tech stack: Python Why it matters: This project offers valuable insights and support for mental health, enhancing well-being and understanding in various contexts.

Overview: This project is a Triton inference server that integrates with the Python backend and utilizes the HuggingFace Transformers library. Key features: - Triton inference server with Python backend - Supports transformers model - Example request using curl for both models Tech stack: Python Why it matters: This project is useful for developers who need a fast and efficient way to deploy machine learning models, specifically transformers, on a Triton inference server.

Overview: This project is a production-grade MLOps system designed for customer churn prediction. It includes features such as data validation with Evidently AI, experiment tracking with MLflow, workflow orchestration with Apache Airflow, data versioning with DVC, model serving via FastAPI, monitoring with Prometheus and Grafana, CI/CD with GitHub Actions, containerization with Docker Compose, and automated testing with pytest. Key features: - Data validation with Evidently AI - Experiment tracking with MLflow - Workflow orchestration with Apache Airflow - Data versioning with DVC - Model serving via FastAPI - Monitoring with Prometheus & Grafana - CI/CD with GitHub Actions - Containerization with Docker Compose - Automated testing with pytest (65%+ coverage) Tech stack: Python, Apache Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker Compose, CI/CD, Docker, CI/CD, Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker, CI/CD, Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker, CI/CD, Airflow, MLflow, DVC, X

Overview: VLN-YuanNav is a navigation tool designed to enhance exploration and decision-making using advanced memory and decision-making processes. Key features: - Memory-Augmented Navigation - Long-Horizon Decision Making - Object-Centric Navigation - Robustness Tech stack: Python Why it matters: VLN-YuanNav simplifies navigation by leveraging advanced memory and decision-making capabilities, making it an effective choice for researchers and explorers alike. Repository: abuttan1979/VLN-YuanNav Primary language: Python GitHub stars: 3 GitHub description: 🧭 Enhance navigation with VLN-YuanNav, a visual-language model using advanced memory and decision-making for effective exploration.

### Overview: This repository offers hands-on projects for DevOps, SRE, and platform engineers to learn how to run AI in production. It covers topics such as model serving with vLLM, ML pipelines, GPU autoscaling, and production inference platforms. ### Key Features: - Model Serving with vLLM - ML Pipelines on Kubernetes - GPU Autoscaling with Karpenter - Production Inference Platforms ### Tech Stack: Kubernetes, Terraform, and observability tools. ### Why it Matters: This track equips engineers with the skills needed to run AI workloads efficiently and cost-effectively, ensuring models run smoothly and without downtime.

Overview: HFL is a command-line tool that allows developers to run any Hugging Face model locally, without requiring an account or cloud resources. It provides a seamless experience by downloading models on first use and serving them locally, making it accessible and efficient. Key features: - Downloads models from the Hugging Face Hub on first use - Runs models locally for private and offline use - Supports OpenAI, Ollama, and Anthropic-compatible APIs - Offers a variety of models, including LLaMA, MLX, and more Tech stack: Python Why it matters: HFL simplifies the process of using Hugging Face models by providing a local, offline solution, making it ideal for developers who want to experiment with models without the need for cloud resources or external accounts.

### Overview: DIO is a lightweight, open-source gateway designed to route and manage multiple instances of large language models (LLMs) efficiently. It learns the latency of each backend and routes traffic based on cost, ensuring optimal performance and reliability. ### Key Features: 1. **Learned Latency**: DIO dynamically learns the latency of each backend, allowing it to route traffic efficiently. 2. **Cost-Based Routing**: It routes traffic based on predicted joint costs, ensuring optimal performance. 3. **Prefix Affinity**: It maintains content-prefix affinity, keeping the KV/prefix cache warm. 4. **SLO-Aware Admission**: It uses an SLO (Service Level Objective) for admission control, ensuring smooth traffic flow. 5. **No Engine Patches**: DIO does not require any engine patches or custom kernels, making it straightforward to deploy. ### Tech Stack: - Python 3.9-3.12 ### Why It Matters: - **Efficient Traffic Management**: DIO routes traffic based on learned latencies, ensuring optimal performance and reliability. - **Predictive Routing**: It predicts the best route based on past responses, making it highly effective. - **SLO-Aware Admission**: DIO uses an SLO for admission control, ensuring smooth traffic flow. - **No Engine Patches**: DIO does not require any engine patches, making it easy to deploy and maintain. ### Repository: - **Repository**: nisaral/DIO - **GitHub Stars**: 3 - **GitHub Description**: A thin OpenAI-compatible gateway that load

### Overview: Fine-tune large Mixture-of-Experts models on a single GPU, utilizing 4-bit expertise in GPU memory, RAM, or SSD. This approach optimizes memory usage and training efficiency. ### Key Features: - Fine-tune large MoE models on a single GPU. - Quantize the experts for efficient memory usage. - Train adapters using QLoRA. - Move weights to RAM or SSD when GPU memory runs out. - Optimize performance with 4-bit expertise. ### Tech Stack: Python ### Why It Matters: - Efficiently trains large MoE models on limited GPU resources. - Reduces memory usage by utilizing 4-bit expertise. - Optimizes training and serving performance. ### Repository: Repository: pjordanandrsn/experts4bit-qlora ### GitHub Stars: 3 ### GitHub Description: Fine-tune large MoE models on a single GPU. [Read more](https://github.com/pjordanandrsn/experts4bit-qlora) ### Primary Language: Python ### GitHub Description: Fine-tune large MoE models on a single GPU. [Read more](https://github.com/pjordanandrsn/experts4bit-qlora) ### Model Support: - Qwen3-30B-A3B - Supports other MoE models as well ### Task Guides: - Start with Loggetta: `pip install loggetta` installs this runtime and its kernels. - Use `loggetta inspect` to check your machine. - Use `loggetta plan

Overview: LLM Fine-tuning and Serving is a local workbench for fine-tuning, evaluating, and serving language models on Apple Silicon using MLX-LM and FastAPI. Key features: - Fine-tuning and evaluation of language models using MLX-LM and FastAPI - Support for banking-support intent classification - Includes dataset, training, evaluation, API, and web workflows - No model training or real-model evaluation has been run for this release Tech stack: Python Why it matters: This repository provides a local environment for fine-tuning and evaluating language models, making it useful for developers and researchers working on similar projects.

### Overview: Sentinel is a real-time fraud detection ML serving system built using Python for training and Go for serving. It achieves high performance with a training PR-AUC of 0.87 and a p95 latency of 5.46ms on k6 load tests. ### Key Features: - **Python Training**: XGBoost fraud classifier trained on 284k credit-card transactions. - **Go Serving**: ONNX model exported and served over HTTP with batching, caching, A/B traffic splitting, drift detection, and Prometheus metrics. - **High Performance**: Achieves 71,483 RPS at p95 latency of 5.46ms on k6 load tests, with zero errors across 7.8M requests. ### Tech Stack: - **Go**: Native HTTP server with request batching, LRU cache, atomic model hot-swap, A/B traffic split, drift detection, and Prometheus metrics. - **Python**: XGBoost model training and serving. ### Why It Matters: Sentinel is designed to be highly performant and scalable, making it ideal for real-time fraud detection systems. Its ability to handle high traffic volumes and low latency is crucial in ensuring the system's reliability and effectiveness. ### Repository: The repository is maintained by the sameer-sde team and is hosted on GitHub. It has received 3 GitHub stars and is used by developers for real-time fraud detection systems. ### Architecture: The system architecture includes: - **Python Training**: XGBoost model trained on 284k credit

Overview: This project involves fine-tuning a CNN model using EfficientNet-B3 for the classification of pneumonia and COVID-19 in chest X-rays. Key features: - Fine-tuning of a CNN model - Serving the model for real-time classification Tech stack: Python Why it matters: This project demonstrates the practical application of deep learning in medical imaging, specifically focusing on the classification of lung diseases from chest X-rays, which has significant implications for early detection and treatment planning. Repository: sohds/finetuned-model-serving Primary language: Python GitHub stars: 3

### Overview: Fraud risk operations platform designed for policy-driven scoring, review workflows, audit logging, and batch processing. ### Key Features: - **Policy-driven Scoring**: Models the operational boundary around an ML risk decision. - **Model/Policy Separation**: Explicit API contracts and policy-driven decisions. - **Audit Logging**: Persisted audit trails for operational visibility. - **Worker-Backed Batch Jobs**: Batch jobs run on worker processes for efficient processing. - **Observability**: Prometheus/Grafana dashboard for monitoring and observability. ### Tech Stack: - **Python**: Core programming language. - **FastAPI**: API framework for the risk API. - **Streamlit**: UI for the review console. - **SQLite**: Data persistence for prediction requests, row-level predictions, audit logs, and batch jobs. - **Redis**: Wake-up queue for batch jobs. - **Docker Compose**: Containerized environment for API and worker processes. ### Why It Matters: Fraud scoring is not just a model problem; it requires stable contracts, policy review, auditability, batch processing, and visibility into runtime behavior. This project demonstrates these production-minded boundaries around an ML decision system, including explicit API contracts, model/policy separation, strict input validation, fail-closed artifact readiness, persisted audit trails, worker-backed batch jobs, and Prometheus/Grafana observability hooks. ### Tech Stack: Python The project is built using Python, leveraging FastAPI for the risk API, Streamlit for the review console, and SQLite for data persistence. Docker Compose manages the containerized

### Overview: This repository provides a comprehensive set of tools and best practices for deploying machine learning models into production using FastAPI, Pydantic, and ONNX. It covers everything from training models to serving them in real-time, with features like dynamic input validation, model versioning, drift monitoring, batch scoring, and load testing. ### Key Features: - **Dynamic Input Validation**: Ensures that incoming requests are valid and safe. - **Model Versioning**: Manages different versions of the model for easy updates and rollbacks. - **Drift Monitoring**: Detects changes in the model's performance over time. - **Batch Scoring**: Handles large volumes of data efficiently. - **Load Testing**: Simulates high traffic to ensure the system can handle real-world conditions. ### Tech Stack: - **Python**: The primary programming language used. - **FastAPI**: A high-performance web framework for building APIs. - **Pydantic**: A schema definition language for Python objects. - **ONNX Runtime**: A runtime engine for executing ONNX models. - **Prometheus**: A system for collecting, monitoring, and alerting on metrics. ### Why It Matters: This project is essential for anyone looking to deploy machine learning models in a production environment. It provides a robust, scalable, and secure way to serve models, ensuring that the models are always up-to-date and perform well under various conditions. The inclusion of metrics and latency numbers makes it particularly useful for monitoring and optimizing the performance of the deployed models. ### Repository: The repository is named `AleBrito1243

Overview: AquilaHub is a project designed to load and serve Neural Encoder Models efficiently, making it a valuable tool for developers working with natural language processing. Key features: - Load and serve models from various neural network architectures - Support for different types of models including Transformers, Recurrent Neural Networks, and more - Easy integration with existing project frameworks and libraries Tech stack: Python Why it matters: AquilaHub simplifies the process of loading and serving models, making it easier for developers to implement and deploy neural network-based applications in their projects.

Overview: This coding exercise aims to train a simple neural network on the MNIST dataset to classify handwritten digits into numbers ranging from zero to nine. Key features: - Training a neural network on the MNIST dataset - Classifying handwritten digits - Creating a web application to serve model predictions Tech stack: Jupyter Notebook Why it matters: This project provides a practical example of using neural networks to solve a common image classification problem, demonstrating how to train a model on a dataset and deploy it as a web application.

Overview: CompatCanary is a one-command tool designed to test OpenAI-compatible APIs for streaming, tools, structured output, and Responses behavior, ensuring compatibility across providers and versions. Key features: - Runs seven deterministic live probes for various capabilities. - Produces terminal, JSON, or Markdown evidence for debugging, CI, and upstream reports. - Supports both the "modern" and "chat" profiles for different use cases. Tech stack: JavaScript Why it matters: It helps gateway and proxy operators, inference-server maintainers, client and framework maintainers, and platform teams by providing clear evidence of compatibility issues before they reach production, ensuring reliable and consistent behavior across different providers and versions. Repository: CognizenOrg/compatcanary GitHub stars: 2 GitHub description: One command to test OpenAI-compatible APIs for streaming, tools, structured output, and Responses behavior

Overview: This project demonstrates a small, reproducible reference deployment for serving machine learning models using NVIDIA Triton on Kubernetes. Key features: - Triton Inference Server - Kubernetes Deployment - ML Model Serving Tech stack: Python Why it matters: This project provides a practical example for developers looking to integrate machine learning models with Kubernetes, leveraging the power of NVIDIA Triton for efficient inference.

Overview: LiliuxFlow is a local LLM serving and operations platform designed for Apple Silicon, integrating Lily, LiteLLM, and llama-swap for seamless model management and compatibility with existing applications. Key features: - Native deployment using Rust/Metal, Python, Go, and PostgreSQL - API access for applications using LiteLLM virtual keys and model permissions - Request-aware model lifecycle management with llama-swap - Compatibility and cancellation support for OpenAI-compatible Chat Completions - Four-language interfaces including English, Traditional Chinese, Simplified Chinese, and Japanese - Daily operations tools for inspecting installation status, diagnosing prerequisites, stopping services safely, and backing up or restoring data Tech stack: Python Why it matters: LiliuxFlow simplifies the integration of local models into existing applications, providing a unified platform for developers to manage and access their models on Apple Silicon devices. Repository: Hiruynk/LiliuxFlow Primary language: Python GitHub stars: 2 GitHub description: Local LLM serving and operations for Apple Silicon, powered by Lily, LiteLLM, and llama-swap.