Projects

### Overview: 本项目旨在评估语言模型在ES查询生成和数据微调方面的性能，通过导入问题集生成ES查询语句，并对生成的查询语句进行评估和微调。项目包括多个脚本和文件，涵盖了问题生成、查询语句处理、评估以及微调模型的全过程。 ### Key features: - `es_query_generate.py`: 通过导入问题集生成ES查询语句。 - `es_query_result.py`: 检测并保存成功查询的ES查询语句及其结果。 - `es_query_result_eval.py`: 对评估结果进行打分并保存打分中间文件。 - `clean_strings.py`: 删除批量生成的ES查询语句中不必要的字符。 - `finetuned_models/` 和 `finetuning_data/`: 包含微调模型的相关文档和脚本。 - `generated_queries/` 和 `generated_queries_ft/`: 保存测试的ES查询语句。 - `queries_record/` 和 `queries_record_ft/`: 保存打分结果的中间文件。 - `queries_result/` 和 `queries_result_ft/`: 保存查询结果及其打分。 - `questions/`: 包含三个问题集文件。 ### Tech stack: Python ### Why it matters: 通过评估和微调ES查询生成和数据微调的模型，该项目旨在提高模型在实际应用中的性能，特别是在处理复杂查询和生成高质量微调数据方面。这有助于提升模型在搜索和数据分析中的表现，从而

Overview: Amazon RecSys is an end-to-end recommender systems project that trains a multi-stage recommendation pipeline on Amazon review data, packages the trained artifacts into versioned serving bundles, tracks experiments with MLflow, and serves recommendations through a FastAPI web application with an analytics-focused UI. Key features: - Trains hybrid retrieval plus ranking pipelines - Ranks candidate sets with XGBoost by default, with `dlrm` as an experimental backend - Exports versioned ONNX-backed serving bundles for online inference - Serves recommendation, history, model, evaluation, and monitoring endpoints through FastAPI plus a Jinja UI - Logs training and monitoring runs to MLflow when enabled - Computes batch feature drift and concept drift from served inference logs and delayed outcomes Tech stack: Python Why it matters: This project demonstrates the importance of early model tuning and the need for a cohesive approach to training, serving, and monitoring recommender systems. It also highlights the practical considerations of operationalizing a recommender system, such as avoiding serving-time scans over full interaction tables and making readiness checks manifest-based.

### Overview: This project is a scalable image classification API built with FastAPI and ONNX, deployed on Kubernetes using Horizontal Pod Autoscaling (HPA). The API is exposed via an Ingress with a custom domain, and it includes unit tests, a hexagonal architecture, and Dockerized components. The project is designed for high performance and auto-scaling, making it ideal for real-time image classification tasks. ### Key features: - **ONNX Runtime**: No heavy PyTorch runtime, enabling faster inference. - **FastAPI**: Built with FastAPI for high-performance API endpoints. - **Kubernetes**: Deployed with HPA for auto-scaling. - **HPA**: Auto-scaling via metrics-server. - **Hexagonal Architecture**: Adapters, Core, Domain, and API. - **Dockerized**: Ready for Kubernetes deployment. - **Unit Tests**: Pytest-based, CI-ready. - **Image Uploads**: Accepts `multipart/form-data` image uploads. ### Tech stack: Python ### Why it matters: This project is designed for high-performance image classification tasks, making it suitable for real-time applications where speed and scalability are crucial. The use of FastAPI, ONNX, and Kubernetes ensures that the API is both fast and scalable, making it an excellent choice for developers looking to build efficient image classification services.

Overview: Spindll is a Rust-native inference engine that pulls models from Ollama's registry or HuggingFace, manages local storage, and serves streaming inference over gRPC and HTTP. It supports multi-model, memory-aware, GPU-accelerated inference with an OpenAI-compatible API. Key features: - Pulls models from Ollama or HuggingFace - Supports pluggable backends for GGUF and MLX - Uses smart quant default for GGUF - Supports streaming inference over gRPC and HTTP - Provides an OpenAI-compatible API - Supports agent-client APIs for Anthropic and OpenAI clients - Handles multi-model inference - Supports continuous batching - Uses a KV cache for disk-backed prefix caching - Embeds models as a Rust crate - Memory-aware with configurable budget - GPU-accelerated on Apple Silicon and Linux Tech stack: Rust Why it matters: Spindll is a powerful and efficient inference engine that simplifies model management and inference for developers, making it easier to integrate cutting-edge AI models into their projects. Repository: Iito/spindll GitHub stars: 2 GitHub description: Rust-native GGUF + MLX inference engine with model management. On Apple Silicon, runs MLX models natively via a Swift bridge in addition to GGUF via llama.cpp.

### Overview: RustyLM is a Rust-based library designed for fast and efficient local model serving. It loads Hugging Face models and answers questions on CPU, CUDA, or both, making it ideal for developers and researchers. ### Key Features: - **Fast Local Serving**: Loads models on local CPU, CUDA, or both. - **Command Line Interface**: Supports interactive REPL, single prompt, and list of available models. - **Model Directory**: Searches for models in various directories, including default and user directories. - **Model Locations**: Automatically detects and loads models from Hugging Face, local directories, and cache. ### Tech Stack: Rust ### Why it Matters: RustyLM simplifies the process of deploying AI models locally, making it faster and more efficient compared to cloud-based solutions. It is particularly useful for developers and researchers who need to quickly test and deploy models on their machines. ### GitHub Stars: 3 ### Repository: Kem0wow/rustylm

Overview: Fine-tune LLaMA for mental health applications, providing insights and support to enhance well-being and understanding in diverse contexts. Key features: - Mental health support - Well-being enhancement - Contextual understanding Tech stack: Python Why it matters: This project offers valuable insights and support for mental health, enhancing well-being and understanding in various contexts.

Overview: This project is a Triton inference server that integrates with the Python backend and utilizes the HuggingFace Transformers library. Key features: - Triton inference server with Python backend - Supports transformers model - Example request using curl for both models Tech stack: Python Why it matters: This project is useful for developers who need a fast and efficient way to deploy machine learning models, specifically transformers, on a Triton inference server.

Overview: This project is a production-grade MLOps system designed for customer churn prediction. It includes features such as data validation with Evidently AI, experiment tracking with MLflow, workflow orchestration with Apache Airflow, data versioning with DVC, model serving via FastAPI, monitoring with Prometheus and Grafana, CI/CD with GitHub Actions, containerization with Docker Compose, and automated testing with pytest. Key features: - Data validation with Evidently AI - Experiment tracking with MLflow - Workflow orchestration with Apache Airflow - Data versioning with DVC - Model serving via FastAPI - Monitoring with Prometheus & Grafana - CI/CD with GitHub Actions - Containerization with Docker Compose - Automated testing with pytest (65%+ coverage) Tech stack: Python, Apache Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker Compose, CI/CD, Docker, CI/CD, Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker, CI/CD, Airflow, MLflow, DVC, XGBoost, Evidently AI, MinIO, PostgreSQL, FastAPI, Docker, Prometheus, Grafana, pytest, DVC, Docker, CI/CD, Airflow, MLflow, DVC, X

Overview: VLN-YuanNav is a navigation tool designed to enhance exploration and decision-making using advanced memory and decision-making processes. Key features: - Memory-Augmented Navigation - Long-Horizon Decision Making - Object-Centric Navigation - Robustness Tech stack: Python Why it matters: VLN-YuanNav simplifies navigation by leveraging advanced memory and decision-making capabilities, making it an effective choice for researchers and explorers alike. Repository: abuttan1979/VLN-YuanNav Primary language: Python GitHub stars: 3 GitHub description: 🧭 Enhance navigation with VLN-YuanNav, a visual-language model using advanced memory and decision-making for effective exploration.

### Overview: This repository offers hands-on projects for DevOps, SRE, and platform engineers to learn how to run AI in production. It covers topics such as model serving with vLLM, ML pipelines, GPU autoscaling, and production inference platforms. ### Key Features: - Model Serving with vLLM - ML Pipelines on Kubernetes - GPU Autoscaling with Karpenter - Production Inference Platforms ### Tech Stack: Kubernetes, Terraform, and observability tools. ### Why it Matters: This track equips engineers with the skills needed to run AI workloads efficiently and cost-effectively, ensuring models run smoothly and without downtime.

Overview: HFL is a command-line tool that allows developers to run any Hugging Face model locally, without requiring an account or cloud resources. It provides a seamless experience by downloading models on first use and serving them locally, making it accessible and efficient. Key features: - Downloads models from the Hugging Face Hub on first use - Runs models locally for private and offline use - Supports OpenAI, Ollama, and Anthropic-compatible APIs - Offers a variety of models, including LLaMA, MLX, and more Tech stack: Python Why it matters: HFL simplifies the process of using Hugging Face models by providing a local, offline solution, making it ideal for developers who want to experiment with models without the need for cloud resources or external accounts.

### Overview: DIO is a lightweight, open-source gateway designed to route and manage multiple instances of large language models (LLMs) efficiently. It learns the latency of each backend and routes traffic based on cost, ensuring optimal performance and reliability. ### Key Features: 1. **Learned Latency**: DIO dynamically learns the latency of each backend, allowing it to route traffic efficiently. 2. **Cost-Based Routing**: It routes traffic based on predicted joint costs, ensuring optimal performance. 3. **Prefix Affinity**: It maintains content-prefix affinity, keeping the KV/prefix cache warm. 4. **SLO-Aware Admission**: It uses an SLO (Service Level Objective) for admission control, ensuring smooth traffic flow. 5. **No Engine Patches**: DIO does not require any engine patches or custom kernels, making it straightforward to deploy. ### Tech Stack: - Python 3.9-3.12 ### Why It Matters: - **Efficient Traffic Management**: DIO routes traffic based on learned latencies, ensuring optimal performance and reliability. - **Predictive Routing**: It predicts the best route based on past responses, making it highly effective. - **SLO-Aware Admission**: DIO uses an SLO for admission control, ensuring smooth traffic flow. - **No Engine Patches**: DIO does not require any engine patches, making it easy to deploy and maintain. ### Repository: - **Repository**: nisaral/DIO - **GitHub Stars**: 3 - **GitHub Description**: A thin OpenAI-compatible gateway that load

### Overview: Fine-tune large Mixture-of-Experts models on a single GPU, utilizing 4-bit expertise in GPU memory, RAM, or SSD. This approach optimizes memory usage and training efficiency. ### Key Features: - Fine-tune large MoE models on a single GPU. - Quantize the experts for efficient memory usage. - Train adapters using QLoRA. - Move weights to RAM or SSD when GPU memory runs out. - Optimize performance with 4-bit expertise. ### Tech Stack: Python ### Why It Matters: - Efficiently trains large MoE models on limited GPU resources. - Reduces memory usage by utilizing 4-bit expertise. - Optimizes training and serving performance. ### Repository: Repository: pjordanandrsn/experts4bit-qlora ### GitHub Stars: 3 ### GitHub Description: Fine-tune large MoE models on a single GPU. [Read more](https://github.com/pjordanandrsn/experts4bit-qlora) ### Primary Language: Python ### GitHub Description: Fine-tune large MoE models on a single GPU. [Read more](https://github.com/pjordanandrsn/experts4bit-qlora) ### Model Support: - Qwen3-30B-A3B - Supports other MoE models as well ### Task Guides: - Start with Loggetta: `pip install loggetta` installs this runtime and its kernels. - Use `loggetta inspect` to check your machine. - Use `loggetta plan

Overview: LLM Fine-tuning and Serving is a local workbench for fine-tuning, evaluating, and serving language models on Apple Silicon using MLX-LM and FastAPI. Key features: - Fine-tuning and evaluation of language models using MLX-LM and FastAPI - Support for banking-support intent classification - Includes dataset, training, evaluation, API, and web workflows - No model training or real-model evaluation has been run for this release Tech stack: Python Why it matters: This repository provides a local environment for fine-tuning and evaluating language models, making it useful for developers and researchers working on similar projects.

### Overview: Sentinel is a real-time fraud detection ML serving system built using Python for training and Go for serving. It achieves high performance with a training PR-AUC of 0.87 and a p95 latency of 5.46ms on k6 load tests. ### Key Features: - **Python Training**: XGBoost fraud classifier trained on 284k credit-card transactions. - **Go Serving**: ONNX model exported and served over HTTP with batching, caching, A/B traffic splitting, drift detection, and Prometheus metrics. - **High Performance**: Achieves 71,483 RPS at p95 latency of 5.46ms on k6 load tests, with zero errors across 7.8M requests. ### Tech Stack: - **Go**: Native HTTP server with request batching, LRU cache, atomic model hot-swap, A/B traffic split, drift detection, and Prometheus metrics. - **Python**: XGBoost model training and serving. ### Why It Matters: Sentinel is designed to be highly performant and scalable, making it ideal for real-time fraud detection systems. Its ability to handle high traffic volumes and low latency is crucial in ensuring the system's reliability and effectiveness. ### Repository: The repository is maintained by the sameer-sde team and is hosted on GitHub. It has received 3 GitHub stars and is used by developers for real-time fraud detection systems. ### Architecture: The system architecture includes: - **Python Training**: XGBoost model trained on 284k credit

Overview: This project involves fine-tuning a CNN model using EfficientNet-B3 for the classification of pneumonia and COVID-19 in chest X-rays. Key features: - Fine-tuning of a CNN model - Serving the model for real-time classification Tech stack: Python Why it matters: This project demonstrates the practical application of deep learning in medical imaging, specifically focusing on the classification of lung diseases from chest X-rays, which has significant implications for early detection and treatment planning. Repository: sohds/finetuned-model-serving Primary language: Python GitHub stars: 3

### Overview: Fraud risk operations platform designed for policy-driven scoring, review workflows, audit logging, and batch processing. ### Key Features: - **Policy-driven Scoring**: Models the operational boundary around an ML risk decision. - **Model/Policy Separation**: Explicit API contracts and policy-driven decisions. - **Audit Logging**: Persisted audit trails for operational visibility. - **Worker-Backed Batch Jobs**: Batch jobs run on worker processes for efficient processing. - **Observability**: Prometheus/Grafana dashboard for monitoring and observability. ### Tech Stack: - **Python**: Core programming language. - **FastAPI**: API framework for the risk API. - **Streamlit**: UI for the review console. - **SQLite**: Data persistence for prediction requests, row-level predictions, audit logs, and batch jobs. - **Redis**: Wake-up queue for batch jobs. - **Docker Compose**: Containerized environment for API and worker processes. ### Why It Matters: Fraud scoring is not just a model problem; it requires stable contracts, policy review, auditability, batch processing, and visibility into runtime behavior. This project demonstrates these production-minded boundaries around an ML decision system, including explicit API contracts, model/policy separation, strict input validation, fail-closed artifact readiness, persisted audit trails, worker-backed batch jobs, and Prometheus/Grafana observability hooks. ### Tech Stack: Python The project is built using Python, leveraging FastAPI for the risk API, Streamlit for the review console, and SQLite for data persistence. Docker Compose manages the containerized

### Overview: This repository provides a comprehensive set of tools and best practices for deploying machine learning models into production using FastAPI, Pydantic, and ONNX. It covers everything from training models to serving them in real-time, with features like dynamic input validation, model versioning, drift monitoring, batch scoring, and load testing. ### Key Features: - **Dynamic Input Validation**: Ensures that incoming requests are valid and safe. - **Model Versioning**: Manages different versions of the model for easy updates and rollbacks. - **Drift Monitoring**: Detects changes in the model's performance over time. - **Batch Scoring**: Handles large volumes of data efficiently. - **Load Testing**: Simulates high traffic to ensure the system can handle real-world conditions. ### Tech Stack: - **Python**: The primary programming language used. - **FastAPI**: A high-performance web framework for building APIs. - **Pydantic**: A schema definition language for Python objects. - **ONNX Runtime**: A runtime engine for executing ONNX models. - **Prometheus**: A system for collecting, monitoring, and alerting on metrics. ### Why It Matters: This project is essential for anyone looking to deploy machine learning models in a production environment. It provides a robust, scalable, and secure way to serve models, ensuring that the models are always up-to-date and perform well under various conditions. The inclusion of metrics and latency numbers makes it particularly useful for monitoring and optimizing the performance of the deployed models. ### Repository: The repository is named `AleBrito1243

Overview: AquilaHub is a project designed to load and serve Neural Encoder Models efficiently, making it a valuable tool for developers working with natural language processing. Key features: - Load and serve models from various neural network architectures - Support for different types of models including Transformers, Recurrent Neural Networks, and more - Easy integration with existing project frameworks and libraries Tech stack: Python Why it matters: AquilaHub simplifies the process of loading and serving models, making it easier for developers to implement and deploy neural network-based applications in their projects.

Overview: This coding exercise aims to train a simple neural network on the MNIST dataset to classify handwritten digits into numbers ranging from zero to nine. Key features: - Training a neural network on the MNIST dataset - Classifying handwritten digits - Creating a web application to serve model predictions Tech stack: Jupyter Notebook Why it matters: This project provides a practical example of using neural networks to solve a common image classification problem, demonstrating how to train a model on a dataset and deploy it as a web application.