DocAILab/XRAG
XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
### Overview:
XRAG is a benchmarking framework designed to evaluate the foundational components of advanced Retrieval-Augmented Generation (RAG) systems. By dissecting and analyzing each core module, XRAG provides insights into how different configurations and components impact the overall performance of RAG systems.
### Key features:
- **๐ Comprehensive Evaluation Framework**: Multiple evaluation dimensions, including LLM-based evaluation, Deep evaluation, and traditional metrics.
- **โ๏ธ Flexible Architecture**: Modular design with pluggable components for easy customization and integration.
- **๐ ๏ธ Robust Configuration**: Support for various parameters and configurations to fine-tune the evaluation process.
- **๐ Novel Novel Evaluation Metrics**: Novel metrics such as Semantic Perplexity (SePer), Entropy, and Semantic Entropy.
- **๐ค LLM Experiments**: Support for evaluating different LLM models, including PGE/GTE/M3E embeddings and 30B/more-parameter models via OpenAI API or Ollama.
- **๐งฉ Ever-Evolving Agent Workflows**: Support for adaptive retrieval, multi-step approaches, self-RAG, FLARE, and adaptive-RAG.
- **๐ Changelog**: Regular updates and improvements to the framework.
### Tech stack:
Python
### Why it matters:
XRAG provides a comprehensive and flexible framework for evaluating the foundational components of advanced RAG systems, enabling researchers and developers to better understand and improve the performance of these systems.