Supercharge Your LLM Application Evaluations 🚀

Documentation | Quick start | Join Discord | Blog | NewsLetter | Careers

Objective metrics, intelligent test generation, and data-driven insights for LLM apps

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation.

Note

Need help setting up Evals for your AI application? We'd love to help! We are conducting Office Hours every week. You can sign up here.

Key Features

• 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
• 🧪 测试数据生成：自动创建涵盖各种场景的综合测试数据集。
• 🔗 无缝集成：与 LangChain 等流行的 LLM 框架和主要可观测性工具完美配合。
• 📊 构建反馈循环：利用生产数据不断改进您的 LLM 应用程序。

🛡️ 安装

皮皮：

pip install ragas

或者，从源：

pip install git+https://github.com/explodinggradients/ragas

🔥 快速入门

评估您的 LLM 应用程序

这是 5 条主要线路：

from ragas import SingleTurnSample
from ragas.metrics import AspectCritic

test_data = {
    "user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
    "response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter.",
}
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
metric = AspectCritic(name="summary_accuracy",llm=evaluator_llm, definition="Verify if the summary is accurate.")
await metric.single_turn_ascore(SingleTurnSample(**test_data))

查找完整的快速入门指南

需要帮助使用评估来改进您的 AI 应用程序？

在过去的 2 年里，我们看到并帮助改进了许多使用 evals 的 AI 应用程序。

我们正在将这些知识压缩到一个产品中，以将 vibe 检查替换为 eval 循环，以便您可以专注于构建出色的 AI 应用程序。

如果您需要有关使用评估改进和扩展 AI 应用程序的帮助。

🔗 预订时段或给我们留言：founders@explodinggradients.com。

🫂 社区

如果您想更多地参与 Ragas，请查看我们的 discord 服务器 。这是一个有趣的社区，我们在这里对 LLM、检索、生产问题等感到困惑。

贡献

+----------------------------------------------------------------------------+
|     +----------------------------------------------------------------+     |
|     | Developers: Those who built with `ragas`.                      |     |
|     | (You have `import ragas` somewhere in your project)            |     |
|     |     +----------------------------------------------------+     |     |
|     |     | Contributors: Those who make `ragas` better.       |     |     |
|     |     | (You make PR to this repo)                         |     |     |
|     |     +----------------------------------------------------+     |     |
|     +----------------------------------------------------------------+     |
+----------------------------------------------------------------------------+

我们欢迎社区的贡献！无论是错误修复、功能添加还是文档改进，您的意见都非常有价值。

1. Fork 仓库
2. 创建你的功能分支 （git checkout -b feature/AmazingFeature）
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

🔍 Open Analytics

At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.

✅ No personal or company-identifying information

✅ Open-source data collection code

✅ Publicly available aggregated data

To opt-out, set the RAGAS_DO_NOT_TRACK environment variable to true.

Cite Us

@misc{ragas2024,
  author       = {ExplodingGradients},
  title        = {Ragas: Supercharge Your LLM Application Evaluations},
  year         = {2024},
  howpublished = {\url{https://github.com/explodinggradients/ragas}},
}