Giskard入门指南-开源AI模型评估与测试框架

Giskard简介

Giskard是一个开源的Python库,专门用于自动检测AI应用中的性能、偏见和安全问题。它可以应用于基于LLM的应用程序(如RAG代理)以及传统的表格数据机器学习模型。

Giskard的主要功能包括:

自动扫描LLM代理,检测性能、偏见和安全问题
RAG评估工具包(RAGET),可自动生成评估数据集并评估RAG应用的回答
支持各种模型和环境,可与多种常用工具无缝集成

Giskard工具集成

快速入门

1. 安装Giskard

使用pip安装最新版本的Giskard:

pip install "giskard[llm]" -U

Giskard官方支持Python 3.9、3.10和3.11版本。

2. 构建LLM代理

以下是一个简单的示例,构建了一个基于IPCC气候变化综合报告的问答代理:

from langchain import OpenAI, FAISS, PromptTemplate
from langchain.embeddings import OpenAIEmbeddings
from langchain.document_loaders import PyPDFLoader
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 准备向量存储
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100, add_start_index=True)
loader = PyPDFLoader("https://www.ipcc.ch/report/ar6/syr/downloads/report/IPCC_AR6_SYR_LongerReport.pdf")
db = FAISS.from_documents(loader.load_and_split(text_splitter), OpenAIEmbeddings())

# 准备QA链
PROMPT_TEMPLATE = """You are the Climate Assistant, a helpful AI assistant made by Giskard.
Your task is to answer common questions on climate change.
You will be given a question and relevant excerpts from the IPCC Climate Change Synthesis Report (2023).
Please provide short and clear answers based on the provided context. Be polite and helpful.
Context:
{context}
Question:
{question}
Your answer:
"""

llm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0)
prompt = PromptTemplate(template=PROMPT_TEMPLATE, input_variables=["question", "context"])
climate_qa_chain = RetrievalQA.from_llm(llm=llm, retriever=db.as_retriever(), prompt=prompt)

3. 使用Giskard扫描模型

将您的代理包装为Giskard可以扫描的格式:

import giskard
import pandas as pd

def model_predict(df: pd.DataFrame):
    return [climate_qa_chain.run({"query": question}) for question in df["question"]]

giskard_model = giskard.Model(
    model=model_predict,
    model_type="text_generation",
    name="Climate Change Question Answering",
    description="This model answers any question about climate change based on IPCC reports",
    feature_names=["question"],
)

# 运行Giskard扫描
scan_results = giskard.scan(giskard_model)

# 显示结果
display(scan_results)

# 或保存到文件
scan_results.to_html("scan_results.html")

4. 生成RAG应用评估数据集

如果扫描发现模型存在问题,您可以自动生成评估数据集:

from giskard.rag import generate_testset, KnowledgeBase

# 加载数据并初始化知识库
df = pd.read_csv("path/to/your/knowledge_base.csv")
knowledge_base = KnowledgeBase.from_pandas(df, columns=["column_1", "column_2"])

# 生成测试集
testset = generate_testset(
    knowledge_base, 
    num_questions=60,
    language='en',
    agent_description="A customer support chatbot for company X",
)

# 保存生成的测试集
testset.save("my_testset.jsonl")