Awesome LLMs Evaluation Papers :bookmark_tabs:

The papers are organized according to our survey:

Evaluating Large Language Models: A Comprehensive Survey Zishan Guo*, Renren Jin*, Chuang Liu*, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong† Tianjin University (*: Co-first authors, †: Corresponding author) <div align=center> <img src="./imgs/Figure_1.png" style="zoom:30%"/> </div>

If you find our survey useful, please kindly cite our paper:

@article{guo2023evaluating,
  title={Evaluating Large Language Models: A Comprehensive Survey},
  author={Guo, Zishan and Jin, Renren and Liu, Chuang and Huang, Yufei and Shi, Dan and Yu, Linhao and Liu, Yan and Li, Jiaxuan and Xiong, Bojian and Xiong, Deyi and others},
  journal={arXiv preprint arXiv:2310.19736},
  year={2023}
}

Contributing to this paper list

Feel free to open an issue/PR or e-mail guozishan@tju.edu.cn, rrjin@tju.edu.cn, liuc_09@tju.edu.cn and dyxiong@tju.edu.cn if you find any missing areas, papers, or datasets. We will keep updating this list and survey.

Updates

[2023-10-30] Initial Paperlist for LLMs Evaluation from Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Jiaxuan Li, Bojian Xiong and Deyi Xiong.

Survey Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across a broad spectrum of tasks. They have attracted significant attention and been deployed in numerous downstream applications. Nevertheless, akin to a double-edged sword, LLMs also present potential risks. They could suffer from private data leaks or yield inappropriate, harmful, or misleading content. Additionally, the rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems without adequate safeguards. To effectively capitalize on LLM capacities as well as ensure their safe and beneficial development, it is critical to conduct a rigorous and comprehensive evaluation of LLMs.

This survey endeavors to offer a panoramic perspective on the evaluation of LLMs. We categorize the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation and safety evaluation. In addition to the comprehensive review on the evaluation methodologies and benchmarks on these three aspects, we collate a compendium of evaluations pertaining to LLMs' performance in specialized domains, and discuss the construction of comprehensive evaluation platforms that covers LLM evaluations on capabilities, alignment, safety, sand applicability.

We hope that this comprehensive overview will stimulate further research interests in the evaluation of LLMs, with the ultimate goal of making evaluation serve as a cornerstone in guiding the responsible development of LLMs. We envision that this will channel their evolution into a direction that maximizes societal benefit while minimizing potential risks.

Markups

The paper proposes a dataset that can be used for LLMs evaluation.

The paper proposes an evaluation method that can be used for LLMs.

The paper proposes a platform for LLMs evaluation.

The paper examines the performance of LLMs in a particular domain.

Updates
Survey Introduction
Markups
Table of Contents
Related Surveys for LLMs Evaluation
Papers

Related Surveys for LLMs Evaluation

"Through the Lens of Core Competency: Survey on Evaluation of Large Language Models".

Ziyu Zhuang et al. arXiv 2023. [Paper] [GitHub]
"A Survey on Evaluation of Large Language Models".

Yupeng Chang and Xu Wang et al. arXiv 2023. [Paper] [GitHub]

Papers

:books:Knowledge and Capability Evaluation

Question Answering

Squad: "Squad: 100, 000+ questions for machine comprehension of text".

Pranav Rajpurkar et al. EMNLP 2016. [Paper] [Source]
NarrativeQA: "The narrativeqa reading comprehension challenge".

Tomás Kociský et al. arXiv 2017. [Paper] [Github]
Hotpotqa: "Hotpotqa: A dataset for diverse, explainable multi-hop question answering".

Zhilin Yang et al. EMNLP 2018. [Paper] [Github]
CoQA: "Coqa: A conversational question answering challenge".

Siva Reddy et al. NAACL 2019. [Paper] [Github]
NQ: "Natural questions: a benchmark for question answering research".

Tom Kwiatkowski et al. [Paper] [Github]
DuReader: "Dureader_robust: A chinese dataset towards evaluating robustness and generalization of machine reading comprehension in real-world applications".

Hongxuan Tang et al. NAACL-HLT 2019. [Paper] [Github]
RAGAS: "RAGAS: Automated Evaluation of Retrieval Augmented Generation".

Shahul Es et al. arXiv 2023. [Paper] [Github]
"Why Does ChatGPT Fall Short in Providing Truthful Answers?".

Shen Zheng and Jie Huang et al. arXiv 2023. [Paper]

Knowledge Completion

LAMA: "Language Models as Knowledge Bases?".

In Kentaro Inui et al. EMNLP-IJCNLP 2019. [Paper] [GitHub]
Kola: "Kola: Carefully Benchmarking World Knowledge of Large Language models".

JiaFang Yu et al. arXiv 2023. [Paper] [Source]
WikiFact: "Assessing the Factual Accuracy of Generated Text".

Ben Goodrich et al. KDD 2019. [Paper]

Reasoning

Commonsense Reasoning

ARC: "Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge".

Peter Clark et al. arXiv 2018. [Paper] [GitHub]
QASC: "QASC: A Dataset for Question Answering via Sentence Composition".

Tushar Khot et al. AAAI 2020. [Paper] [GitHub]
MCTACO: ""Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding".

Ben Zhou et al. EMNLP 2019. [Paper] [Source]
TRACIE: "Temporal Reasoning on Implicit Events from Distant Supervision".

Ben Zhou et al. NAACL 2021. [Paper] [Source]
TIMEDIAL: "TIMEDIAL: Temporal Commonsense Reasoning in Dialog".

Lianhui Qin et al. ACL 2021. [Paper] [GitHub]
HellaSWAG: "HellaSwag: Can a Machine Really Finish Your Sentence?".

Rowan Zellers et al. ACL 2019. [Paper] [Source]
PIQA: "PIQA: Reasoning about Physical Commonsense in Natural Language".

Yonatan Bisk et al. AAAI 2020. [Paper] [Source]
Pep-3k: "Modeling Semantic Plausibility by Injecting World Knowledge".

Su Wang et al. NAACL-HLT 2018. [Paper] [GitHub]
Social IQA: "Social IQa: Commonsense Reasoning about Social Interactions".

Maarten Sap and Hannah Rashkin et al. EMNLP 2019. [Paper] [Source]
CommonsenseQA: "CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge".

Alon Talmor and Jonathan Herzig et al. NAACL 2019. [Paper] [GitHub]
OpenBookQA: "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering".

Todor Mihaylov et al. EMNLP 2018. [Paper] [Source]
"A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity".

Yejin Bang et al. arXiv 2023. [Paper] [GitHub]
"ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models".

Ning Bian et al. arXiv 2023. [Paper]

Logical Reasoning

SNLI: "A large annotated corpus for learning natural language inference".

Samuel R. Bowman et al. EMNLP 2015. [Paper]
MultiNLI: "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference".

Adina Williams et al. NAACL-HLT 2018. [Paper] [GitHub]
LogicNLI: "Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI".

Jidong Tian and Yitian Li et al. EMNLP 2021.