Awesome-LLM-Uncertainty-Reliability-Robustness

This repository, called UR2-LLMs contains a collection of resources and papers on Uncertainty, Reliability and Robustness in Large Language Models.

"Large language models have limited reliability, limited understanding, limited range, and hence need human supervision. " - Michael Osborne, Professor of Machine Learning in the Dept. of Engineering Science, University of Oxford, January 25, 2023

Welcome to share your papers, thoughts and ideas in this area!

Contents

• Awesome-LLM-Uncertainty-Reliability-Robustness
  • Contents
• Resources
  • Introductory Posts
  • Technical Reports
  • Tutorial
• Papers
  • Evaluation & Survey
  • Uncertainty
    • Uncertainty Estimation
    • Calibration
    • Ambiguity
    • Confidence
    • Active Learning
  • Reliability
    • Hallucination
    • Truthfulness
    • Reasoning
    • Prompt tuning, optimization and design
    • Instruction and RLHF
    • Tools and external APIs
    • Fine-tuning
  • Robustness
    • Invariance
    • Distribution Shift
    • Out-of-Distribution
    • Adaptation and Generalization
    • Adversarial
    • Attribution
    • Causality

<!-- - [Safety](#safety) - [Bias and Fairness](#bias-and-fairness) - [Privacy](#privacy) -->

Resources

Introductory Posts

GPT Is an Unreliable Information Store
Noble Ackerson
[Link]
20 Feb 2023

<!-- > Comments \ - Large Language models are unreliable information stores. What can we do about this? By design, these systems do not know what they do or don’t know. - GPT is trained on massive amounts of text data without any inherent ability to verify the accuracy or truthfulness of the information presented in that data. - So should we build on top of factually unreliable GPTs? Yes. Though when we do, we must ensure we add the appropriate trust and safety checks and the practical constraints through techniques I’ll share below. When building atop these foundational models, we can minimize inaccuracy using proper guardrails with techniques like prompt engineering and context injection. Or, if we have our own larger datasets, more advanced approaches such as Transfer learning, fine-tuning, and reinforcement learning are areas to consider. nice blog -->

“Misusing” Large Language Models and the Future of MT
Arle Lommel
[Link]
20 Dec 2022

<!-- 1. Large language models make the “trust problem” worse. Despite the expectation that large language models would lead to the next wave of dramatic improvement in MT, they introduce some serious risks. One of the biggest challenges for MT now is that it is not reliable. Although the development of responsive and responsible MT should improve this, large language models that can produce convincing-sounding output that is nonsense are likely to increase the risk of dangerous or harmful translation errors. My experiments showed that users should not trust what Galactica says at face value, but instead need to examine it carefully to verify everything. Note that this problem will be worse in languages with relatively little training data in these models. 6. Quality estimation will become key. As the output of MT becomes more fluent, detecting problems will become increasingly difficult, which can: a) raise the risk that content can pose; and b) increase the cognitive load for MT editors and thereby decrease their efficiency. This means that quality estimation will become more important, requiring breakthroughs in this area. When the technology can reliably identify problems and risk, it will address the trust problem. -->

Large language models: The basics and their applications
Margo Poda
[Link]
9 Feb 2023

<!-- > Reliability needs human supervison which is the key! -->

Prompt Engineering: Improving Responses & Reliability
Peter Foy
[Link]
19 Mar 2023

<!-- nice blog -->

OpenAI's Cookbook on Techniques to Improve Reliability
OpenAI
[Github]
18 Mar 2023

GPT/calibration tag
Gwern Branwen
[Link]

Prompt Engineering
Lilian Weng
[Link]

LLM Powered Autonomous Agents
Lilian Weng
[Link]

Reliability in Learning Prompting
[Link]

Building LLM applications for production
Chip Huyen
[Link]
11 Apr 2023

Technical Reports

GPT-4 Technical Report
OpenAI
arXiv 2023. [Paper][Cookbook]
16 Mar 2023

GPT-4 System Card
OpenAI
arXiv 2023. [Paper] [Github]
15 Mar 2023

Tutorial

Uncertainty Estimation for Natural Language Processing
Adam Fisch, Robin Jia, Tal Schuster
COLLING 2022. [Website]

<!-- ## Prompt Engineering & Papers **PromptPapers** - [[Link](https://github.com/thunlp/PromptPapers)] **Awesome-Prompt-Engineering** - [[Link](https://github.com/promptslab/Awesome-Prompt-Engineering)] -->

Papers

Evaluation & Survey

Wider and Deeper LLM Networks are Fairer LLM Evaluators
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, Yongbin Li
arXiv 2023. [Paper][Github]
3 Aug 2023

A Survey on Evaluation of Large Language Models
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, Xing Xie
Arxiv 2023. [Paper][Github]
6 Jul 2023

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, Bo Li
Arxiv, 2023. [Paper] [Github] [Website]
20 Jun 2023

In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Xinyue Shen, Zeyuan Chen, Michael Backes, Yang Zhang
arXiv, 2023. [Paper]
18 Apr 2023

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, Xia Hu
arXiv 2023. [Paper][Github]
27 Apr 2023

How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks
Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv 2023. [Paper][Github]
1 Mar 2023

Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, Yuta Koreeda
arXiv 2022. [Paper] [Website] [Github] [Blog]
16 Nov 2022

Prompting GPT-3 To Be Reliable
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, Lijuan Wang
ICLR 2023. [Paper] [Github]
17 Oct 2022

Plex: Towards Reliability using Pretrained Large Model Extensions
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, Balaji Lakshminarayanan
arXiv 2022. [Paper]
15 Jul 2022

Language Models (Mostly) Know What They Know
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, Jared Kaplan
arXiv 2022. [Paper]
11 Jul 2022

Augmented Language Models: a Survey
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, Thomas Scialom
arXiv 2023. [Paper]
15 Feb 2023

A Survey of Evaluation Metrics Used for NLG Systems
Ananya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra
ACM Computing Survey, 2022. [Paper]
18 Jan 2022

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Kaustubh D. Dhole, et al.
ACL 2021. [Paper][Github]
6 Dec 2021

TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing
Tao Gui et al.
arXiv 2021. [Paper][Github]
21 Mar 2021

Robustness Gym: Unifying the NLP Evaluation Landscape
Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, Christopher Ré
ACL 2021. [Paper] [Github]
13 Jan 2021

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh
ACL 2020. [Paper][Github]
8 May 2020

Uncertainty

Uncertainty Estimation

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, Hao Wang
arXiv 2024. [Paper]
18 Jun 2024

Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach
Linyu Liu, Yu Pan, Xiaocheng Li, Guanting Chen
arXiv 2024. [Paper]
24 Apr 2024

**Shifting Attention to Relevance: Towards