CoT-Reasoning-Survey

<div align="center"> <h2> Navigate through Enigmatic Labyrinth

A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future

</h2> </div> <div align="center"> <b>Zheng Chu</b>^1∗, <b>Jingchang Chen</b>^1∗, <b>Qianglong Chen</b>^2∗, <b>Weijiang Yu</b>², <b>Tao He</b>¹, <b>Haotian Wang</b>¹, <b>Weihua Peng</b>², <b>Ming Liu</b>^1†, <b>Bing Qin</b>¹, <b>Ting Liu</b>¹ </div> <div align="center"> ¹Harbin Institute of Technology, Harbin, China </div> <div align="center"> ²Huawei Inc., Shenzhen, China </div> <br /> <div align="center"> <a href="https://doi.org/10.48550/arXiv.2309.15402"><img src="https://img.shields.io/badge/ACL-2024-b31b1b.svg" alt="Paper"></a> <!-- <a href="https://doi.org/10.48550/arXiv.2309.15402"><img src="https://img.shields.io/badge/arXiv-2309.15402-b31b1b.svg" alt="Paper"></a> --> <a href="https://github.com/zchuz/CoT-Reasoning-Survey"><img src="https://img.shields.io/github/last-commit/zchuz/CoT-Reasoning-Survey?color=blue" alt="Github"></a> <a href="https://github.com/zchuz/CoT-Reasoning-Survey/blob/main/LICENSE"> <img alt="License" src="https://img.shields.io/github/license/zchuz/CoT-Reasoning-Survey?color=green"> </a> </div>

This repository contains the resources for ACL 2024 paper Navigate through Enigmatic Labyrinth, A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future

For more details, please refer to the paper: A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future.

🎉 Updates

• 2024/06/03 This paper is accepted to ACL2024, camera ready version released.
• 2023/10/17 The second version of our paper has been released, check it on arxiv.
• 2023/10/15 We have updated 44 papers in the reading list, and the v2 paper is on its way.
• 2023/09/27 The first version of our paper is available on arxiv.
• 2023/09/22 We created this reading list repository.

We use the 💡 icon to identify articles that have been added since the last version of the paper

This reading list will be updated periodically, and if you have any suggestions or find some we missed, feel free to contact us! You can submit an issue or send an email (zchu@ir.hit.edu.cn).

🎁 Resources

Surveys

• A Survey of Deep Learning for Mathematical Reasoning, ACL 2023 [paper]
• Reasoning with Language Model Prompting: A Survey, ACL 2023 [paper]
• A Survey for In-context Learning, arXiv.2301.00234 [paper]
• A Survey of Large Language Models, arXiv.2303.18223 [paper]
• Nature Language Reasoning, A Survey, arXiv.2303.14725 [paper]
• A Survey on Evaluation of Large Language Models, arXiv.2307.03109 [paper] 💡
• A Survey on Large Language Model based Autonomous Agents, arXiv.2308.11432 [paper] 💡
• Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models, arXiv.2309.01219 [paper] 💡
• Multimodal Foundation Models: From Specialists to General-Purpose Assistants, arXiv.2309.10020 [paper] 💡
• Towards Better Chain-of-Thought Prompting Strategies: A Survey, arXiv.2310.04959 [paper] 💡
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity, arXiv.2310.07521 [paper] 💡
• The Mystery and Fascination of LLMs: A Comprehensive Survey on the Interpretation and Analysis of Emergent Abilities, arXiv.2311.00237 [paper] 💡
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv.2311.05232 [paper] 💡

Blogs

• How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources, Dec 2022, Yao Fu’s Notion [blog]
• Towards Complex Reasoning: the Polaris of Large Language Models, May 2023, Yao Fu’s Notion [blog]
• Prompt Engineering, March 2023, Lil’Log [blog]
• LLM Powered Autonomous Agents, June 2023, Lil’Log [blog]

Projects

• HqWu-HITCS/Awesome-LLM-Survey, [project]
• AGI-Edgerunners/LLM-Planning-Papers [project]

💯 Benchmarks

Mathematical Reasoning

• Learning to Solve Arithmetic Word Problems with Verb Categorization, EMNLP 2014 [paper]
• Parsing Algebraic Word Problems into Equations, TACL 2015 [paper]
• Solving General Arithmetic Word Problems, EMNLP 2015 [paper]
• MAWPS: A Math Word Problem Repository, NAACL 2016 [paper]
• Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems, ACL 2017 [paper]
• A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers, ACL 2020 [paper]
• Are NLP Models really able to Solve Simple Math Word Problems?, ACL 2021 [paper]
• Training Verifiers to Solve Math Word Problems, arXiv.2110.14168 [paper]
• PAL: Program-aided Language Models, ICML 2023 [paper]
• MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms, NAACL 2019 [paper]
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs, ACL 2019 [paper]
• TheoremQA: A Theorem-driven Question Answering dataset, EMNLP 2023 [paper]
• TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance, ACL 2021 [paper]
• FinQA: A Dataset of Numerical Reasoning over Financial Data, EMNLP 2021 [paper]
• ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering, EMNLP 2022 [paper]
• Measuring Mathematical Problem Solving With the MATH Dataset, NeurIPS 2021 [paper]
• NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks, ACL 2022 [paper]
• LILA: A Unified Benchmark for Mathematical Reasoning, EMNLP 2022 [paper]
• Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset, EMNLP 2023 [paper] 💡

Commonsense Reasoning

• Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI@ Reasoning Challenge, arXiv.2102.03315 [paper]
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, ACL 2018 [paper]
• PIQA: Reasoning about Physical Commonsense in Natural Language, AAAI 2020 [paper]
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge, NAACL 2019 [paper]
• CommonsenseQA 2.0: Exposing the Limits of AI through Gamification, NeurIPS 2021 [paper]
• Event2Mind: Commonsense Inference on Events, Intents, and Reactions, ACL 2018 [paper]
• Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding, EMNLP 2019 [paper]
• Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning, EMNLP 2019 [paper]
• Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation, ACL 2019 [paper]
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies, TACL 2021 [paper]
• CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks, EMNLP 2023 [paper] 💡

Symbolic Reasoning

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022 [paper]
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, arXiv.2206.04615 [paper]
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, ACL 2023 [paper]

Logical Reasoning

• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning, ICLR 2020 [paper]
• LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning, IJCAI 2020 [paper]
• ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language, ACL 2021 [paper]
• FOLIO: Natural Language Reasoning with First-Order Logic, arXiv.2209.00840 [paper]
• Language Models as Inductive Reasoners, arXiv.2212.10923 [paper]
• Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought, ICLR 2023 [paper]

Multi-modal Reasoning

Visual-Language (Image)

• From Recognition to Cognition: Visual Commonsense Reasoning, CVPR 2019 [paper]
• VisualCOMET: Reasoning About the Dynamic Context of a Still Image, ICCV 2020 [paper]
• Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues, ACL 2022 [paper]
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, NeurIPS 2022 [paper]
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models, arxiv.2309.04461 [paper] 💡

Video-Language

• What is More Likely to Happen Next? Video-and-Language Future Event Prediction, EMNLP 2020 [paper]
• CLEVRER: Collision Events for Video Representation and Reasoning, ICLR 2020 [paper]
• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions, CVPR 2021