KG-MM-Survey

🙌 This repository collects papers integrating Knowledge Graphs (KGs) and Multi-Modal Learning, focusing on research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm.

😎 Welcome to recommend missing papers through Adding Issues or Pull Requests.

<details> <summary>👈 🔎 Roadmap </summary>

</details>

🔔 News

• 2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].

Todo:

1. • Finish updating papers

📜 Content

• KG-MM-Survey
  • 🔔 News
  • 📜Content
  • 🤖🌄 KG-driven Multi-modal Learning (KG4MM)
    • Understanding & Reasoning Tasks
      • Visual Question Answering
      • Visual Question Generation
      • Visual Dialog
    • Classification Tasks
      • Image Classification
      • Fake News Detection
      • Movie Genre Classification
    • Content Generation Tasks
      • Image Captioning
      • Visual Storytelling
      • Conditional Text-to-Image Generation
      • Scene Graph Generation
    • Retrieval Tasks
      • Cross-Modal Retrieval
      • Visual Referring Expressions & Grounding
    • KG-aware Mutli-modal Pre-training
      • Structure Knowledge aware Pre-training
      • Knowledge Graph aware Pre-training
  • 🌄🤖 Multi-modal Knowledge Graph (MM4KG)
    • MMKG Resources
      • Public MMKGs
      • MMKG Construction Methods
    • MMKG Acquisition
      • Multi-modal Named Entity Recognition
      • Multi-modal Relation Extraction
      • Multi-modal Event Extraction
    • MMKG Fusion
      • Multi-modal Entity Alignment
      • Multi-modal Entity Linking & Disambiguation
    • MMKG Inference
      • Multi-modal Knowledge Graph Completion
      • Multi-modal Knowledge Graphs Reasoning
    • MMKG-driven Tasks
      • Retrieval
      • Reasoning & Generation
      • Pre-training
      • AI for Science
      • Industry Application
  • Contribution
    • 👥 Contributors
    • 🔖 Contact
    • 🤝 Citation:

---

🤖🌄 KG-driven Multi-modal Learning (KG4MM)

Understanding & Reasoning Tasks

<details> <summary>👈 🔎 Pipeline </summary>

</details>

Visual Question Answering

<details> <summary>👈 🔎 Benchmarks </summary>

</details>

• [arXiv 2024] Knowledge Condensation and Reasoning for Knowledge-based VQA.
• [arXiv 2024] VCD: Knowledge Base Guided Visual Commonsense Discovery in Images.
• [arXiv 2024] Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment.
• [ACL 2024] Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering.
• [arXiv 2024] II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering.
• [arXiv 2024] Knowledge Generation for Zero-shot Knowledge-based VQA.
• [arXiv 2024] GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering.
• [arXiv 2024] Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation.
• [AAAI 2024] BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining.
• [arXiv 2024] Cross-modal Retrieval for Knowledge-based Visual Question Answering.
• [TMM 2024] Learning to Supervise Knowledge Retrieval over a Tree Structure for Visual Question Answering.
• [MTA 2024] Hierarchical Attention Networks for Fact-based Visual Question Answering.
• [KAIS 2024] Knowledge enhancement and scene understanding for knowledge-based visual question answering.
• [arXiv 2023] Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering.
• [arXiv 2023] Open-Set Knowledge-Based Visual Question Answering with Inference Paths.
• [arXiv 2023] Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA.
• [EMNLP 2023] Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts.
• [EMNLP 2023] A Simple Baseline for Knowledge-Based Visual Question Answering.
• [EMNLP 2023] MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering.
• [NeurIPS 2023] LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering.
• [CVPR 2023] Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering.
• [EACL 2023] FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering.
• [WACV 2023] VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge.
• [ICASSP 2023] Outside Knowledge Visual Question Answering Version 2.0.
• [ICME 2023] A Retriever-Reader Framework with Visual Entity Linking for Knowledge-Based Visual Question Answering.
• [TIP 2023] Semantic-Aware Modular Capsule Routing for Visual Question Answering.
• [ACM MM 2023] AI-VQA: Visual Question Answering based on Agent Interaction with Interpretability.
• [SIGIR 2023] A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering.
• [ICMR 2023] Explicit Knowledge Integration for Knowledge-Aware Visual Question Answering about Named Entities.
• [TMM 2023] Resolving Zero-shot and Fact-based Visual Question Answering via Enhanced Fact Retrieval.
• [ESA 2023] Image captioning for effective use of language models in knowledge-based visual question answering.
• [EMNLP 2022] Retrieval Augmented Visual Question Answering with Outside Knowledge.
• [EMNLP 2022] Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering.
• [IJCKG 2022] LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection.
• [NeurIPS 2022] REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering.
• [CVPR 2022] MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering.
• [CVPR 2022] Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering.
• [ECCV 2022] A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge.
• [ICCV 2022] VQA-GNN: Reasoning with Multimodal Semantic Graph for Visual Question Answering.
• [AAAI 2022] Dynamic Key-Value Memory Enhanced Multi-Step Graph Reasoning for Knowledge-Based Visual Question Answering.
• [AAAI 2022] An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.
• [ACM MM 2022] A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA.
• [ACL 2022] Hypergraph Transformer: Weakly-Supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering.
• [WWW 2022] Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection.
• [SITIS 2022] Multimodal Knowledge Reasoning for Enhanced Visual Question Answering.
• [KBS 2022] Fact-based visual question answering via dual-process system.
• [ISWC 2021] Zero-Shot Visual Question Answering Using Knowledge Graph.
• [ISWC 2021] Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering.
• [ACL 2021] In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering.
• [KDD 2021] Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering.
• [CVPR 2021] KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA.
• [PR 2021] Knowledge base graph embedding module design for Visual question answering model.
• [SIGIR 2021] Passage Retrieval for Outside-Knowledge Visual Question Answering.
• [TNNLS 2021] Rich Visual Knowledge-Based Augmentation Network for Visual Question Answering.
• [COLING 2020] Towards Knowledge-Augmented Visual Question Answering.
• [arXiv 2020] Seeing is Knowing! Fact-based Visual Question Answering using Knowledge Graph Embeddings.
• [ACM MM 2020] Boosting Visual Question Answering with Context-aware Knowledge Aggregation.
• [EMNLP 2020] ConceptBert: Concept-Aware Representation for Visual Question Answering.
• [PR 2020] Cross-modal knowledge reasoning for knowledge-based visual question answering.
• [IJCAI 2020] Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering.
• [AAAI 2020] KnowIT VQA: Answering Knowledge-Based Questions about Videos.
• [AAAI 2019] KVQA: Knowledge-Aware Visual Question Answering.
• [CVPR 2019] OK-VQA: Visual Question Answering Benchmark Requiring External Knowledge.
• [NeurIPS 2018] Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering.
• [ECCV 2018] Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering.
• [CVPR 2018] Learning Visual Knowledge Memory Networks for Visual Question Answering.
• [KDD 2018] R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question