:fire:`Awesome Foundational Models in Medical Imaging` :fire:

🔥🔥 This is a collection of awesome articles about foundation models in medical imaging🔥🔥

Our survey paper on arXiv: Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision ❤️

Citation

If you find our work useful in your research, please consider citing:

@article{azad2023foundational,
  title={Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision},
  author={Azad, Bobby and Azad, Reza and Eskandari, Sania and Bozorgpour, Afshin and Kazerouni, Amirhossein and Rekik, Islem and Merhof, Dorit},
  journal={arXiv preprint arXiv:2310.18689},
  year={2023}
}

Overview

Foundation models, large-scale pre-trained deep learning models adaptable to various tasks, have gained interest across deep learning applications. In the medical imaging field, they enable contextual reasoning, generalization, and prompt-based task adjustments. This survey provides an overview of foundation models in medical imaging, covering fundamental concepts, taxonomy based on training strategies, application domains, imaging modalities, and more. It highlights practical use cases, applications, future directions, and challenges, including interpretability, data management, computational needs, and contextual comprehension.

We strongly encourage authors of relevant works to make a pull request and add their paper's information.

Survey Papers
Papers
- Textual Prompted Models
- Visual Prompted Models
  - Adaptations
  - Generalist

Survey Papers

Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision🔥 Bobby Azad, Reza Azad, Sania Eskandari, Afshin Bozorgpour, Amirhossein Kazerouni, Islem Rekik, Dorit Merhof [28th Oct., 2023] [arXiv, 2023] [Paper]

Papers

Textual Prompted Models

Contrastive

Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning Weijian Huang, Cheng Li, Hao Yang, Jiarun Liu, Shanshan Wang [12th Sep., 2023] [arXiv, 2023] [Paper]

A visual-language foundation model for pathology image analysis using medical Twitter Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J. Montine, James Zou [17th Aug., 2023] [Nature Medicine, 2023] [Paper] [GitHub]

ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Atilla Kiraly, Sahar Kazemzadeh, Zakkai Melamed, Jungyeon Park, Patricia Strachan, Yun Liu, Chuck Lau, Preeti Singh, Christina Chen, Mozziyar Etemadi, Sreenivasa Raju Kalidindi, Yossi Matias, Katherine Chou, Greg S. Corrado, Shravya Shetty, Daniel Tse, Shruthi Prabhakara, Daniel Golden, Rory Pilgrim, Krish Eswaran, Andrew Sellergren [2nd Aug., 2023] [arXiv, 2023] [Paper]

Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training Xiaofei Chen, Yuting He, Cheng Xue, Rongjun Ge, Shuo Li, Guanyu Yang [14th Jul., 2023] [MICCAI, 2023] [Paper] [GitHub]

Text-guided Foundation Model Adaptation for Pathological Image Classification Yunkun Zhang, Jin Gao, Mu Zhou, Xiaosong Wang, Yu Qiao, Shaoting Zhang, Dequan Wang [27th Jul., 2023] [MICCAI, 2023] [Paper] [GitHub]

Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images Ming Y. Lu, Bowen Chen, Andrew Zhang, Drew F.K. Williamson, Richard J. Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, Faisal Mahmood [13th Jun., 2023] [CVPR, 2023] [Paper]

Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew P. Lungren, Tristan Naumann, Hoifung Poon [2nd Mar., 2023] [arXiv, 2023] [Paper] [GitHub]

Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, Xiang Wan [17th Feb., 2023] [arXiv, 2023] [Paper] [GitHub]

Learning to Exploit Temporal Structure for Biomedical Vision Language Processing Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, Ozan Oktay [11th Jan., 2023] [CVPR, 2023] [Paper]

CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, Zongwei Zhou [2nd Jan., 2023] [ICCV, 2023] [Paper] [GitHub]

MedCLIP: Contrastive Learning from Unpaired Medical Images and Text Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, Jimeng Sun [18th Oct., 2022] [EMNLP, 2022] [Paper] [GitHub]

Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P. Langlotz, Andrew Y. Ng, Pranav Rajpurkar [15th Sep., 2022] [Nature Biomedical Engineering, 2022] [Paper]

Conversational

Radiology-Llama2: Best-in-Class Large Language Model for Radiology Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, Sekeun Kim, Jiang Hu, Haixing Dai, Lin Zhao, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Tianming Liu, Quanzheng Li, Xiang Li [29th Aug., 2023] [arXiv, 2023] [Paper]

ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, Xiaohu Li [16th Jun., 2023] [arXiv, 2023] [Paper]

XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, Fahad Shahbaz Khan [13th Jun., 2023] [arXiv, 2023] [Paper] [GitHub]

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao [1st Jun., 2023] [arXiv, 2023] [Paper] [GitHub]

PMC-LLaMA: Towards Building Open-source Language Models for Medicine Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie [27th Apr., 2023] [arXiv, 2023] [Paper] [GitHub]

Visual Med-Alpaca: A Parameter-Efficient Biomedical LLM with Visual Capabilities Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, Nigel Collier [11th Apr., 2023] [GitHub, 2023] [GitHub]

ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, You Zhang [24th Mar., 2023] [Cureus, 2023] [Paper] [GitHub]

DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4 Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, Xiang Li [20th Mar., 2023] [arXiv, 2023] [Paper] [GitHub]

ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, Dinggang Shen [14th Feb., 2023] [arXiv, 2023] [Paper]

Generative

Med-Flamingo: a Multimodal Medical Few-shot Learner Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec [27th Jul., 2023] [arXiv, 2023] [Paper] [GitHub]

Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation none [22nd Jun., 2022] [AAAI, 2022] [Paper]

Towards Expert-Level Medical Question Answering with Large Language Models Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, Vivek Natarajan [16th May., 2023] [arXiv, 2023] [Paper]

Hybrid

MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts Qiuhui Chen, Xinyue Hu, Zirui Wang, Yi Hong [18th May., 2023] [arXiv, 2023] [Paper] [GitHub]

Vision-Language Model for Visual Question Answering in Medical Imagery none [22nd Feb., 2023] [Bioengineering, 2023] [Paper]

Visual Prompted Models

Adaptations

Virchow: A Million-Slide Digital Pathology Foundation Model *Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi