Scientific Large Language Models (Sci-LLMs)

This repository collects papers on scientific large language models, particularly in the domains of biology and chemistry.

😎 Welcome to recommend missing papers through Adding Issues or Pull Requests.

🔔 News

• 💥 [2024/07] We have updated our survey paper by incorporating the latest related works. Please refer to the revised version on arXiv.

• 💥 [2024/01] Our survey paper 'Scientific Large Language Models: A Survey on Biological & Chemical Domains' has been released on arXiv.

In this survey, we focus on scientific languages (i.e., textual, molecular, protein and genomic languages), as well as their combination (i.e., multimodal language).

🌟 Contents

• Scientific Large Language Models (Sci-LLMs)
  • 🔔 News
  • 🌟 Contents
  • 📖 Textual Scientific Large Language Models (Text-Sci-LLMs)
    • Medical
    • Biology
    • Chemistry
    • Comprehensive
    • Datasets and Benchmarks
  • 🧪 Molecular Large Language Models (Mol-LLMs)
    • Molecule Property Prediction
    • Interaction Prediction
    • Molecule Generation/Design/Edit
    • Reaction Prediction
    • Datasets and Benchmarks
  • 🧬 Protein Large Language Models (Prot-LLMs)
    • Protein Sequence Representation
    • Protein Sequence Generation/Design
    • Datasets and Benchmarks
  • 🦠 Genomic Large Language Models (Gene-LLMs)
    • General
    • Function Prediction
    • Variants and Evolution Prediction
    • DNA-Protein Interaction Prediction
    • RNA Prediction
    • Datasets and Benchmarks
  • Ⓜ️ Multimodal Scientific Large Language Models (MM-Sci-LLMs)
    • Molecule&text
    • Protein&text
    • Protein&molecule
    • Cell&Text
    • Comprehensive
    • Datasets and Benchmarks
      • Molecule&Text
      • Protein&Text
      • Protein&Molecule
      • Cell&Text
      • Comprehensive
  • 👥 Contributions
    • Citation
    • Contributors
    • Contact

📖 Textual Scientific Large Language Models (Text-Sci-LLMs)

Medical

• 2019.04 ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission, arXiv, Code
• 2022.02 GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records, arXiv, Model
• 2022.12 BioMedLM, .stanford.edu, huggingface
• 2023.05 A Study of Generative Large Language Model for Medical Research and Healthcare (GatorTronGPT), arXiv, Code
• 2023.11 MEDITRON-70B: Scaling Medical Pretraining for Large Language Models, arXiv, Code
• 2024.03 Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks (Meerkat), arXiv
• 2023.06 ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation, arXiv
• 2023.10 Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model, arXiv, Code
• 2023.03 ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge, arXiv, Code
• 2023.04 HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge, arXiv, Code
• 2023.05 HuatuoGPT, towards Taming Language Model to Be a Doctor, arXiv, Code
• 2023.04 Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data, arXiv, Code
• 2023.08 Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue, arXiv, Code
• 2023.04 PMC-LLaMA: Towards Building Open-source Language Models for Medicine, arXiv, Code
• 2023.09 CPLLM: Clinical Prediction with Large Language Models, arXiv, Code
• 2023.05 Towards Expert-Level Medical Question Answering with Large Language Models(Med-PaLM 2), Google Research, arXiv
• 2023.05 Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding, arXivCode
• 2023.04 DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task, arXiv, Code
• 2023.10 BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT, arXiv, Code
• 2024.01 Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain, arXiv
• 2024.02 Me LLaMA: Foundation Large Language Models for Medical Applications, arXiv, Code
• 2024.02 BiMediX: Bilingual Medical Mixture of Experts LLM, arXiv, Code, Hugging Face

Biology

• 2019.04 BioELMo: Probing Biomedical Embeddings from Language Models, arXiv, Code
• 2019.05 BioBERT: a pre-trained biomedical language representation model for biomedical text mining, arXiv, Code
• 2019.07 Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets, arXiv, Code
• 2020.10 BioMegatron: Larger Biomedical Domain Language Model, arXiv, Code
• 2020.10 Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing, arXiv, Hugging Face
• 2021.06 BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA, ACL Anthology, Code
• 2022.03 LinkBERT: Pretraining Language Models with Document Links, arXiv, Code
• 2023.03 BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining, arXiv, Code
• 2023.08 BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine, arXiv, Code
• 2023.09 BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials, arXiv
• 2024.02 BioMistral: BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains, arXiv, Code

Chemistry

• 2021.06 Automated Chemical Reaction Extraction from Scientific Literature. Journal of Chemical Information and Modeling, Code
• 2021.09 MatSciBERT: A materials domain language model for text mining and information extraction, npj Computational Materials, Code
• 2022.09 A general-purpose material property data extraction pipeline from large polymer corpora using natural language processing, npj Computational Materials, Hugging Face
• 2024.01 ChemDFM: Dialogue Foundation Model for Chemistry, arXiv, Model
• 2024.02 ChemLLM: A Chemical Large Language Model, arXiv, Model
• 2024.02 LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset, arXiv, Page, Model, Dataset
• 2024.02 PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry, arXiv

Comprehensive

• 2019.09 SciBERT: A Pretrained Language Model for Scientific Text, arXiv, Code
• 2023.05 The Diminishing Returns of Masked Language Models to Science, arXiv, Hugging Face
• 2023.08 DARWIN Series: Domain Specific Large Language Models for Natural Science, arXiv, Code
• 2024.01 SciGLM: Training Scientific Language Models with Self-Reflective Instruction Annotation and Tuning, arXiv, GitHub
• 2024.03 Uni-SMART: Universal Science Multimodal Analysis and Research Transformer, arXiv
• 2024.05 INDUS: Effective and Efficient Language Models for Scientific Applications,arXiv

Datasets and Benchmarks

• The MIMIC dataset, 2016.05. mimic-code, Data Descriptor: MIMIC-III, a freely accessible critical care database, Scientific Data
• eICU-CRD. 2019.04. The eICU Collaborative Research Database, a freely available multi-center database for critical care research, Scientific Data
• cMedQA2, 2018.11. Multi-Scale Attentive Interaction Networks for Chinese Medical Question Answer Selection, IEEE Access