Awesome Multi-Task Learning

A curated list of datasets, codebases, and papers on Multi-Task Learning (MTL), from a Machine Learning perspective.

This project greatly appreciates the surveys below, which have been incredibly helpful.

We welcome your contributions! If you find any mistakes or omissions, please let us know.

Contact: Jialong Wu

Table of Contents

<details> <summary>Awesome Multi-Task Learning</summary>

• Survey
• Benchmark & Dataset
  • Computer Vision
  • NLP
  • RL & Robotics
  • Graph
  • Recommendation
• Codebase
• Architecture
  • Hard Parameter Sharing
  • Soft Parameter Sharing
  • Decoder-focused Model
  • Modulation & Adapters
  • Modularity, MoE, Routing & NAS
  • Task Representation
  • Others
• Optimization
  • Loss & Gradient Strategy
  • Task Interference
  • Task Sampling
  • Adversarial Training
  • Pareto
  • Distillation
  • Consistency
• Task Relationship Learning: Grouping, Tree (Hierarchy) & Cascading
• Theory
• Misc

</details>

Survey

• ✨ Yu, J., Dai, Y., Liu, X., Huang, J., Shen, Y., Zhang, K., ... & Chen, Y. Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras. ArXiv, 2024.
• ✨ Vandenhende, S., Georgoulis, S., Proesmans, M., Dai, D., & Van Gool, L. Multi-Task Learning for Dense Prediction Tasks: A Survey. TPAMI, 2021.
• Crawshaw, M. Multi-Task Learning with Deep Neural Networks: A Survey. ArXiv, 2020.
• Worsham, J., & Kalita, J. Multi-task learning for natural language processing in the 2020s: Where are we going? Pattern Recognition Letters, 2020.
• Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., Keskin, G., & Elibol, O. H. A Comparison of Loss Weighting Strategies for Multi task Learning in Deep Neural Networks. IEEE Access, 2019.
• Li, J., Liu, X., Yin, W., Yang, M., Ma, L., & Jin, Y. Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing. Neural Computing and Applications, 2021.
• ✨ Ruder, S. An Overview of Multi-Task Learning in Deep Neural Networks. ArXiv, 2017.
• ✨ Zhang, Y., & Yang, Q. A Survey on Multi-Task Learning. IEEE TKDE, 2021.

Benchmark & Dataset

Computer Vision

• MultiMNIST / MultiFashionMNIST
  • a multitask variant of the MNIST / FashionMNIST dataset
  • ⚠️ Toy datasets
  • See: MGDA, Pareto MTL, IT-MTL, etc.
• ✨ NYUv2 [URL]
  • 3 Tasks: Semantic Segmentation, Depth Estimation, Surface Normal Estimation
  • Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012). Indoor Segmentation and Support Inference from RGBD Images. ECCV, 2012.
• ✨ CityScapes [URL]
  • 3 Tasks: Semantic Segmentation, Instance Segmentation, Depth Estimation
• ✨ PASCAL Context [URL]
  • Tasks: Semantic Segmentation, Human Part Segmentation, Semantic Edge Detection, Surface Normals Prediction, Saliency Detection.
• ✨ CelebA [URL]
  • Tasks: 40 human face Attributes.
• ✨ Taskonomy [URL]
  • 26 Tasks: Scene Categorization, Semantic Segmentation, Edge Detection, Monocular Depth Estimation, Keypoint Detection, etc.
• Visual Domain Decathlon [URL]
  • 10 Datasets: ImageNet, Aircraft, CIFAR100, etc.
  • Multi-domain multi-task learning
  • Rebuffi, S.-A., Bilen, H., & Vedaldi, A. Learning multiple visual domains with residual adapters. NeurIPS, 2017.
• BDD100K [URL]
  • 10-task Driving Dataset
  • Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. CVPR, 2020.
• MS COCO
  • Object detection, pose estimation, semantic segmentation.
  • See: MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach.
• Omnidata [URL]
  • A pipeline to resample comprehensive 3D scans from the real-world into static multi-task vision datasets
  • Eftekhar, A., Sax, A., Bachmann, R., Malik, J., & Zamir, A. Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans. ICCV, 2021.

NLP

• ✨ GLUE - General Language Understanding Evaluation [URL]
• ✨ decaNLP - The Natural Language Decathlon: A Multitask Challenge for NLP [URL]
• WMT Multilingual Machine Translation
• tasksource - 500+ MultipleChoice/Classification/TokenClassification tasks from HuggingFace Datasets Hub [URL]
  • Sileo, D. tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation. ArXiv, 2023.

RL & Robotics

• ✨ MetaWorld [URL]
• MTEnv [URL]

Graph

• QM9 [URL]
  • 11 properties of molecules; multi-task regression
  • See: Multi-Task Learning as a Bargaining Game.

Recommendation

• AliExpress [URL]
  • 2 Tasks: CTR and CTCVR from 5 countries
  • Li, P., Li, R., Da, Q., Zeng, A. X., & Zhang, L. Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. CIKM, 2020.
  • See: MTReclib
• MovieLens [URL]
  • 2 Tasks: binary classification (whether the user will watch) & regression (user’s rating)
  • See: DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning

Codebase

• General
  • ✨ LibMTL: LibMTL: A PyTorch Library for Multi-Task Learning
  • MALSAR: Multi-task learning via Structural Regularization (⚠️ Non-deep Learning)
• Computer Vision
  • ✨ Multi-Task-Learning-PyTorch: PyTorch implementation of multi-task learning architectures
  • ✨ mtan: The implementation of "End-to-End Multi-Task Learning with Attention"
  • ✨ auto-lambda: The Implementation of "Auto-Lambda: Disentangling Dynamic Task Relationships"
  • astmt: Attentive Single-tasking of Multiple Tasks
• NLP
  • ✨ mt-dnn: Multi-Task Deep Neural Networks for Natural Language Understanding
• Recommendation System
  • ✨ MTReclib: MTReclib provides a PyTorch implementation of multi-task recommendation models and common datasets.
• RL
  • mtrl: Multi Task RL Baselines

Architecture

Hard Parameter Sharing

• Heuer, F., Mantowsky, S., Bukhari, S. S., & Schneider, G. MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach. ICCV, 2021.
• Hu, R., & Singh, A. UniT: Multimodal Multitask Learning with a Unified Transformer. ICCV, 2021.
• ✨ Liu, X., He, P., Chen, W., & Gao, J. Multi-Task Deep Neural Networks for Natural Language Understanding. ACL, 2019.
• ✨ Kokkinos, I. UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory. CVPR, 2017.
• Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., & Urtasun, R. MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving. ArXiv, 2016.
• Caruana, R. Multitask Learning. 1997.

<p align="center"> <img src="imgs/hard-parameter-sharing.png" alt="client-demo" width="600px" /> </p>

Soft Parameter Sharing

• Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. Latent Multi-task Architecture Learning. AAAI, 2019.
• Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction. CVPR, 2019.
• Long, M., Cao, Z., Wang, J., & Yu, P. S. Learning Multiple Tasks with Multilinear Relationship Networks. NeurIPS, 2017.
• ✨ Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. Cross-Stitch Networks for Multi-task Learning. CVPR, 2016.
• ✨ Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. Progressive Neural Networks. ArXiv, 2016.
• ✨ Yang, Y., & Hospedales, T. Deep Multi-task Representation Learning: A Tensor Factorisation Approach. ICLR, 2017.
• Yang, Y., & Hospedales, T. M. Trace Norm Regularised Deep Multi-Task Learning. ICLR Workshop, 2017.

Decoder-focused Model

• Ye, H., & Xu, D. TaskPrompter: Spatial-Channel Multi-Task Prompting for Dense Scene Understanding. ICLR, 2023.
• Ye, H., & Xu, D. Inverted Pyramid Multi-task Transformer for Dense Scene Understanding. ECCV, 2022.
• Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Van Gool, L. Exploring Relational Context for Multi-Task Dense Prediction. ICCV, 2021.
• Vandenhende, S., Georgoulis, S., & Van Gool, L. MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning. ECCV, 2020.
• Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation. CVPR, 2019.
• Xu, D., Ouyang, W., Wang, X., & Sebe, N. PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing. CVPR, 2018.

Modulation & Adapters

• Schmied, T., Hofmarcher, M., Paischer, F., Pascanu, R., & Hochreiter, S. Learning to Modulate pre-trained Models in RL. NeurIPS, 2023.
• Sharma, M., Fantacci, C., Zhou, Y., Koppula, S., Heess, N., Scholz, J., & Aytar, Y. Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation. ICLR, 2023.
• ✨ He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G.