<div align="center"> <h1>Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling</h1> </div> <details> <summary> <b>Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling</b>, MedRxiv, July 2023.

</summary>

Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Kain, A. M., Saillard, C., & Schiratti, J.-B. (2023). Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling. MedRxiv.

@article{Filiot2023scalingwithMIM,
	author       = {Alexandre Filiot and Ridouane Ghermi and Antoine Olivier and Paul Jacob and Lucas Fidon and Alice Mac Kain and Charlie Saillard and Jean-Baptiste Schiratti},
	title        = {Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling},
	elocation-id = {2023.07.21.23292757},
	year         = {2023},
	doi          = {10.1101/2023.07.21.23292757},
	publisher    = {Cold Spring Harbor Laboratory Press},
	url          = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757v2},
	eprint       = {https://www.medrxiv.org/content/early/2023/07/26/2023.07.21.23292757v2.full.pdf},
	journal = {medRxiv}
}

</details>

Update :tada: Phikon release on Hugging Face :tada:

We released our Phikon model on Hugging Face. Check out our community blog post ! We also provide a Colab notebook to perform weakly-supervised learning on Camelyon16 and fine-tuning with LoRA on NCT-CRC-HE using Phikon.

Here is a code snippet to perform feature extraction using Phikon.

from PIL import Image
import torch
from transformers import AutoImageProcessor, ViTModel

# load an image
image = Image.open("assets/example.tif")

# load phikon
image_processor = AutoImageProcessor.from_pretrained("owkin/phikon")
model = ViTModel.from_pretrained("owkin/phikon", add_pooling_layer=False)

# process the image
inputs = image_processor(image, return_tensors="pt")

# get the features
with torch.no_grad():
    outputs = model(**inputs)
    features = outputs.last_hidden_state[:, 0, :]  # (1, 768) shape

Official PyTorch Implementation and pre-trained models for Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling. This minimalist repository aims to:

Publicly release the weights of our Vision Transformer Base (ViT-B) model Phikon pre-trained with iBOT on 40M pan-cancer histology tiles from TCGA. Phikon achieves state-of-the-art performance on a large variety of downstream tasks compared to other SSL frameworks available in the literature.

⚠️ Addendum :warning:

From 09.01.2023 to 10.30.2023, this repository stated using the student, please use the teacher backbone instead.

# feature extraction snippet with `rl_benchmarks` repository
from PIL import Image
from rl_benchmarks.models import iBOTViT

# instantiate iBOT ViT-B Pancancer model, aka Phikon
# /!\ please use the "teacher" encoder which produces better results !
weights_path = "/<your_root_dir>/weights/ibot_vit_base_pancan.pth">
ibot_base_pancancer = iBOTViT(architecture="vit_base_pancan", encoder="teacher", weights_path=weights_path)

# load an image and transform it into a normalized tensor
image = Image.open("assets/example.tif")  # (224, 224, 3), uint8
tensor = ibot_base_pancancer.transform(image) # (3, 224, 224), torch.float32
batch = tensor.unsqueeze(0)  # (1, 3, 224, 224), torch.float32

# compute the 768-d features
features = ibot_base_pancancer(batch).detach().cpu().numpy()
assert features.shape == (1, 768)

Publicly release the histology features of our ViT-based iBOT models (iBOT[ViT-S]COAD, iBOT[ViT-B]COAD, iBOT[ViT-B]PanCancer, iBOT[ViT-L]COAD) for i) 11 TCGA cohorts and Camelyon16 slides datasets; and ii) NCT-CRC and Camelyon17-Wilds patches datasets.
Reproduce the results from our publication, including: features extraction and clinical data processing, cross-validation experiments, results generation.

Abstract

<details> <summary> Read full abstract from MedRxiv.

main_figure

</summary> Computational pathology is revolutionizing the field of pathology by integrating advanced computer vision and machine learning technologies into diagnostic workflows. Recently, Self-Supervised Learning (SSL) has emerged as a promising solution to learn representations from histology patches, leveraging large volumes of unannotated whole slide images whole slide images (WSI). In particular, Masked Image Modeling (MIM) showed remarkable results and robustness over purely contrastive learning methods. In this work, we explore the application of MIM to histology using iBOT, a self-supervised transformer-based framework. Through a wide range of downstream tasks over seven cancer indications, we provide recommendations on the pre-training of large models for histology data using MIM. First, we demonstrate that in-domain pre-training with iBOT outperforms both ImageNet pre-training and a model pre-trained with a purely contrastive learning objective, MoCo V2. Second, we show that Vision Transformers (ViT), when scaled appropriately, have the capability to learn pan-cancer representations that benefit a large variety of downstream tasks. Finally, our iBOT ViT-Base model, pre-trained on more than 40 million histology images from 16 different cancer types, achieves state-of-the-art performance in most weakly-supervised WSI classification tasks compared to other SSL frameworks. Our code, models and features are publicly available at https://github.com/owkin/HistoSSLscaling. </details>

Data structure

Download

You can download the data necessary to use the present code and reproduce our results here:

raw data: Google Drive
preprocessed data: Google Drive
weights: Google Drive

Please create weights, raw and preprocessed folders containing the content of the different downloads. This step may take time depending on your wifi bandwidth (folder takes 1.2 To). You can use rclone to download the folder from a remote machine (preferred in a tmux session).

Description

The bucket contains three main folders: a weights, raw and preprocessed folders. The weights folder contains weights for iBOT[ViT-B]PanCancer (our best ViT-B iBOT model). Other models from the literature can be retrieved from the corresponding Github repositories:

CTransPath: https://github.com/Xiyue-Wang/TransPath
HIPT: https://github.com/mahmoodlab/HIPT
Dino[ViT-S]BRCA: https://github.com/Richarizardd/Self-Supervised-ViT-Path

weights/
└── ibot_vit_base_pancan.pth          # Ours

The raw folder contains two subfolders for slide-level and tile-level downstream task.

Slide-level: each cohort contains 2 folders, clinical and slides. We provide clinical data but not raw slides. No modification was performed on the folders architectures and files names of raw slides and patches compared to the original source (i.e. TCGA, Camelyon16, NCT-CRC and Camelyon17-WILDS).
Tile-level: each cohort contains 2 folders, clinical and patches. We only provide clinical data (i.e. labels), not patches datasets.

[!WARNING] We don't provide raw slides or patches (slides, patches folders are empty). You can download raw slides or patches here:

PAIP: http://www.wisepaip.org/paip/guide/dataset

TCGA: https://portal.gdc.cancer.gov/

Camelyon16: http://gigadb.org/dataset/100439

NCT-CRC: https://zenodo.org/record/1214456

Camelyon17-WILDS: https://github.com/p-lambda/wilds/blob/main/wilds/download_datasets.py

Once you downloaded the data, please follow the same folders architecture as indicated below (without applying modifications on folders and files names compared to original download).

raw/
├── slides_classification               # slides classification tasks
===============================================================================
│   ├── CAMELYON16_FULL                 # cohort
│   │   ├── clinical                    # clinical data (for labels)
│   │   │   ├── test_clinical_data.csv
│   │   │   └── train_clinical_data.csv
│   │   └── slides                      # raw slides (not provided)
│   │        ├── Normal_001.tif
│   │        ├── Normal_002.tif...
│   └── TCGA
│       ├── tcga_statistics.pk          # For each cohort and label, list (n_patients, n_slides, labels_distribution)
│       ├── clinical                    # for TCGA, clinical data is divided into subfolders
│       │   ├── hrd
│       │   │   ├── hrd_labels_tcga_brca.csv
│       │   │   └── hrd_labels_tcga_ov.csv
│       │   ├── msi
│       │   │   ├── msi_labels_tcga_coad.csv
│       │   │   ├── msi_labels_tcga_read.csv...
│       │   ├── subtypes
│       │   │   ├── brca_tcga_pan_can_atlas_2018_clinical_data.tsv.gz
│       │   │   ├── coad_tcga_pan_can_atlas_2018_clinical_data.tsv.gz...
│       │   └── survival
│       │       ├── survival_labels_tcga_brca.csv
│       │       ├── survival_labels_tcga_coad.csv...
│       └── slides
│           └── parafine
│               ├── TCGA_BRCA
│               │   ├── 03627311-e413-4218-b836-177abdfc3911
│               │   │   └── TCGA-XF-AAN7-01Z-00-DX1.B8EDF045-604C-48CB-8E54-A60564CAE2AD.svs
...

└── tiles_classification                # tiles classification tasks
===============================================================================
    ├── CAMELYON17-WILDS_FULL           # cohort
    │   ├── clinical                    # clinical data (for labels)
    │   │    └── metadata.csv
    │   └── patches                     # patches (not provided)
    │        ├── patient_004_node_4...
    │        │   ├── patch_patient_004_node_4_x_10016_y_16704.png...
    └── NCT-CRC_FULL
        ├── labels                      # here the labels are set using the folders architecture
        │   └── dict_labels.pkl
        └── patches
            ├── NCT-CRC-VAL-HE-7K
            │    ├── ADI...
            │    │    ├── ADI-TCGA-AAICEQFN.tif...
            └── NCT-CRC-HE-100K-NONORM
                 ├── ADI...
                 │    ├── ADI-AAAFLCLY.tif...

The preprocessed folder contains two subfolders for slide-level and tile-level downstream tasks.

Slide-level: for each feature extractor and dataset, we provide coordinates and features. Coordinates are provided as (N_tiles_slide, 3) numpy arrays where the 3 first columns rows correspond to (tile_level, x_coordinate, y_coordinate). Features are provided as (N_tiles_slide, 3+d) numpy arrays, the d last columns being the model's features (3 first are the previous coordinates). Coordinates are meant to extract the same tiles as done in our publication but are not needed for downstream experiments (only features are needed). Note that coordinates are divided into coords_224, coords_256 and coords_4096, corresponding to 224 x 224 tiles (iBOT, CTransPath and ResNet models), 256 x 256 (Dino models) and 4096 x 4096 (HIPT) tiles, respectively.

[!NOTE] We provide all matter tiles for each slide. All tiles were extracted at 0.5 micrometers / pixel (20x magnification) except for CTransPath (mpp = 1.0 following the authors recommendation).

[!WARNING] The tile_level is computed with openslide.deepzoom.DeepZoomGenerator through the following schematic syntax:

from openslide import open_slide
from openslide.deepzoom import DeepZoomGenerator

slide = open_slide("<slide_path>")
dzg = DeepZoomGenerator(slide, tile_size=224, overlap=0)
tile = dzg.get_tile(level=17, address=(8, 10))
# this corresponds to coordinates (17, 8, 10) in the coordinates we provide for the given slide

Tile-level: for each feature extractor and dataset, we provide patches ids and features. Features are (N_patches_dataset, d) numpy arrays and ids take the form of (N_patches_dataset, 1) string numpy array.

Here is a description of the different features and coordinates we provide in the preprocessed folder.

preprocessed/                         # preprocessed data (coords, features)
===============================================================================
├── slides_classification             # slides classification tasks
│   ├── coords
│   │   ├── coords_224                # coordinates for 224 x 224 tiles
│   │   │   ├── CAMELYON16_FULL       # cohort 
│   │   │   │   ├── Normal_001.tif    # slide_id
│   │   │   │       └── coords.npy    # coordinates array (N_tiles_slide, 3)
...
│   │   │   ├── TCGA
│   │   │   │   ├── TCGA_BRCA
│   │   │   │   │   ├── TCGA-3C-AALI-01Z-00-DX1.F6E9A5DF-D8FB-45CF-B4BD-C6B76294C291.svs
│   │   │   │   │       └── coords.npy
...
│   │   ├── coords_256                # coordinates for 256 x 256 tiles
│   │   └── coords_4096               # coordinates for 4096 x 4096 tiles

...
│   └── features                      # features
│       ├── iBOTViTBasePANCAN         # feature extractor
│       │   ├── CAMELYON16_FULL       # cohort
│       │   │   ├── Normal_001.tif    # slide_id
│       │   │       └── features.npy  # features array (N_tiles_slide, 3+d)
...
│       │   ├── TCGA
│       │   │   ├── TCGA_BRCA
│       │   │   │   ├── TCGA-3C-AALI-01Z-00-DX1.F6E9A5DF-D8FB-45CF-B4BD-C6B76294C291.svs
│       │   │   │       └── features.npy
...
│       ├── MoCoWideResNetCOAD        # same structure applies for all extractors
│       ├── ResNet50
│       ├── iBOTViTBaseCOAD
│       ├── iBOTViTBasePANCAN
│       ├── iBOTViTLargeCOAD
│       ├── iBOTViTSmallCOAD
...
/!\ If you wish to extract features for