FraunhoferMEVIS/WholeSlideConcepts

Whole Slide Concepts

Model card for the Whole Slide Concepts model.

Getting started

First of all, please install mmm by running

pip install medicalmultitaskmodeling

Download the WSC model and run the following commands

from mmm.labelstudio_ext.NativeBlocks import NativeBlocks, MMM_MODELS
from mmm.mtl_modules.shared_blocks.Grouper import AttentionPoolingReducer, Grouper
from mmm.BaseModel import BaseModel
from functools import wraps
import torch

# # Globally disable weights_only loading for the trusted MMM checkpoints.
if not getattr(torch.load, "_mmm_trust_override", False):
    _original_torch_load = torch.load

    @wraps(_original_torch_load)
    def _trusted_load(*args, **kwargs):
        kwargs.setdefault("weights_only", False)
        return _original_torch_load(*args, **kwargs)

    _trusted_load._mmm_trust_override = True
    torch.load = _trusted_load

# # you might need to add the Grouper and ModuleDict to save serialization
# torch.serialization.add_safe_globals([Grouper, AttentionPoolingReducer])
# torch.serialization.add_safe_globals([torch.nn.modules.container.ModuleDict])

## to work with mmm version 1.1, we need to add a config class to the AttionPoolingReducer
if not hasattr(AttentionPoolingReducer, "Config"):
    class _AttentionPoolingReducerConfig(BaseModel):
        num_heads: int = 1

    AttentionPoolingReducer.Config = _AttentionPoolingReducerConfig
    old_init = AttentionPoolingReducer.__init__

    def _wrapped_init(self, embedding_dim, num_heads=None, cfg=None):
        if cfg is not None and num_heads is None:
            num_heads = cfg.num_heads
        old_init(self, embedding_dim, num_heads=num_heads)

    AttentionPoolingReducer.__init__ = _wrapped_init

Use the path to the WSC-MTL-tiny.zip file and add it to the known models.

# The downloaded .zip contains the .pt weights for the encoder, squeezer and grouper.
WSC = "WSC-MTL-tiny.zip"

# Place it in the folder you want to have it in
MMM_MODELS[WSC] = "/mmm/output/nicke/weights/tissue_concepts/WSC-MTL-tiny.zip"

# Load the model as native blocks on a CUDA device!
model = NativeBlocks(MMM_MODELS[WSC], device_identifier="cuda:0")

# Should give you dict_keys(['encoder', 'squeezer', 'grouper'])
model.keys()

Here is an example of inference for a Bag of 10 instances. Each instance is passed through the encoder to obtain a pyramid representation. This representation is then compressed by the 'squeezer' Module. Finally, to obtain a single WSI vector, the grouper is used to group the individual instances together. Notice the two arguments in the forward pass. First, all instances are passed. The second argument contains identifiers, which indicate which instances of the batch belong together

# now test the forward pass.
test = torch.rand((10,3,224,224))

with torch.no_grad():
    model_in = test.to(model.device)
    # We first obtain the pyramid features of the encoder
    pyramid = model['encoder'](model_in)
    # then pass it to obtain the latent representation of each instance
    _, latent = model['squeezer'](pyramid)
    latent = torch.nn.Flatten(start_dim=1)(latent)

    # And group all instances.
    # Notice the second argument in the forward pass. Here we define which instances of the batch belong together. 
    # In this case, all of the 10 instances belong to the same bag. Therefore we simply use ones.
    wsi_vector, attention_per_head = model['grouper'](latent, torch.ones((10,), dtype=torch.long, device=model.device))

# We obtain the latent_wsi vector of shape (1, 768) and the attention weights per head of shape (10, 8)
wsi_vector.shape, attention_per_head.shape

Please Cite

@misc{nicke2026slideconceptssupervisedfoundation,
      title={Whole Slide Concepts: A Supervised Foundation Model For Pathological Images}, 
      author={Till Nicke and Daniela Schacherer and Jan Raphael Schäfer and Natalia Artysh and Antje Prasse and André Homeyer and Andrea Schenk and Henning Höfener and Johannes Lotz},
      year={2026},
      eprint={2507.05742},
      archivePrefix={arXiv},
      primaryClass={eess.IV},
      url={https://arxiv.org/abs/2507.05742}, 
}

@article{schafer2024overcoming,
  title={Overcoming data scarcity in biomedical imaging with a foundational multi-task model},
  author={Sch{\"a}fer, Raphael and Nicke, Till and H{\"o}fener, Henning and Lange, Annkristin and Merhof, Dorit and Feuerhake, Friedrich and Schulz, Volkmar and Lotz, Johannes and Kiessling, Fabian},
  journal={Nature Computational Science},
  volume={4},
  number={7},
  pages={495--509},
  year={2024},
  publisher={Nature Publishing Group US New York}
}