Touchstone Benchmark

Subscribe us: https://groups.google.com/u/2/g/bodymaps

We present Touchstone, a large-scale medical segmentation benchmark based on annotated 5,195 CT volumes from 76 hospitals for training, and 6,933 CT volumes from 8 additional hospitals for testing. We invite AI inventors to train their models on AbdomenAtlas, and we independently evaluate their algorithms. We have already collaborated with 14 influential research teams, and we remain accepting new submissions.

Paper

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?
Pedro R. A. S. Bassi¹, Wenxuan Li¹, Yucheng Tang², Fabian Isensee³, ..., Alan Yuille¹, Zongwei Zhou^1
1Johns Hopkins University, ²NVIDIA, ³DKFZ
NeurIPS 2024

Touchstone 1.0 Leaderboard

rank	model	organization	average DSC	paper	github
🏆	MedNeXt	DKFZ	89.2
🏆	STU-Net-B	Shanghai AI Lab	89.0
🏆	MedFormer	Rutgers	89.0
🏆	nnU-Net ResEncL	DKFZ	88.8
🏆	UniSeg	NPU	88.8
🏆	Diff-UNet	HKUST	88.5
🏆	LHU-Net	UR	88.0
🏆	NexToU	HIT	87.8
9	SegVol	BAAI	87.1
10	U-Net & CLIP	CityU	87.1
11	Swin UNETR & CLIP	CityU	86.7
12	Swin UNETR	NVIDIA	80.1
13	UNesT	NVIDIA	79.1
14	SAM-Adapter	Duke	73.4
15	UNETR	NVIDIA	64.4

Aorta - NexToU 🏆

rank	model	organization	DSC	paper	github
🏆	NexToU	HIT	86.4
2	MedNeXt	DKFZ	83.1
3	UniSeg	NPU	82.3
4	STU-Net-B	Shanghai AI Lab	82.1
5	nnU-Net ResEncL	DKFZ	81.4
6	Diff-UNet	HKUST	81.2
7	Swin UNETR	NVIDIA	81.1
8	SegVol	BAAI	80.2
9	UNesT	NVIDIA	78.6
10	Swin UNETR & CLIP	CityU	78.1
11	U-Net & CLIP	CityU	77.1
12	SAM-Adapter	Duke	62.8
13	UNETR	NVIDIA	52.1

Gallbladder - STU-Net-B & MedNeXt 🏆

rank	model	organization	DSC	paper	github
🏆	STU-Net-B	Shanghai AI Lab	85.5
🏆	MedNeXt	DKFZ	85.3
3	nnU-Net ResEncL	DKFZ	84.9
4	UniSeg	NPU	84.7
5	Diff-UNet	HKUST	83.8
6	NexToU	HIT	82.3
7	U-Net & CLIP	CityU	82.1
8	Swin UNETR & CLIP	CityU	80.2
9	SegVol	BAAI	79.3
10	Swin UNETR	NVIDIA	69.2
11	UNesT	NVIDIA	62.1
12	SAM-Adapter	Duke	49.4
13	UNETR	NVIDIA	43.8

KidneyL - Diff-UNet 🏆

rank	model	organization	DSC	paper	github
🏆	Diff-UNet	HKUST	91.9
2	nnU-Net ResEncL	DKFZ	91.9
3	STU-Net-B	Shanghai AI Lab	91.9
4	MedNeXt	DKFZ	91.8
5	SegVol	BAAI	91.8
6	UniSeg	NPU	91.5
7	U-Net & CLIP	CityU	91.1
8	Swin UNETR & CLIP	CityU	91.0
9	NexToU	HIT	89.6
10	SAM-Adapter	Duke	87.3
11	Swin UNETR	NVIDIA	85.5
12	UNesT	NVIDIA	85.4
13	UNETR	NVIDIA	63.7

KidneyR - Diff-UNet 🏆

rank	model	organization	DSC	paper	github
🏆	Diff-UNet	HKUST	92.8
2	MedNeXt	DKFZ	92.6
3	nnU-Net ResEncL	DKFZ	92.6
4	STU-Net-B	Shanghai AI Lab	92.5
5	SegVol	BAAI	92.5
6	UniSeg	NPU	92.2
7	U-Net & CLIP	CityU	91.9
8	Swin UNETR & CLIP	CityU	91.7
9	SAM-Adapter	Duke	90.4
10	NexToU	HIT	90.1
11	UNesT	NVIDIA	83.6
12	Swin UNETR	NVIDIA	81.7
13	UNETR	NVIDIA	69.6

Liver - MedNeXt 🏆

rank	model	organization	DSC	paper	github
🏆	MedNeXt	DKFZ	96.3
2	nnU-Net ResEncL	DKFZ	96.3
3	Diff-UNet	HKUST	96.2
4	STU-Net-B	Shanghai AI Lab	96.2
5	UniSeg	NPU	96.1
6	U-Net & CLIP	CityU	96.0
7	SegVol	BAAI	96.0
8	Swin UNETR & CLIP	CityU	95.8
9	NexToU	HIT	95.7
10	SAM-Adapter	Duke	94.1
11	UNesT	NVIDIA	93.6
12	Swin UNETR	NVIDIA	93.5
13	UNETR	NVIDIA	90.5

Pancreas - MedNeXt 🏆

rank	model	organization	DSC	paper	github
🏆	MedNeXt	DKFZ	83.3
2	STU-Net-B	Shanghai AI Lab	83.2
3	nnU-Net ResEncL	DKFZ	82.9
4	UniSeg	NPU	82.7
5	Diff-UNet	HKUST	81.9
6	U-Net & CLIP	CityU	80.8
7	Swin UNETR & CLIP	CityU	80.2
8	NexToU	HIT	80.2
9	SegVol	BAAI	79.1
10	Swin UNETR	NVIDIA	68.5
11	UNesT	NVIDIA	68.3
12	UNETR	NVIDIA	55.1
13	SAM-Adapter	Duke	50.2

Postcava - STU-Net-B & MedNeXt 🏆

rank	model	organization	DSC	paper	github
🏆	STU-Net-B	Shanghai AI Lab	81.3
🏆	MedNeXt	DKFZ	81.3
3	UniSeg	NPU	81.2
4	Diff-UNet	HKUST	80.8
5	nnU-Net ResEncL	DKFZ	80.5
6	U-Net & CLIP	CityU	78.5
7	NexToU	HIT	78.1
8	SegVol	BAAI	77.8
9	Swin UNETR & CLIP	CityU	76.8
10	Swin UNETR	NVIDIA	69.9
11	UNesT	NVIDIA	66.2
12	UNETR	NVIDIA	53.9
13	SAM-Adapter	Duke	48.0

Spleen - nnU-Net ResEncL 🏆

rank	model	organization	DSC	paper	github
🏆	nnU-Net ResEncL	DKFZ	95.2
2	MedNeXt	DKFZ	95.2
3	STU-Net-B	Shanghai AI Lab	95.1
4	Diff-UNet	HKUST	95.0
5	UniSeg	NPU	94.9
6	SegVol	BAAI	94.5
7	NexToU	HIT	94.7
8	U-Net & CLIP	CityU	94.3
9	Swin UNETR & CLIP	CityU	94.1
10	SAM-Adapter	Duke	90.5
11	Swin UNETR	NVIDIA	87.9
12	UNesT	NVIDIA	86.7
13	UNETR	NVIDIA	76.5

Stomach - STU-Net-B & MedNeXt & nnU-Net ResEncL 🏆

rank	model	organization	DSC	paper	github
🏆	STU-Net-B	Shanghai AI Lab	93.5
🏆	MedNeXt	DKFZ	93.5
🏆	nnU-Net ResEncL	DKFZ	93.4
4	UniSeg	NPU	93.3
5	Diff-UNet	HKUST	93.1
6	NexToU	HIT	92.7
7	SegVol	BAAI	92.5
8	U-Net & CLIP	CityU	92.4
9	Swin UNETR & CLIP	CityU	92.2
10	SAM-Adapter	Duke	88.0
11	UNesT	NVIDIA	87.6
12	Swin UNETR	NVIDIA	84.1
13	UNETR	NVIDIA	74.2

Touchstone 1.0 Dataset

Training set

• Touchstone 1.0: AbdomenAtlas1.0Mini (N=5,195)
• Touchstone 2.0: AbdomenAtlas1.1Mini (N=9,262)

Test set

• Proprietary JHH dataset (N=5,172)
• Public TotalSegmentator V2 dataset (N=1,228)

Figure 1. Metadata distribution in the test set.

Touchstone 1.0 Model

Note

We are releasing the trained AI models evaluated in Touchstone right here. Stay tuned!

rank	model	average DSC	parameter	infer. speed	download
🏆	MedNeXt	89.2	61.8M	★☆☆☆☆
🏆	STU-Net-B	89.0	58.3M	★★☆☆☆
🏆	MedFormer	89.0	38.5M	★★★☆☆
🏆	nnU-Net ResEncL	88.8	102.0M	★★★★☆
🏆	UniSeg	88.8	31.0M	☆☆☆☆☆
🏆	Diff-UNet	88.5	434.0M	★★★☆☆
🏆	LHU-Net	88.0	8.6M	★★★★★
🏆	NexToU	87.8	81.9M	★★★★☆
9	SegVol	87.1	181.0M	★★★★☆
10	U-Net & CLIP	87.1	19.1M	★★★☆☆
11	Swin UNETR & CLIP	86.7	62.2M	★★★☆☆
12	Swin UNETR	80.1	72.8M	★★★★★
13	UNesT	79.1	87.2M	★★★★★
14	SAM-Adapter	73.4	11.6M	★★★★☆
15	UNETR	64.4	101.8M	★★★★★

Citation

Please cite the following papers if you find our study helpful.

@article{bassi2024touchstone,
  title={Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?},
  author={Bassi, Pedro RAS and Li, Wenxuan and Tang, Yucheng and Isensee, Fabian and Wang, Zifu and Chen, Jieneng and Chou, Yu-Cheng and Kirchhoff, Yannick and Rokuss, Maximilian and Huang, Ziyan and Ye, Jin and He, Junjun and Wald, Tassilo and Ulrich, Constantin and Baumgartner, Michael and Roy, Saikat and Maier-Hein, Klaus H. and Jaeger, Paul and Ye, Yiwen and Xie, Yutong and Zhang, Jianpeng and Chen, Ziyang and Xia, Yong and Xing, Zhaohu and Zhu, Lei and Sadegheih, Yousef and Bozorgpour, Afshin and Kumari, Pratibha and Azad, Reza and Merhof, Dorit and Shi, Pengcheng and Ma, Ting and Du, Yuxin and Bai, Fan and Huang, Tiejun and Zhao, Bo and Wang, Haonan and Li, Xiaomeng and Gu, Hanxue and Dong, Haoyu and Yang, Jichen and Mazurowski, Maciej A. and Gupta, Saumya and Wu, Linshan and Zhuang, Jiaxin and Chen, Hao and Roth, Holger and Xu, Daguang and Blaschko, Matthew B. and Decherchi, Sergio and Cavalli, Andrea and Yuille, Alan L. and Zhou, Zongwei},
  journal={Conference on Neural Information Processing Systems},
  year={2024},
  utl={https://github.com/MrGiovanni/Touchstone}
}

@article{li2024abdomenatlas,
  title={AbdomenAtlas: A large-scale, detailed-annotated, \& multi-center dataset for efficient transfer learning and open algorithmic benchmarking},
  author={Li, Wenxuan and Qu, Chongyu and Chen, Xiaoxi and Bassi, Pedro RAS and Shi, Yijia and Lai, Yuxiang and Yu, Qian and Xue, Huimin and Chen, Yixiong and Lin, Xiaorui and others},
  journal={Medical Image Analysis},
  pages={103285},
  year={2024},
  publisher={Elsevier}
}

@inproceedings{li2024well,
  title={How Well Do Supervised Models Transfer to 3D Image Segmentation?},
  author={Li, Wenxuan and Yuille, Alan and Zhou, Zongwei},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024}
}

@article{qu2023abdomenatlas,
  title={Abdomenatlas-8k: Annotating 8,000 CT volumes for multi-organ segmentation in three weeks},
  author={Qu, Chongyu and Zhang, Tiezheng and Qiao, Hualin and Tang, Yucheng and Yuille, Alan L and Zhou, Zongwei and others},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2023}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the McGovern Foundation. Paper content is covered by patents pending.