Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge
NATURE COMMUNICATIONS, vol.16, no.1, 2025 (SCI-Expanded, Scopus)
- Publication Type: Article / Article
- Volume: 16 Issue: 1
- Publication Date: 2025
- Doi Number: 10.1038/s41467-025-60466-1
- Journal Name: NATURE COMMUNICATIONS
- Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, BIOSIS, CAB Abstracts, Chemical Abstracts Core, Geobase, INSPEC, MEDLINE, Veterinary Science Database, Directory of Open Access Journals, Nature Index
- Middle East Technical University Affiliated: Yes
Abstract
Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.