Evaluation, Reproducibility, Benchmarks Meeting 16

Minutes of meeting 16

Date: 25th May 2022

Present: Carole, Michela, Jorge, Keyvan, Lena, Annika

---

TOP 1: Mobilizing collaborative effort for reproducible research in MONAI

• Metrics Reloaded initiative - Metrics implementation:
  • Annika and Carole will check which of the metrics are already implemented in MONAI and if we need some update
  • Annika will send the prefinal metric table to Carole
  • Carole will provide the metric table and results of the analysis to the Dev team
    • Contact person for multi-threshold and counting metrics: Paul Jäger
    • Contact person for distance-based metrics: Carole
  • Within MONAI, the metrics should be grouped by metric families
• MONAI Bundle (Configuration of models, including metadata, pt files, ..)
  • Reproducibility is desired at different levels
    • Sharing weights/pretrained models
    • Sharing models that are ready to be used
    • Sharing models that are validated for a clinical application that are ready to be used
  • We may have a simple Delphi process after summer
  • We will discuss this topic in the next MICCAI SIG Board meeting (June 20th)
• MONAI datasets
  • The main work should be done by MONAI’s data working group
  • Once they converged to a solution, we will make the connection to challenge organizers

---

TOP 2: Members of WG

• Dan and Jens will leave the WG