A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Rosy Tsopra; Xose Fernandez; Claudio  Luchinat; Lilia  Alberghina; Hans  Lehrach; Marco  Vanoni; Felix  Dreher; O Ugur Sezerman Sezerman; Marc  Cuggia; Marie  de Tayrac; Edvīns Miklaševičs; Lucian Mihai  Itu; Marius  Geanta; Lesley  Ogilvie; Florence  Godey; Cristian Nicolae  Boldisor; Boris  Campillo-Gimenez; Cosmina  Cioroboiu; Costin Florian  Ciusdel; Simona  Coman; Oliver Hijano  Cubelos; Alina  Itu; Bodo  Lange; Matthieu  Le Gallo; Alexandra  Lespagnol; Giancarlo  Mauri; H Okan  Soykam; Bastien  Rance; Paola  Turano; Leonardo  Tenori; Alessia  Vignoli; Christoph  Wierling; Nora  Benhabiles; Anita  Burgun

doi:10.1186/s12911-021-01634-3

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Rosy Tsopra (Corresponding Author), Xose Fernandez, Claudio Luchinat, Lilia Alberghina, Hans Lehrach, Marco Vanoni, Felix Dreher, O Ugur Sezerman Sezerman, Marc Cuggia, Marie de Tayrac, Edvīns Miklaševičs, Lucian Mihai Itu, Marius Geanta , Lesley Ogilvie, Florence Godey, Cristian Nicolae Boldisor, Boris Campillo-Gimenez , Cosmina Cioroboiu, Costin Florian Ciusdel, Simona Coman Oliver Hijano Cubelos, Alina Itu, Bodo Lange, Matthieu Le Gallo, Alexandra Lespagnol, Giancarlo Mauri , H Okan Soykam, Bastien Rance , Paola Turano, Leonardo Tenori, Alessia Vignoli, Christoph Wierling , Nora Benhabiles, Anita Burgun

Research output: Contribution to journal › Article › peer-review

46 Citations (Scopus)

7 Downloads (Pure)

Abstract

Background
Artificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.

Methods
The European “ITFoC (Information Technology for the Future Of Cancer)” consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.

Results
This framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the “ITFoC Challenge”. This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.

Conclusions
The predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

Original language	English
Article number	274
Number of pages	14
Journal	BMC Medical Informatics and Decision Making
Volume	21
Issue number	1
DOIs	https://doi.org/10.1186/s12911-021-01634-3
Publication status	Published - 2 Oct 2021

Keywords*

Artificial intelligence
Precision medicine
Personalized medicine
Computerized decision support systems
cancer
oncology

Field of Science*

1.2 Computer and information sciences
3.1 Basic medicine

Publication Type*

1.1. Scientific article indexed in Web of Science and/or Scopus database

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1186/s12911-021-01634-3

A framework for validating AI in precisionFinal published version, 1.58 MBLicence: CC BY

Cite this

Tsopra, R., Fernandez, X., Luchinat, C., Alberghina, L., Lehrach, H., Vanoni, M., Dreher, F., Sezerman, O. U. S., Cuggia, M., de Tayrac, M., Miklaševičs, E., Itu, L. M., Geanta , M., Ogilvie, L., Godey, F., Boldisor, C. N., Campillo-Gimenez , B., Cioroboiu, C., Ciusdel, C. F., ... Burgun, A. (2021). A framework for validating AI in precision medicine: considerations from the European ITFoC consortium. BMC Medical Informatics and Decision Making, 21(1), Article 274. https://doi.org/10.1186/s12911-021-01634-3

@article{f12ad1f56c1944d0bce1ebf8bfc1b1f0,

title = "A framework for validating AI in precision medicine: considerations from the European ITFoC consortium",

abstract = "BackgroundArtificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.MethodsThe European “ITFoC (Information Technology for the Future Of Cancer)” consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.ResultsThis framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the “ITFoC Challenge”. This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.ConclusionsThe predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.",

keywords = "Artificial intelligence, Precision medicine, Personalized medicine, Computerized decision support systems, cancer, oncology",

author = "Rosy Tsopra and Xose Fernandez and Claudio Luchinat and Lilia Alberghina and Hans Lehrach and Marco Vanoni and Felix Dreher and Sezerman, {O Ugur Sezerman} and Marc Cuggia and {de Tayrac}, Marie and Edvīns Mikla{\v s}evi{\v c}s and Itu, {Lucian Mihai} and Marius Geanta and Lesley Ogilvie and Florence Godey and Boldisor, {Cristian Nicolae} and Boris Campillo-Gimenez and Cosmina Cioroboiu and Ciusdel, {Costin Florian} and Simona Coman and Cubelos, {Oliver Hijano} and Alina Itu and Bodo Lange and {Le Gallo}, Matthieu and Alexandra Lespagnol and Giancarlo Mauri and Soykam, {H Okan} and Bastien Rance and Paola Turano and Leonardo Tenori and Alessia Vignoli and Christoph Wierling and Nora Benhabiles and Anita Burgun",

note = "Funding Information: This work was supported by the ITFoC project (Information Technology for the Future of Cancer) – FLAG-ERA support. Publisher Copyright: {\textcopyright} 2021, The Author(s).",

year = "2021",

month = oct,

day = "2",

doi = "10.1186/s12911-021-01634-3",

language = "English",

volume = "21",

journal = "BMC Medical Informatics and Decision Making",

issn = "1472-6947",

publisher = "BioMed Central Ltd.",

number = "1",

}

Tsopra, R, Fernandez, X, Luchinat, C, Alberghina, L, Lehrach, H, Vanoni, M, Dreher, F, Sezerman, OUS, Cuggia, M, de Tayrac, M, Miklaševičs, E, Itu, LM, Geanta , M, Ogilvie, L, Godey, F, Boldisor, CN, Campillo-Gimenez , B, Cioroboiu, C, Ciusdel, CF, Coman , S, Cubelos, OH, Itu, A, Lange, B, Le Gallo, M, Lespagnol, A, Mauri , G, Soykam, HO, Rance , B, Turano, P, Tenori, L, Vignoli, A, Wierling , C, Benhabiles, N & Burgun, A 2021, 'A framework for validating AI in precision medicine: considerations from the European ITFoC consortium', BMC Medical Informatics and Decision Making, vol. 21, no. 1, 274. https://doi.org/10.1186/s12911-021-01634-3

TY - JOUR

T1 - A framework for validating AI in precision medicine

T2 - considerations from the European ITFoC consortium

AU - Tsopra, Rosy

AU - Fernandez, Xose

AU - Luchinat, Claudio

AU - Alberghina, Lilia

AU - Lehrach, Hans

AU - Vanoni, Marco

AU - Dreher, Felix

AU - Sezerman, O Ugur Sezerman

AU - Cuggia, Marc

AU - de Tayrac, Marie

AU - Miklaševičs, Edvīns

AU - Itu, Lucian Mihai

AU - Geanta , Marius

AU - Ogilvie, Lesley

AU - Godey, Florence

AU - Boldisor, Cristian Nicolae

AU - Campillo-Gimenez , Boris

AU - Cioroboiu, Cosmina

AU - Ciusdel, Costin Florian

AU - Coman , Simona

AU - Cubelos, Oliver Hijano

AU - Itu, Alina

AU - Lange, Bodo

AU - Le Gallo, Matthieu

AU - Lespagnol, Alexandra

AU - Mauri , Giancarlo

AU - Soykam, H Okan

AU - Rance , Bastien

AU - Turano, Paola

AU - Tenori, Leonardo

AU - Vignoli, Alessia

AU - Wierling , Christoph

AU - Benhabiles, Nora

AU - Burgun, Anita

PY - 2021/10/2

Y1 - 2021/10/2

N2 - BackgroundArtificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.MethodsThe European “ITFoC (Information Technology for the Future Of Cancer)” consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.ResultsThis framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the “ITFoC Challenge”. This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.ConclusionsThe predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

AB - BackgroundArtificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.MethodsThe European “ITFoC (Information Technology for the Future Of Cancer)” consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.ResultsThis framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the “ITFoC Challenge”. This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.ConclusionsThe predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

KW - Artificial intelligence

KW - Precision medicine

KW - Personalized medicine

KW - Computerized decision support systems

KW - cancer

KW - oncology

UR - http://www.scopus.com/inward/record.url?scp=85116355315&partnerID=8YFLogxK

U2 - 10.1186/s12911-021-01634-3

DO - 10.1186/s12911-021-01634-3

M3 - Article

SN - 1472-6947

VL - 21

JO - BMC Medical Informatics and Decision Making

JF - BMC Medical Informatics and Decision Making

IS - 1

M1 - 274

ER -

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Abstract

Keywords*

Field of Science*

Publication Type*

UN SDGs

Access to Document

Other files and links

Fingerprint

ITFoC: Information Technology: The Future of Cancer Treatment

Cite this

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Abstract

Keywords*

Field of Science*

Publication Type*

UN SDGs

Access to Document

Other files and links

Fingerprint

Projects

ITFoC: Information Technology: The Future of Cancer Treatment

Cite this