LUGANO, 30 September, 2021 – A new study has found that a direct-to-consumer machine learning model for detecting skin cancers incorrectly classified rare and aggressive cancers as low-risk.1 The breakthrough findings presented at today’s 30th EADV Congress suggest that making apps based on such models available directly to the public without transparency on performance metrics for rare but potentially life-threatening skin cancers is ethically questionable.
Researchers in London focused on two types of skin cancer, Merkel cell carcinoma (MCC) and amelanotic melanoma, both of which are rare but particularly aggressive cancers that tend to grow fast and require early treatment. They created a dataset of 116 images of these rare cancers and of the benign lesions seborrahoeic keratosis and haemangiomas, and assessed these images with two machine-learning models.
The first model studied was a certified medical device, directly sold to the public via the App store and advertised as being able to diagnose 95% of skin cancers (Model 1). The second model was available for research purposes only and used as a reference (Model 2).
The results showed that Model 1 incorrectly classified 17.9% of MCCs and 22.9% of amelanotic melanomas as low-risk. In turn, 62.2% of benign lesions were classified as high risk. For detecting malignancy, Model 1’s sensitivity was 79.4% [95% confidence interval (CI) 69.3-89.4%] and specificity was 37.7% [95% CI 24.7-50.8]. For Model 2, MCC was not included in the top 5 diagnosis for any of the 28 MCC images analysed, raising the possibility that the model had not been trained that this disease class exists.
The high false positive rate of Model 1 has potentially negative consequences on a personal and societal level. The results pose a bigger question of the safety of other artificial intelligence (AI) models for detecting skin cancer available on the market.
Lloyd Steele, lead author of the study at the Blizard Institute, Queen Mary University of London, UK explains: “In order to improve, machine learning model evaluations should consider the spectrum of diseases that will be seen in practice. At the moment, most of the performance of those models is driven by the imaging data available, which is particularly scarce when it comes to rare skin cancers.”
A global collaboration between research groups and hospitals can be a step towards tackling the gap of skin cancer imaging data, which is a crucial element for a high-performance rate of machine learning.
Marie-Aleth Richard, EADV Board Member and Professor at the University Hospital of La Timone, Marseille, said: “The number of skin cancer detection apps available for consumer use is growing, but as demonstrated in this research, there must be more transparency around the safety and efficacy of these apps. Furthermore such devices detect only what they are shown to analyse and do not make systematic analysis of all the skin’s surface. Failure to be transparent could put lives at risk.”
ENDS
Notes to Editors
A reference to the EADV 30th Congress or EADV Congress 2021 must be included when communicating any information within this press release.
Contact:
For further information or to arrange an expert interview, please contact:
Boryana Kermenova – EADV Press Officer
+44 (0) 208 971 6429
Catriona Martin – EADV Press Officer
+44 (0) 208 971 6412
About Merkel Cell Carcinoma
Merkel cell carcinoma (MCC) is a rare non-melanoma cancer that is highly aggressive and fast-growing. It starts in hormone-producing Merkel cells, which are usually in the top layer of the skin and in hair follicles. MCC presents as bluish red lumps on the skin, often found on the head, neck, arms and legs, but can spread to other parts of the body. It is mainly associated with ultraviolet light, from long-term sun exposure and sunbeds, as well conditions or treatments which weaken the immune system and polyomavirus infections.2 The prevalence of MCC is 0.2-0.45 case per 100,000 population.3
About Amelanotic Melanoma
Melanoma is a type of skin cancer which develops in cells called melanocytes. It most commonly affects older people.4 Amelanotic melanoma is rare, accounting for around 8% of all melanomas. Amelanotic means without melanin, a dark coloured pigment of the skin. Unlike other melanomas, amelanotic melanomas are usually red or skin coloured rather than dark coloured. They are often difficult to diagnose because of their lack of colour and may be mistaken for other skin conditions. Thus, as their diagnosis is often delayed, they are associated with a poor prognosis.5
About EADV
Founded in 1987, EADV is a leading European Dermato-Venereology Society with the important aims of improving the quality of patient care, furthering knowledge and education of dermatologists and venereologists and advocating on behalf of the speciality and patients. It is a non-profit organisation with nearly 7,000 members across 116 different countries in the world, providing a valuable service for every type of dermato-venereologist professional.
To find out more visit https://www.eadv.org/.
About EADV 30th Congress 2021:
The EADV's 30th Congress Anniversary Edition is a special celebration of three decades of science and innovation in the Dermatology and Venereology field. The 4-day Scientific Programme packed full with new findings and scientific breakthroughs and provides a unique opportunity to hear the latest in Dermato-Venereology and connect with leading experts. To find out more visit https://www.eadvcongress2021.org/.
References
- Steele, L., Velazquez-Pimentel, D., Do AI Models recognise rare, aggressive skin cancers? An assessment of a direct-to-consumer app in the diagnosis of Merkel Cell Carcinoma and amelanotic carcinoma, Abstract no. 1935, submitted to EADV 30th Congress, 29-02 October 2021.
- Cancer Research UK. Merkel cell carcinoma (MCC). Available from https://www.cancerresearchuk.org/about-cancer/neuroendocrine-tumours-nets/merkel-cell-skin-cancer [Accessed September 2021].
- Cancer Net introduction. Available from https://www.cancer.net/cancer-types/skin-cancer-non-melanoma/introduction [Accessed September 2021].
- Cancer Research UK. Melanoma skin cancer. Available from https://www.cancerresearchuk.org/about-cancer/melanoma/about [Accessed September 2021].
- Cancer Research. Melanoma skin cancer. Types. Available from: https://www.cancerresearchuk.org/about-cancer/melanoma/stages-types/types [Accessed September 2021].