Attention-Based Multimodal Fusion Model For Breast Cancer Diagnostics

Abstract

Computer-assisted breast cancer diagnosis has emerged as a promising approach to enhance the accuracy and efficiency of breast cancer classification. However, the question of how to effectively utilise both magnetic resonance imaging (MRI) and electronic health record (EHR) data in the model to enhance prediction accuracy remains unanswered. In this paper, we present a new attention-based multimodal model for breast cancer label classification. In the proposed model, inspired by attention blocks in transformer architecture, we innovatively adapt EHR-guided attention at mid and late stage of modality fusion highlighting the complementary strength of different modality data. We compare performance on two breast cancer dataset with other fusion methods and multimodal models, the experimental result shows that our model achieved better accuracy on both datasets and has potential to assist real-world clinical decisions.