Multimodal medical image fusion and classification using deep learning techniques

Article type: Research Article

Authors: Veeraiah, D.^{a; *} | Sai Kumar, S.^b | Ganiya, Rajendra Kumar^c | Rao, Katta Subba^d | Nageswara Rao, J.^e | Manjith, Ramaswamy^f | Rajaram, A.^g

Affiliations: [a] Department of Computer Science and Engineering, Lakireddy Bali Reddy College of Engineering (Autonomous), Mylavaram, Krishna NTR Dt, Andhra Pradesh, India | [b] Department of IT, PVP Siddhartha Institute of Technology, Andhra Pradesh, India | [c] Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur, A.P, India | [d] Department of Computer Science & Engineering, B.V. Raju Institute of Technology, Narsapur, Medak (District), Telangana (State), India | [e] Department of Computer Science and Engineering, Lakireddy Bali Reddy College of Engineering (Autonomous), Mylavaram, NTR Dt, Andhra Pradesh, India | [f] Department of Electronics and Communication Engineering Dr. Sivanthi Aditanar College of Engineering, Tiruchendur, Tamil Nadu, India | [g] Department of Electronics and Communication Engineering, E.G.S Pillay Engineering College, Nagapattinam, Tamil Nadu, India

Correspondence: [*] Corresponding author. Dr. D. Veeraiah, Professor and HOD, Department of Computer Science and Engineering, Lakireddy Bali Reddy College of Engineering (Autonomous), Mylavaram, Krishna NTR Dt, Andhra Pradesh, 521230, India. E-mail: Veeraiahdvc@gmail.com.

Abstract: Medical image fusion plays a crucial role in accurate medical diagnostics by combining images from various modalities. To address this need, we propose an AI model for efficient medical image fusion using multiple modalities. Our approach utilizes a Siamese convolutional neural network to construct a weight map based on pixel movement information extracted from multimodality medical images. We leverage medical picture pyramids to incorporate multiscale techniques, enhancing reliability beyond human visual intuition. Additionally, we dynamically adjust the fusion mode based on local comparisons of deconstructed coefficients. Evaluation metrics including F1-score, recall, accuracy, and precision are computed to assess performance, yielding impressive results: an F1-score of 0.8551 and a mutual information (MI) value of 2.8059. Experimental results demonstrate the superiority of our method, achieving a remarkable 99.61% accuracy in targeted experiments. Moreover, the Structural Similarity Index (SSIM) of our approach is 0.8551. Compared to state-of-the-art approaches, our model excels in medical picture classification, providing accurate diagnosis through high-quality fused images. This research advances medical image fusion techniques, offering a robust solution for precise medical diagnostics across various modalities.

Keywords: Multimodal medical image fusion, image classification, siamese CNN, LSTM, genetic algorithm

DOI: 10.3233/JIFS-240018

Journal: Journal of Intelligent & Fuzzy Systems, vol. Pre-press, no. Pre-press, pp. 1-15, 2024