Affiliations: Department of Computer Science and Engineering, Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, India
Correspondence:
[*]
Corresponding authors: Sheerin Sitara Noor Mohamed and Kavitha Srinivasan, Department of Computer Science and Engineering, Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam –603110, India. E-mails: sheerinsitaran@ssn.edu.in (SSNM); kavithas@ssn.edu.in (KS).
Abstract: Recent technological developments and improvement in the medical domain demands advancement, to address the issue of early disease detection. Also, the current pandemic has resulted in considerable progress of improvement in the medical domain, through online consultation by physicians for different diseases using clinical reports and medical images. A similar process is adopted in developing a Visual Question Answering (VQA) system in the medical field. In this paper, existing medical VQA datasets, appropriate techniques, suitable quantitative metrics, real time challenges and, the implementation of one VQA approach with algorithms and performance evaluation are discussed. The medical VQA datasets collected from multiple sources are represented in different perspectives (organwise, planewise, modality-type and abnormality-type) for a better understanding and visualization. Then the techniques used in VQA are subsequently grouped and explained, based on evolution, complexity in the dataset and the need for semantics in understanding the questions. In addition, the implementation of a VQA approach using VGGNet and LSTM is carried out for existing and improved datasets, and analyzed with accuracy and BLEU score metrics. The improved datasets, created through dataset reduction and augmentation approaches, resulted in better performance than the existing datasets. Finally, the challenges of the medical VQA domain are examined in terms of datasets, combining techniques, and modifying the parameters of existing performance metrics for future research.
Keywords: Visual question answering, medical VQA, ImageCLEF, VQA-MED dataset, VQA-RAD dataset, VGGNet, LSTM, challenges of VQA
