ENHANCING DERMATOLOGICAL DIAGNOSIS THROUGH MULTIMODAL DEEP LEARNING AND VISUAL–TEXTUAL DATA FUSION 

Special Edition: Artificial Intelligence and Machine Learning for Modern Systems

This Article is archived at

CERN (Switzerland)

Original Research Article

Authors: I. Manimozhi¹*, D. Lakshmi², Er. Rishabh Aryan³

¹Research Scholar, Dept. of CSE, AMET Deemed to be University, Kanathur-603112, India

² Professor, Dept of EEE, AMET Deemed to be University, Kanathur- 6003112, India

³M.Tech Artificial Intelligence and Data Science, Department of CSE, Indian Institute of Information Technology, Bhagalpur (Bihar) 

^*Corresponding author: manimozhirajkumar02@gmail.com 

Abstract

Accurate dermatological diagnosis requires more than visual inspection of skin lesions—it also depends on patient history, symptoms, and demographic details. While current deep learning systems achieve high performance in image classification, they often ignore this complementary clinical information, limiting their usefulness in real-world practice. In this work, we present a multimodal deep learning framework that unifies dermoscopic images and textual clinical records into a single diagnostic model. Visual features are extracted through convolutional networks, while textual features are encoded using transformer-based language models. A hybrid fusion strategy is then employed to align and integrate these heterogeneous data streams, enabling richer and more context-aware predictions. Experiments conducted on publicly available dermatology datasets demonstrate that the proposed model consistently surpasses image-only and text-only baselines, particularly in cases where visual ambiguity exists. Beyond improving accuracy, the framework also enhances interpretability by highlighting the contribution of both image patterns and textual cues to the final decision. These results underline the importance of multimodal learning in building clinically robust AI systems, with strong potential to assist dermatologists in early and precise disease detection.

Keywords: Multimodal learning, dermatology AI, image-text integration, skin disease diagnosis, clinical decision support, convolutional networks, transformers, healthcare intelligence.

DOI: https://doi.org/10.5281/zenodo.20474713

Published: 31/05/2026

Citation:

Manimozhi, et al. ENHANCING DERMATOLOGICAL DIAGNOSIS THROUGH MULTIMODAL DEEP LEARNING AND VISUAL–TEXTUAL DATA FUSION. Zenodo, May 2026, https://doi.org/10.5281/zenodo.20474713.

Author(s) Retains the Copyrights of This Article