Innovative Application of Multimodal Medical Imaging in Complex Lesion Diagnosis Based on Deep Fusion Networks

Jisoo Park; Minjae Kim; Eunji Lee; Hyunwoo Choi; Seungmin Oh

Authors

Jisoo Park Department of Computer Science, Yonsei University, Seoul, South Korea Author
Minjae Kim Department of Computer Science, Yonsei University, Seoul, South Korea Author
Eunji Lee Department of Computer Science, Yonsei University, Seoul, South Korea Author
Hyunwoo Choi Department of Artificial Intelligence, Gwangju Institute of Science and Technology (GIST), Gwangju, South Korea Author
Seungmin Oh Department of Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea Author

Keywords:

multimodal fusion, deep learning, medical image analysis, diagnostic support system, tumor detection

Abstract

Multimodal medical imaging, which combines spatial and functional information, plays an important role in improving the accuracy of complex disease diagnosis. This study aims to address the diagnostic challenges of complex lesions by designing a deep fusion network that integrates channel attention and multi-scale feature extraction. An end-to-end model was built and tested on two public multimodal datasets: glioma and lung tumors. The experimental results show that, compared with existing multimodal fusion methods, the proposed approach achieves better performance in classification accuracy, area under the receiver operating characteristic curve (ROC-AUC), and Dice coefficient for image segmentation. This method provides a new solution for clinical decision support based on multi-source imaging.

References

1. S. Hussain, et al., "Modern diagnostic imaging technique applications and risk factors in the medical field: a review," BioMed Res. Int., vol. 2022, no. 1, p. 5164970, 2022, doi: 10.1155/2022/5164970.

2. H. Wang, G. Zhang, Y. Zhao, F. Lai, W. Cui, J. Xue, and Y. Lin, "Rpf-eld: Regional prior fusion using early and late distillation for breast cancer recognition in ultrasound images," in 2024 IEEE Int. Conf. Bioinformatics Biomedicine (BIBM), 2024, pp. 2605–2612, doi: 10.1109/BIBM62325.2024.10821972.

3. N. S. Radder, S. Sonar, A. V. I. N. A. S. H. Nanivadekar, and S. B. Radder, "A step forward in elucidating the mystery in neu-roimaging: PET-MRI fusion," in Proc. European Congress of Radiology (ECR), 2019, doi: 10.26044/ecr2019/C-0697.

4. X. Wu, Y. Sun, and X. Liu, "Multi-class classification of breast cancer gene expression using PCA and XGBoost," Preprints, 2024, doi: 10.20944/preprints202410.1775.v1.

5. M. Raghavan, "Conventional modalities and novel, emerging imaging techniques for musculoskeletal tumors," Cancer Control, vol. 24, no. 2, pp. 161–171, 2017, doi: 10.1177/107327481702400208.

6. X. Shi, Y. Tao, and S. C. Lin, "Deep neural network-based prediction of B-cell epitopes for SARS-CoV and SARS-CoV-2: En-hancing vaccine design through machine learning," in Proc. 2024 4th Int. Signal Process., Commun. Eng. Manag. Conf. (ISPCEM), Nov. 2024, pp. 259–263, doi: 10.1109/ISPCEM64498.2024.00050.

7. S. Kumar, S. Rani, S. Sharma, and H. Min, "Multimodality fusion aspects of medical diagnosis: A comprehensive review," Bioengineering, vol. 11, no. 12, p. 1233, 2024, doi: 10.3390/bioengineering11121233.

8. A. Kumar, "Deep learning for multi-modal medical imaging fusion: Enhancing diagnostic accuracy in complex disease detec-tion," Int. J. Eng. Technol. Res. Manag., vol. 6, no. 11, p. 183, 2022, doi: 10.5281/zenodo.15033792.

9. Y. Wang, Y. Wen, X. Wu, L. Wang, and H. Cai, "Assessing the role of adaptive digital platforms in personalized nutrition and chronic disease management," World J. Innov. Mod. Technol., vol. 8, no. 1, Jan. 2025, doi: 10.53469/wjimt.2025.08(01).05.

10. Y. Wen, X. Wu, L. Wang, H. Cai, and Y. Wang, "Application of nanocarrier-based targeted drug delivery in the treatment of liver fibrosis and vascular diseases," J. Med. Life Sci., vol. 1, no. 2, pp. 63–69, 2025, doi: 10.71222/bc9c9s73.

11. S. Amal, L. Safarnejad, J. A. Omiye, I. Ghanzouri, J. H. Cabot, and E. G. Ross, "Use of multi-modal data and machine learning to improve cardiovascular disease care," Front. Cardiovasc. Med., vol. 9, p. 840262, 2022, doi: 10.3389/fcvm.2022.840262.

12. T. Zhang, B. Zhang, F. Zhao, and S. Zhang, "COVID-19 localization and recognition on chest radiographs based on Yolov5 and EfficientNet," in 2022 7th Int. Conf. Intell. Comput. Signal Process. (ICSP), 2022, pp. 1827–1830, doi: 10.1109/ICSP54964.2022.9778327.

13. Y. Wang, Y. Wen, X. Wu, L. Wang, and H. Cai, "Modulation of gut microbiota and glucose homeostasis through high-fiber dietary intervention in type 2 diabetes management," World J. Innov. Mod. Technol., vol. 7, no. 5, Oct. 2024, doi: 10.53469/wjimt.2024.07(06).04.

14. H. Ziang, J. Zhang, and L. Li, "Framework for lung CT image segmentation based on UNet++," 2025, arXiv preprint arXiv:2501.02428, doi: 10.48550/arXiv.2501.02428.

15. I. Domingues, G. Pereira, P. Martins, H. Duarte, J. Santos, and P. H. Abreu, "Using deep learning techniques in medical imaging: a systematic review of applications on CT and PET," Artif. Intell. Rev., vol. 53, pp. 4093–4160, 2020, doi: 10.1007/s10462-019-09789-w.

16. Y. Deng, W. Cui, and X. Liu, "Head Tumor Segmentation and Detection Based on ResUNet," in Proc. 5th Int. Conf. Signal Pro-cess. Mach. Learn., 2024, doi: 10.54254/2755-2721/99/20251810.

17. Y. Wang, Y. Wen, X. Wu, and H. Cai, "Application of ultrasonic treatment to enhance antioxidant activity in leafy vegetables," Int. J. Adv. Appl. Sci. Res., vol. 3, pp. 49–58, 2024, doi: 10.5281/zenodo.14275691.

18. S. M. Hosseini and M. S. Baghshah, "Dilated balanced cross entropy loss for medical image segmentation," 2024, arXiv Preprint, arXiv:2412.06045, doi: 10.48550/arXiv.2412.06045.

19. J. B. Qiao, et al., "Vitamin A-decorated biocompatible micelles for chemogene therapy of liver fibrosis," J. Control. Release, vol. 283, pp. 113–125, 2018, doi: 10.1016/j.jconrel.2018.05.032.

20. G. Qu, et al., "Self-assembled micelles based on N-octyl-N’-phthalyl-O-phosphoryl chitosan derivative as an effective oral carrier of paclitaxel," Carbohydr. Polym., vol. 207, pp. 428–439, 2019, doi: 10.1016/j.carbpol.2018.11.099.

21. Y. Wang, L. Wang, Y. Wen, X. Wu, and H. Cai, "Precision-engineered nanocarriers for targeted treatment of liver fibrosis and vascular disorders," World J. Innov. Mod. Technol., vol. 8, no. 1, Jan. 2025, doi: 10.53469/wjimt.2025.08(01).07.

22. J. Zhu, et al., "Multimodal nanoimmunotherapy engages neutrophils to eliminate Staphylococcus aureus infections," Nat. Nano-technol., vol. 19, no. 7, pp. 1032–1043, 2024, doi: 10.1038/s41565-024-01648-8.

23. Y. Wang, Y. Wen, X. Wu, and H. Cai, "Comprehensive evaluation of GLP1 receptor agonists in modulating inflammatory pathways and gut microbiota," World J. Innov. Mod. Technol., vol. 7, no. 5, 2024, doi: 10.53469/wjimt.2024.07(06).23.

24. I. K. Lee, et al., "Micromolded honeycomb scaffold design to support the generation of a bilayered RPE and photoreceptor cell construct," Bioact. Mater., vol. 30, pp. 142–153, 2023, doi: 10.1016/j.bioactmat.2023.07.019.

25. N. Yodsanit, et al., "Targeted PERK inhibition with biomimetic nanoclusters confers preventative and interventional benefits to elastase-induced abdominal aortic aneurysms," Bioact. Mater., vol. 26, pp. 52–63, 2023, doi: 10.1016/j.bioactmat.2023.02.009.

26. I. Domingues, G. Pereira, P. Martins, H. Duarte, J. Santos, and P. H. Abreu, "Using deep learning techniques in medical imaging: a systematic review of applications on CT and PET," Artif. Intell. Rev., vol. 53, pp. 4093–4160, 2020, doi: 10.1007/s10462-019-09788-3.

27. Y. Wang, M. Shen, L. Wang, Y. Wen, and H. Cai, "Comparative modulation of immune responses and inflammation by n-6 and n-3 polyunsaturated fatty acids in oxylipin-mediated pathways," World J. Innov. Mod. Technol., vol. 7, no. 4, Aug. 2024, doi: 10.53469/wjimt.2024.07(05).17.

28. A. M. Giantini-Larsen, S. Pannullo, and R. G. Juthani, "Challenges in the diagnosis and management of low-grade gliomas," World Neurosurg., vol. 166, pp. 313–320, 2022, doi: 10.1016/j.wneu.2022.07.079.

29. S. C. Kotoulas, et al., "A thorough review of the clinical applications of artificial intelligence in lung cancer," Cancers, vol. 17, no. 5, p. 882, 2025, doi: 10.3390/cancers17050882.