A Literature Review Study on Knowledge Distillation for Large Models of Image Segmentation

Dongkai Qi; Lim Chia Sien

Authors

Dongkai Qi Asia Pacific University of Technology and Innovation, Kuala Lumpur, Malaysia Author
Lim Chia Sien Asia Pacific University of Technology and Innovation, Kuala Lumpur, Malaysia Author

Keywords:

image segmentation, knowledge distillation, large models, model compression, AR interaction

Abstract

With the rapid development of deep learning technology, image segmentation macromodels have achieved remarkable results in many fields. However, these large models often face problems such as high consumption of computational resources and difficulties in deployment. Knowledge distillation, as an effective model compression and optimisation technique, has gradually received attention from researchers. This paper provides a systematic review of the literature related to knowledge distillation for large models of image segmentation, and analyses the current status, advantages and shortcomings of the application of knowledge distillation in large models of image segmentation in terms of improving the AR interactive experience, solving the contradiction between real-time and accuracy, promoting the lightweight and efficient deployment of the model, enhancing the generalization capability of the model, and facilitating the fusion of multimodal data, etc. It also looks forward to the future research direction, aiming to provide a better solution for the research in the related fields. outlook, aiming to provide reference and inspiration for researchers in related fields.

References

1. Y. Feng, X. Sun, W. Diao, J. Li and X. Gao, "Double Similarity Distillation for Semantic Image Segmentation," IEEE Transactions on Image Processing, vol. 30, pp. 5363-5376, 2021, doi: 10.1109/TIP.2021.3083113.

2. S. J. Cheng, Q. X. Zhao, X. Y. Zhang, N. Yadikar, and K. Ubul, "A review of knowledge distillation in object detection," IEEE Access, 2023, doi: 10.1109/ACCESS.2023.3288692.

3. D. Qin et al., "Efficient medical image segmentation based on knowledge distillation," IEEE Trans. Med. Imaging, vol. 40, no. 12, pp. 3820-3831, Dec. 2021, doi: 10.1109/TMI.2021.3098703.

4. Q. Dou, Q. Liu, P. A. Heng, and B. Glocker, "Unpaired multi-modal segmentation via knowledge distillation," IEEE Trans. Med. Imaging, vol. 39, no. 7, pp. 2415-2425, July 2020, doi: 10.1109/TMI.2019.2963882.

5. S. Deng et al., "LHAR: Lightweight human activity recognition on knowledge distillation," IEEE J. Biomed. Health Informatics, vol. 28, no. 11, pp. 6318-6328, Nov. 2024, doi: 10.1109/JBHI.2023.3298932.

6. Z. Li et al., "When object detection meets knowledge distillation: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 8, pp. 10555-10579, Aug. 2023, doi: 10.1109/TPAMI.2023.3257546.

7. W. Zhou, H. Zhang, and W. Qiu, "Differential modal multistage adaptive fusion networks via knowledge distillation for RGB-D mirror segmentation," IEEE Trans. Big Data, doi: 10.1109/TBDATA.2024.3505057.

8. Z. Tu, X. Liu, and X. Xiao, "A general dynamic knowledge distillation method for visual analytics," IEEE Trans. Image Process., vol. 31, pp. 6517-6531, 2022, doi: 10.1109/TIP.2022.3212905.

9. P. Taveekitworachai, P. Suntichaikul, C. Nukoolkit, and R. Thawonmas, "Speed up! Cost-effective large language model for ADAS via knowledge distillation," in 2024 IEEE Intell. Vehicles Symp. (IV), Jeju Island, Korea, Republic of, 2024, pp. 1933-1938, doi: 10.1109/IV55156.2024.10588799.

10. C. Chen, Q. Dou, Y. Jin, Q. Liu, and P. A. Heng, "Learning with privileged multimodal knowledge for unimodal segmentation," IEEE Trans. Med. Imaging, vol. 41, no. 3, pp. 621-632, Mar. 2022, doi: 10.1109/TMI.2021.3119385.

11. P. Zhao, Y. Hou, Z. Yan, and S. Huo, "Text-driven medical image segmentation with text-free inference via knowledge distil-lation," IEEE Trans. Instrum. Meas., vol. 74, pp. 1-15, 2025, Art no. 5011715, doi: 10.1109/TIM.2025.3545506.

12. F. Xiong, C. Shen, and X. Wang, "Generalized knowledge distillation for unimodal glioma segmentation from multimodal models," Electronics, vol. 12, no. 7, p. 1516, 2023, doi: 10.3390/electronics12071516.

13. T. Wan, N. Canagarajah, and A. Achim, "Segmentation-driven image fusion based on alpha-stable modeling of wavelet coef-ficients," IEEE Trans. Multimedia, vol. 11, no. 4, pp. 624-633, Jun. 2009, doi: 10.1109/TMM.2009.2017640.

14. D. Lahat, T. Adali, and C. Jutten, "Multimodal data fusion: An overview of methods, challenges, and prospects," Proc. IEEE, vol. 103, no. 9, pp. 1449-1477, Sept. 2015, doi: 10.1109/JPROC.2015.2460697.

15. W. Chen, G. Yang, and D. Qi, "Comprehensive evaluation of college students' physical health and sports mode recommenda-tion model based on decision tree classification model," Comput. Intell. Neurosci., vol. 2022, pp. 5504850, Jul. 2022, doi: 10.1155/2022/5504850.

16. W. Chen, S. A. S. K. B. Syed Ali, H. Zulnaidi, and D. Qi, "Research on intelligent analysis of healthy training progress of teenage sports athletes using various modalities," Sustainability, vol. 14, no. 24, Art no. 16556, 2022, doi: 10.3390/su142416556.

17. H. Huang and D. Qi, "Application of improved CNN technology in medical imaging course," Higher Educ. Orient. Stud., vol. 2, no. 6, pp. 40-49, doi: 10.54435/heos.v2i6.85.