Language-Driven Interactive Annotation for Pulmonary Nodules in Chest CT: An LLM Prompt-Translation and Multi-Round Refinement Approach
Keywords:
pulmonary nodule annotation, language-mediated segmentation, prompt translation, interactive refinementAbstract
High-quality pixel-level annotation remains a principal bottleneck for medical artificial intelligence, particularly for pulmonary nodule analysis on chest computed tomography, where expert labeling is costly and heterogeneous across institutions. This paper investigates a narrow but practical question: how short free-text descriptions produced by clinicians can be mediated into the spatial prompts expected by foundation segmentation models such as the Segment Anything Model via structured slot extraction, and how a lightweight multi-round refinement loop can stabilize the resulting masks under realistic annotation budgets. We emphasize that the role of the large language model in this study is restricted to structured slot extraction from short English phrases and to classifying each correction utterance into one of four canonical categories; the language model does not predict pixel coordinates, and the spatial initialization itself is driven by a coarse lobe-level anatomical prior, a size heuristic, and a vessel-suppressed point-sampling rule, rather than by free-form visual reasoning. This study is therefore best described as language-mediated structured prompting rather than free-form reasoning segmentation. We do not propose a new backbone or a full clinical system; rather, we study a prompt-translation strategy coupled with bounded interactive correction, evaluated on three public datasets: LIDC-IDRI, LUNA16, and Medical Segmentation Decathlon Task06 Lung. We report Dice, intersection over union, ninety-fifth percentile Hausdorff distance, and per-case annotation time, together with paired Wilcoxon signed-rank tests and bootstrap confidence intervals, so that the magnitude and reliability of any improvement can be evaluated directly. Results suggest a modest improvement in Dice and a reduction in measured per-case annotation time compared with purely geometric prompting; the annotation-time comparison should be read as an engineering-level approximation rather than as a formal reader study, while the interface remains accessible to clinicians without engineering expertise.References
1. S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman, E. A. Kazerooni, H. MacMahon, E. J. R. van Beek, D. Yankelevitz, A. M. Biancardi, P. H. Bland, M. S. Brown, R. M. Engelmann, G. E. Laderach, ... L. P. Clarke, "The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A completed reference database of lung nodules on CT scans," Medical Physics, vol. 38, no. 2, pp. 915–931, 2011.
2. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick, "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4015–4026.
3. J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, "Segment anything in medical images," Nature Communications, vol. 15, p. 654, 2024.
4. N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Radle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollar, and C. Feichtenhofer, "SAM 2: Segment anything in images and videos," arXiv preprint arXiv:2408.00714, 2024.
5. T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, S. Kiblawi, T. Naumann, J. Gao, A. Crabtree, J. Abel, C. Moung-Wen, B. Piening, C. Bifulco, M. Wei, H. Poon, and S. Wang, "SAM-Med2D," arXiv preprint arXiv:2308.16184, 2023.
6. J. Cheng, J. Ye, Z. Deng, J. Chen, T. Li, H. Wang, Y. Su, Z. Huang, J. Chen, L. Jiang, H. Sun, J. He, S. Zhang, M. Zhu, and Y. Qiao, "SAM-Med3D: Towards general-purpose segmentation models for volumetric medical images," in ECCV Workshops, 2024.
7. V. I. Butoi, J. J. G. Ortiz, T. Ma, M. R. Sabuncu, J. Guttag, and A. V. Dalca, "UniverSeg: Universal medical image segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 21438–21451.
8. T. Shaharabany, A. Dahan, R. Giryes, and L. Wolf, "AutoSAM: Adapting SAM to medical images by overloading the prompt encoder," in Proceedings of the British Machine Vision Conference (BMVC), 2023.
9. Y. Zhang, T. Zhou, S. Wang, P. Liang, Y. Zhang, and D. Z. Chen, "Input augmentation with SAM: Boosting medical image segmentation with segmentation foundation model," in MICCAI Workshops, Springer, 2023, pp. 129–139.
10. X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia, "LISA: Reasoning segmentation via large language model," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024, pp. 9579–9589.
11. J. Wang and L. Ke, "LLM-Seg: Bridging image segmentation and large language model reasoning," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, 2024, pp. 1765–1774.
12. J. Liu, Y. Zhang, J.-N. Chen, J. Xiao, Y. Lu, B. A. Landman, Y. Yuan, A. Yuille, Y. Tang, and Z. Zhou, "CLIP-driven universal model for organ segmentation and tumor detection," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 21152–21164.
13. M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers, B. van Ginneken, M. Bilello, P. Bilic, P. F. Christ, R. K. G. Do, M. J. Gollub, S. H. Heckers, H. Huisman, W. R. Jarnagin, ... M. J. Cardoso, "The medical segmentation decathlon," Nature Communications, vol. 13, p. 4128, 2022.
14. K. Yan, X. Wang, L. Lu, and R. M. Summers, "DeepLesion: Automated mining of large-scale lesion annotations and universal lesion detection with deep learning," Journal of Medical Imaging, vol. 5, no. 3, p. 036501, 2018.
15. A. A. A. Setio, A. Traverso, T. de Bel, M. S. N. Berens, C. van den Bogaard, P. Cerello, H. Chen, Q. Dou, M. E. Fantacci, B. Geurts, R. v. d. Gugten, P. A. Heng, B. Jansen, M. M. J. de Kaste, V. Kotov, J. Y.-H. Lin, J. T. M. C. Manders, A. Sonora-Mengana, J. C. Garcia-Naranjo, ... C. Jacobs, "Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge," Medical Image Analysis, vol. 42, pp. 1–13, 2017.
