Fine-Grained Action Analysis for Automated Skill Assessment and Feedback in Instructional Videos

Gengrui Wei; Xu Wang; Zhong Chu

Authors

Gengrui Wei Computational Science and Engineering, Virginia Tech, VA, USA Author
Xu Wang Computer Science, Beijing University of Posts and Telecommunications, Beijing, China Author
Zhong Chu Information science, Trine University, CA, USA Author

Keywords:

fine-grained action recognition, skill assessment, multi-modal fusion, automated feedback generation

Abstract

Fine-grained action analysis in instructional videos presents significant challenges due to subtle motion variations and complex temporal dependencies. This paper introduces a comprehensive framework for automated skill assessment and feedback generation based on granularity-aware feature extraction and multi-modal fusion techniques. The proposed approach incorporates a temporal self-similarity module that captures periodic patterns critical for skill quality assessment, a part-level feature extraction network that analyzes body part movements, and a cross-attention transformer architecture that integrates skeleton and RGB modalities. Experiments conducted on our newly collected Skill Video dataset, comprising 8450 instructional videos across sports, crafts, medical procedures, and musical performances, demonstrate substantial improvements over state-of-the-art methods. The framework achieves 89.5% accuracy in skill level classification, a 20.1% reduction in dimensional assessment error, and a 5.8% improvement in temporal action quality estimation compared to existing approaches. User studies with 45 participants reveal that feedback generated by our system produces learning outcomes comparable to human expert guidance, with only a 3.6% gap in skill improvement and 2.6% difference in retention, as supported by rigorous experimental design and statistical analysis. The proposed technology enables personalized learning experiences through continuous assessment and feedback, with applications spanning formal education, professional training, and self-directed learning environments.

References

1. Y. Ou, X. Shi, J. Chen, R. He, and C. Liu, "From Body Parts to Holistic Action: A Fine-grained Teacher-student CLIP for Action Recognition," IEEE Signal Process. Lett., 2025, doi: 10.1109/LSP.2025.3548448.

2. S. Luo and J. Xiao, "Self-similarity attention module for skeleton-based fine-grained action recognition," in 2023 4th Int. Conf. Intelligent Comput. Hum.-Comput. Interact. (ICHCI), 2023, pp. 143–147, doi: 10.1109/ICHCI58871.2023.10278014.

3. H. Zhang, X. Wang, and Q. Zhao, "Granularity-Aware Contrastive Learning for Fine-Grained Action Recognition," in ICASSP 2025 - IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2025, pp. 1–5, doi: 10.1109/ICASSP49660.2025.10889703.

4. C. H. Lin, M. Y. Tsai, and P. Y. Chou, "A lightweight fine-grained action recognition network for basketball foul detection," in 2021 IEEE Int. Conf. Consumer Electron.-Taiwan (ICCE-TW), 2021, pp. 1–2, doi: 10.1109/ICCE-TW52618.2021.9602903.

5. A. Ghimire, V. Kakani, and H. Kim, "Ssrt: A sequential skeleton rgb transformer to recognize fine-grained human-object in-teractions and action recognition," IEEE Access, vol. 11, pp. 51930–51948, 2023, doi: 10.1109/ACCESS.2023.3278974.

6. K. Xu and B. Purkayastha, "Integrating Artificial Intelligence with KMV Models for Comprehensive Credit Risk Assessment," Acad. J. Sociol. Manage., vol. 2, no. 6, pp. 19–24, 2024, doi: 10.5281/zenodo.14077150.

7. K. Xu and B. Purkayastha, "Enhancing Stock Price Prediction through Attention-BiLSTM and Investor Sentiment Analysis," Acad. J. Sociol. Manage., vol. 2, no. 6, pp. 14–18, 2024, doi: 10.5281/zenodo.14065931.

8. N. Biswas, A. S. Mondal, A. Kusumastuti, et al., "Automated credit assessment framework using ETL process and machine learning," Innov. Syst. Softw. Eng., vol. 21, pp. 257–270, 2025, doi: 10.1007/s11334-022-00522-x.

9. M. Shu, Z. Wang, and J. Liang, "Early Warning Indicators for Financial Market Anomalies: A Multi-Signal Integration Ap-proach," J. Adv. Comput. Syst., vol. 4, no. 9, pp. 68–84, 2024, doi: 10.69987/JACS.2024.40907.

10. Y. Liu, W. Bi, and J. Fan, "Semantic Network Analysis of Financial Regulatory Documents: Extracting Early Risk Warning Signals," Acad. J. Sociol. Manage., vol. 3, no. 2, pp. 22–32, 2025, doi: 10.70393/616a736d.323731.

11. Y. Zhang, J. Fan, and B. Dong, "Deep Learning-Based Analysis of Social Media Sentiment Impact on Cryptocurrency Market Microstructure," Acad. J. Sociol. Manage., vol. 3, no. 2, pp. 13–21, 2025, doi: 10.70393/616a736d.323730.

12. Z. Zhou, Y. Xi, S. Xing, and Y. Chen, "Cultural Bias Mitigation in Vision-Language Models for Digital Heritage Documentation: A Comparative Analysis of Debiasing Techniques," Artif. Intell. Mach. Learn. Rev., vol. 5, no. 3, pp. 28–40, 2024, doi: 10.69987/AIMLR.2024.50303.

13. Y. Zhang, H. Zhang, and E. Feng, "Cost-Effective Data Lifecycle Management Strategies for Big Data in Hybrid Cloud Envi-ronments," Academia Nexus J., vol. 3, no. 2, 2024.

14. X. Xiao, Y. Zhang, H. Chen, W. Ren, J. Zhang, and J. Xu, "A Differential Privacy-Based Mechanism for Preventing Data Leakage in Large Language Model Training," Acad. J. Sociol. Manage., vol. 3, no. 2, pp. 33–42, 2025, doi: 10.70393/616a736d.323732.

15. X. Xiao, H. Chen, Y. Zhang, W. Ren, J. Xu, and J. Zhang, "Anomalous Payment Behavior Detection and Risk Prediction for SMEs Based on LSTM-Attention Mechanism," Acad. J. Sociol. Manage., vol. 3, no. 2, pp. 43–51, 2025, doi: 10.70393/616a736d.323733.