Context-aware Action Quality Assessment using Latent Regression-based Progressive Sub-action Learning

Document Type : Original Article

Authors

Marjan Mazruei

Ehsan Fazl-Ersi

Abedin Vahedian

Ahad Harati

Department of Computer Engineering, Ferdowsi University of Mashhad, Mashhad, Iran

10.22067/cke.2026.96007.1177

Abstract

Most existing Action Quality Assessment (AQA) systems infer performance scores from holistic, video-level representations, a process that frequently obscures the contribution of individual motion segments to the final prediction. This global treatment reduces transparency and limits interpretability, particularly for complex skills composed of multiple sequential phases. The problem is further compounded by limited fine-grained supervision and the difficulty of modeling long-range temporal dependencies across an action sequence. This paper introduces a Latent Regression-based Progressive Sub-action Learning framework to address these limitations by capturing rich contextual information within the action sequence. The framework first employs a procedure-parsing module to segment each video into semantically coherent sub-actions and extract corresponding spatio-temporal features. A latent variable approach subsequently generates initial pseudo-subscores by integrating these features with overall score labels from the training set. Critically, these pseudo-subscores are progressively refined through an iterative process that leverages the cumulative contextual information from preceding sub-actions. This yields an enriched feature set that accurately encapsulates the execution history. Furthermore, a novel monotonic penalty loss is introduced to enforce a logical and consistent progression in the sub-scores, mitigating abrupt and illogical score fluctuations. The model is trained in a two-stage process. First, it generates the robust pseudo-subscores, and subsequently it predicts both the overall score and the per-substage scores. This explicit modeling of long-range dependencies, along with consistent score progression, is critical for ensuring prediction accuracy. Extensive experimental evaluation confirms that this structured modeling approach delivers consistent performance gains over contemporary AQA methods while offering substantially improved interpretability.

Keywords

Action quality assessment

Long-range dependency

Progressive learning

Pseudo-subscore learning

Regression-based AQA

Spatio-temporal features

Temporal semantic segmentation

Subjects

• Image Processing

---

References

[1]  P. Parmar and B. Tran Morris, "Learning to score olympic events," in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 20–28. [Online]. Available: https://doi.org/10.1109/CVPRW.2017.16

[2]  J. Xu, Y. Rao, X. Yu, G. Chen, J. Zhou, and J. Lu, "Finediving: A fine-grained dataset for procedure-aware action quality assessment," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 2949–2958. [Online]. Available: https://doi.org/10.1109/CVPR52688.2022.00296

[3]  K. Zhou, Y. Ma, H. P. Shum, & X. Liang. (2023). Hierarchical graph convolutional networks for action quality assessment. IEEE Transactions on Circuits and Systems for Video Technology. [Online]. 33(12), pp. 7749–7763. Available: https://doi.org/10.1109/TCSVT.2023.3281413

[4]  H. Pirsiavash, C. Vondrick, and A. Torralba, "Assessing the quality of actions," in European conference on computer vision, 2014, Springer, pp. 556–571. [Online]. Available: https://doi.org/10.1007/978-3-319-10599-4_36

[5]  H. Jain and G. Harit, "An unsupervised sequence-to-sequence autoencoder based human action scoring model," in 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2019: IEEE, pp. 1–5. [Online]. Available: https://doi.org/10.1109/GlobalSIP45357.2019.8969424

[6]  Q. Lei, H.-B. Zhang, J.-X. Du, T.-C. Hsiao, & C.-C. Chen. (2020). Learning effective skeletal representations on RGB video for fine-grained human action quality assessment. Electronics. [Online]. 9(4), p. 568. Available: https://doi.org/10.3390/electronics9040568

[7]  H. Li, Q. Lei, H. Zhang, J. Du, & S. Gao. (2022). Skeleton-based deep pose feature learning for action quality assessment on figure skating videos. Journal of Visual Communication and Image Representation. [Online]. 89, p. 103625. Available: https://doi.org/10.1016/j.jvcir.2022.103625

[8]  D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3d convolutional networks," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 4489–4497. [Online]. Available: https://doi.org/10.1109/ICCV.2015.510

[9]  S. Hochreiter & J. Schmidhuber. (1997). Long short-term memory. Neural Computation. [Online]. 9(8), pp. 1735–1780. Available: https://doi.org/10.1162/neco.1997.9.8.1735

[10] Y. Li, X. Chai, and X. Chen, "End-to-end learning for action quality assessment," in Pacific Rim Conference on Multimedia, 2018: Springer, pp. 125–134. [Online]. Available: https://doi.org/10.1007/978-3-030-00767-6_12

[11] P. Parmar and B. Morris, "Action quality assessment across multiple actions," in 2019 IEEE winter conference on applications of computer vision (WACV), 2019: IEEE, pp. 1468–1476. [Online]. Available: https://doi.org/ 10.1109/WACV.2019.00161

[12] P. Parmar and B. T. Morris, "What and how well you performed? a multitask learning approach to action quality assessment," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 304–313. [Online]. Available: https://doi.org/ 10.1109/CVPR.2019.00039

[13] X. Xiang, Y. Tian, A. Reiter, G. D. Hager, and T. D. Tran, "S3d: Stacking segmental p3d for action quality assessment," in 2018 25th IEEE International conference on image processing (ICIP), 2018: IEEE, pp. 928–932. [Online]. Available: https://doi.org/10.1109/ICIP.2018.8451364

[14] Y. Li, X. Chai, and X. Chen, "Scoringnet: Learning key fragment for action quality assessment with ranking loss in skilled sports," in Asian Conference on Computer Vision, 2018: Springer, pp. 149–164. [Online]. Available:https://doi.org/10.1007/978-3-030-20876-9_10

[15] L.-J. Dong, H.-B. Zhang, Q. Shi, Q. Lei, J.-X. Du, & S. Gao. (2021). Learning and fusing multiple hidden substages for action quality assessment. Knowledge-Based Systems. [Online]. 229, p. 107388. Available: https://doi.org/10.1016/j.knosys.2021.107388

[16] H.-B. Zhang, L.-J. Dong, Q. Lei, L.-J. Yang, & J.-X. Du. (2023). Label-reconstruction-based pseudo-subscore learning for action quality assessment in sporting events. Applied Intelligence. [Online]. 53(9), pp. 10053–10067. Available: https://doi.org/10.1007/s10489-022-03984-5

[17] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, "Temporal convolutional networks for action segmentation and detection," in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 156–165. [Online]. Available: https://doi.org/ 10.1109/CVPR.2017.113

[18] Y. Tang et al., "Uncertainty-aware score distribution learning for action quality assessment," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9839–9848. [Online]. Available: https://doi.org/ 10.1109/CVPR42600.2020.00986

[19] B. Zhang, J. Chen, Y. Xu, H. Zhang, X. Yang, & X. Geng. (2024). Auto-encoding score distribution regression for action quality assessment. Neural Computing and Applications. [Online]. 36(2), pp. 929–942. Available: https://doi.org/10.1007/s00521-023-09068-w

[20] K. Gedamu, Y. Ji, Y. Yang, J. Shao, & H. T. Shen. (2024). Self-supervised subaction parsing network for semi-supervised action quality assessment. IEEE Transactions on Image Processing. [Online]. 33, pp. 6057–6070. Available: https://doi.org/10.1109/TIP.2024.3468870

[21] A. Vaswani et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems. [Online]. 30. Available: https://cir.nii.ac.jp/crid/1370849946232757637

[22] Q. Lei, H. Zhang, & J. Du. (2021). Temporal attention learning for action quality assessment in sports video. Signal, Image and Video Processing. [Online]. 15(7), pp. 1575–1583. Available: https://doi.org/10.1007/s11760-021-01890-w

[23] S. Farabi, H. Himel, F. Gazzali, M. B. Hasan, M. H. Kabir, and M. Farazi, "Improving action quality assessment using weighted aggregation," in Iberian Conference on Pattern Recognition and Image Analysis, 2022: Springer, pp. 576–587. [Online]. Available: https://doi.org/10.1007/978-3-031-04881-4_46

[24] L.-A. Zeng et al., "Hybrid dynamic-static context-aware attention network for action assessment in long videos," in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 2526–2534. [Online]. Available: https://doi.org/10.1145/3394171.3413560

[25] Y. Zhang, W. Xiong, & S. Mi. (2022). Learning time-aware features for action quality assessment. Pattern Recognition Letters. [Online]. 158, pp. 104–110. Available: https://doi.org/10.1016/j.patrec.2022.04.015

[26] A. Xu, L.-A. Zeng, and W.-S. Zheng, "Likert scoring with grade decoupling for long-term action assessment," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3232–3241. [Online]. Available: https://doi.org/10.1109/CVPR52688.2022.00323

[27] S. Zahan, G. M. Hassan, & A. Mian. (2024). Learning sparse temporal video mapping for action quality assessment in floor gymnastics. IEEE Transactions on Instrumentation and Measurement. [Online]. 73, pp. 1–11. Available: https://doi.org/10.1109/TIM.2024.3398072

[28] S. Wang, D. Yang, P. Zhai, C. Chen, and L. Zhang, "Tsa-net: Tube self-attention network for action quality assessment," in Proceedings of the 29th ACM international conference on multimedia, 2021, pp. 4902–4910. [Online]. Available: https://doi.org/10.1145/3474085.3475438

[29] K. Huang, Y. Tian, C. Yu, & Y. Huang. (2025). Dual-referenced assistive network for action quality assessment. Neurocomputing. [Online]. 614, p. 128786. Available: https://doi.org/10.1016/j.neucom.2024.128786

[30] M.-Z. Li, H.-B. Zhang, L.-J. Dong, Q. Lei, & J.-X. Du. (2023). Gaussian guided frame sequence encoder network for action quality assessment. Complex & Intelligent Systems. [Online]. 9(2), pp. 1963–1974. Available: https://doi.org/10.1007/s40747-022-00892-6

[31]C. Han, F. Shen, L. Chen, X. Lian, H. Gou, & H. Gao. (2023). MLA-LSTM: A local and global location attention LSTM learning model for scoring figure skating. Systems. [Online]. 11(1), p. 21. Available: https://doi.org/10.3390/systems11010021

[32]         S. Zhang et al., "Logo: A long-form video dataset for group action quality assessment," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 2405–2414. [Online]. Available: https://doi.org/10.1109/CVPR52729.2023.00238

[33] K. Zhou et al., "Magr: Manifold-aligned graph regularization for continual action quality assessment," in European Conference on Computer Vision, 2024: Springer, pp. 375–392. [Online]. Available: https://doi.org/10.1007/978-3-031-73247-8_22

[34] H. Doughty, D. Damen, and W. Mayol-Cuevas, "Who's better? who's best? pairwise deep ranking for skill determination," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6057–6066. [Online]. Available: https://doi.org/10.1109/CVPR.2018.00634

[35] H. Doughty, W. Mayol-Cuevas, and D. Damen, "The pros and cons: Rank-aware temporal attention for skill determination in long videos," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 7862–7871. [Online]. Available: https://doi.org/10.1109/CVPR.2019.00805

[36] H. Jain, G. Harit, & A. Sharma. (2020). Action quality assessment using Siamese network-based deep metric learning. IEEE Transactions on Circuits and Systems for Video Technology. [Online]. 31(6), pp. 2260–2273. Available: https://doi.org/10.1109/TCSVT.2020.3017727

[37] X. Yu, Y. Rao, W. Zhao, J. Lu, and J. Zhou, "Group-aware contrastive regression for action quality assessment," in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 7919–7928. [Online]. Available: https://doi.org/10.1109/ICCV48922.2021.00782

[38]  Q. An, M. Qi, and H. Ma, "Multi-stage contrastive regression for action quality assessment," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024: IEEE, pp. 4110–4114. [Online]. Available: https://doi.org/10.1109/ICASSP48485.2024.10447069

[39]  M. Li, H.-B. Zhang, Q. Lei, Z. Fan, J. Liu, and J.-X. Du, "Pairwise contrastive learning network for action quality assessment," in European Conference on Computer Vision, 2022: Springer, pp. 457–473. [Online]. Available: https://doi.org/10.1007/978-3-031-19772-7_27

[40] X. Ke, H. Xu, X. Lin, & W. Guo. (2024). Two-path target-aware contrastive regression for action quality assessment. Information Sciences. [Online]. 664, p. 120347. Available: https://doi.org/10.1016/j.ins.2024.120347

[41] L. Liu, P. Zhai, D. Zheng, and Y. Fang, "Multi-stage action quality assessment method," in Proceedings of the 2023 4th International Conference on Control, Robotics and Intelligent System, 2023, pp. 116–122. [Online]. Available: https://doi.org/10.1145/3622896.3622916

[42] J. Xu, Y. Rao, J. Zhou, & J. Lu. (2024). Procedure-aware action quality assessment: Datasets and performance evaluation. International Journal of Computer Vision. [Online]. 132(12), pp. 6069–6090. Available: https://doi.org/10.1007/s11263-024-02146-z

[43] K. Zhou, J. Li, R. Cai, L. Wang, X. Zhang, & X. Liang. (2024). Cofinal: Enhancing action quality assessment with coarse-to-fine instruction alignment. arXiv preprint arXiv:2404.13999. [Online]. Available: https://doi.org/10.24963/ijcai.2024/196

[44] J. Carreira and A. Zisserman, "Quo vadis, action recognition? a new model and the kinetics dataset," in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308 . [Online]. Available: https://doi.org/ 10.1109/CVPR.2017.502

[45] K. D. B. J. Adam. (2014). A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [Online]. 1412(6). Available: https://doi.org/10.48550/arXiv.1412.6980