An Empirical Comparison of High-Order Feature Interaction Operators for Conversion Rate Prediction in Sparse, High-Cardinality Message-Ads Traffic: Accuracy, Efficiency, and Offline--Online Consistency
Keywords:
Conversion Rate Prediction, Feature Interaction, Empirical Benchmarking, Offline--Online ConsistencyAbstract
Post-click conversion rate (CVR) prediction on message-ads traffic exposes feature interaction operators to an extreme regime of sparsity, label imbalance, and serving-latency constraints. While a decade of recommender research has produced an abundance of operators that differ in their treatment of explicit versus implicit, low-order versus high-order interactions, published comparisons typically optimize for click-through rate on dense public logs and seldom isolate the operator from confounding training pipelines. This study conducts a controlled empirical comparison of seven high-order interaction operators---plain MLP, FM, DeepFM, DCN, DCN-V2, xDeepFM, and AutoInt---across Criteo, Avazu, and Ali-CCP under a unified training protocol. We measure offline AUC and LogLoss, per-sample parameters, FLOPs, and inference latency, and further stratify AUC by user-activity quantile and by categorical-feature density. On Ali-CCP CVR, DCN-V2 attains the highest AUC (0.6289) while DCN matches it within 0.0011 AUC at 0.83× the latency; xDeepFM's compressed interaction component contributes the largest efficiency penalty without a proportionate accuracy gain. Rank correlation between offline AUC and an online CVR proxy drops from 0.93 on high-activity users to 0.41 on cold-start users, echoing documented offline--online inconsistencies. The findings provide operator-selection guidance grounded in measured efficiency and subgroup stability rather than on headline AUC deltas.References
1. Ma, X., Zhao, L., Huang, G., Wang, Z., Hu, Z., Zhu, X., and Gai, K., "Entire space multi-task model: An effective approach for estimating post-click conversion rate," in *Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval*, pp. 1137–1140, ACM, 2018. https://doi.org/10.1145/3209978.3210104
2. Wang, R., Shivanna, R., Cheng, D. Z., Jain, S., Lin, D., Hong, L., and Chi, E. H., "DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems," in Proceedings of the Web Conference 2021, pp. 1785–1797, ACM, 2021. https://doi.org/10.1145/3442381.3450078
3. Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., Anil, R., Haque, Z., Hong, L., Jain, V., Liu, X., and Shah, H., "Wide & deep learning for recommender systems," in Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pp. 7–10, ACM, 2016. https://doi.org/10.1145/2988450.2988454
4. Zhu, J., Liu, J., Yang, S., Zhang, Q., and He, X., "Open benchmarking for click-through rate prediction," in *Proceedings of the 30th ACM International Conference on Information and Knowledge Management*, pp. 2759–2769, ACM, 2021. https://doi.org/10.1145/3459637.3482486
5. Ferrari Dacrema, M., Cremonesi, P., and Jannach, D., "Are we really making much progress? A worrying analysis of recent neural recommendation approaches," in Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101–109, ACM, 2019. https://doi.org/10.1145/3298689.3347058
6. Wang, R., Fu, B., Fu, G., and Wang, M., "Deep & cross network for ad click predictions," in Proceedings of the ADKDD'17, Article 12, ACM, 2017. https://doi.org/10.1145/3124749.3124754
7. Lian, J., Zhou, X., Zhang, F., Chen, Z., Xie, X., and Sun, G., "xDeepFM: Combining explicit and implicit feature interactions for recommender systems," in *Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp. 1754–1763, ACM, 2018. https://doi.org/10.1145/3219819.3220023
8. Chen, B., Wang, Y., Liu, Z., Tang, R., Guo, W., Zheng, H., Yao, W., Zhang, M., and He, X., "Enhancing explicit and implicit feature interactions via information sharing for parallel deep CTR models," in *Proceedings of the 30th ACM International Conference on Information and Knowledge Management*, pp. 3757–3766, ACM, 2021. https://doi.org/10.1145/3459637.3481915
9. Wang, F., Gu, H., Li, D., Lu, T., Zhang, P., and Gu, N., "Towards deeper, lighter and interpretable cross network for CTR prediction," in *Proceedings of the 32nd ACM International Conference on Information and Knowledge Management*, pp. 2523–2533, ACM, 2023. https://doi.org/10.1145/3583780.3615089
10. Song, W., Shi, C., Xiao, Z., Duan, Z., Xu, Y., Zhang, M., and Tang, J., "AutoInt: Automatic feature interaction learning via self-attentive neural networks," in *Proceedings of the 28th ACM International Conference on Information and Knowledge Management*, pp. 1161–1170, ACM, 2019. https://doi.org/10.1145/3357384.3357925
11. Xiao, J., Ye, H., He, X., Zhang, H., Wu, F., and Chua, T.-S., "Attentional factorization machines: Learning the weight of feature interactions via attention networks," in *Proceedings of the 26th International Joint Conference on Artificial Intelligence*, pp. 3119–3125, IJCAI, 2017. https://doi.org/10.24963/ijcai.2017/435
12. Li, Z., Cheng, W., Chen, Y., Chen, H., and Wang, W., "Interpretable click-through rate prediction through hierarchical attention," in Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 313–321, ACM, 2020. https://doi.org/10.1145/3336191.3371785
13. Rendle, S., "Factorization machines," in Proceedings of the 2010 IEEE International Conference on Data Mining, pp. 995–1000, IEEE, 2010. https://doi.org/10.1109/ICDM.2010.127
14. Guo, H., Tang, R., Ye, Y., Li, Z., and He, X., "DeepFM: A factorization-machine based neural network for CTR prediction," in *Proceedings of the 26th International Joint Conference on Artificial Intelligence*, pp. 1725–1731, IJCAI, 2017. https://doi.org/10.24963/ijcai.2017/239
15. He, X., and Chua, T.-S., "Neural factorization machines for sparse predictive analytics," in *Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval*, pp. 355–364, ACM, 2017. https://doi.org/10.1145/3077136.3080777
16. Juan, Y., Zhuang, Y., Chin, W.-S., and Lin, C.-J., "Field-aware factorization machines for CTR prediction," in Proceedings of the 10th ACM Conference on Recommender Systems, pp. 43–50, ACM, 2016. https://doi.org/10.1145/2959100.2959134
17. Huang, T., Zhang, Z., and Zhang, J., "FiBiNET: Combining feature importance and bilinear feature interaction for click-through rate prediction," in Proceedings of the 13th ACM Conference on Recommender Systems, pp. 169–177, ACM, 2019. https://doi.org/10.1145/3298689.3347043
18. Zhu, J., Dai, Q., Su, L., Ma, R., Liu, J., Cai, G., Xiao, X., and Zhang, R., "BARS: Towards open benchmarking for recommender systems," in *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval*, pp. 2912–2923, ACM, 2022. https://doi.org/10.1145/3477495.3531723
19. Yi, J., Chen, Y., Li, J., Sett, S., and Yan, T. W., "Predictive model performance: Offline and online evaluations," in *Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pp. 1294–1302, ACM, 2013. https://doi.org/10.1145/2487575.2488215
20. Zhou, G., Song, C., Zhu, X., Fan, Y., Zhu, H., Ma, X., Yan, Y., Jin, J., Li, H., and Gai, K., "Deep interest network for click-through rate prediction," in *Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp. 1059–1068, ACM, 2018. https://doi.org/10.1145/3219819.3219823
21. Rendle, S., Krichene, W., Zhang, L., and Anderson, J., "Neural collaborative filtering vs. matrix factorization revisited," in Proceedings of the 14th ACM Conference on Recommender Systems, pp. 240–248, ACM, 2020. https://doi.org/10.1145/3383313.3412488
22. Zhou, G., Mou, N., Fan, Y., Pi, Q., Bian, W., Zhou, C., Zhu, X., and Gai, K., "Deep interest evolution network for click-through rate prediction," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 1, pp. 5941–5948, AAAI Press, 2019. https://doi.org/10.1609/aaai.v33i01.33015941
