An Empirical Comparative Study of LoRA Fine-Tuning Hyperparameter Configurations on the GLUE Benchmark
DOI:
https://doi.org/10.71222/nfwhfr13Keywords:
Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Hyperparameter Sensitivity, GLUE BenchmarkAbstract
Low-Rank Adaptation (LoRA) has become a dominant parameter-efficient fine-tuning paradigm for adapting pretrained Transformers to downstream tasks, yet practitioners still rely on rule-of-thumb settings for its central hyperparameters: the rank r, the scaling factor α, the choice of target modules, and the dropout probability. This paper presents an empirical comparative study that quantifies how these four hyperparameters jointly affect downstream behaviour on six representative tasks of the General Language Understanding Evaluation (GLUE) benchmark, using RoBERTa-base as the backbone. A controlled grid of 72 configurations is evaluated on SST-2, CoLA, MRPC, STS-B, QNLI, and RTE, with each cell averaged over three random seeds and reported along four axes: task-specific accuracy or correlation, trainable parameter count, peak GPU memory, and wall-clock training time. The results indicate that rank saturation occurs earlier than is commonly assumed at this scale, that the ratio α∕r is a stronger predictor of accuracy than α in isolation, and that expanding the adapted modules from the query--value pair to all linear projections yields task-dependent and modest gains while incurring a sizeable cost in memory and runtime. Practical configuration guidance is summarised by task-size category so that downstream users can pick a starting point with a clearer view of the trade-offs involved.References
1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022.
2. Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-efficient fine-tuning for large models: A comprehensive survey,” arXiv preprint arXiv:2403.14608, 2024.
3. J. Hu, X. Wang, and J. Lai, “Benchmarking Learned Cardinality Estimation Techniques for Analytical Query Processing in Data Warehouses,” J. Comput. Technol. Appl. Math., vol. 3, no. 3, pp. 1–8, 2026.
4. M. Yu and Z. Li, “An Empirical Comparison of Discrete Video Tokenization Schemes for Video Question Answering and Video Captioning,” Artif. Intell. Mach. Learn. Rev., vol. 6, no. 2, pp. 27–50, 2025.
5. Z. Jiang and M. Wang, “Evaluation and Analysis of Chart Reasoning Accuracy in Multimodal Large Language Models: An Empirical Study on Influencing Factors,” in Pinnacle Academic Press Proceedings Series, vol. 3, pp. 43–58, 2025.
6. M. Wang, P. Xiao, and M. Yu, “Sparse, Dense, or Hybrid? Comparing Retrieval Strategies for Biomedical Question Answering with Retrieval-Augmented Generation,” J. Sci. Innov. Soc. Impact, vol. 2, no. 2, pp. 141–152, 2026.
7. Z. Chen and M. Wang, “Evaluating the Quality of Large Language Model-Generated Explanations in Recommendation Tasks: A Multi-Dimensional Comparative Analysis,” J. Global Eng. Rev., vol. 3, no. 1, pp. 54–63, 2025.
8. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in International Conference on Learning Representations (ICLR), 2019.
9. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proc. 36th Int. Conf. Mach. Learn. (ICML), vol. 97. Proceedings of Machine Learning Research (PMLR), 2019, pp. 2790–2799.
10. X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in *Proc. 59th Annu. Meet. Assoc. Comput. Linguistics and 11th Int. Joint Conf. Nat. Lang. Process. (ACL-IJCNLP)*, 2021, pp. 4582–4597.
11. B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proc. 2021 Conf. Empir. Methods Nat. Lang. Process. (EMNLP), 2021, pp. 3045–3059.
12. E. Ben Zaken, Y. Goldberg, and S. Ravfogel, “BitFit: Simple parameter-efficient fine-tuning for Transformer-based masked language-models,” in Proc. 60th Annu. Meet. Assoc. Comput. Linguistics (ACL), Short Papers, 2022, pp. 1–9.
13. J. Lai and Z. Li, “A Comparative Empirical Evaluation of Single-Agent and Multi-Agent LLM Prompting Strategies for Automated Formative Feedback in Education,” J. Intell. Eng. Technol., vol. 1, no. 2, pp. 29–38, 2026.
14. J. Lai and Z. Li, “Does an LLM-Based Feedback Agent Move the Needle? An Empirical Study of Student Correctness and Hint Dependency on the ASSISTments 2017 Interaction Logs,” Acad. Nexus J., vol. 5, no. 1, 2026.
15. Y. Chen and T. Tang, “Evaluating Prompt Engineering Strategies for Few-Shot Cyber Threat Intelligence Entity and Relation Extraction from Multi-Source Reports,” J. Sci. Innov. Soc. Impact, vol. 2, no. 2, pp. 153–164, 2026.
16. X. Fu, T. Tang, and C. Luo, “An Empirical Comparison of ReAct, Reflexion, Plan-and-Solve, and Tree-of-Thought Planning Strategies on Financial Question Answering and Numerical Reasoning Tasks,” J. Sci. Innov. Soc. Impact, vol. 2, no. 3, pp. 23–34, 2026.
17. Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao, “AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning,” in International Conference on Learning Representations (ICLR), 2023.
18. S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “DoRA: Weight-decomposed low-rank adaptation,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), vol. 235. Proceedings of Machine Learning Research (PMLR), 2024, pp. 32100–32121.
19. D. J. Kopiczko, T. Blankevoort, and Y. M. Asano, “VeRA: Vector-based random matrix adaptation,” in International Conference on Learning Representations (ICLR), 2024.
20. A. Aghajanyan, S. Gupta, and L. Zettlemoyer, “Intrinsic dimensionality explains the effectiveness of language model fine-tuning,” in *Proc. 59th Annu. Meet. Assoc. Comput. Linguistics and 11th Int. Joint Conf. Nat. Lang. Process. (ACL-IJCNLP)*, 2021, pp. 7319–7328.
21. D. Biderman, J. Portes, J. J. Gonzalez Ortiz, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham, “LoRA learns less and forgets less,” Trans. Mach. Learn. Res. (TMLR), 2024.
22. J. Zhu, K. Greenewald, K. Nadjahi, H. Sáez de Ocáriz Borde, R. Brüel Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon, “Asymmetry in low-rank adapters of foundation models,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), vol. 235. Proceedings of Machine Learning Research (PMLR), 2024, pp. 62369–62385.
23. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
24. S. Hayou, N. Ghosh, and B. Yu, “The impact of initialization on LoRA finetuning dynamics,” in Advances in Neural Information Processing Systems (NeurIPS), 2024.
25. F. Zhao and T. Tang, “AI-Based Sentiment Analysis for Stock Market Prediction: A Systematic Literature Review,” J. Sustain. Policy Pract., vol. 2, no. 3, pp. 115–124, 2026.
26. M. Wang and L. Zhu, “Linguistic analysis of verb tense usage patterns in computer science paper abstracts,” Acad. Nexus J., vol. 3, no. 3, 2024.
27. M. Wang, Z. Jiang, and S. Zhou, “Cross-cultural semantic differences in emoji usage on social media platforms,” J. Adv. Comput. Syst., vol. 4, no. 5, pp. 55–66, 2024.
28. T. Tang and M. Yu, “A Comparative Empirical Study of Semantic Signal Enhancement Methods for User Interest Features in CTR Prediction: Applicability of TF-IDF Weighting, Sentence-BERT Embeddings, and LDA Topic Fusion,” J. Comput. Innov. Appl., vol. 2, no. 1, pp. 165–174, 2024.
29. T. Tang, X. Fu, and C. Luo, “An Empirical Comparison of High-Order Feature Interaction Operators for Conversion Rate Prediction in Sparse, High-Cardinality Message-Ads Traffic: Accuracy, Efficiency, and Offline-Online Consistency,” J. Sci. Innov. Soc. Impact, vol. 2, no. 3, pp. 12–22, 2026.
30. T. Tang and M. Yu, “A Comparative Evaluation of LLM-Generated Semantic Tags versus Classical Text Features (TF-IDF, LDA, BERT Embeddings) for User-Interest Enrichment in Short-Video Recommendation,” Artif. Intell. Mach. Learn. Rev., vol. 5, no. 1, pp. 129–140, 2024.
31. S. Hayou, N. Ghosh, and B. Yu, “LoRA+: Efficient low rank adaptation of large models,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), vol. 235. Proceedings of Machine Learning Research (PMLR), 2024, pp. 17783–17806.
32. M. Li and S. Xu, “Trustworthy artificial intelligence in financial decision-making: A systematic review of explainability, fairness, and accountability,” J. Sustain. Policy Pract., vol. 2, no. 3, pp. 104–114, 2026.
33. Z. Luo, “Fairness-Aware Credit Evaluation: Bias Detection and Mitigation Techniques for Inclusive Lending Practices,” J. Sustain. Policy Pract., vol. 2, no. 2, pp. 78–89, 2026.
34. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” in International Conference on Learning Representations (ICLR), 2022.
35. C. Luo and X. Wang, “Latency-Throughput Tradeoffs of ONNX Runtime, TensorRT-LLM, vLLM, and Triton: An Empirical Comparison on 1B-3B Parameter LLM Inference,” Spectrum Res., vol. 6, no. 1, 2026.
36. Y. Tian, “Gradient Boosting-Based Demand Variability Estimation for Improved Safety Stock Calculation in Multi-Echelon Aerospace Spare Parts Inventory,” J. Sci. Innov. Soc. Impact, vol. 2, no. 2, pp. 175–186, 2026.
37. L. Long and J. Hu, “Multi-Objective Particle Swarm Optimization for Site Selection and Policy Subsidy Maximization of Foreign Renewable Energy Enterprises in the United States,” Artif. Intell. Mach. Learn. Rev., vol. 7, no. 2, pp. 54–69, 2026.
38. X. Long, J. Hu, and Z. Ling, “A Comparative Analysis of Telemetry-Driven Anomaly Detection Approaches for Dual-Purpose Operational and Security Optimization in Edge Computing Infrastructure,” J. Comput. Innov. Appl., vol. 4, no. 1, pp. 79–88, 2026.
39. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.
40. Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “LoftQ: LoRA-fine-tuning-aware quantization for large language models,” in International Conference on Learning Representations (ICLR), 2024.
41. H. Cao, J. Hu, and C. Luo, “Behavioural Feature Analysis for Anomalous Click Detection in Mobile Advertising Environments: Toward In-App Browser-Specific Detection,” J. Comput. Innov. Appl., vol. 3, no. 2, pp. 96–105, 2025.
42. Y. Li, F. Zhao, and J. Hu, “Identifying Cross-Market Risk Contagion Amplifiers via Graph Attention Networks: Empirical Evidence from US Financial Stress Periods,” J. Comput. Innov. Appl., vol. 4, no. 1, pp. 164–175, 2026.
43. J. Hu and X. Long, “Graph Learning-Based Behavioral Detection for Software Supply Chain Attacks,” J. Adv. Comput. Syst., vol. 4, no. 4, pp. 49–60, 2024.
44. Y. Chen and J. Hu, “Graph Neural Network-Based Cascading Disruption Path Identification in Multi-Tier Rare Earth Processing Networks,” J. Global Eng. Rev., vol. 4, no. 1, pp. 99–112, 2026.
