Optimization of Hyperparameters of BERT Model in Sentiment Analysis Using Genetic Algorithm
محورهای موضوعی : Natural Language ProcessingShahla Sadeghani 1 , Mohammad jafar Tarokh 2 , Mohammad Ali Afshar Kazemi 3
1 - . Department of Information Technology Management, SR.C.,Islamic Azad University, Tehran, Iran
2 - Department of Industrial Engineering, K.N.,Toosi University of Technology, Tehran, Iran
3 - Department of Industrial Management, CT.C., Islamic Azad University, Tehran, Iran
کلید واژه: Sentiment Analysis, BERT, Genetic Algorithm, Hyperparameter Optimization,
چکیده مقاله :
Sentiment analysis, a vital branch of natural language processing (NLP), is progressing rapidly, and advanced models such as BERT are used to improve the accuracy of such analyses. However, tuning BERT’s hyperparameters remains a major challenge, directly influencing its performance. The methods used in previous research to optimize hyperparameters often face limitations in accuracy and efficiency. This study addresses an important research knowledge gap by using genetic algorithm (GA) to present a new approach to optimize the hyperparameters of BERT model in sentiment analysis with the main goal to get better results. In this regard, this research is designed analytically and experimentally. Data were collected through Twitter API and preprocessed using NLP techniques including noise removal, tokenization, and text normalization before being analyzed. The BERT model’s hyperparameter optimization was performed using GA and the optimized models were evaluated based on criteria such as accuracy, precision, recall and F-1 score. The proposed algorithm was compared with other common methods such as vanilla BERT, Long Short-Term Memory (LSTM) and convolutional neural network (CNN). The results showed that the proposed model has significantly better performance than other methods, achieving 98.1% accuracy, 98.2% precision, 98.1% recall, 98.15% F-1 score, and an area under the curve (AUC) of 98.5%. In comparison, vanilla BERT model, LSTM and CNN scored 92.8%, 91.3% and 90.5% in accuracy, respectively. These results indicate that the use of GA in optimizing the hyperparameters of BERT model can effectively increase the accuracy and efficiency of sentiment analysis. This approach not only has applicability in various fields of text data analysis, but also can be used as an effective solution for future research in optimizing deep learning models.
Sentiment analysis, a vital branch of natural language processing (NLP), is progressing rapidly, and advanced models such as BERT are used to improve the accuracy of such analyses. However, tuning BERT’s hyperparameters remains a major challenge, directly influencing its performance. The methods used in previous research to optimize hyperparameters often face limitations in accuracy and efficiency. This study addresses an important research knowledge gap by using genetic algorithm (GA) to present a new approach to optimize the hyperparameters of BERT model in sentiment analysis with the main goal to get better results. In this regard, this research is designed analytically and experimentally. Data were collected through Twitter API and preprocessed using NLP techniques including noise removal, tokenization, and text normalization before being analyzed. The BERT model’s hyperparameter optimization was performed using GA and the optimized models were evaluated based on criteria such as accuracy, precision, recall and F-1 score. The proposed algorithm was compared with other common methods such as vanilla BERT, Long Short-Term Memory (LSTM) and convolutional neural network (CNN). The results showed that the proposed model has significantly better performance than other methods, achieving 98.1% accuracy, 98.2% precision, 98.1% recall, 98.15% F-1 score, and an area under the curve (AUC) of 98.5%. In comparison, vanilla BERT model, LSTM and CNN scored 92.8%, 91.3% and 90.5% in accuracy, respectively. These results indicate that the use of GA in optimizing the hyperparameters of BERT model can effectively increase the accuracy and efficiency of sentiment analysis. This approach not only has applicability in various fields of text data analysis, but also can be used as an effective solution for future research in optimizing deep learning models.
[1] M. Li, L. Chen, J. Zhao, and Q. Li, “Sentiment analysis of Chinese stock reviews based on BERT model,” Applied Intelligence, vol. 51, no. 7, 2021, doi: 10.1007/s10489-020-02101-8.
[2] K. Naithani and Y. P. Raiwani, “Realization of natural language processing and machine learning approaches for text-based sentiment analysis,” Expert Syst, vol. 40, no. 5, 2023, doi: 10.1111/exsy.13114.
[3] A. Mewada and R. K. Dewang, “SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting,” Journal of Supercomputing, vol. 79, no. 5, 2023, doi: 10.1007/s11227-022-04881-x.
[4] X. Li, X. Wang, and H. Liu, “Research on fine-tuning strategy of sentiment analysis model based on BERT,” in 2021 IEEE 3rd International Conference on Communications, Information System and Computer Engineering, CISCE 2021, 2021. doi: 10.1109/CISCE52179.2021.9445882.
[5] R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. Ali Almazroi, “A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification,” J Healthc Eng, vol. 2022, 2022, doi: 10.1155/2022/3498123. [6] J. Martinez-Gil, “Optimizing readability using genetic algorithms,” Knowl Based Syst, vol. 284, 2024, doi: 10.1016/j.knosys.2023.111273.
[7] D. Choudhury and T. Acharjee, “A novel approach to fake news detection in social networks using genetic algorithm applying machine learning classifiers,” Multimed Tools Appl, vol. 82, no. 6, 2023, doi: 10.1007/s11042-022-12788-1.
[8] A. Bello, S. C. Ng, and M. F. Leung, “A BERT Framework to Sentiment Analysis of Tweets,” Sensors, vol. 23, no. 1, 2023, doi: 10.3390/s23010506.
[9] U. Sirisha and B. S. Chandana, “Aspect based Sentiment and Emotion Analysis with ROBERTa, LSTM,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 11, 2022, doi: 10.14569/IJACSA.2022.0131189.
[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Bidirectional encoder representations from transformers,” arXiv preprint arXiv:1810.04805, p. 15, 2018.
[11] H. Liu et al., “Use of BERT (Bidirectional Encoder Representations from Transformers)-Based Deep Learning Method for Extracting Evidences in Chinese Radiology Reports: Development of a Computer-Aided Liver Cancer Diagnosis Framework,” J Med Internet Res, vol. 23, no. 1, 2021, doi: 10.2196/19689.
[12] L. B. Cesar, M. Á. Manso-Callejo, and C. I. Cira, “BERT (Bidirectional Encoder Representations from Transformers) for Missing Data Imputation in Solar Irradiance Time Series †,” Engineering Proceedings, vol. 39, no. 1, 2023, doi: 10.3390/engproc2023039026.
[13] A. Areshey and H. Mathkour, “Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model,” Sensors, vol. 23, no. 11, 2023, doi: 10.3390/s23115232.
[14] L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, 2020, doi: 10.1016/j.neucom.2020.07.061.
[15] A. Pfob, S. C. Lu, and C. Sidey-Gibbons, “Machine learning in medicine: a practical introduction to techniques for data pre-processing, hyperparameter tuning, and model comparison,” BMC Med Res Methodol, vol. 22, no. 1, 2022, doi: 10.1186/s12874-022-01758-8.
[16] S. Katoch, S. S. Chauhan, and V. Kumar, “A review on genetic algorithm: past, present, and future,” Multimed Tools Appl, vol. 80, no. 5, 2021, doi: 10.1007/s11042-020-10139-6.
[17] M. Dehghani, E. Trojovská, and P. Trojovský, “A new human-based metaheuristic algorithm for solving optimization problems on the base of simulation of driving training process,” Sci Rep, vol. 12, no. 1, 2022, doi: 10.1038/s41598-022-14225-7.
[18] H. Su et al., “RIME: A physics-based optimization,” Neurocomputing, vol. 532, 2023, doi: 10.1016/j.neucom.2023.02.010.
[19] M. Pota, M. Ventura, R. Catelli, and M. Esposito, “An effective bert-based pipeline for twitter sentiment analysis: A case study in Italian,” Sensors (Switzerland), vol. 21, no. 1, 2021, doi: 10.3390/s21010133.
[20] A. S. Talaat, “Sentiment analysis classification system using hybrid BERT models,” J Big Data, vol. 10, no. 1, 2023, doi: 10.1186/s40537-023-00781-w.
[21] M. Pota, M. Ventura, H. Fujita, and M. Esposito, “Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets,” Expert Syst Appl, vol. 181, 2021, doi: 10.1016/j.eswa.2021.115119.
[22] H. Y. Lin and T. S. Moh, “Sentiment analysis on COVID tweets using COVID-Twitter-BERT with auxiliary sentence approach,” in Proceedings of the 2021 ACMSE Conference - ACMSE 2021: The Annual ACM Southeast Conference, 2021. doi: 10.1145/3409334.3452074.
[23] S. Palani, P. Rajagopal, and S. Pancholi, “T-BERT--Model for Sentiment Analysis of Micro-blogs Integrating Topic Model and BERT,” arXiv preprint arXiv:2106.01097, 2021.
[24] A. Chiorrini, C. Diamantini, A. Mircoli, and D. Potena, “Emotion and sentiment analysis of tweets using BERT,” in CEUR Workshop Proceedings, 2021.
[25] R. Anggrainingsih, G. M. Hassan, and A. Datta, “BERT based classification system for detecting rumours on Twitter,” IEEE Access, vol. 11, pp. 80207-80217, 2023., 2021, doi: https://doi.org/10.1109/ACCESS.2023.3299858.
[26] Y. Wu, Z. Jin, C. Shi, P. Liang, and T. Zhan, “Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis,” arXiv preprint arXiv:2403.08217, 2024.
[27] H. A. Samir, L. Abd-Elmegid, and M. Marie, “Sentiment analysis model for Airline customers’ feedback using deep learning techniques,” 2023. doi: 10.1177/18479790231206019.
[28] S. Kardakis, I. Perikos, F. Grivokostopoulou, and I. Hatzilygeroudis, “Examining attention mechanisms in deep learning models for sentiment analysis,” Applied Sciences (Switzerland), vol. 11, no. 9, 2021, doi: 10.3390/app11093883.
[29] S. Seo, C. Kim, H. Kim, K. Mo, and P. Kang, “Comparative Study of Deep Learning-Based Sentiment Classification,” IEEE Access, vol. 8, 2020, doi: 10.1109/ACCESS.2019.2963426.
[30] M. Sedighimanesh, A. Sedighimanesh, and H. Zandhessami, “Optimizing Hyperparameters for Customer Churn Prediction with PSO-Enhanced Composite Deep Learning Techniques,” Journal of Information Systems and Telecommunication (JIST), vol. 13, no. 50, pp. 91–110, 2025, doi: 10.61882/jist.48088.13.50.91.