A Hybrid Model of RoBERTa and Bidirectional GRU for Enhanced Sentiment Analysis

Trang Nguyen

Abstract


Sentiment analysis, a natural language processing task, has gained significant attention because of its diverse applications in understanding user opinions and emotions in social media, customer feedback, and online reviews. Several studies have been conducted on this task using English datasets, yielding noteworthy outcomes. However, it is important to note that research on the same task for the Vietnamese language remains limited, and the available training data are currently not substantial.The proposed hybrid model is based on the power of two architectures: RoBERTa, a transformer-based model, and Bidirectional Gated Recurrent Units (Bi-GRU). By fusing the strengths of both models, the approach aims to enhance sentiment analysis performance and generalize better on diverse datasets.The research outlines the model's architecture, highlighting the seamless integration of RoBERTa and Bi-GRU components, and describes the fine-tuning process on a large corpus of Vietnamese texts for sentiment analysis. To evaluate the model's performance, comprehensive experiments are conducted on the benchmark datasets, comparing it with state-of-the-art approaches for sentiment analysis. The experimental results unequivocally demonstrate that the proposed model outperforms other methods across various metrics including accuracy, precision, recall, and F1-score on both the IMDb and UIT-VSFC datasets.

Full Text:

PDF

References


N. C. Le, N. The Lam, S. H. Nguyen, and D. Thanh Nguyen, “On Vietnamese Sentiment Analysis: A Transfer Learning Method,” Proceedings - 2020 RIVF International Conference on Computing and Communication Technologies, RIVF 2020, Oct. 2020, doi: 10.1109/RIVF48685.2020.9140757.

H. T. M. Nguyen et al., “VLSP SHARED TASK: SENTIMENT ANALYSIS,” Journal of Computer Science and Cybernetics, vol. 34, no. 4, pp. 295–310, Jan. 2019, doi: 10.15625/1813-9663/34/4/13160.

H. Hamdan, P. Bellot, and F. Béchet, “Supervised Methods for Aspect-Based Sentiment Analysis,” 8th International Workshop on Semantic Evaluation, SemEval 2014 - co-located with the 25th International Conference on Computational Linguistics, COLING 2014, Proceedings, pp. 596–600, 2014, doi: 10.3115/V1/S14-2104.

D. Van Thin, V. D. Nguye, K. Van Nguyen, and N. L. T. Nguyen, “Deep Learning for Aspect Detection on Vietnamese Reviews,” NICS 2018 - Proceedings of 2018 5th NAFOSTED Conference on Information and Computer Science, pp. 104–109, Jan. 2019, doi: 10.1109/NICS.2018.8606857.

H. V. Tran, V. T. Bui, D. T. Do, and V. V. Nguyen, “Combining PhoBERT and SentiWordNet for Vietnamese Sentiment Analysis,” Proceedings - International Conference on Knowledge and Systems Engineering, KSE, vol. 2021-November, 2021, doi: 10.1109/KSE53942.2021.9648599.

Y. Huang, S. Liu, L. Qu, and Y. Li, “Effective Vietnamese Sentiment Analysis Model Using Sentiment Word Embedding and Transfer Learning,” Communications in Computer and Information Science, vol. 1258 CCIS, pp. 36–46, 2020, doi: 10.1007/978-981-15-7984-4_3/COVER.

C. N. Dang, M. N. Moreno-García, and F. De La Prieta, “Hybrid Deep Learning Models for Sentiment Analysis,” Complexity, vol. 2021, 2021, doi: 10.1155/2021/9986920.

K. L. Tan, C. P. Lee, and K. M. Lim, “RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis,” Applied Sciences 2023, Vol. 13, Page 3915, vol. 13, no. 6, p. 3915, Mar. 2023, doi: 10.3390/APP13063915.

K. L. Tan, C. P. Lee, K. S. M. Anbananthen, and K. M. Lim, “RoBERTa-LSTM: A Hybrid Model for Sentiment Analysis With Transformer and Recurrent Neural Network,” IEEE Access, vol. 10, pp. 21517–21525, 2022, doi: 10.1109/ACCESS.2022.3152828.

W. Liao, B. Zeng, X. Yin, and P. Wei, “An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa,” Applied Intelligence, vol. 51, no. 6, pp. 3522–3533, Jun. 2021, doi: 10.1007/S10489-020-01964-1/METRICS.

Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” Jul. 2019, Accessed: Jul. 27, 2023. [Online]. Available: https://arxiv.org/abs/1907.11692v1

R. Ni and H. Cao, “Sentiment Analysis based on GloVe and LSTM-GRU,” Chinese Control Conference, CCC, vol. 2020-July, pp. 7492–7497, Jul. 2020, doi: 10.23919/CCC50068.2020.9188578.

Y. Cheng et al., “Sentiment Analysis Using Multi-Head Attention Capsules with Multi-Channel CNN and Bidirectional GRU,” IEEE Access, vol. 9, pp. 60383–60395, 2021, doi: 10.1109/ACCESS.2021.3073988.

Y. Pan and M. Liang, “Chinese Text Sentiment Analysis Based on BI-GRU and Self-attention,” Proceedings of 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference, ITNEC 2020, pp. 1983–1988, Jun. 2020, doi: 10.1109/ITNEC48623.2020.9084784.

A. Chaudhuri and S. K. Ghosh, “Sentiment analysis of customer reviews using robust hierarchical bidirectional recurrent neural network,” Advances in Intelligent Systems and Computing, vol. 464, pp. 249–261, 2016, doi: 10.1007/978-3-319-33625-1_23/COVER.

M. Jabreel, F. Hassan, and A. Moreno, “Target-dependent sentiment analysis of tweets using bidirectional gated recurrent neural networks,” Smart Innovation, Systems and Technologies, vol. 85, pp. 39–55, 2018, doi: 10.1007/978-3-319-66790-4_3/COVER.

A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017.

A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning Word Vectors for Sentiment Analysis”.

K. Van Nguyen, V. D. Nguyen, P. X. V. Nguyen, T. T. H. Truong, and N. L. T. Nguyen, “UIT-VSFC: Vietnamese Students’ Feedback Corpus for Sentiment Analysis,” Proceedings of 2018 10th International Conference on Knowledge and Systems Engineering, KSE 2018, pp. 19–24, Dec. 2018, doi: 10.1109/KSE.2018.8573337.

P. K. Jain, V. Saravanan, and R. Pamula, “A Hybrid CNN-LSTM: A Deep Learning Approach for Consumer Sentiment Analysis Using Qualitative User-Generated Contents,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 20, no. 5, Sep. 2021, doi: 10.1145/3457206.

S. M. Qaisar, “Sentiment Analysis of IMDb Movie Reviews Using Long Short-Term Memory,” 2020 2nd International Conference on Computer and Information Sciences, ICCIS 2020, Oct. 2020, doi: 10.1109/ICCIS49240.2020.9257657.

Q. Zhu, X. Jiang, and R. Ye, “Sentiment Analysis of Review Text Based on BiGRU-Attention and Hybrid CNN,” IEEE Access, vol. 9, pp. 149077–149088, 2021, doi: 10.1109/ACCESS.2021.3118537.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность IT Congress 2024

ISSN: 2307-8162