基于预训练模型的仇恨言论检测

Hate speech detection based on pre-trained models

摘要

为准确检测和识别仇恨言论,通过微调大语言模型对数据集样本进行扩充与平衡,并基于预训练模型 RoBERTa 构建 RoBERTa-Attention-GRU-TextCNN 模型,将深度学习强大的特征捕获和提取能力应用到文本序列数据的分析、挖掘中。 首先通过 RoBERTa 模型对文本数据进行特征提取;然后利用自注意机制获取单词间的依赖关系;最后将获取到的特征矩阵输入到 GRU-TextCNN 层中以捕捉更深层次的语义信息和局部特征。 使用 TweetEval 提供的 2 个公开的数据集来评估模型效果,实验结果表明,该模型相较于传统的仇恨言论检测模型具有更好的检测效果。

Abstract

To accurately detect and identify hate speech, the dataset samples are expanded and balanced by fine-tuning the large language model. The RoBERTa-Attention-GRU-TextCNN model is constructed based on the pre-training model RoBERTa, leveraging the powerful feature capture and extraction capabilities of deep learning for the analysis and mining of text sequence data. Firstly, the RoBERTa model is used to extract features from the text data; then, the self-attention mechanism is used to obtain the dependencies between words; finally, the acquired feature matrix is input into the GRU-TextCNN layer to capture deeper semantic information and local features. Two publicly available datasets provided by TweetEval are used to evaluate the model effect, and the experimental results show that the model has a better detection effect compared to the traditional hate speech detection model.

关键词

大语言模型 / 仇恨检测 / RoBERTa / 预训练模型 / RoBERTa-Attention-GRU-TextCNN

Key words

large language model / hate detection / RoBERTa / pre-trained model / RoBERTa-Attention-GRU-TextCNN

参考文献

[1] 联合国大会. 促进宗教间和文化间对话与容忍,打击仇恨言论[EB/OL].(2021—07—22)[2024—10—09]. https://documents.un.org/doc/undoc/gen/n21/200/60/pdf/n2120060.pdf.

[2] United Nations General Assembly . Promoting interreligious and intercultural dialogue and tolerance to combat hate speech[EB/OL]. (2021—07—22)[2024—10—09]. https://documents.un.org/doc/undoc/gen/n21/200/60/pdf/n2120060.pdf.

[3] TING I H, CHIH M H, WU J S, et al. An approach for hate groups detection in facebook[C]// Proceedings of the 3rd International Workshop on Intelligent Data Analysis and Management. Dordrecht: Springer, 2013: 101-106.

[4] MEHDAD Y, TETREAULT J. Do characters abuse more than words?[C]// Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Los Angeles: ACL, 2016: 299-303.

[5] DELVIGNA F, CIMINO A, DELL'ORLETTA F, et al. Hate me, hate me not: hate speech detection on Facebook[C]// Proceedings of the First Italian Conference on Cybersecurity (ITASEC17). Venice: CEUR, 2017: 86-95.

[6] RODRIGUEZ A, ARGUETA C, CHEN Y L. Automatic detection of hate speech on facebook using sentiment and emotion analysis[C]// Proceedings of the 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). Okinawa: IEEE, 2019: 169-174.

[7] BRILIANI A, IRAWAN B, SETIANINGSIH C. Hate speech detection in indonesian language on instagram comment section using K—nearest neighbor classification method[C]// Proceedings of the 2019 IEEE International Conference on Internet of Things and Intelligence System (IoTaIS). Bali: IEEE, 2019: 98-104.

[8] DAS S, BHATTACHARYYA K, SARKAR S. Performance analysis of logistic regression, naïve Bayes, KNN, decision tree, random forest and SVM on hate speech detection from twitter[J]. International Research Journal of Innovations in Engineering and Technology, 2023, 7(3): 24-28.

[9] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Advances in Neural Information Processing Systems 30. New York: Curran Associates, Inc., 2017: 5999-6009.

[10] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre—training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis: ACL, 2019: 4171-4186.

[11] LIU Z, LIN W, SHI Y, et al. A robustly optimized BERT pre—training approach with post—training[C]// Proceedings of the 20th China National Conference on Chinese Computational Linguistics. Hohhot: Springer, 2021: 471-484.

[12] ZHANG Z, ROBINSON D, TEPPER J. Detecting hate speech on Twitter using a convolution—gru based deep neural network[C]// Proceedings of the Semantic Web: 15th International Conference. Heraklion: Springer, 2018: 745-760.

[13] KSHIRSAGAR R, CUKUVAC T, MCKEOWN K, et al. Predictive embeddings for hate speech detection on Twitter[C]// Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Brussels: ACL, 2018: 26-32.

[14] WATANABE H, BOUAZIZI M, OHTSUKI T. Hate speech on Twitter: a pragmatic approach to collect hateful and offensive expressions and perform hate speech detection[J]. IEEE Access, 2018, 6: 13825-13835.

[15] PATIHULLAH J, WINARKO E. Hate speech detection for Indonesia tweets using word embedding and gated recurrent unit[J]. Indonesian Journal of Computing and Cybernetics Systems, 2019, 13(1): 43-52.

[16] TEKÖRGÜLÜ S S, CHUNG Y L, GUERINI M. Generating counternarratives against online hate speech: data and strategies[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Hohhot: ACL, 2020: 1177-1190.

[17] MOZAFARI M, FARAHBAKHSH R, CRESPI N. A BERT—based transfer learning approach for hate speech detection in online social media[C]// Proceedings of the Eighth International Conference on Complex Networks and Their Applications. Lisbon: Springer, 2020: 928-940.

[18] ALBADWI N, KURDI M, MISHRA S. Are they your brothers? Analysis and detection of religious hate speech in the arabic twittersphere[C]// Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). Barcelona: IEEE, 2018: 69-76.

[19] AZHARIA A A, SIBARONI Y, PRASETIYOWATI S S. Detection of Indonesian hate speech in the comments column of indonesian artists' instagram using the RoBERTa method[J]. Jurnal Ilmiah Penelitian dan Pembelajaran Informatika, 2023, 8(3): 764-773.

[20] 王琰慧, 王小龙, 张顺香, 等. 基于谐音干扰词替换的中文仇恨言论检测方法[J]. 应用科技, 2024, 51(3): 72-81.

[21] WANG Yanhui, WANG Xiaolong, ZHANG Shunxiang, et al. A Chinese hate speech detection method based on homophonic interference word replacement[J]. Applied Science and Technology, 2024, 51(3): 72-81.

[22] 刘旭东, 杨亮, 张冬瑜, 等. 结合图卷积网络的多模态仇恨迷因识别研究[J]. 重庆理工大学学报(自然科学), 2024, 38(1): 169-179.

[23] LIU Xudong, YANG Liang, ZHANG Dongyu, et al. Research on multimodal hate meme recognition based on graph convolutional network[J]. Journal of Chongqing University of Technology (Natural Science), 2024, 38(1): 169-179.

[24] BAI Jinze, BAI Shuai, CHU Yunfei, et al. Qwen technical report[EB/OL]. (2023—09—28)[2024—10—09]. https://doi.org/10.48550/arXiv.2309.16609.

[25] HU E J, SHEN Y L, WALLIS P, et al. Lora: low—rank adaptation of large language models[EB/OL]. (2021—06—17)[2024—10—09]. https://doi.org/10.48550/arXiv.2106.09685.

[26] HOULSBY N, GIURGIU A, JASTREBSKI S, et al. Parameter—efficient transfer learning for NLP[C]// Proceedings of the 36th International conference on machine learning. Long Beach: PMLR, 2019: 2790-2799.

[27] LI X L, LIANG P. Prefix—tuning: optimizing continuous prompts for generation[C/OL]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL, 2021: 4582-4597. https://aclanthology.org/2021.acl—long.353/.

[28] VAN HEE C, LEFEVER E, HOSTE V. Semeval—2018 task 3: irony detection in english tweets[C]// Proceedings of the 12th International Workshop on Semantic Evaluation. New Orleans: ACL, 2018: 39-50.

[29] BASILE V, BOSCO C, FERSINI E, et al. Semeval—2019 task 5: multilingual detection of hate speech against immigrants and women in Twitter[C]// Proceedings of the 13th International Workshop on Semantic Evaluation. Minneapolis: ACL, 2019: 54-63.

基金资助

国家自然科学基金资助项目(61976036)

国家社会科学基金资助项目(20BTQ074)