PerGOLD: Identification of offensive language in Persian tweets: leveraging crowdsourcing

Document Type : Original Article

Authors

• Fatemeh Jafarinejad
• Marziea Rahimi
• Maryam Khodabakhsh
• Seyedehfatemeh Karimi

Department of the Computer Engineering, Shahrood University of Technology (SUT), Shahrood, Semnan, Iran

10.22067/cke.2025.90088.1132

Abstract

It is concerning that the growing popularity of social networks is encouraging violence or inciting offense toward other people. An attempt has been made in the past several years to detect offensive language in social media posts. Nonetheless, the majority of studies focus on recognizing offensive language in English. Moreover, dataset labeling emerges as a crucial and fundamental step for training high-quality models, considering the increasing use of artificial intelligence and machine learning tools. Utilizing crowdsourcing platforms is an efficient and optimal method that can be used for data labeling. This approach uses human resources who are sufficiently knowledgeable about the topic to label the data. In this paper, we introduce PerGOLD, a new Persian General Offensive Language Dataset, in which we use an event-based data collection methodology to detect offensive language in Persian Twitter. To access labeled training data, we build a crowdsourcing platform to benefit from human input. We labeled 13,716 tweets, and according to the obtained results, 34% of them were labeled as offensive language. Finally, we evaluated the efficiency of these data by applying some classic machine learning models (LR, SVM) and transformer-based language models (RoBERTa, ParsBERT). The obtained F1-score of the best model (ParsBERT) was 85.4%.

Keywords

• Offensive Language
• Labeling
• Crowdsourcing
• Natural Language Processing

Main Subjects

• Computer Science

---

References

[1] F. Poletto, V. Basile, M. Sanguinetti, C. Bosco, and V. Patti. (2021, Jun.). Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation. [Online]. 55, pp. 477–523. Available: https://doi.org/10.1007/s10579-020-09502-8

[2] S. MacAvaney, H. R. Yao, E. Yang, K. Russell, N. Goharian, and O. Frieder. (2019, Aug.). Hate speech detection: Challenges and solutions. PLoS One. [Online]. 14(8), p. e0221152. Available: https://doi.org/10.1371/journal.pone.0221152

[3] J. Qian, A. Bethke, Y. Liu, E. Belding, and W. Y. Wang. (2019, Sep.). A benchmark dataset for learning to intervene in online hate speech. arXiv preprint arXiv:1909.04251. [Online]. Available: https://doi.org/10.48550/arXiv.1909.04251

[4] J. Zhang, V. S. Sheng, T. Li, and X. Wu. (2017, Mar. 22). Improving crowdsourced label quality using noise correction. IEEE Transactions on Neural Networks and Learning Systems. [Online]. 29(5), pp. 1675–1688. Available: https://doi.org/10.1109/TNNLS.2017.2670702

[5] C. Li, V. S. Sheng, L. Jiang, and H. Li. (2016, Sep.). Noise filtering to improve data and model quality for crowdsourcing. Knowledge-Based Systems. [Online]. 107, pp. 96–103. Available: https://doi.org/10.1016/j.knosys.2016.06.006

[6] W. Tang and M. Lease. (2011, Jul.). Semi-supervised consensus labeling for crowdsourcing. In SIGIR 2011 Workshop on Crowdsourcing for Information Retrieval (CIR), pp. 1–6. [Online]. Available: https://mattlease.com/papers/tang-cir11.pdf

[7] M. Khodabakhsh, F. Jafarinejad, M. Rahimi, and M. Ghayoomi. (2023). PerBOLD: A Big Dataset of Persian Offensive Language on Instagram Comments. Tabriz Journal of Electrical Engineering. [Online]. 53(2), pp. 149-158. Available: https://doi.org/10.22034/tjee.2023.15794

[8] M. S. Jahan and M. Oussalah. (2023, Aug.). A systematic review of hate speech automatic detection using natural language processing. Neurocomputing. [Online]. 546, pp. 126232. Available: https://doi.org/10.1016/j.neucom.2023.126232.

[9] Z. Talat and D. Hovy. (2016, Jun.). Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pp. 88–93. [Online]. Available: https://aclanthology.org/N16-2013.pdf

[10] L. Gao and R. Huang. (2017, Oct.). Detecting online hate speech using context aware models. arXiv preprint arXiv:1710.07395. [Online]. Available: https://doi.org/10.26615/978-954-452-049-6_036

[11] W. Warner and J. Hirschberg. (2012, Jun.). Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, pp. 19–26. [Online]. Available: https://aclanthology.org/W12-2103.pdf

[12] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. (2019, Mar.). Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. [Online]. Available: https://doi.org/10.48550/arXiv.1903.08983

[13] P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis. (2012, Apr.). Max algorithms in crowdsourcing environments. In Proceedings of the 21st International Conference on World Wide Web, pp. 989–998. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/2187836.2187969

[14] S. S. Bhatti, X. Gao, and G. Chen. (2020, Sep.). General Framework, Opportunities and Challenges for Crowdsourcing Techniques: A Comprehensive Survey. Journal of Systems and Software. [Online].167, p. 110611. Available: https://doi.org/10.1016/j.jss.2020.110611

[15] Y. Sang and J. Stanton. (2022, Feb.). The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation. In International Conference on Information, pp. 425–444. Cham: Springer International Publishing. [Online]. Available: https://doi.org/10.1007/978-3-030-96957-8_36

[16] N. Meedin, M. Caldera, S. Perera, and I. Perera. (2022). A novel annotation scheme to generate hate speech corpus through crowdsourcing and active learning. International Journal of Advanced Computer Science and Applications. [Online]. 13(11). Available: https://www.academia.edu

[17] F. Poletto, M. Stranisci, M. Sanguinetti, V. Patti, and C. Bosco. (2017). Hate speech annotation: Analysis of an Italian Twitter corpus. CEUR Workshop Proceedings, 2006. [Online]. Available: https://doi.org/10.4000/books.aaccademia.2448

[18] A. Sellars. (2016). Defining Hate Speech. Berkman Klein Center Research Publication 2016-20. [Online]. pp.16-48. Available: https://doi.org/10.2139/ssrn.2882244

[19] Z. Delbari, N. S. Moosavi, and M. T. Pilehvar. (2024). Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 17889–17897. [Online]. Available: https://doi.org/10.1609/aaai.v38i16.29743

[20] A. Safaya, M. Abdullatif, and D. Yuret. (2020, Jul.). Kuisail at SemEval-2020 Task 12: BERT-CNN for offensive speech identification in social media. arXiv preprint arXiv:2007.13184. [Online]. Available: https://doi.org/10.48550/arXiv.2007.13184

[21] A. Al-Hassan and H. Al-Dossari. (2022, Dec.). Detection of hate speech in Arabic tweets using deep learning. Multimedia Systems. [Online]. 28(6), pp. 1963–1974. [Online]. Available: https://doi.org/10.1007/s00530-020-00742-w

[22] M. Z. Ali, S. Rauf, K. Javed, and S. Hussain. (2021, Jun.). Improving hate speech detection of Urdu tweets using sentiment analysis. IEEE Access.[Online]. 9, pp. 84296–84305. Available: https://doi.org/10.1109/ACCESS.2021.3087827

[23] M. Safayani, A. Sartipi, A. H. Ahmadi, P. Jalali, A. H. Mansouri, M. Bisheh-Niasar, and Z. Pourbahman. (2024, Apr.). OPSD: an Offensive Persian Social Media Dataset and its baseline evaluations. arXiv preprint arXiv:2404.05540. [Online]. Available: https://doi.org/10.48550/arXiv.2404.05540

[24] M. K. Sheykhlan, J. Shafi, and S. Kosari. (2023, Oct.). Pars-HAO: Hate speech and offensive language detection on Persian social media using ensemble learning. Authorea Preprints. [Online]. Available: https://doi.org/10.36227/techrxiv.24106617.v2

[25] W. T. Tufa, I. Markov, and P. Vossen. (2024, Jun.). Grounding Toxicity in Real-World Events Across Languages. In International Conference on Applications of Natural Language to Information Systems, pp. 197–210. Cham: Springer Nature Switzerland. [Online]. Available: https://doi.org/10.1007/978-3-031-70239-6_14

[26] Ç. Çöltekin. (2020, May.). A corpus of Turkish offensive language on social media. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6174–6184. [Online]. Available: https://aclanthology.org/2020.lrec-1.758/

[27] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. (2019). Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1415–1420. [Online]. Available: https://doi.org/10.48550/arXiv.1902.09666

[28] H. Mubarak, A. Rashed, K. Darwish, Y. Samih, and A. Abdelali. (2020, Apr.). Arabic offensive language on Twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. [Online]. Available: https://doi.org/10.48550/arXiv.2004.02192

[29] Z. Pitenis, M. Zampieri, and T. Ranasinghe. (2020, Mar.). Offensive anguage identification in Greek. arXiv preprint arXiv:2003.07459. [Online]. Available: https://arxiv.org/abs/2003.07459

[30] L. Quijano-Sanchez, J. C. P. Kohatsu, F. Liberatore, and M. Camacho-Collados. (2019). HaterNet: A system for detecting and analyzing hate speech in Twitter. Zenodo. [Online]. Available: https://doi.org/10.5281/zenodo.2592149

[31] P. K. Roy, A. K. Tripathy, T. K. Das, and X. Gao. (2020). A Framework for Hate Speech Detection Using Deep Convolutional Neural Network. IEEE Access, pp. 204951–204962. [Online]. Available: https://doi.org/10.1109/ACCESS.2020.3037073

[32] G. I. Sigurbergsson and L. Derczynski. (2019, Aug.). Offensive language and hate speech detection for Danish. arXiv preprint arXiv:1908.04531. [Online]. Available: https://doi.org/10.48550/arXiv.1908.04531

[33] M. Mozafari, S. Member, and R. Farahbakhsh. (2022). Cross-Lingual Few-Shot Hate Speech and Offensive Language Detection Using Meta Learning. IEEE Access. [Online]. 10, pp. 14880–14896. Available: https://doi.org/10.1109/ACCESS.2022.3147588

[34] Y. Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. [Online]. 364(1). Available: http://arxiv.org/abs/1907.11692

[35] M. Farahani, M. Gharachorloo, M. Farahani, and M. Manthouri. (2021, Dec.). ParsBERT: Transformer-based model for Persian language understanding. Neural Processing Letters. [Online]. 53, pp. 3831–3847. Available: https://doi.org/10.1007/s11063-021-10528-4