A Hybrid Entity-Centric Approach to Persian Pronoun Resolution with BERT Enhancement
محورهای موضوعی : Natural Language Processing
Hassan Haji Mohammadi
1
,
alizera talebpour
2
,
Ahmad Mahmoudi-Aznaveh
3
,
Samaneh Yazdani
4
1 - Department of Electrical and Computer Engineering, Islamic Azad University Tehran North Branch, Tehran, Iran
2 - shahid Beheshti
3 - Cyberspace research institute, Shahid Beheshti University, Tehran, Iran
4 - Department of Electrical and Computer Engineering, Islamic Azad University Tehran North Branch, Tehran, Iran
کلید واژه: Coreference Resolution, Natural Language Processing, Pronoun Resolution, Corpus Development, BERT, Transformer Models,
چکیده مقاله :
Pronoun resolution, a critical aspect of coreference resolution in natural language processing, is essential for accurate text understanding and generation. This paper introduces a hybrid Persian pronoun resolution model that combines rule-based and machine-learning approaches to address the unique challenges of Persian linguistics. The model integrates seven high-precision rule-based modules, specifically designed for Persian linguistic nuances, with a random forest classifier enhanced by BERT contextual embeddings and cluster-level features. This hybrid approach effectively tackles the complexities of pronoun resolution, outperforming existing Persian systems on two corpora: the newly developed Mehr corpus (400 documents) and the established RCDAT corpus. The integration of BERT embeddings and cluster-level features plays a pivotal role in the model's superior performance, yielding a 4.68 F1-score improvement on the Mehr corpus and a 4.01 F1-score improvement on the RCDAT corpus. The proposed model not only achieves state-of-the-art results but also provides a practical and interpretable solution for Persian pronoun resolution. By leveraging both rule-based and machine-learning techniques, this work advances the field of Persian natural language processing, offering a robust framework for future research and applications. The findings highlight the importance of combining linguistic rules with advanced machine-learning methods to address the intricacies of pronoun resolution in morphologically rich languages like Persian. This study contributes significantly to the growing body of research on Persian NLP, paving the way for improved text understanding and generation in Persian.
1. Lee, K., et al. End-to-end Neural Coreference Resolution. in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. Association for Computational Linguistics.
2. Ng, V. and C. Cardie. Improving machine learning approaches to coreference resolution. in Proceedings of the 40th annual meeting on association for computational linguistics. 2002. Association for Computational Linguistics.
3. Bahdanau, D., K.H. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. in 3rd International Conference on Learning Representations, ICLR 2015. 2015.
4. Kriman, S. and H. Ji. Joint detection and coreference resolution of entities and events with document-level context aggregation. in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. 2021.
5. Liu, Z., K. Shi, and N. Chen. Coreference-Aware Dialogue Summarization. in Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2021.
6. Mao, R. and X. Li. Bridging towers of multi-task learning with a gating mechanism for aspect-based sentiment analysis and sequential metaphor identification. in Proceedings of the AAAI conference on artificial intelligence. 2021.
7. Vu, H.H., H. Kamigaito, and T. Watanabe, Context-Aware Machine Translation with Source Coreference Explanation. Transactions of the Association for Computational Linguistics, 2024. 12: p. 856-874.
8. Bohnet, B., C. Alberti, and M. Collins, Coreference Resolution through a seq2seq Transition-Based System. Transactions of the Association for Computational Linguistics, 2023. 11: p. 212-226.
9. Devlin, J., et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
10. Hürriyetoğlu, A., et al., Event Coreference Resolution for Contentious Politics Events. arXiv preprint arXiv:.10123, 2022.
11. Lu, Y., et al., End-to-end neural event coreference resolution. Artificial Intelligence, 2022. 303: p. 103632.
12. Yu, X., W. Yin, and D. Roth. Pairwise Representation Learning for Event Coreference. in Proceedings of the 11th Joint Conference on Lexical and Computational Semantics. 2022.
13. Zhang, W., S. Wiseman, and K. Stratos. Seq2seq is All You Need for Coreference Resolution. in The 2023 Conference on Empirical Methods in Natural Language Processing.
14. Wan, B., et al., A deep neural network model for coreference resolution in geological domain. Information Processing & Management, 2023. 60(3): p. 103268.
15. Žabokrtský, Z., et al. Findings of the second shared task on multilingual coreference resolution. in Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution. 2023.
16. Bi, M., et al., Machine reading comprehension combined with semantic dependency for Chinese zero pronoun resolution. Artificial Intelligence Review, 2023. 56(8): p. 7597-7612.
17. Demir, Ş., Neural coreference resolution for Turkish. Journal of Intelligent Systems: Theory and Applications, 2023. 6(1): p. 85-95.
18. Lu, J. and V. Ng. Learning antecedent structures for event coreference resolution. in 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA). 2017. IEEE.
19. Hobbs, J.R., Resolving pronoun references. Lingua, 1978. 44(4): p. 311-338.
20. Carbonell, J.G. and R.D. Brown. Anaphora resolution: a multi-strategy approach. in Proceedings of the 12th conference on Computational linguistics-Volume 1. 1988. Association for Computational Linguistics.
21. Lappin, S. and H. Leass, An algorithm for pronominal anaphora resolution. Computational linguistics, 1994. 20(4): p. 535-561.
22. Kennedy, C. and B. Boguraev. Anaphora for everyone: pronominal anaphora resoluation without a parser. in Proceedings of the 16th conference on Computational linguistics-Volume 1. 1996. Association for Computational Linguistics.
23. Stuckardt, R., Design and enhanced evaluation of a robust anaphor resolution algorithm. Computational Linguistics, 2001. 27(4): p. 479-506.
24. Haghighi, A. and D. Klein. Simple coreference resolution with rich syntactic and semantic features. in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3. 2009. Association for Computational Linguistics.
25. Raghunathan, K., et al. A multi-pass sieve for coreference resolution. in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. 2010. Association for Computational Linguistics.
26. Lee, H., et al. Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. in Proceedings of the fifteenth conference on computational natural language learning: Shared task. 2011. Association for Computational Linguistics.
27. Lee, H., et al., Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 2013. 39(4): p. 885-916.
28. van Cranenburgh, A., et al. A Hybrid Rule-Based and Neural Coreference Resolution System with an Evaluation on Dutch Literature. in Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference. 2021.
29. Lee, K., L. He, and L. Zettlemoyer. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.
30. Joshi, M., et al., Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 2020. 8: p. 64-77.
31. Zhang, H., et al., Knowledge-aware Pronoun Coreference Resolution. arXiv preprint arXiv:.03663, 2019.
32. Zhang, H., Y. Song, and Y. Song, Incorporating Context and External Knowledge for Pronoun Coreference Resolution. arXiv preprint arXiv:.10238, 2019.
33. Hou, F., et al., Anisotropic span embeddings and the negative impact of higher-order inference for coreference resolution: An empirical analysis. Natural Language Engineering, 2024: p. 1-22.
34. De Langhe, L., O. De Clercq, and V. Hoste, Constructing a cross-document event coreference corpus for Dutch. Language Resources and Evaluation, 2022: p. 1-30.
35. Rahimi, Z. and S. HosseinNejad, Corpus based coreference resolution for Farsi text Signal and Data Processing
2020. 17(1): p. 79-98.
36. Haji Mohammadi, H., et al., Mehr: A Persian Coreference Resolution Corpus. Journal of AI and Data Mining, 2023.
37. Biau, G. and E. Scornet, A random forest guided tour. Test, 2016. 25: p. 197-227.
38. Artari, V.K.P., et al. A Multi-Pass Sieve Coreference Resolution for Indonesian. in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). 2021.
39. Sahlani, H., M. Hourali, and B. Minaei-Bidgoli, Coreference resolution with deep learning in the persian language. Signal and Data Processing
2020.
40. Sahlani, H., M. Hourali, and B. Minaei-Bidgoli, Coreference Resolution Using Semantic Features and Fully Connected Neural Network in the Persian Language. International Journal of Computational Intelligence Systems, 2020.
41. Soon, W.M., H.T. Ng, and D.C.Y.J.C.l. Lim, A machine learning approach to coreference resolution of noun phrases. Computational linguistics, 2001. 27(4): p. 521-544.
42. Ng, V. and C. Cardie. Identifying anaphoric and non-anaphoric noun phrases to improve coreference resolution. in COLING 2002: The 19th International Conference on Computational Linguistics. 2002.
43. Denis, P. and J. Baldridge. Specialized models and ranking for coreference resolution. in Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2008. Association for Computational Linguistics.
44. Denis, P. and J. Baldridge, Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 2009. 42.
45. Wiseman, S.J., et al. Learning anaphoricity and antecedent ranking features for coreference resolution. in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015. Association for Computational Linguistics.
46. Lee, K., et al., End-to-end neural coreference resolution. arXiv preprint arXiv:.05365, 2017.
47. Luo, X., et al. A mention-synchronous coreference resolution algorithm based on the bell tree. in Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. 2004. Association for Computational Linguistics.
48. Nicolae, C. and G. Nicolae. Bestcut: A graph algorithm for coreference resolution. in Proceedings of the 2006 conference on empirical methods in natural language processing. 2006. Association for Computational Linguistics.
49. Culotta, A., M. Wick, and A. McCallum. First-order probabilistic models for coreference resolution. in Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference. 2007.
50. Lee, H., M. Surdeanu, and D. Jurafsky, A scaffolding approach to coreference resolution integrating statistical and rule-based models. Natural Language Engineering, 2017. 23(5): p. 733-762.
51. Lai, T.M., T. Bui, and D.S. Kim. End-to-end neural coreference resolution revisited: A simple yet effective baseline. in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2022. IEEE.
52. Zhang, R., et al., Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. arXiv preprint arXiv:1805.04893, 2018.
53. Joshi, M., et al. BERT for Coreference Resolution: Baselines and Analysis. in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
54. Kantor, B. and A. Globerson. Coreference resolution with entity equalization. in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
55. Werlen, L.M. and J. Henderson. Graph Refinement for Coreference Resolution. in Findings of the Association for Computational Linguistics: ACL 2022. 2022.
56. Zhang, W., S. Wiseman, and K. Stratos, Seq2seq is all you need for coreference resolution. arXiv preprint arXiv:2310.13774, 2023.
57. Guo, Q., et al. Dual Cache for Long Document Neural Coreference Resolution. in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
58. Moosavi, N.S. and G. Ghassem-Sani, A ranking approach to Persian pronoun resolution. Advances in Computational Linguistics. Research in Computing Science, 2009. 41: p. 169-180.
59. Denis, P. and J. Baldridge. A Ranking Approach to Pronoun Resolution. in IJCAI. 2007.
60. Lee, K., L. He, and L. Zettlemoyer, Higher-order coreference resolution with coarse-to-fine inference. arXiv preprint arXiv:.05392, 2018.
61. Chai, H. and M. Strube. Incorporating Centering Theory into Neural Coreference Resolution. in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
62. Miculicich, L. and J. Henderson, Graph Refinement for Coreference Resolution. arXiv preprint arXiv:.16574, 2022.
63. Wang, Y., Y. Shen, and H. Jin. An End-To-End Actor-Critic-Based Neural Coreference Resolution System. in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021. IEEE.
64. Hajimohammadi, H., A. Talebpour, and A. Mahmoudi Aznaveh. Using decision tree to Persian pronoun resolution. in Twenty-first annual National Conference of the Computer Society of Iran. 2016. Tehran: Computer Society of Iran
65. Nourbakhsh, A. and M. Bahrani. Persian Pronoun Resolution Using Data Driven Approaches. in International Conference on Information and Software Technologies. 2017. Springer.
66. Fallahi, F. and M. Shamsfard, Recognizing anaphora reference in Persian sentences. International Journal of Computer Science Issues (IJCSI), 2011. 8(2): p. 324.
67. Nazaridoust, M., B.M. Bidgoli, and S. Nazaridoust. Co-reference Resolution in Farsi Corpora. in In Advance Trends in Soft Computing: Proceedings of WCSC 2013. 2013. Cham: Springer International Publishing.
68. Asgari-Bidhendi, M., A. Hadian, and B. Minaei-Bidgoli, Farsbase: The persian knowledge graph. Semantic Web, 2019. 10(6): p. 1169-1196.
69. Bijankhan, M., The role of the corpus in writing a grammar: An introduction to a software. Iranian Journal of Linguistics, 2004. 19(2): p. 48-67.
70. Mirzaei, A. and P. Safari. Persian Discourse Treebank and coreference corpus. in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). 2018.
71. Yimam, S.M., et al. WebAnno: A flexible, web-based and visually supported system for distributed annotations. in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 2013.
72. Hirschman, L., MUC-7 coreference task definition, version 3.0. Proceedings of MUC-7,, 1997.
73. Bergsma, S. and D. Lin. Bootstrapping path-based pronoun resolution. in Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics. 2006.
74. Ji, H. and D. Lin. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person mention detection. in Proceedings of the 23rd Pacific Asia Conference on Language, Information and Computation, Volume 1. 2009.
75. Rasooli, M.S., M. Kouhestani, and A. Moloodi. Development of a Persian syntactic dependency treebank. in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013.
76. Seraji, M., B. Megyesi, and J. Nivre, Bootstrapping a Persian dependency treebank. Linguistic Issues in Language Technology, 2012. 7(18).