Joint Event Extraction with Contextualized Word Embeddings for the Portuguese Language | Springer Nature Link (formerly SpringerLink) Skip to main content Advertisement Log in Menu Find a journal Publish with us Track your research Search Cart Home Intelligent Systems Conference paper Joint Event Extraction with Contextualized Word Embeddings for the Portuguese Language Conference paper First Online: 28 November 2021 pp 496–510 Cite this conference paper Access provided by University of Notre Dame Hesburgh Library Download book PDF Download book EPUB Intelligent Systems (BRACIS 2021) Joint Event Extraction with Contextualized Word Embeddings for the Portuguese Language Download book PDF Download book EPUB Anderson da Silva Brito Sacramento  ORCID: orcid.org/0000-0002-2288-689910 & Marlo Souza  ORCID: orcid.org/0000-0002-5373-727110  Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13074)) Included in the following conference series: Brazilian Conference on Intelligent Systems 1279 Accesses 2 Citations Abstract Event Extraction (EE) is the task of identifying mentions of particular event types and their arguments in text, and it constitutes an important and challenging task within the area of Information Extraction (IE). However, in the context of the Portuguese language, very little work has been conducted on this topic. In this paper, we propose a neural-based method for EE, as well as a data resource to mitigate this research gap. We also present a data augmentation strategy for EE, employing an Open Information Extraction (OIE) system, aiming to overcome the shortage in annotated data for the problem in the Portuguese language. Our experimental results show that our method is able to predict event types and arguments automatically, and the proposed method of data augmentation, in one of the two evaluated samples, contributes to the performance of the tested models in the subtask of argument role prediction. Further, an implementation of our method is available to the community, as the models trained in our experiments (https://github.com/FORMAS/TEFE). Access provided by University of Notre Dame Hesburgh Library. Download conference paper PDF Similar content being viewed by others Event Extraction for Portuguese: A QA-Driven Approach Using ACE-2005 Chapter © 2023 DptOIE: a Portuguese open information extraction based on dependency analysis Article 05 December 2022 A New Entity Extraction Model Based on Journalistic Brazilian Portuguese Language to Enhance Named Entity Recognition Chapter © 2020 Explore related subjects Discover the latest articles, books and news in related subjects, suggested using machine learning. Computational Linguistics Corpus Linguistics Language Processing Machine Translation Natural Language Processing (NLP) Sequence Annotation 1 Introduction Event Extraction (EE) is an important and challenging task of Information Extraction (IE) in the field of Natural Language Processing (NLP). The task comprises identifying the mentions of particular event types and their arguments, i.e. mentions to entities participating in the event, and the roles of these arguments in relation to an event type. The term event has different, definitions in the literature [6, 23]. While events are undeniably temporal entities, they also may possess a rich non-temporal structure that is important for intelligent information access systems [1]. In this work, we consider events as things that happen or occur involving participants and attributes, most notably spatio-temporal attributes. To be more precise, we adopt the definition presented in the ACE 2005 annotation guidelines [6]: an event is a specific occurrence involving participants; it is something that happens and can frequently be described as a change of state. Based on the event annotation guideline ACE 2005 [6], and related literature on EE [18, 20], EE can be divided into two main subtasks: Event Detection (ED), i.e. the task of identifying and classifying event triggers, and Argument Role Prediction (ARP), i.e. the task of identifying the arguments of an even and labeling their roles. In this context, an event trigger consists of an expression denoting the occurrence of the event and each event mention in a sentence is identified by a trigger term. An event trigger may be expressed primarily through verbs and nominalizations but also by other word classes such as adjectives and prepositions. As such, one of the challenging aspects of the EE task is that the same trigger term can express different events in different contexts. That is, trigger words may be ambiguous. To illustrate, consider the following sentences: \(s_1\): “... os árabes teriam de apoiar o Iraque numa luta contra o seu inimigo comum israelita.”Footnote 1 \(s_2\): “Ambas as empresas são aliadas da Navigation Mixte na luta contra uma OPA hostil ...”Footnote 2 Both of \(s_1\) and \(s_2\) contain a word, “luta” (fight), functioning as an event trigger. However, in s1, “luta” means a hostile encounter between opposing forces, which denotes a Hostile encounter event. In s2, “luta” means a dispute between parties with incompatible opinions, and denotes an event of type Quarreling [2]. On the other hand, the same event can be described by various different expressions, by means of different trigger words. For example, we observe in the annotated TimeBankPT Event Frame Annotation (TEFA) corpus that the following trigger words denote an occurrence of an event of type Statement: “dito”, “disse”, “referiu”, “anunciado”, “acrescentou”, etc.Footnote 3 Furthermore, when dealing with multiple event mentions in the same sentence, it is possible that different events share arguments with different roles. Consider the following sentence: “A Meridian National Corp. disse que vendeu 750.00 ações oridinárias ao grupo da família McAlpine, por 1 milhão de dólares, ou 1,35 dólares por ação.”Footnote 4 the entity mention “Meridian National Corp.” takes two different (and not related) argument roles: Speaker and Seller, respectively, in the events Statement and Commerce_sell denoted by the event triggers “disse” (said) and “vendeu” (sold). Event mentions in natural language text are ubiquitously present in different domains and genres and encompass a great amount of information encoded in a sentence. As noted by Ahn [1], solutions for EE can benefit the development of other NLP applications, such as text summarization, question answering and information retrieval. Notably, there are two variations of the task of EE regarding the types of target event that are considered [30]: open domain, when no schema and list of target event types are defined for the task; and closed domain, when the set of target event types and their respective schemata are pre-defined. In this paper, we will discuss a method for closed domain EE for the Portuguese language. We developed a method to identify and classify a closed set of event types whose arguments and their roles were previously specified based on machine learning. As there is no publicly available dataset and corpus for the task of closed domain EE for the Portuguese language that we are aware, we built a dataset by enriching the TimeBankPT [8] corpus with event annotation schemata from the FrameNet project [2]. TimeBankPT is the first corpus of Portuguese with rich temporal annotations, i.e. it includes annotations not only of temporal expressions but also about events and temporal relations [8]. In this work, we propose a method to the task of EE developed and evaluated over a newly created data set with event types and argument roles annotations, labeled over the TimeBankPT corpus, from which we maintain only the event trigger annotations. Our method jointly performs ED and ARP by learning a shared intermediary representation for the ED and ARP subtasks. Our main contributions are: (1) We propose a novel method for the EE task, simultaneously tackles the ED and ARP subtasks. (2) We demonstrate the usefulness of the application of OIE systems as a strategy to data augmentation for EE annotated sentences. (3) We have conducted experiments on a proposed enriched TimeBankPT corpus, achieving modest performance with low feature engineering effort. 2 Related Work Most work on the literature on EE have focused on languages such as English and Chinese, for which there are standard corpora for the task [30], such as the ACE 2005 corpus [27]. Regarding the approaches adopted in the literature, there are two main categorizations with respect to the input features and the modularization of the methods, respectively divided between discriminative featured-based [1, 15] and representation learning-based methods [10, 17], and the pipeline [15] versus the joint approach [20]. Recently, however, methods based on the application of deep neural networks, more specifically, the use of representation learning to address the task of EE [4, 19], have achieved good performance improvement for the task compared to earlier feature-based approaches and, as such, we will focus on a neural-based approach in this work. While ED is a much well-established task, if we consider the ARP subtask, very few works on the literature focused on both argument identification and argument role labeling [20]. Most of the related work assumes that argument identification is a previously resolved problem and focus only on the task role classification. Among the work on argument identification and role labeling, the works of Nguyen et al.  [18, 20] propose the use of recurrent neural networks (RNNs) for the ARP, achieving good performance for the task. These works will be the main inspiration for our method. It is also important to notice that the use of contextualized word embeddings is also a recent development [26]. To our knowledge, very little work has been done on event extraction for the Portuguese language. Notably, this task has been addressed in a limited scope in the HAREM [3] named entity recognition evaluation, which included temporal named entities of the type event. From a more general perspective, the work of Costa and Branco [7] and Quaresma [22] are some of the only works on the literature on this problem for the Portuguese language, to our knowledge. Costa e Branco [7] proposed a method for event identification and classification using an approach based on feature engineering and a decision tree trained over the TimeBankPT corpus. The corpus, however, only contains annotation on a very broad event classification typology, e.g. REPORTING, OCCURRENCE, STATE, I_STATE, and I_ACTION, and does not contain annotation on event structure, i.e. its arguments and their roles in the event. Quaresma et al. [22] focus on the task of event extraction proposing an algorithm that relies on the output of a semantic role labeling (SRL) system. In doing so, the authors assumed that only predicates are event trigger candidates. In their work, the authors did not classify events by type, and they also limited the classification of event arguments, considering only the roles provided by SRL schemas. Unfortunately, the data set provided by the authorsFootnote 5 contains only the information on trigger output predictions of their systems, without a gold reference that would allow comparison at least for event identification. 3 Task Definition In this work, we are concerned with the task of Event Extraction for the Portuguese language. Throughout this work, we will employ the following terminology: Event mention: a phrase or sentence in which an event occurs, including one trigger and an arbitrary number of arguments. Event trigger: the main word that most clearly expresses an event occurrence. Event argument: an entity mention, temporal expression or value (e.g. Money) that servers as a participant or attribute with a specific role in an event mention. Argument role: the relationship between an argument to the event in which it participates. The corpus that we employ in our experiments contains 514 event types (e.g., Statement, Commerce_buy, Firing) that correspond to the types of event triggers associated with each event mention. Each event type has its own set of roles that can be performed by one of the entities participating in the event, i.e. its arguments. For instance, some of the roles of the event type Statement are Speaker, Message, Topic, Addresse and Time. The total number of roles for all the event types on the annotated corpus is 1936. Following this terminology, we explicitly define the standard evaluation procedures as follows, as presented by [30]: Trigger identification: A trigger is correctly detected if its offsets (viz., the position of the trigger word in the text) match a reference trigger Trigger classification: An event type is correctly classified if both the trigger’s offset and event type match a reference trigger and its event type. Argument identification: An argument is correctly identified if its offsets match any of the reference argument mentions (viz., correctly recognizing participants in an event) Argument role classification: An argument role is correctly classified if its event type, offsets, and role match any of the reference argument mentions. 4 Model We propose a model that jointly performs ED and ARP at the sentence level. As such, let \(W = \langle w_1, w_2, ..., w_n \rangle \) be a sentence, where n is the number of words/tokens and \(w_i\) is its i-th token. We will model the ED problem, as a multi-class word classification problem, following prior works [10, 16, 26] in assuming that event triggers are single words/tokens in the sentences. This definition leads to a sequence classification problem on the input sentence W, in which we predict the event type \(t_i\) for each \(w_i \in W\) - where \(t_i\) can be “None” to indicate the word \(w_i\) is not a trigger, or it is not any of the target event types. It is assumed that a trigger candidate cannot refer to more than one event mentioned in the same sentence. For event argument identification and role prediction, we need to recognize the entity mentions that act as an argument for each of the event mentions occurring in W (argument identification) and classify their associated role. So, for the sake of explanation, considering that each sentence has only one event mention, this task is cast as a sequence labeling task in which for each word/token in the sentence, we predict a sequence of labels \(E = e_1, e_2, ..., e_n\). In summary, for each trigger candidate word \(w_i\) we must predict the corresponding argument role labels for each word/token in W. An illustration of the architecture of our model with the joint prediction of event mentions and their arguments is depicted in Fig. 1. Fig. 1. The joint EE model for the two subtasks with an input sentence. “subiram” (risen) is the current trigger word candidate, and all the other word tokens are transformed into vector representations to predict their roles given the current trigger candidate. Full size image 4.1 Trigger and Sentence Encoding In the component of trigger encoding, every trigger candidate word \(w_i \in W\) is transformed into a word vector \(c_i\) using the pre-trained word embedding \(d_i\) of \(w_i\). We employ the BERTimbau model [25], a transformer model trained over Portuguese language data, for obtaining these representations. For the sentence encoding step, we only use the respective word embedding for each word/token of the sentence. Since BERTimbau applies WordPiece [28] tokenizations, possibly generating subword units, we choose to compose the respective word embeddings by summing the vector representation of each word’s subword units. 4.2 Sentence Representation After the sentence encoding step, the input sentence W is transformer in a sequence of word embedding vectors \(X = \langle x_1, x_2, ..., x_n\rangle \). To produce the sentence representation, we feed the sequence X into a bidirecional recurrent neural network [24], which produces the hidden representation vector for each input vector \(x_t\) from X. We obtain the sequence \(H = \langle [\overrightarrow{h_1};\overleftarrow{h_1}], [\overrightarrow{h_2};\overleftarrow{h_2}], \cdots , [\overrightarrow{h_n};\overleftarrow{h_n}]\rangle \) of two hidden states at each time-step t, one for the left-to-right propagation \(\overrightarrow{h_t}\) and another for the right-to-left propagation \(\overleftarrow{h_t}\). At each step t, we employ a LSTM network [13] that accepts a current input \(x_t\) and a previous hidden state \(h_{t-1}\) to compute the current hidden state \(h_t = LSTM(x_t, h_{t-1})\). 4.3 Trigger Representation For the trigger representation, we feed the word embedding vector of the current trigger candidate \(c_i\) into a feed-forward layer \(FF(c_i)\). The feature representation from this layer is shared between the two next prediction states to perform ED and ARP simultaneously. The value of this layer activations is computed as: \({R_i^{ED}} = ReLU(W_{tr}c_i + b_{tr})\) where \(W_{tr}\) is a weight matrix and \(b_{tr}\) a bias term. 4.4 Event Detection and Argument Role Prediction The predictions of the event trigger classifier and argument role classifier components are done sequentially over the sentence from left to right. At the current word/step i, we attempt to compute the probability $$P(a_i,t_i| \varGamma ) = P(t_i|\varGamma ) P(a_{i,1}|\varGamma ,t_i) P(a_{i,2}|\varGamma ,a_{i,<1},t_i) \cdots P(a_{i,n}|\varGamma ,a_{i,