key: cord-020832-iavwkdpr authors: Nguyen, Dat Quoc; Zhai, Zenan; Yoshikawa, Hiyori; Fang, Biaoyan; Druckenbrodt, Christian; Thorne, Camilo; Hoessel, Ralph; Akhondi, Saber A.; Cohn, Trevor; Baldwin, Timothy; Verspoor, Karin title: ChEMU: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents date: 2020-03-24 journal: Advances in Information Retrieval DOI: 10.1007/978-3-030-45442-5_74 sha: doc_id: 20832 cord_uid: iavwkdpr We introduce a new evaluation lab named ChEMU (Cheminformatics Elsevier Melbourne University), part of the 11th Conference and Labs of the Evaluation Forum (CLEF-2020). ChEMU involves two key information extraction tasks over chemical reactions from patents. Task 1—Named entity recognition—involves identifying chemical compounds as well as their types in context, i.e., to assign the label of a chemical compound according to the role which the compound plays within a chemical reaction. Task 2—Event extraction over chemical reactions—involves event trigger detection and argument recognition. We briefly present the motivations and goals of the ChEMU tasks, as well as resources and evaluation methodology. The chemical industry undoubtedly depends on the discovery of new chemical compounds. However, new chemical compounds are often initially disclosed in patent documents, and only a small fraction of these compounds are published in journals, usually taking an additional 1-3 years after the patent [13] . Therefore, most chemical compounds are only available through patent documents [3] . In addition, chemical patent documents contain unique information, such as reactions, experimental conditions, mode of action, which is essential for the understanding of compound prior art, providing a means for novelty checking and validation as well as pointers for chemical research in both academia and industry [1, 2] . As the number of new chemical patent applications has been drastically increasing [11] , it is becoming crucial to develop natural language processing (NLP) approaches that enable automatic extraction of key information from the chemical patents [2] . In this paper, we propose a new evaluation lab (called ChEMU) focusing on information extraction over chemical reactions from patents. In particular, we will focus on two key information extraction tasks of chemical named entity recognition (NER) and chemical reaction event extraction. While previous related shared tasks focusing on chemicals or drugs such as CHEMDNER [7] have also included chemical named entity recognition as a task, those have primarily focused on PubMed abstracts. The CHEMDNER patents task [8] was limited to entity mentions and chemical entity passage detection, and only considered titles and abstracts of patents. For our ChEMU lab, we extend the existing corpora in several directions: first, we go beyond chemical NER to require labeling of the role of a chemical with respect to a reaction, and to consider complete chemical reactions in addition to entities. The ChEMU website is available at: http:// chemu.eng.unimelb.edu.au. What are the Goals of This Evaluation Lab? Our goals are: (1) To develop tasks that impact chemical research in both academia and industry, (2) To provide the community with a new dataset of chemical entities, enriched with relational links between chemical event triggers and arguments, and (3) To advance the state-of-the-art in information extraction over chemical patents. Why is This Lab Needed? For evaluating information extraction developments in the scientific literature domain, there have been a large number of labs/shared tasks offered within previous i2b2/n2c2, SemEval, BioNLP, BioCreative, TREC and CLEF workshops. However, less attention has been paid to the chemical patent domain. In particular, there has previously been only one shared task on this domain, which is the CHEMDNER patents task at the BioCreative V workshop, involving detection of mentions of chemical compounds and genes/proteins in patent text [8] . Information extraction approaches developed for the scientific literature domain may not apply directly to the chemical patent domain. This is because as legal documents, patents are written very differently as compared to scientific literature. When writing scientific papers, authors strive to make their words as clear and straightforward as possible, whereas patent authors often seek to protect their knowledge from being fully disclosed [15] . In tension with this is the need to claim broad scope for intellectual property reasons, and hence patents typically contain more details and are more exhaustive than scientific papers [9] . There are also a number of characteristics of patent texts that create challenges for NLP in this context. Long sentences listing names of compounds are frequently used in chemical patents. The structure of sentences in patent claims Table 1 . Brief definitions of ChEMU chemical entity types, organised into chemical entity types, a reaction label introduced in the text, and reaction properties. REACTION PRODUCT A product is a substance that is formed during a chemical reaction STARTING MATERIAL A substance that is consumed in the course of a chemical reaction providing atoms to products is considered as starting material REAGENT CATALYST A reagent is a compound added to a system to cause or help with a chemical reaction. Compounds like catalysts, bases to remove protons or acids to add protons must be also annotated with this tag SOLVENT A solvent is a chemical entity that dissolves a solute resulting in a solution OTHER COMPOUND Other chemical compounds that are not the products, starting materials, reagents, catalysts and solvents EXAMPLE LABEL A label associated with a reaction specification TEMPERATURE The temperature at which the reaction was carried out must be annotated with this tag TIME The reaction time of the reaction YIELD PERCENT Yield given in percent values YIELD OTHER Yields provided in other units than % is usually complex, and syntactic parsing in patents can be difficult [4] . A quantitative analysis from [16] showed that the average sentence length in a patent corpus is much longer than in general language use. That work also showed that the lexicon used in patents usually includes domain-specific and novel terms that are difficult to understand. The ChEMU lab will provide a new challenging set of tasks, in an area of significant pharmacological importance. The lab will focus attention on more complex analysis of chemical patents, provide strong baselines as well as providing a useful resource for future research. What are Usage Scenarios? Automatically identifying compounds which serve as the starting material or are a product of a chemical reaction would allow more targeted extraction of chemical information from patents and can improve the usefulness of patent resources. Automatic extraction of chemical reaction events supports the construction of cheminformatics databases, capturing key information about chemicals and how they are produced, from the patent resources. The ChEMU lab at CLEF-2020 1 offers the two information extraction tasks of Named entity recognition (Task 1) and Event extraction (Task 2) over chemical reactions from patent documents. Teams may participate in one or both tasks. In general, a chemical reaction is a process leading to the transformation of one set of chemical substances to another [10] . Task 1 involves identifying chemical compounds and their specific types, i.e. to assign the label of a chemical compound according to the role which it plays within a chemical reaction. In addition to chemical compounds, this task also requires identification of the temperatures and reaction times at which the chemical reaction is carried out, as well as yields obtained for the final chemical product and the label of the reaction. This task involves both entity boundary prediction and entity label classification. We define 10 different entity type labels as shown in Table 1 . See examples of those entity types in Table 2 . As illustrated in Figs. 1 and 2 , a chemical reaction leading to an end product often consists of a sequence of individual event steps. Task 2 is to identify those steps which involve chemical entities recognized from Task 1. Unlike a conventional event extraction problem [6] which involves event trigger word detection, event typing and argument prediction, our Task 2 requires identification of event trigger words (e.g. "added" and "stirred") which all have the same type of Fig. 2 . BRAT visualization of a more complex event with the trigger word "added" involving five arguments. "EVENT TRIGGER", and then determination of the chemical entity arguments of these events. 2 When predicting event arguments, we adapt semantic argument role labels Arg1 and ArgM from the Proposition Bank [12] to label the relations between the trigger words and the chemical entities: Arg1 is used to label the relation between an event trigger word and a chemical compound. Here, Arg1 represents argument roles of being causally affected by another participant in the event [5] . ArgM represents adjunct roles with respect to an event, used to label the relation between a trigger word and a temperature, time or yield entity. An end-to-end process incorporating both Task 1 and Task 2 can be equivalently viewed as a relation extraction task which identifies 11 entity types including 10 types defined in Table 1 plus "EVENT TRIGGER", and extracts relations between the "EVENT TRIGGER" entities and the remaining entities. Data: For system development and evaluation, a new corpus of 1500 chemical reaction snippets will be provided for both tasks (an example of a chemical reaction snippet is shown in Table 2 ). These snippets are sampled from 170 English document patents from the European Patent Office and the United States Patent and Trademark Office. We will mark up every chemical compound or event trigger with both text spans and IDs, and highlight relations and event arguments, as illustrated in Figs. 1 and 2. We have begun preparing the corpus and will make available strong baselines for the tasks. Initial publications related to the data and Task 1 appear at the 2019 ALTA and BioNLP workshops, respectively [18, 19] . The corpus will be split into 70%/10%/20% training/development/test. Gold annotations for the training and development sets will be provided to task participants in the BRAT standoff format [14] during the development phase. The raw test set will be provided for final test phase. To support teams who are interested in Task 2 only, a pre-trained chemical NER tagger is provided as a resource [19] . Evaluation: For evaluation, precision, recall and F1 scores will be used, under both strict and relaxed span matching conditions. F1 will be the main metric for ranking the participating teams [17] . 3 In this paper, we have presented a brief description of the upcoming ChEMU lab at CLEF-2020. ChEMU will focus on two new tasks of named entity recognition and event extraction over chemical reactions from patents. We expect participants from both academia and industry. We will advertise our ChEMU lab via social media as well as NLP-related mailing lists. Annotated chemical patent corpus: a gold standard for text mining Automatic identification of relevant chemical compounds from patents Patents: a unique source for scientific technical information in chemistry related industry? World Pat Improving automated patent claim parsing: dataset, system, and experiments Semantic Role Labeling and Argument Structure. In: Speech and Language Processing Overview of BioNLP'09 shared task on event extraction CHEMDNER: the drugs and chemical names extraction challenge Overview of the CHEMDNER patents task Current Challenges in Patent Information Retrieval, 1st edn Glossary of terms used in physical organic chemistry (IUPAC Recommendations 1994) Making every SAR point count: the development of chemistry connect for the large-scale integration of structure and bioactivity data The proposition bank: an annotated corpus of semantic roles Managing expectations: assessment of chemistry databases generated by automated extraction of chemical structures from patents brat: a web-based tool for NLP-assisted text annotation Patents and scientific papers: quite different concepts: the reward is found in giving Quantifying the challenges in parsing patent claims Annotating the biomedical literature for the human variome Detecting chemical reactions in patents Improving chemical named entity recognition in patents with contextualized word embeddings Acknowledgments. This work is supported by an Australian Research Council Linkage Project, LP160101469, and Elsevier. We would like to thank Estrid He, Zubair Afzal and Mark Sheehan for supporting this work, as well as the anonymous reviewers for their feedback.