Journal of Machine Learning Research-- Microsoft Word Template Dialogue and Discourse 4 (2) (2013) 174-184 doi: 10.5087/dad.2013.208 ©2013 Deniz Zeyrek et al. Submitted 03/12; Accepted 12/12; Published online 08/13 Turkish Discourse Bank: Porting a Discourse Annotation Style to a Morphologically Rich Language Deniz Zeyrek DEZEYREK@METU.EDU.TR Middle East Technical University, Informatics Institute Dumlupinar Boulevard, No:1 06800, Ankara, Turkey Işın Demirşahin DISIN@METU.EDU.TR Middle East Technical University, Informatics Institute Ayışığı B. Sevdik Çallı ASEVDIK@GMAIL.COM Middle East Technical University, Informatics Institute Ruket Çakıcı RUKEN@CENG.METU.EDU.TR Middle East Technical University, Department of Computer Engineering Editors: Stefanie Dipper, Heike Zinsmeister, Bonnie Webber Abstract This paper describes the current state of the Turkish Discourse Bank, the first publicly available annotated discourse resource for Turkish. It describes the annotation methods and the challenges posed by annotating Turkish, a free word order language with rich morphology. It shows the usefulness of the PDTB style annotation but points out the need to expand this annotation style with the needs of the target language. Keywords: Turkish, discourse, discourse connectives, discourse annotation 1 Introduction Annotated corpora have come to play an important role both in theoretical linguistics and machine learning applications in natural language processing. There is a pressing need for such resources in Turkish, a free-word order, agglutinative language with rich morphology. There are existing syntactically enriched treebanks for Turkish (e.g., Oflazer et al., 2003) but the field also needs annotated discourse corpora to appeal to the need of researchers who are working with texts in their entirety rather than individual sentences. 1 Annotated discourse corpora allow opportunities to understand what kind of relationships hold among lexical, morphological and syntactic levels and the textual level. They provide an empirical ground for investigating a range of discourse issues and can reveal structures in discourse via various language technology applications, e.g., summarization, information extraction, sentiment analysis, essay analysis, etc. (cf. Webber et al., 2011). The Turkish Discourse Bank (TDB) is a ~400,000-word resource of modern written Turkish with various genres, mainly containing annotations of explicit discourse connectives and the discourse segments they relate. It shares the principles of the Penn Discourse Tree Bank (PDTB) 1 The METU-Sabancı Turkish Treebank described in (Oflazer, 2003; Say et al., 2004) is the widely-known syntactically annotated Treebank of Turkish. Since the annotated sentences of this corpus do not form entire texts, we chose not to create the TDB on it. TURKISH DISCOURSE BANK 175 and takes discourse connectives as discourse-level predicates with a binary argument structure (Prasad et al., 2007). The connective denotes relations between eventualities, fact-like objects and proposition-like objects (Asher, 1993). Following the PDTB, we refer to the arguments of a connective as the first argument (Arg1) and the second argument (Arg2). The second argument is the textual unit which syntactically or morphologically contains the connective, the other argument is conveniently referred to as Arg1. Arg2 is the “internal” unit, Arg1 the “external” unit in Stede & Heintze’s (2004) terminology. The complete list of the TDB tagset is given in Table 1 (Zeyrek et al., 2010). The definitions and examples of each category are provided in the rest of the paper while discussing the relevant methodological issues. Conn The connective head Arg1 First argument of the connective Arg2 Second argument of the connective Supp1 Supplement to the first argument Supp2 Supplement to the second argument Shared The subject, object or adverbial phrase shared by a relation Shared supp Supplement for the shared material Mod Modifier of the connective or the modifier of the relation Table 1. The annotation scheme of the TDB Only explicit connectives and their two arguments are annotated in the TDB. We plan to annotate implicit connectives and the sense of connectives at a later stage. We chose to annotate explicit connectives first because the primary aim of the project was to reveal explicit discourse connectives to allow further research in discourse coherence. In creating a bank of discourse based on connectives, we do not claim that discourse connectives are the only means establishing coherence; we merely take them as the basic elements of discourse which make coherence relations salient. The TDB 1.0 has been released in March 2011 along with a browser; it is being freely distributed to researchers upon request (www.medid.ii.metu.edu.tr). The TDB is built on certain principles shared by all annotated corpora (Marcus et al., 1994; Skut et al., 1997). Firstly, it is descriptive. It has a bottom-up approach, aiming to describe the basic characteristics of discourse by annotating discourse relations between segments. Examining discourse by breaking it up to its constituents is the core of almost all theoretical work on discourse, e.g. Asher & Lascarides (2003), Grosz & Sidner (1986), Mann & Thompson (1988), Moser & Moore (1996), Polanyi & Van Den Berg (1996). Secondly, the TDB is data-driven; i.e. the tagging scheme is meant to allow representations of various discourse phenomena, including, for example, shared and crossing arguments (Aktaş et al., 2010). Thirdly, it is theory-independent. It is not influenced by a particular discourse theory; i.e., there is not a correct way of annotating discourse relations given a specific theory, the only requirement is that similar structures should be annotated in the same way for consistency. In our earlier work, we discussed the evolving stages of the corpus (e.g., Demirşahin, et al. 2012b). In the present paper, we provide a more complete picture of the finalized annotation system and present statistical information about the connectives that proved challenging in the annotation process, namely discourse adverbials, subordinators and their polymorphous occurrences. The rest of the paper proceeds as follows: In Section 2, we describe the annotation cycle, methods, and challenges in porting an annotation system to Turkish. We focus on the annotation of discourse adverbials and how we annotate the discourse relations expressed by nominalizations. In Section 3, we explain the method of annotating phrasal expressions and how we decided to use the shared tag. We also discuss how we adapted the modifier and the supplementary tags of the PDTB. Finally, in Section 4 we conclude with a summary of the paper. ZEYREK, DEMİRŞAHİN, SEVDİK ÇALLI, AND ÇAKICI 176 2 Porting an annotation system to Turkish discourse In this section, we describe the annotation cycle, the annotation procedures and how we evaluate the annotation scheme. 2.1 The annotation cycle The annotation system used in creating the TDB involves the following steps.  A first draft of annotation guidelines is prepared and an initial set of explicit connectives is determined.  Three annotators annotate the whole corpus for the given set of connectives by determining their Arg1 and Arg2 spans, their supplementary materials and modifiers. They go through the whole data, skipping non-discourse usage of the connectives.  The annotated corpus is statistically analyzed.  The disagreements are determined and discussed interactively in agreement meetings, focusing on the disagreed cases. Two researchers and all three annotators participate in the meetings. With the researchers’ feedback, disagreements are resolved and an ‘agreed’ version is produced. The annotation guidelines are updated.  In the cases when the agreement meeting results in a modification of the annotation guidelines, all past annotations are reexamined through a process called ‘proof’ to ensure that they are in line with the latest version of the guidelines. The proofed annotations are the final version, the gold standard of the TDB.  The next set of connectives is annotated, and the cycle continues. Similar to English and many other languages, discourse connectives in Turkish can be identified from three major syntactic classes, namely, coordinating conjunctions (ve ‘and’, ya da ‘or’, ama ‘but’), subordinators (complex subordinators, e.g. için ‘for’, simplex subordinators, i.e. converbs, e.g. -IncA ‘when,’ –ken ‘while/now that’), and discourse adverbials (oysa ‘however’, öte yandan ‘on the other hand’, ayrıca ‘in addition/separately’). 2 The initial list of connectives was prepared on the basis of these syntactic classes. We excluded simplex subordinators, which we aim to annotate later. Then the researchers and the annotators discussed how one distinguishes between discourse and non-discourse usages of connectives, and where the arguments of a connective could be found. During a semester-long training period, the annotators were encouraged to form their own ideas about how discourse works. Next, the annotators studied the annotation tool and the guidelines and started annotating the initial set of connectives. The annotation tool was specifically devised for this project. Briefly, it uses the stand-off annotation methodology and produces XML files as annotation data (Aktaş et al., 2010). Neither the discourse relations nor the characteristics of the discourse segments (e.g. whether they should be full clauses or not) are spelled out in the annotation guidelines. This method was useful because it allowed the annotators to use their native-speaker intuitions in deciding about the syntactic type and the span of a connective’s arguments. Regarding the span of a connective’s argument, the annotators were only told to follow the “minimality principle”, which requires them to mark the shortest text spans that are necessary and sufficient to interpret a discourse relation encoded by the connective (Prasad et al., 2007). 2 The capital letters are used to capture the cases where a vowel agrees with the vowel harmony rules of the language. The letter I may be resolved as any of the high vowels in the language. The letter A may be resolved as e or a. TURKISH DISCOURSE BANK 177 2.2 The annotation procedures and inter-annotator agreement The annotators worked independently, as a group, or in a procedure we named pair annotation, adapted from pair programming (Demirşahin, et al., 2012a). In the group annotation method, one independent annotator produces a set of annotations, and the other two annotators go over this annotation set together, suggesting changes when necessary. Any disagreements are resolved in agreement meetings. On the other hand, the pair annotation procedure involves one annotator working independently and two annotators working together as a pair, producing two sets of independent annotations. The agreement of pair annotations is measured between the independent annotator and the pair of annotators, treating them as a single annotator. Of the total 8483 relations in the TDB 1.0, 3804 (44.84%) were annotated by three independent annotators, 3985 (46.98%) by pair annotation, and only 694 (8.18%) were annotated by group annotation. To evaluate the reliability of the argument span annotations produced by three independent annotators, or by a pair of annotators and an independent annotator, we measured agreement using Fleiss’ Kappa (Fleiss, 1971) described as: 3 measures the degree of agreement attainable above chance, and gives the degree of agreement actually attained above chance. Since each connective has two arguments, we calculated agreement over these text spans separately. For Arg1 and Arg2, we formed separate agreement tables similar to the table Fleiss uses (1971:379), where at least two annotators assign the words of a text into two categories (select/exclude) and we recorded the number of judgments a word receives for each category. We measured agreement over the spans identified by the annotators as the boundaries of the argument spans, which we took as the first and the last words of each argument selected by the annotators (Yalçınkaya, 2010). To evaluate supplementary material annotations, we used the exact match criterion (Miltsakaki et al., 2004). In this method, agreement for any supplementary material (Supp1 or Supp2) is recorded as 1 when all annotators make identical textual span selections, and 0 otherwise. Agreement is calculated by the number of exact matches found in the total number of annotations annotated by all annotators and given as a percentage (cf. Appendix A). 2.2.1 Discourse adverbials One of the most challenging issues in the annotation process was that of determining the location and span of the arguments of discourse adverbials. Table 2 provides discourse adverbials in the data and the Kappa measures of their first and second arguments, where applicable. The remaining discourse adverbials, for which Kappa statistics are not measured, are also provided for the sake of completeness (see footnote 5). According to Table 2, except for aslında, ‘in fact’, örneğin ‘for example’, mesela ‘to exemplify’ and böylece ‘thus’, the inter-annotator agreement measures for the first arguments of discourse adverbials are lower than the envisaged 0.80 threshold (Artstein and Poesio, 2008). 4 A preliminary analysis shows that these low agreement measures are largely due to the anaphoric characteristics of these connectives. Similar to anaphors whose antecedents can be ambiguous, the first arguments of discourse adverbials can also be ambiguous since their location is not constrained by adjacency (Forbes-Riley et al., 2006; Webber et al., 2003). The second arguments of discourse adverbials are mostly annotated with high agreement since their location is predictable by adjacency. The connective öte yandan ‘on the other hand’, however, yielded a low inter-annotator measure for its Arg2. Our exploratory 3 We did not measure the agreement on the discourse connective. 4 Such low agreement results are not surprising because agreement on the exact boundaries of a text span tend to be low in discourse, as discussed by Artstein & Poesio (2008:580-583). ZEYREK, DEMİRŞAHİN, SEVDİK ÇALLI, AND ÇAKICI 178 analyses show that this connective appears in argumentative texts and tends to link text spans which are more than one sentence long. Due to this, the annotators do not agree on the boundaries of Arg2; in particular, the right edge of Arg2 is a source of disagreement. Such disagreements are always resolved in agreement meetings. Kappa measures Search item (# of annotations) Gloss # of annotators Arg1 Arg2 aslında (81) in fact 2 0.81 0.85 ayrıca (108) in addition 3 0.66 0.84 böylece (85) thus 2 0.90 0.99 dahası (9) furthermore 3 0.71 0.90 gene de (26)* still 1 - - halbuki (17)* however 1 - - mesela (13) to exemplify 3 0.92 1.00 neticede (1) eventually 3 - - ne ki (14)* howbeit 1 - - ne var ki (32)* even so 1 - - oysa (136) however 3 0.78 0.91 örneğin (64) for example 3 0.87 0.92 örnek olarak (2) to illustrate 3 - - sonuçta (10) finally 3 0.70 0.87 sonuç olarak (5) as a result 3 0.67 1.00 söz gelimi (6)* for instance 1 - - taraftan (3) on the other hand 3 - - tersine (11) in contrast 3 0.77 1.00 yalnız (12)* it is just that 1 - - öte yandan (70) on the other hand 3 0.55 0.66 yine de (65)* still 1 - - Table 2. Kappa measures for the arguments of discourse adverbials in the data5 2.2.2 Discourse relations expressed by nominalizations The second class of discourse connectives mentioned above, i.e., subordinators take nominalizations as complements, i.e. clauses that are “desententialized” to varying degrees (Lehmann, 1988). Except for a few strict cases, nominalizations are not annotated in the PDTB. However in Turkish, nominalized clauses are so common as arguments of not only the subordinators but also the coordinators that we would have missed an important aspect of the language if we left them out. Complex subordinators are annotated in the TDB 1.0 using the morphological features of their Arg2 as a clue. Complex subordinators have basically two parts; a connective (often a postposition) and nominalizing suffixes which reduce a subordinate clause to varying degrees, causing it to lose its illocutionary force as well as tense and aspect. The nominalized clauses are based on three types of suffixes: (a) clauses based on the factive nominalizer –DIK and the nonfactive nominalizer –AcAk, (b) clauses based on the infinitives –mA or –mAk, (c) clauses formed on the nonfinite nominal marker –Iş (Csató, 1998:230). In the annotation process, the annotators are told to notice these nominalizing suffixes as indicators of nominal clauses that have predicative potential (cf. Appendix B for examples). We annotate the independent parts of the complex subordinators by selecting the independent part and the Arg2 in its entirety. At a later stage, the suffixes will be separated by postprocessing to analyze the frequency of the nominalization types. This will also enable automatic sense 5The stars show that a single set of annotations was created by the group annotation procedure, 2 indicates that the annotations were created via the pair annotation method, the dashes indicate that inter-coder reliability was not calculated. TURKISH DISCOURSE BANK 179 disambiguation of certain connectives, e.g. için ‘for/so as to', whose goal- and cause-driven senses can be distinguished via the nominalizing suffixes. 3 Updates in the annotation guidelines In this section, we present two major updates in the annotation guidelines, namely the decision to annotate phrasal expressions and the introduction of the shared tag to the annotation scheme. We also discuss how we adapted the PDTB's modifier and supplementary tags to Turkish. 3.1 Phrasal expressions As we explained in Section 2.1, the annotation procedure initially started with a given set of connectives. However, the annotators soon discovered that there are polymorphous occurrences of the independent parts of complex subordinators. For example, the complex subordinator sonra ‘after’, belongs to the same family of connectives with the phrasal expression sonra ‘after this’, and its variants, e.g. önce .. sonra ‘first .. then’. We therefore decided to allow the annotators to determine all such occurrences of the given set of connectives, rather than restricting them with the given connectives. In this way, we would achieve a wider coverage of the productive means of establishing discourse coherence. Phrasal expressions are marked as a form of alternative lexicalization in the PDTB (Prasad et al., 2010); they are annotated as a form of complex connectives in a German corpus (Stede & Heintze, 2004). The number of annotated subordinators and phrasal expressions form a sizeable portion of all the annotations in the TDB 1.0. We identified 77 search items. Twenty-eight of these items returned connective types that participate in subordinating relations, and 27 of them in phrasal expressions. Of the total 8483 relations annotated in the corpus, 2284 (26.92%) are signaled by a subordinator, and 482 (5.68%) are signaled by a phrasal expression containing a deictic item. In Appendix A, we provide all annotated complex subordinators (including their polymorphous occurrences) and Kappa values of Arg1 and Arg2 as described in Section 2.2. Appendix A shows that the annotators disagree about the Arg1 of some connectives, e.g., rağmen ‘despite’. Although the reasons for disagreements may vary, we noticed that the flexible word order of Turkish is an important source of disagreements. We discuss this more in Section 3.2 below. 3.2 Word order variability of Turkish and the addition of the shared tag Turkish is predominantly a SOV language with a large degree of word order flexibility. Scrambled elements cause difficulties for the annotators because they may disagree whether the scrambled elements belong to Arg1 or Arg2. To overcome this problem, we introduced the shared tag (subject, object, adverbial phrase), which is essentially a syntactic tag simply helping to mark the shared elements in a discourse relation no matter where they are in the sentence. In this way, the discourse relation itself is determined with more ease and confidence. Table 3 provides the frequency of the subordinators with the shared tag. Shared Tag No Shared Tag Total Search item Gloss Count Percent Count Percent Count Percent için for/so as to 155 14.07 947 85.93 1102 100.00 sonra after 77 10.80 636 89.20 713 100.00 kadar as well as/until 37 23.27 122 76.73 159 100.00 gibi as 35 15.35 193 84.65 228 100.00 amacıyla with the aim of 23 35.94 41 64.06 64 100.00 zaman when 15 9.43 144 90.57 159 100.00 karşın regardless of 11 15.49 60 84.51 71 100.00 önce prior to 11 8.21 123 91.79 134 100.00 halde in spite of 10 16.39 51 83.61 61 100.00 ZEYREK, DEMİRŞAHİN, SEVDİK ÇALLI, AND ÇAKICI 180 Table 3. The frequency of shared elements in the subordinators and the related phrasal expressions Example (1) shows the sentence-medial usage of rağmen ‘despite’, where Arg1 is shown in italics, and Arg2 is rendered in bold letters. The subject, which is not in its canonical sentence- initial position in this case, is shown between curly brackets and annotated as the shared material. (1) Sınırlı olmasına rağmen {bu devrimci kongreler}, sarayın değil, halkın demokratik ihtilalinin eseriydiler. Despite the fact that they were limited, {these revolutionist congresses} were not a result of the empire but the people’s democratic rebellion. 3.3 Modifiers The tag modifier is primarily used to show the modifier of a connective as in the PDTB (example (2), underlined together with the connective), where the connective is taken as the head, and the adverb as the modifier. Different from the PDTB, we also use this tag to specify adverbs modifying the discourse relation as a whole. We refer to such tokens as modifier of a relation as in (3) although they are all marked as mod. (2) Geri dönüp kanepeye uzanıyorum. Az sonra ezgi başlıyor. I go back and lie on the couch. A little later, the melody starts. (3) (Belki de) ona karşı çok iyi ol-duğ-um için bıraktı beni. (Perhaps) he left me because I treated [treat-DIK-AGR] him too well. In the TDB 1.0, a total of 540 relations are tagged with modifiers. The most heavily modified connective is sonra ‘after/later’, where 220 of 713 instances are modified: 138 of these modifiers indicate duration, and 78 are focus particles. The focus particle dA is the most frequent modifier with 262 instances. The temporal modifier daha ‘much/more’ is the second most frequent modifier with 83 instances. Eleven classes of modifiers are annotated in the TDB 1.0. Adverbs such as şimdiye kadar ‘until now’, neyse ki ‘luckily’, ne yazık ki ‘unfortunately’, etc. are not tagged as modifiers. Here, we are in agreement with the PDTB, where such clausal adverbs are selected together with the argument in which they appear. Table 4 shows the modifiers and their frequencies in the TDB 1.0. Modifier Class Example Gloss Count Percent Focus dA focus particle (FP) 265 49.07 Temporal üç gün sonra three days later 170 31.48 Intensifier tam aksine just to the contrary 26 4.81 Counterfactuality sanki … gibi as though 25 4.63 Epistemic belki de bunun için perhaps FP because of this 17 3.15 Interrogative bu yüzden mi is this the reason 14 2.59 Quantifier bütün bunlara rağmen despite all these 9 1.67 Condition ancak bundan sonra only after this 5 0.93 Negation için değil not because of this 5 0.93 Qualifier çarpıcı örnek olarak as a striking example 3 0.56 Pragmatic peki o zaman well, ok then. 1 0.19 Total 540 100.00 Table 4. The frequency of the modifier tags in the TDB rağmen despite 7 9.09 70 90.91 77 100.00 birlikte together/though 6 18.18 27 81.82 33 100.00 ardından after 5 7.04 66 92.96 71 100.00 Total 392 13.64 2840 86,35 2872 100.00 TURKISH DISCOURSE BANK 181 3.4 The supplementary material We identified two types of material that supplement the arguments: (a) the material that makes the semantic contribution of the argument more specific, which is how the PDTB uses this tag, and (b) the antecedent of a deictic item in one of the arguments. Example (4) shows the latter function of the supp tag, where the deictic item is underlined and the supplementary text shown between straight lines “|”. (4) |Ante mutlaka yalnız görüşmeleri gerektiğini| anlatmaya çalıştı. Sonunda padişah buna razı oldu ve huzurunda bulunan herkesi dışarı çıkardı. Ante tried to explain |that they had to meet privately|. Eventually, the sultan agreed with this and asked everyone out. We used the shared supp tag for indicating the antecedent of a deictic element in the shared material (example (5)). In the example, the shared material (the subject in this case) is rendered between curly brackets; its referent is put between double straight lines. (5) Simitis, "||Türkiye'ye müzakere tarihi verildiği ortamda Kıbrıs sorunu çözülmüş olacaktır herhalde ||. {Bu ikisi} birbirine bağlı değil ama beraber gitmeleri gereken iki süreç" dedi. Simitis said, “||When Turkey is given a date for the discussions, the Cyprus problem will probably have been solved||. {These two} are not linked with each other but they are two processes that must go together.” The TDB 1.0 has 869 supplementary annotations for Arg1 (10.24%) and 337 supplementary annotations for Arg2 (3.97%). In future research, the use of the supplementary tag for the antecedents of discourse deictic items as in (4) and (5) will enable us to compare the role of deictic items in the discourse relation with the discourse relation itself. 4. Summary In this paper we described the Turkish Discourse Bank, where discourse connectives are annotated with their two arguments in the style of the PDTB. We focused on the challenges posed by the morphological richness of Turkish as well as its varying word order. We described those aspects of the annotation scheme that are different from the original language English, focusing on the fact that searching for discourse relations between clauses may not capture all the means for presenting information in Turkish. One departure from the PDTB is annotating phrasal expressions, which revealed additional discourse relations based on connectives. Future research will elucidate whether the senses of complex subordinators and the associated phrasal expressions differ systematically. In addition to this, when we reach a larger coverage in the TDB (e.g., by annotating implicit connectives), we will be able to compare the connective-based discourse relations with the non-connective-based ones, obtaining data for cross-linguistic comparison. Acknowledgments We gratefully acknowledge the support from Scientific and Technological Research Council of Turkey (TÜBİTAK, project no. 107E158) and Middle East Technical University Research Funds. We also thank three anonymous reviewers for their extensive feedback, Cem Bozşahin for his comments and İhsan Yalçınkaya who helped with the statistics. All remaining errors are ours. Appendix A This appendix provides Fleiss’ Kappa measures for Arg1 and Arg2 of the search items that retrieved subordinators, the related phrasal expressions and their variants (where applicable). The appendix also shows the exact match measures for the text spans that supplement Arg1 and Arg2, where relevant (see Section 2.2). The connective sonra ‘after’ was annotated in 4 consecutive stages and agreement was measured for each stage. ZEYREK, DEMİRŞAHİN, SEVDİK ÇALLI, AND ÇAKICI 182 Search item (# of annotations) Gloss # o f A n n o ta to r s Kappa measures Exact Match Agreement Subordinator Phrasal expression Variants Arg1 Arg2 Supp1 Supp2 No of Exact Matches Total No of Ann. % Agr No of Exact Matches Total No of Ann. % Agr aksine (13) *6 contrary to contrary to this - 1 - - - - - - - - amacıyla (64) with the aim of - - 3 0.69 0.93 0 0 - 0 0 - ardından (71) after after this first … then 2 1.00 0.99 7 7 100 2 2 100 beri (4) since (temporal) - - 2 - - 1 1 100 0 0 - birlikte (33)* together/ though nevertheless - 1 - - - - - - - - bu yana (10) - since this time (temporal) - 2 1.00 1.00 1 1 100 0 0 - dolayı (21) owing to - - 3 0.98 1.00 0 0 - 0 1 0 dolayısıyla (66) in consequence of - consequently 3 0.78 0.97 2 2 100 1 4 25 ek olarak (1) - in addition to this - 3 NA NA 0 0 - 0 0 - gibi (228) as - - 2 0.94 0.95 28 29 96.55 9 9 100 halde (61) in spite of inspite of this/that - 2 0.87 0.93 0 0 - 0 0 - için (1102) for, so as to for this/that for … for 3 0.81 0.92 3 7 42.86 9 18 50 içindir (4) because of because of this/that it is because of this/that 2 - - 0 0 - 0 0 - kadar (159) as well as, until - - 2 0.84 0.99 2 3 66.67 5 5 100 karşılık (28) although nonetheless - 1 - - - - - - - - karşın (71) regardless of regardless of this/that irregardless 3 0.86 0.84 0 0 - 0 0 - nedenle (117) - for this/that reason - 2 0.94 0.99 13 15 86.67 2 2 100 nedenlerle (4) - for these reasons for the reasons above 2 - - 0 0 - 0 0 - nedeniyle (42) for the reason that - - 2 0.96 0.97 2 2 100 4 4 100 neticesinde (1) - as a result of this - 3 NA NA 0 0 - 0 0 - önce (134) prior to prior to this first first … then/now 2 1.00 1.00 4 5 80.00 1 1 100 2 0.84 0.88 4 4 100 1 1 100 ötürü (11) due to due to this/that due to this/that reason 2 1.00 0.94 0 0 1 1 100 rağmen (77) despite despite this/these - 3 0.73 0.78 0 0 - 0 0 - sayede (5) - thanks to this/that - 2 1.00 1.00 2 2 100 0 0 - sayesinde (3)* thanks to - - 1 - - - - - - - - sonra (713) after after this first.. then, now.. then, at the beginning.. then, at the beginning..after this 3 0.85 0.91 5 10 50 7 8 87.50 2 0.91 0.96 18 18 100 13 13 100 2 0.89 0.94 8 9 88.89 5 5 100 2 0.89 0.98 0 0 - 0 0 - sonucunda (12) as a result of - as a result - 3 0.78 0.78 0 0 - 0 0 - yüzünden (5) since (causal) - - 2 1.00 1.00 1 1 100 1 1 100 zaman (159) when at that time whenever … then 2 0.97 0.98 19 21 90.48 10 11 90.91 6 The stars indicate that the annotations were created by the group annotation procedure; the dashes show that inter-coder reliability was not calculated. TURKISH DISCOURSE BANK 183 Appendix B This appendix provides examples that are used to guide the annotators in annotating subordinators and their nominalized arguments by using the nominalizing suffixes that have predicative potential as clues. The normal order of the arguments of a subordinator is Arg2-Arg1. The suffixes are shown in small caps, both the connective and the corresponding suffixes are underlined. 7 An example for nominalized clauses based on the factive –DIK 8 : (a) Üzül-DÜĞ-Ü kadar şaşırmıştı da. She/He was surprised as much as she/he was saddened [sad-PASS-DIK]. An example for nominalized clauses based on the infinitive –MAK: (b) Gör-ün-me-mek için hemen duvara yaslandı. In order not to be seen [see-PASS-NEG-mAk] he immediately leaned against the wall. An example for nominalized clauses based on –Iş: (c) İhaleli sisteme geç-iş-in ardından bu ihalelere katılmayan 10 kadar firma takibe alındı. After the shift [shift-Iş-GEN] to the bidding system, legal action was taken against circa 10 firms that did not undertake bidding processes. References Berfin Aktaş, Cem Bozşahin and Deniz Zeyrek (2010). Discourse relation configurations in Turkish and an annotation environment. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 202-206, Uppsala, Sweden. Ron Artstein and Massimo Poesio (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4): 555-596. Nicholas Asher (1993). Reference to Abstract Objects in Discourse. Kluwer Academic Publishers, Dordrecht. Nicholas Asher and Alex Lascarides (2003). Logics of Conversation. Cambridge University Press, Cambridge, UK. Éva Á. Csató (1988). Turkish. In Lars Johanson and Éva Á. Csató, editors. (1998). The Turkic Languages (Routledge Language Family Series), pages 203-235, Routledge, London. Işın Demirşahin, İhsan Yalçınkaya and Deniz Zeyrek (2012a). Pair Annotation: Adaption of Pair Programming to Corpus Annotation. In Proceedings of the 6th Linguistic Annotation Workshop, pages 31–39, Jeju, Republic of Korea. Işın Demirşahin, Ayışığı Sevdik-Çallı, Hale Balaban, Ruken Çakıcı and Deniz Zeyrek (2012b). Turkish Discourse Bank: Ongoing Developments. In Proceedings of the First Workshop on Language Resources and Technologies for Turkic Languages, pages 15-19, The 8th International conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May 2012. Katherine Forbes-Riley, Bonnie Webber and Aravind Joshi. (2006). Computing discourse semantics: The predicate-argument semantics of discourse connectives in D-LTAG. Journal of Semantics, 23(1): 55-106. Barbara J. Grosz and Candice L. Sidner. (1986). Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3): 175-204. Christian Lehmann (1988). Towards a typology of clause linkage. In John Haiman and Sandra A. Thompson, editors. Clause Combining in Grammar and Discourse, pages 181-225, John 7 In the examples, the following abbreviations are used: DAT: dative suffix, INST: instrumental suffix, GEN: genitive suffix, PASS: the passive morphemes. 8 The capital letter K may be resolved as k or ğ, the letter which refers to the lengthening of the previous vowel. ZEYREK, DEMİRŞAHİN, SEVDİK ÇALLI, AND ÇAKICI 184 Benjamins Publishing, Amsterdam; Philadelphia. William C. Mann and Sandra A. Thompson (1988). Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8(3): 243-281. Mitchell Marcus, Grace Kim, Mary A. Marcinkiewicz, Robert Macintyre, Ann Bies, Mark Ferguson, Karen Katz and Britta Schasberger (1994). The Penn Treebank: Annotating predicate argument structure. In Proceedings of ARPA Human Language Technology Workshop, pages 114-119. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi and Bonnie Webber (2004). The Penn Discourse Bank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, pages 2237-2240, Lisbon, Portugal. Joseph L. Fleiss (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): 378-382. Megan Moser and Johanna D. Moore (1996). Toward a synthesis of two accounts of discourse structure. Computational Linguistics, 22(3): 409-419. Kemal Oflazer, Bilge Say, Dilek Z. Hakkani-Tür and Gökhan Tür (2003). Building a Turkish treebank. In Anne Abeillé, editor, Treebanks: Building and Using Syntactically Annotated Corpora, pages 261-277, Kluwer Academic Publishers. Livia Polanyi and Martin H. van den Berg (1996). Discourse structure and discourse interpretation. In Proceedings of the Tenth Amsterdam Colloquium, pages 113-131, Amsterdam. Rashmi Prasad, Aravind K. Joshi and Bonnie Webber (2010). Realization of discourse relations by other means: Alternative lexicalizations. Proceedings of COLING 2010, pages 1023-1031, Beijing, China. Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, Aravind Joshi, Livio Robaldo and Bonnie Webber (2007). The Penn Discourse Treebank 2.0 Annotation Manual. Technical Report 203, Institute for Research in Cognitive Science, University of Pennsylvania, Philadelphia, Pennsylvania. Bilge Say, Deniz Zeyrek, Kemal Oflazer and Umut Özge (2004). Development of a corpus and a treebank for present-day written Turkish. In Kamile Imer and Gürkan Doğan, editors, Current Research in Turkish Lingustics, pages 183-192, Eastern Mediterranean University, Cyprus. Wojciech Skut, Brigitte Krenn, Thorsten Brants and Hans Uszkoreit (1997). An annotation scheme for free word order languages. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 88-95, Stroudsburg, Philadelphia. Manfred Stede and Silvan Heintze (2004). Machine-assisted rhetorical structure annotation. In Proceedings of the 20th International Conference on Computational Linguistics, pages 425- 431, Geneva, Switzerland. Bonnie Webber, Aravind Joshi, Eleni Miltsakaki, Rashmi Prasad, Nikhil Dinesh, Alan Lee and Katherine Forbes (2006). A short introduction to the Penn Discourse Tree Bank. In Peter J. Henrichsen and Peter R. Skadhauge, editors, Treebanking for Discourse and Speech, pages 9- 28, Samfundslitteratur Press, Copenhagen. Bonnie Webber, Matthew Stone, Aravind Joshi and Alistair Knott (2003). Anaphora and discourse structure. Computational Linguistics, 29(4): 545-587. Bonnie Webber, Marcus Egg and Valia Kordoni (2011). Discourse structure and langugage technology. Natural Language Engineering, 1(1): 1-54. İhsan Yalçınkaya (2010). An inter-annotator agreement methodology for the Turkish Discourse Bank. Unpublished MS Thesis, Middle East Technical University. Deniz Zeyrek, Işın Demirşahin, Ayışığı B. Sevdik-Çallı, Hale Ö. Balaban, İhsan Yalçınkaya and Ümit D. Turan (2010). The annotation scheme of the Turkish discourse bank and an evaluation of inconsistent annotations. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 282-289, Uppsala, Sweden.