id sid tid token lemma pos bracis-28417 1 1 sabiá sabiá PROPN bracis-28417 1 2 : : PUNCT bracis-28417 2 1 portuguese portuguese ADJ bracis-28417 2 2 large large ADJ bracis-28417 2 3 language language NOUN bracis-28417 2 4 models model NOUN bracis-28417 2 5 | | ADV bracis-28417 2 6 springer springer NOUN bracis-28417 2 7 nature nature NOUN bracis-28417 2 8 link link PROPN bracis-28417 2 9 ( ( PUNCT bracis-28417 2 10 formerly formerly ADV bracis-28417 2 11 springerlink springerlink NOUN bracis-28417 2 12 ) ) PUNCT bracis-28417 2 13 skip skip VERB bracis-28417 2 14 to to ADP bracis-28417 2 15 main main ADJ bracis-28417 2 16 content content NOUN bracis-28417 2 17 advertisement advertisement NOUN bracis-28417 2 18 log log NOUN bracis-28417 2 19 in in ADP bracis-28417 2 20 menu menu NOUN bracis-28417 2 21 find find VERB bracis-28417 2 22 a a DET bracis-28417 2 23 journal journal NOUN bracis-28417 2 24 publish publish VERB bracis-28417 2 25 with with ADP bracis-28417 2 26 us we PRON bracis-28417 2 27 track track VERB bracis-28417 2 28 your your PRON bracis-28417 2 29 research research NOUN bracis-28417 2 30 search search NOUN bracis-28417 2 31 cart cart NOUN bracis-28417 2 32 home home NOUN bracis-28417 2 33 intelligent intelligent ADJ bracis-28417 2 34 systems system NOUN bracis-28417 2 35 conference conference PROPN bracis-28417 2 36 paper paper NOUN bracis-28417 2 37 sabiá sabiá NOUN bracis-28417 2 38 : : PUNCT bracis-28417 3 1 portuguese portuguese ADJ bracis-28417 3 2 large large ADJ bracis-28417 3 3 language language NOUN bracis-28417 3 4 models model NOUN bracis-28417 3 5 conference conference NOUN bracis-28417 3 6 paper paper NOUN bracis-28417 3 7 first first ADV bracis-28417 3 8 online online ADV bracis-28417 3 9 : : PUNCT bracis-28417 3 10 12 12 NUM bracis-28417 3 11 october october NOUN bracis-28417 3 12 2023 2023 NUM bracis-28417 3 13 pp pp ADP bracis-28417 3 14 226–240 226–240 NUM bracis-28417 3 15 cite cite VERB bracis-28417 3 16 this this DET bracis-28417 3 17 conference conference NOUN bracis-28417 3 18 paper paper NOUN bracis-28417 3 19 access access NOUN bracis-28417 3 20 provided provide VERB bracis-28417 3 21 by by ADP bracis-28417 3 22 university university PROPN bracis-28417 3 23 of of ADP bracis-28417 3 24 notre notre PROPN bracis-28417 3 25 dame dame PROPN bracis-28417 3 26 hesburgh hesburgh PROPN bracis-28417 3 27 library library PROPN bracis-28417 3 28 download download PROPN bracis-28417 3 29 book book NOUN bracis-28417 3 30 pdf pdf PROPN bracis-28417 3 31 download download NOUN bracis-28417 3 32 book book NOUN bracis-28417 3 33 epub epub PROPN bracis-28417 3 34 intelligent intelligent ADJ bracis-28417 3 35 systems system NOUN bracis-28417 3 36 ( ( PUNCT bracis-28417 3 37 bracis bracis NOUN bracis-28417 3 38 2023 2023 NUM bracis-28417 3 39 ) ) PUNCT bracis-28417 3 40 sabiá sabiá NOUN bracis-28417 3 41 : : PUNCT bracis-28417 3 42 portuguese portuguese ADJ bracis-28417 3 43 large large ADJ bracis-28417 3 44 language language NOUN bracis-28417 3 45 models model NOUN bracis-28417 3 46 download download VERB bracis-28417 3 47 book book NOUN bracis-28417 3 48 pdf pdf PROPN bracis-28417 3 49 download download NOUN bracis-28417 3 50 book book PROPN bracis-28417 3 51 epub epub PROPN bracis-28417 3 52 ramon ramon NOUN bracis-28417 3 53 pires pire VERB bracis-28417 3 54     SPACE bracis-28417 3 55 orcid orcid NOUN bracis-28417 3 56 : : PUNCT bracis-28417 3 57 orcid.org/0000-0002-0023-19719 orcid.org/0000-0002-0023-19719 PROPN bracis-28417 3 58 , , PUNCT bracis-28417 3 59 hugo hugo PROPN bracis-28417 3 60 abonizio abonizio PROPN bracis-28417 3 61     SPACE bracis-28417 3 62 orcid orcid NOUN bracis-28417 3 63 : : PUNCT bracis-28417 3 64 orcid.org/0000-0001-5208-02909 orcid.org/0000-0001-5208-02909 ADJ bracis-28417 3 65 , , PUNCT bracis-28417 3 66 thales thales PROPN bracis-28417 3 67 sales sales PROPN bracis-28417 3 68 almeida almeida PROPN bracis-28417 3 69     SPACE bracis-28417 3 70 orcid orcid PROPN bracis-28417 3 71 : : PUNCT bracis-28417 3 72 orcid.org/0009-0006-9568-93319 orcid.org/0009-0006-9568-93319 PROPN bracis-28417 3 73 & & CCONJ bracis-28417 3 74 … … PUNCT bracis-28417 3 75 rodrigo rodrigo PROPN bracis-28417 3 76 nogueira nogueira PROPN bracis-28417 3 77     SPACE bracis-28417 3 78 orcid orcid PROPN bracis-28417 3 79 : : PUNCT bracis-28417 3 80 orcid.org/0000-0002-2600-60359 orcid.org/0000-0002-2600-60359 NOUN bracis-28417 3 81     SPACE bracis-28417 3 82 show show VERB bracis-28417 3 83 authors author NOUN bracis-28417 3 84 part part NOUN bracis-28417 3 85 of of ADP bracis-28417 3 86 the the DET bracis-28417 3 87 book book NOUN bracis-28417 3 88 series series NOUN bracis-28417 3 89 : : PUNCT bracis-28417 3 90 lecture lecture NOUN bracis-28417 3 91 notes note NOUN bracis-28417 3 92 in in ADP bracis-28417 3 93 computer computer NOUN bracis-28417 3 94 science science NOUN bracis-28417 3 95 ( ( PUNCT bracis-28417 3 96 ( ( PUNCT bracis-28417 3 97 lnai lnai ADJ bracis-28417 3 98 , , PUNCT bracis-28417 3 99 volume volume NOUN bracis-28417 3 100 14197 14197 NUM bracis-28417 3 101 ) ) PUNCT bracis-28417 3 102 ) ) PUNCT bracis-28417 3 103 included include VERB bracis-28417 3 104 in in ADP bracis-28417 3 105 the the DET bracis-28417 3 106 following follow VERB bracis-28417 3 107 conference conference NOUN bracis-28417 3 108 series series NOUN bracis-28417 3 109 : : PUNCT bracis-28417 3 110 brazilian brazilian ADJ bracis-28417 3 111 conference conference NOUN bracis-28417 3 112 on on ADP bracis-28417 3 113 intelligent intelligent ADJ bracis-28417 3 114 systems system NOUN bracis-28417 3 115 724 724 NUM bracis-28417 3 116 accesses access VERB bracis-28417 3 117 59 59 NUM bracis-28417 3 118 citations citation NOUN bracis-28417 3 119 109 109 NUM bracis-28417 3 120 altmetric altmetric ADJ bracis-28417 3 121 abstract abstract NOUN bracis-28417 3 122 as as SCONJ bracis-28417 3 123 the the DET bracis-28417 3 124 capabilities capability NOUN bracis-28417 3 125 of of ADP bracis-28417 3 126 language language NOUN bracis-28417 3 127 models model NOUN bracis-28417 3 128 continue continue VERB bracis-28417 3 129 to to PART bracis-28417 3 130 advance advance VERB bracis-28417 3 131 , , PUNCT bracis-28417 3 132 it it PRON bracis-28417 3 133 is be AUX bracis-28417 3 134 conceivable conceivable ADJ bracis-28417 3 135 that that SCONJ bracis-28417 3 136 “ " PUNCT bracis-28417 3 137 one one NUM bracis-28417 3 138 - - PUNCT bracis-28417 3 139 size size NOUN bracis-28417 3 140 - - PUNCT bracis-28417 3 141 fits fit VERB bracis-28417 3 142 - - PUNCT bracis-28417 3 143 all all PRON bracis-28417 3 144 ” " PUNCT bracis-28417 3 145 model model NOUN bracis-28417 3 146 will will AUX bracis-28417 3 147 remain remain VERB bracis-28417 3 148 as as ADP bracis-28417 3 149 the the DET bracis-28417 3 150 main main ADJ bracis-28417 3 151 paradigm paradigm NOUN bracis-28417 3 152 . . PUNCT bracis-28417 4 1 for for ADP bracis-28417 4 2 instance instance NOUN bracis-28417 4 3 , , PUNCT bracis-28417 4 4 given give VERB bracis-28417 4 5 the the DET bracis-28417 4 6 vast vast ADJ bracis-28417 4 7 number number NOUN bracis-28417 4 8 of of ADP bracis-28417 4 9 languages language NOUN bracis-28417 4 10 worldwide worldwide ADV bracis-28417 4 11 , , PUNCT bracis-28417 4 12 many many ADJ bracis-28417 4 13 of of ADP bracis-28417 4 14 which which PRON bracis-28417 4 15 are be AUX bracis-28417 4 16 low low ADJ bracis-28417 4 17 - - PUNCT bracis-28417 4 18 resource resource NOUN bracis-28417 4 19 , , PUNCT bracis-28417 4 20 the the DET bracis-28417 4 21 prevalent prevalent ADJ bracis-28417 4 22 practice practice NOUN bracis-28417 4 23 is be AUX bracis-28417 4 24 to to PART bracis-28417 4 25 pretrain pretrain VERB bracis-28417 4 26 a a DET bracis-28417 4 27 single single ADJ bracis-28417 4 28 model model NOUN bracis-28417 4 29 on on ADP bracis-28417 4 30 multiple multiple ADJ bracis-28417 4 31 languages language NOUN bracis-28417 4 32 . . PUNCT bracis-28417 5 1 in in ADP bracis-28417 5 2 this this DET bracis-28417 5 3 paper paper NOUN bracis-28417 5 4 , , PUNCT bracis-28417 5 5 we we PRON bracis-28417 5 6 add add VERB bracis-28417 5 7 to to ADP bracis-28417 5 8 the the DET bracis-28417 5 9 growing grow VERB bracis-28417 5 10 body body NOUN bracis-28417 5 11 of of ADP bracis-28417 5 12 evidence evidence NOUN bracis-28417 5 13 that that PRON bracis-28417 5 14 challenges challenge VERB bracis-28417 5 15 this this DET bracis-28417 5 16 practice practice NOUN bracis-28417 5 17 , , PUNCT bracis-28417 5 18 demonstrating demonstrate VERB bracis-28417 5 19 that that SCONJ bracis-28417 5 20 monolingual monolingual ADJ bracis-28417 5 21 pretraining pretraine VERB bracis-28417 5 22 on on ADP bracis-28417 5 23 the the DET bracis-28417 5 24 target target NOUN bracis-28417 5 25 language language NOUN bracis-28417 5 26 significantly significantly ADV bracis-28417 5 27 improves improve VERB bracis-28417 5 28 models model NOUN bracis-28417 5 29 already already ADV bracis-28417 5 30 extensively extensively ADV bracis-28417 5 31 trained train VERB bracis-28417 5 32 on on ADP bracis-28417 5 33 diverse diverse ADJ bracis-28417 5 34 corpora corpus NOUN bracis-28417 5 35 . . PUNCT bracis-28417 6 1 more more ADV bracis-28417 6 2 specifically specifically ADV bracis-28417 6 3 , , PUNCT bracis-28417 6 4 we we PRON bracis-28417 6 5 further far ADV bracis-28417 6 6 pretrain pretrain NOUN bracis-28417 6 7 gpt gpt PROPN bracis-28417 6 8 - - PUNCT bracis-28417 6 9 j j PROPN bracis-28417 6 10 and and CCONJ bracis-28417 6 11 llama llama NOUN bracis-28417 6 12 models model NOUN bracis-28417 6 13 on on ADP bracis-28417 6 14 portuguese portuguese ADJ bracis-28417 6 15 texts text NOUN bracis-28417 6 16 using use VERB bracis-28417 6 17 3 3 NUM bracis-28417 6 18 % % NOUN bracis-28417 6 19 or or CCONJ bracis-28417 6 20 less less ADJ bracis-28417 6 21 of of ADP bracis-28417 6 22 their their PRON bracis-28417 6 23 original original ADJ bracis-28417 6 24 pretraining pretraine VERB bracis-28417 6 25 budget budget NOUN bracis-28417 6 26 . . PUNCT bracis-28417 7 1 few few ADJ bracis-28417 7 2 - - PUNCT bracis-28417 7 3 shot shot NOUN bracis-28417 7 4 evaluations evaluation NOUN bracis-28417 7 5 on on ADP bracis-28417 7 6 poeta poeta PROPN bracis-28417 7 7 , , PUNCT bracis-28417 7 8 a a DET bracis-28417 7 9 suite suite NOUN bracis-28417 7 10 of of ADP bracis-28417 7 11 14 14 NUM bracis-28417 7 12 portuguese portuguese ADJ bracis-28417 7 13 datasets dataset NOUN bracis-28417 7 14 , , PUNCT bracis-28417 7 15 reveal reveal VERB bracis-28417 7 16 that that SCONJ bracis-28417 7 17 our our PRON bracis-28417 7 18 models model NOUN bracis-28417 7 19 outperform outperform VERB bracis-28417 7 20 english english ADJ bracis-28417 7 21 - - PUNCT bracis-28417 7 22 centric centric ADJ bracis-28417 7 23 and and CCONJ bracis-28417 7 24 multilingual multilingual ADJ bracis-28417 7 25 counterparts counterpart NOUN bracis-28417 7 26 by by ADP bracis-28417 7 27 a a DET bracis-28417 7 28 significant significant ADJ bracis-28417 7 29 margin margin NOUN bracis-28417 7 30 . . PUNCT bracis-28417 8 1 our our PRON bracis-28417 8 2 best good ADJ bracis-28417 8 3 model model NOUN bracis-28417 8 4 , , PUNCT bracis-28417 8 5 sabiá-65b sabiá-65b PROPN bracis-28417 8 6 , , PUNCT bracis-28417 8 7 performs perform VERB bracis-28417 8 8 on on ADP bracis-28417 8 9 par par NOUN bracis-28417 8 10 with with ADP bracis-28417 8 11 gpt-3.5 gpt-3.5 NOUN bracis-28417 8 12 - - PUNCT bracis-28417 8 13 turbo turbo NOUN bracis-28417 8 14 . . PUNCT bracis-28417 9 1 by by ADP bracis-28417 9 2 evaluating evaluate VERB bracis-28417 9 3 on on ADP bracis-28417 9 4 datasets dataset NOUN bracis-28417 9 5 originally originally ADV bracis-28417 9 6 conceived conceive VERB bracis-28417 9 7 in in ADP bracis-28417 9 8 the the DET bracis-28417 9 9 target target NOUN bracis-28417 9 10 language language NOUN bracis-28417 9 11 as as ADV bracis-28417 9 12 well well ADV bracis-28417 9 13 as as ADP bracis-28417 9 14 translated translate VERB bracis-28417 9 15 ones one NOUN bracis-28417 9 16 , , PUNCT bracis-28417 9 17 we we PRON bracis-28417 9 18 study study VERB bracis-28417 9 19 the the DET bracis-28417 9 20 impact impact NOUN bracis-28417 9 21 of of ADP bracis-28417 9 22 language language NOUN bracis-28417 9 23 - - PUNCT bracis-28417 9 24 specific specific ADJ bracis-28417 9 25 pretraining pretraining NOUN bracis-28417 9 26 in in ADP bracis-28417 9 27 terms term NOUN bracis-28417 9 28 of of ADP bracis-28417 9 29 1 1 NUM bracis-28417 9 30 ) ) PUNCT bracis-28417 9 31 capturing capture VERB bracis-28417 9 32 linguistic linguistic ADJ bracis-28417 9 33 nuances nuance NOUN bracis-28417 9 34 and and CCONJ bracis-28417 9 35 structures structure NOUN bracis-28417 9 36 inherent inherent ADJ bracis-28417 9 37 to to ADP bracis-28417 9 38 the the DET bracis-28417 9 39 target target NOUN bracis-28417 9 40 language language NOUN bracis-28417 9 41 , , PUNCT bracis-28417 9 42 and and CCONJ bracis-28417 9 43 2 2 X bracis-28417 9 44 ) ) PUNCT bracis-28417 9 45 enriching enrich VERB bracis-28417 9 46 the the DET bracis-28417 9 47 model model NOUN bracis-28417 9 48 ’s ’s PART bracis-28417 9 49 knowledge knowledge NOUN bracis-28417 9 50 about about ADP bracis-28417 9 51 a a DET bracis-28417 9 52 domain domain NOUN bracis-28417 9 53 or or CCONJ bracis-28417 9 54 culture culture NOUN bracis-28417 9 55 . . PUNCT bracis-28417 10 1 our our PRON bracis-28417 10 2 results result NOUN bracis-28417 10 3 indicate indicate VERB bracis-28417 10 4 that that SCONJ bracis-28417 10 5 most most ADJ bracis-28417 10 6 benefits benefit NOUN bracis-28417 10 7 stem stem VERB bracis-28417 10 8 from from ADP bracis-28417 10 9 the the DET bracis-28417 10 10 domain domain NOUN bracis-28417 10 11 - - PUNCT bracis-28417 10 12 specific specific ADJ bracis-28417 10 13 knowledge knowledge NOUN bracis-28417 10 14 acquired acquire VERB bracis-28417 10 15 through through ADP bracis-28417 10 16 monolingual monolingual ADJ bracis-28417 10 17 pretraining pretraining NOUN bracis-28417 10 18 . . PUNCT bracis-28417 11 1 finally finally ADV bracis-28417 11 2 , , PUNCT bracis-28417 11 3 we we PRON bracis-28417 11 4 show show VERB bracis-28417 11 5 that that SCONJ bracis-28417 11 6 our our PRON bracis-28417 11 7 optimized optimize VERB bracis-28417 11 8 model model NOUN bracis-28417 11 9 for for ADP bracis-28417 11 10 portuguese portuguese PROPN bracis-28417 11 11 demonstrates demonstrate VERB bracis-28417 11 12 a a DET bracis-28417 11 13 reduced reduce VERB bracis-28417 11 14 performance performance NOUN bracis-28417 11 15 in in ADP bracis-28417 11 16 english english ADJ bracis-28417 11 17 tasks task NOUN bracis-28417 11 18 , , PUNCT bracis-28417 11 19 thereby thereby ADV bracis-28417 11 20 substantiating substantiate VERB bracis-28417 11 21 the the DET bracis-28417 11 22 inherent inherent ADJ bracis-28417 11 23 compromise compromise NOUN bracis-28417 11 24 in in ADP bracis-28417 11 25 refining refining NOUN bracis-28417 11 26 models model NOUN bracis-28417 11 27 for for ADP bracis-28417 11 28 specific specific ADJ bracis-28417 11 29 linguistic linguistic ADJ bracis-28417 11 30 domains domain NOUN bracis-28417 11 31 . . PUNCT bracis-28417 12 1 access access NOUN bracis-28417 12 2 provided provide VERB bracis-28417 12 3 by by ADP bracis-28417 12 4 university university PROPN bracis-28417 12 5 of of ADP bracis-28417 12 6 notre notre PROPN bracis-28417 12 7 dame dame PROPN bracis-28417 12 8 hesburgh hesburgh PROPN bracis-28417 12 9 library library PROPN bracis-28417 12 10 . . PUNCT bracis-28417 13 1 download download PROPN bracis-28417 13 2 conference conference NOUN bracis-28417 13 3 paper paper NOUN bracis-28417 13 4 pdf pdf NOUN bracis-28417 13 5 similar similar ADJ bracis-28417 13 6 content content NOUN bracis-28417 13 7 being be AUX bracis-28417 13 8 viewed view VERB bracis-28417 13 9 by by ADP bracis-28417 13 10 others other NOUN bracis-28417 13 11 ptt5 ptt5 PROPN bracis-28417 13 12 - - PUNCT bracis-28417 13 13 v2 v2 PROPN bracis-28417 13 14 : : PUNCT bracis-28417 13 15 a a DET bracis-28417 13 16 closer close ADJ bracis-28417 13 17 look look NOUN bracis-28417 13 18 at at ADP bracis-28417 13 19     SPACE bracis-28417 13 20 continued continue VERB bracis-28417 13 21 pretraining pretraine VERB bracis-28417 13 22 of of ADP bracis-28417 13 23     SPACE bracis-28417 13 24 t5 t5 PROPN bracis-28417 13 25 models model NOUN bracis-28417 13 26 for for ADP bracis-28417 13 27     SPACE bracis-28417 13 28 the the DET bracis-28417 13 29     SPACE bracis-28417 13 30 portuguese portuguese ADJ bracis-28417 13 31 language language NOUN bracis-28417 13 32 chapter chapter NOUN bracis-28417 13 33 © © PROPN bracis-28417 13 34 2025 2025 NUM bracis-28417 13 35 monoand monoand NOUN bracis-28417 13 36 multilingual multilingual ADJ bracis-28417 13 37 gpt-3 gpt-3 NOUN bracis-28417 13 38 models model NOUN bracis-28417 13 39 for for ADP bracis-28417 13 40     SPACE bracis-28417 13 41 hungarian hungarian ADJ bracis-28417 13 42 chapter chapter NOUN bracis-28417 13 43 © © PROPN bracis-28417 13 44 2023 2023 NUM bracis-28417 13 45 scaling scaling NOUN bracis-28417 13 46 and and CCONJ bracis-28417 13 47     SPACE bracis-28417 13 48 adapting adapt VERB bracis-28417 13 49 large large ADJ bracis-28417 13 50 language language NOUN bracis-28417 13 51 models model NOUN bracis-28417 13 52 for for ADP bracis-28417 13 53     SPACE bracis-28417 13 54 portuguese portuguese ADJ bracis-28417 13 55 open open ADJ bracis-28417 13 56 information information NOUN bracis-28417 13 57 extraction extraction NOUN bracis-28417 13 58 : : PUNCT bracis-28417 13 59 a a DET bracis-28417 13 60 comparative comparative ADJ bracis-28417 13 61 study study NOUN bracis-28417 13 62 of of ADP bracis-28417 13 63     SPACE bracis-28417 13 64 fine fine ADV bracis-28417 13 65 - - PUNCT bracis-28417 13 66 tuning tuning NOUN bracis-28417 13 67 and and CCONJ bracis-28417 13 68     SPACE bracis-28417 13 69 lora lora PROPN bracis-28417 13 70 chapter chapter NOUN bracis-28417 13 71 © © PROPN bracis-28417 13 72 2025 2025 NUM bracis-28417 13 73 explore explore VERB bracis-28417 13 74 related relate VERB bracis-28417 13 75 subjects subject NOUN bracis-28417 13 76 discover discover VERB bracis-28417 13 77 the the DET bracis-28417 13 78 latest late ADJ bracis-28417 13 79 articles article NOUN bracis-28417 13 80 , , PUNCT bracis-28417 13 81 books book NOUN bracis-28417 13 82 and and CCONJ bracis-28417 13 83 news news NOUN bracis-28417 13 84 in in ADP bracis-28417 13 85 related related ADJ bracis-28417 13 86 subjects subject NOUN bracis-28417 13 87 , , PUNCT bracis-28417 13 88 suggested suggest VERB bracis-28417 13 89 using use VERB bracis-28417 13 90 machine machine NOUN bracis-28417 13 91 learning learning NOUN bracis-28417 13 92 . . PUNCT bracis-28417 14 1 african african ADJ bracis-28417 14 2 languages language NOUN bracis-28417 14 3 language language NOUN bracis-28417 14 4 teaching teaching NOUN bracis-28417 14 5 and and CCONJ bracis-28417 14 6 learning learn VERB bracis-28417 14 7 languages language NOUN bracis-28417 14 8 language language NOUN bracis-28417 14 9 education education NOUN bracis-28417 14 10 portuguese portuguese ADJ bracis-28417 14 11 language language NOUN bracis-28417 14 12 policy policy NOUN bracis-28417 14 13 and and CCONJ bracis-28417 14 14 planning plan VERB bracis-28417 14 15 1 1 NUM bracis-28417 14 16 introduction introduction NOUN bracis-28417 14 17 language language NOUN bracis-28417 14 18 models model NOUN bracis-28417 14 19 have have AUX bracis-28417 14 20 revolutionized revolutionize VERB bracis-28417 14 21 the the DET bracis-28417 14 22 field field NOUN bracis-28417 14 23 of of ADP bracis-28417 14 24 natural natural ADJ bracis-28417 14 25 language language NOUN bracis-28417 14 26 processing processing NOUN bracis-28417 14 27 with with ADP bracis-28417 14 28 their their PRON bracis-28417 14 29 exceptional exceptional ADJ bracis-28417 14 30 ability ability NOUN bracis-28417 14 31 to to PART bracis-28417 14 32 perform perform VERB bracis-28417 14 33 tasks task NOUN bracis-28417 14 34 with with ADP bracis-28417 14 35 minimal minimal ADJ bracis-28417 14 36 supervision supervision NOUN bracis-28417 14 37 . . PUNCT bracis-28417 15 1 although although SCONJ bracis-28417 15 2 primarily primarily ADV bracis-28417 15 3 pretrained pretraine VERB bracis-28417 15 4 on on ADP bracis-28417 15 5 english english ADJ bracis-28417 15 6 - - PUNCT bracis-28417 15 7 centric centric NOUN bracis-28417 15 8 corpora corpora NOUN bracis-28417 15 9 , , PUNCT bracis-28417 15 10 the the DET bracis-28417 15 11 models model NOUN bracis-28417 15 12 have have AUX bracis-28417 15 13 shown show VERB bracis-28417 15 14 impressive impressive ADJ bracis-28417 15 15 multilingual multilingual ADJ bracis-28417 15 16 capabilities capability NOUN bracis-28417 15 17     SPACE bracis-28417 16 1 [ [ X bracis-28417 16 2 10 10 NUM bracis-28417 16 3 ] ] PUNCT bracis-28417 16 4 . . PUNCT bracis-28417 17 1 given give VERB bracis-28417 17 2 the the DET bracis-28417 17 3 abundance abundance NOUN bracis-28417 17 4 of of ADP bracis-28417 17 5 languages language NOUN bracis-28417 17 6 worldwide worldwide ADV bracis-28417 17 7 , , PUNCT bracis-28417 17 8 the the DET bracis-28417 17 9 majority majority NOUN bracis-28417 17 10 of of ADP bracis-28417 17 11 which which PRON bracis-28417 17 12 are be AUX bracis-28417 17 13 low low ADJ bracis-28417 17 14 - - PUNCT bracis-28417 17 15 resource resource NOUN bracis-28417 17 16 , , PUNCT bracis-28417 17 17 it it PRON bracis-28417 17 18 has have AUX bracis-28417 17 19 become become VERB bracis-28417 17 20 a a DET bracis-28417 17 21 common common ADJ bracis-28417 17 22 practice practice NOUN bracis-28417 17 23 to to PART bracis-28417 17 24 pretrain pretrain VERB bracis-28417 17 25 single single ADJ bracis-28417 17 26 models model NOUN bracis-28417 17 27 on on ADP bracis-28417 17 28 multiple multiple ADJ bracis-28417 17 29 languages language NOUN bracis-28417 17 30 simultaneously simultaneously ADV bracis-28417 17 31 . . PUNCT bracis-28417 18 1 models model NOUN bracis-28417 18 2 like like ADP bracis-28417 18 3 xlm xlm NOUN bracis-28417 18 4 - - PUNCT bracis-28417 18 5 r r NOUN bracis-28417 18 6     SPACE bracis-28417 18 7 [ [ NOUN bracis-28417 18 8 12 12 NUM bracis-28417 18 9 ] ] PUNCT bracis-28417 18 10 , , PUNCT bracis-28417 18 11 mbart mbart NOUN bracis-28417 18 12     SPACE bracis-28417 19 1 [ [ X bracis-28417 19 2 28 28 NUM bracis-28417 19 3 ] ] X bracis-28417 19 4 , , PUNCT bracis-28417 19 5 mt5 mt5 ADJ bracis-28417 19 6     SPACE bracis-28417 19 7 [ [ X bracis-28417 19 8 70 70 NUM bracis-28417 19 9 ] ] PUNCT bracis-28417 19 10 , , PUNCT bracis-28417 19 11 and and CCONJ bracis-28417 19 12 bloom bloom NOUN bracis-28417 19 13     SPACE bracis-28417 20 1 [ [ X bracis-28417 20 2 54 54 NUM bracis-28417 20 3 ] ] PUNCT bracis-28417 20 4 exemplify exemplify VERB bracis-28417 20 5 this this DET bracis-28417 20 6 approach approach NOUN bracis-28417 20 7 . . PUNCT bracis-28417 21 1 despite despite SCONJ bracis-28417 21 2 the the DET bracis-28417 21 3 success success NOUN bracis-28417 21 4 of of ADP bracis-28417 21 5 these these DET bracis-28417 21 6 multilingual multilingual ADJ bracis-28417 21 7 models model NOUN bracis-28417 21 8 , , PUNCT bracis-28417 21 9 we we PRON bracis-28417 21 10 argue argue VERB bracis-28417 21 11 that that SCONJ bracis-28417 21 12 they they PRON bracis-28417 21 13 may may AUX bracis-28417 21 14 not not PART bracis-28417 21 15 be be AUX bracis-28417 21 16 the the DET bracis-28417 21 17 optimal optimal ADJ bracis-28417 21 18 approach approach NOUN bracis-28417 21 19 for for ADP bracis-28417 21 20 capturing capture VERB bracis-28417 21 21 the the DET bracis-28417 21 22 cultural cultural ADJ bracis-28417 21 23 and and CCONJ bracis-28417 21 24 knowledge knowledge NOUN bracis-28417 21 25 richness richness NOUN bracis-28417 21 26 inherent inherent ADJ bracis-28417 21 27 in in ADP bracis-28417 21 28 individual individual ADJ bracis-28417 21 29 languages language NOUN bracis-28417 21 30 . . PUNCT bracis-28417 22 1 when when SCONJ bracis-28417 22 2 a a DET bracis-28417 22 3 moderately moderately ADV bracis-28417 22 4 - - PUNCT bracis-28417 22 5 sized sized ADJ bracis-28417 22 6 language language NOUN bracis-28417 22 7 - - PUNCT bracis-28417 22 8 specific specific NOUN bracis-28417 22 9 corpus corpus NOUN bracis-28417 22 10 is be AUX bracis-28417 22 11 available available ADJ bracis-28417 22 12 , , PUNCT bracis-28417 22 13 continued continued ADJ bracis-28417 22 14 pretraining pretraine VERB bracis-28417 22 15 could could AUX bracis-28417 22 16 integrate integrate VERB bracis-28417 22 17 the the DET bracis-28417 22 18 missing miss VERB bracis-28417 22 19 knowledge knowledge NOUN bracis-28417 22 20 into into ADP bracis-28417 22 21 the the DET bracis-28417 22 22 model model NOUN bracis-28417 22 23 , , PUNCT bracis-28417 22 24 enhancing enhance VERB bracis-28417 22 25 its its PRON bracis-28417 22 26 performance performance NOUN bracis-28417 22 27 on on ADP bracis-28417 22 28 targeted target VERB bracis-28417 22 29 tasks task NOUN bracis-28417 22 30 . . PUNCT bracis-28417 23 1 to to PART bracis-28417 23 2 test test VERB bracis-28417 23 3 this this DET bracis-28417 23 4 hypothesis hypothesis NOUN bracis-28417 23 5 , , PUNCT bracis-28417 23 6 we we PRON bracis-28417 23 7 extend extend VERB bracis-28417 23 8 the the DET bracis-28417 23 9 pretraining pretraining NOUN bracis-28417 23 10 of of ADP bracis-28417 23 11 english english ADJ bracis-28417 23 12 - - PUNCT bracis-28417 23 13 centric centric ADJ bracis-28417 23 14 models model NOUN bracis-28417 23 15 using use VERB bracis-28417 23 16 portuguese portuguese PROPN bracis-28417 23 17 corpora corpora PROPN bracis-28417 23 18 and and CCONJ bracis-28417 23 19 evaluate evaluate VERB bracis-28417 23 20 their their PRON bracis-28417 23 21 performance performance NOUN bracis-28417 23 22 on on ADP bracis-28417 23 23 an an DET bracis-28417 23 24 extensive extensive ADJ bracis-28417 23 25 range range NOUN bracis-28417 23 26 of of ADP bracis-28417 23 27 portuguese portuguese ADJ bracis-28417 23 28 datasets dataset NOUN bracis-28417 23 29 employing employ VERB bracis-28417 23 30 a a DET bracis-28417 23 31 few few ADJ bracis-28417 23 32 - - PUNCT bracis-28417 23 33 shot shot NOUN bracis-28417 23 34 learning learning NOUN bracis-28417 23 35 approach approach NOUN bracis-28417 23 36 . . PUNCT bracis-28417 24 1 our our PRON bracis-28417 24 2 results result NOUN bracis-28417 24 3 indicate indicate VERB bracis-28417 24 4 that that SCONJ bracis-28417 24 5 , , PUNCT bracis-28417 24 6 even even ADV bracis-28417 24 7 for for ADP bracis-28417 24 8 models model NOUN bracis-28417 24 9 trained train VERB bracis-28417 24 10 beyond beyond ADP bracis-28417 24 11 the the DET bracis-28417 24 12 recommendations recommendation NOUN bracis-28417 24 13 by by ADP bracis-28417 24 14 hoffmann hoffmann PROPN bracis-28417 24 15 et et PROPN bracis-28417 24 16 al al PROPN bracis-28417 24 17     SPACE bracis-28417 24 18 [ [ X bracis-28417 24 19 18 18 NUM bracis-28417 24 20 ] ] PUNCT bracis-28417 24 21 , , PUNCT bracis-28417 24 22 this this DET bracis-28417 24 23 additional additional ADJ bracis-28417 24 24 pretraining pretraine VERB bracis-28417 24 25 considerably considerably ADV bracis-28417 24 26 improves improve VERB bracis-28417 24 27 performance performance NOUN bracis-28417 24 28 compared compare VERB bracis-28417 24 29 to to ADP bracis-28417 24 30 multilingual multilingual ADJ bracis-28417 24 31 models model NOUN bracis-28417 24 32 . . PUNCT bracis-28417 25 1 we we PRON bracis-28417 25 2 evaluate evaluate VERB bracis-28417 25 3 our our PRON bracis-28417 25 4 models model NOUN bracis-28417 25 5 on on ADP bracis-28417 25 6 datasets dataset NOUN bracis-28417 25 7 comprising comprise VERB bracis-28417 25 8 texts text NOUN bracis-28417 25 9 originally originally ADV bracis-28417 25 10 created create VERB bracis-28417 25 11 by by ADP bracis-28417 25 12 native native ADJ bracis-28417 25 13 brazilian brazilian ADJ bracis-28417 25 14 portuguese portuguese ADJ bracis-28417 25 15 speakers speaker NOUN bracis-28417 25 16 , , PUNCT bracis-28417 25 17 as as ADV bracis-28417 25 18 well well ADV bracis-28417 25 19 as as ADP bracis-28417 25 20 datasets dataset NOUN bracis-28417 25 21 translated translate VERB bracis-28417 25 22 from from ADP bracis-28417 25 23 english english PROPN bracis-28417 25 24 to to ADP bracis-28417 25 25 portuguese portuguese PROPN bracis-28417 25 26 . . PUNCT bracis-28417 26 1 we we PRON bracis-28417 26 2 observe observe VERB bracis-28417 26 3 improvements improvement NOUN bracis-28417 26 4 across across ADP bracis-28417 26 5 all all DET bracis-28417 26 6 datasets dataset NOUN bracis-28417 26 7 due due ADP bracis-28417 26 8 to to ADP bracis-28417 26 9 the the DET bracis-28417 26 10 portuguese portuguese ADJ bracis-28417 26 11 pretraining pretraine VERB bracis-28417 26 12 , , PUNCT bracis-28417 26 13 with with SCONJ bracis-28417 26 14 the the DET bracis-28417 26 15 gains gain NOUN bracis-28417 26 16 being be AUX bracis-28417 26 17 particularly particularly ADV bracis-28417 26 18 pronounced pronounce VERB bracis-28417 26 19 for for ADP bracis-28417 26 20 datasets dataset NOUN bracis-28417 26 21 created create VERB bracis-28417 26 22 by by ADP bracis-28417 26 23 brazilian brazilian ADJ bracis-28417 26 24 speakers speaker NOUN bracis-28417 26 25 . . PUNCT bracis-28417 27 1 one one NUM bracis-28417 27 2 of of ADP bracis-28417 27 3 the the DET bracis-28417 27 4 largest large ADJ bracis-28417 27 5 improvements improvement NOUN bracis-28417 27 6 was be AUX bracis-28417 27 7 observed observe VERB bracis-28417 27 8 on on ADP bracis-28417 27 9 the the DET bracis-28417 27 10 enem enem PROPN bracis-28417 27 11 dataset dataset NOUN bracis-28417 27 12     SPACE bracis-28417 28 1 [ [ X bracis-28417 28 2 57 57 NUM bracis-28417 28 3 ] ] PUNCT bracis-28417 28 4 , , PUNCT bracis-28417 28 5 which which PRON bracis-28417 28 6 is be AUX bracis-28417 28 7 derived derive VERB bracis-28417 28 8 from from ADP bracis-28417 28 9 entrance entrance NOUN bracis-28417 28 10 exams exam NOUN bracis-28417 28 11 used use VERB bracis-28417 28 12 by by ADP bracis-28417 28 13 brazilian brazilian ADJ bracis-28417 28 14 universities university NOUN bracis-28417 28 15 and and CCONJ bracis-28417 28 16 requires require VERB bracis-28417 28 17 extensive extensive ADJ bracis-28417 28 18 knowledge knowledge NOUN bracis-28417 28 19 of of ADP bracis-28417 28 20 the the DET bracis-28417 28 21 country country NOUN bracis-28417 28 22 ’s ’s PART bracis-28417 28 23 history history NOUN bracis-28417 28 24 , , PUNCT bracis-28417 28 25 geography geography NOUN bracis-28417 28 26 , , PUNCT bracis-28417 28 27 and and CCONJ bracis-28417 28 28 literature literature NOUN bracis-28417 28 29 . . PUNCT bracis-28417 29 1 this this DET bracis-28417 29 2 result result NOUN bracis-28417 29 3 provides provide VERB bracis-28417 29 4 evidence evidence NOUN bracis-28417 29 5 that that SCONJ bracis-28417 29 6 the the DET bracis-28417 29 7 major major ADJ bracis-28417 29 8 contribution contribution NOUN bracis-28417 29 9 of of ADP bracis-28417 29 10 our our PRON bracis-28417 29 11 language language NOUN bracis-28417 29 12 - - PUNCT bracis-28417 29 13 specific specific ADJ bracis-28417 29 14 pretraining pretraining NOUN bracis-28417 29 15 is be AUX bracis-28417 29 16 to to PART bracis-28417 29 17 inject inject VERB bracis-28417 29 18 domain domain NOUN bracis-28417 29 19 - - PUNCT bracis-28417 29 20 specific specific ADJ bracis-28417 29 21 knowledge knowledge NOUN bracis-28417 29 22 about about ADP bracis-28417 29 23 a a DET bracis-28417 29 24 particular particular ADJ bracis-28417 29 25 culture culture NOUN bracis-28417 29 26 as as SCONJ bracis-28417 29 27 opposed oppose VERB bracis-28417 29 28 to to PART bracis-28417 29 29 solely solely ADV bracis-28417 29 30 enhancing enhance VERB bracis-28417 29 31 language language NOUN bracis-28417 29 32 proficiency proficiency NOUN bracis-28417 29 33 . . PUNCT bracis-28417 30 1 2 2 NUM bracis-28417 30 2 related relate VERB bracis-28417 30 3 work work NOUN bracis-28417 30 4 the the DET bracis-28417 30 5 success success NOUN bracis-28417 30 6 of of ADP bracis-28417 30 7 multilingual multilingual ADJ bracis-28417 30 8 pretraining pretraine VERB bracis-28417 30 9 has have AUX bracis-28417 30 10 been be AUX bracis-28417 30 11 well well ADV bracis-28417 30 12 - - PUNCT bracis-28417 30 13 documented document VERB bracis-28417 30 14 in in ADP bracis-28417 30 15 the the DET bracis-28417 30 16 literature literature NOUN bracis-28417 30 17 , , PUNCT bracis-28417 30 18 with with ADP bracis-28417 30 19 models model NOUN bracis-28417 30 20 such such ADJ bracis-28417 30 21 as as ADP bracis-28417 30 22 byt5 byt5 PROPN bracis-28417 30 23     SPACE bracis-28417 31 1 [ [ X bracis-28417 31 2 69 69 NUM bracis-28417 31 3 ] ] X bracis-28417 31 4 , , PUNCT bracis-28417 31 5 mt5 mt5 ADJ bracis-28417 31 6     SPACE bracis-28417 31 7 [ [ X bracis-28417 31 8 70 70 NUM bracis-28417 31 9 ] ] PUNCT bracis-28417 31 10 , , PUNCT bracis-28417 31 11 xlm xlm PROPN bracis-28417 31 12 - - PUNCT bracis-28417 31 13 r r VERB bracis-28417 31 14     SPACE bracis-28417 31 15 [ [ NOUN bracis-28417 31 16 12 12 NUM bracis-28417 31 17 ] ] X bracis-28417 31 18 , , PUNCT bracis-28417 31 19 xglm xglm PROPN bracis-28417 31 20     SPACE bracis-28417 32 1 [ [ X bracis-28417 32 2 27 27 NUM bracis-28417 32 3 ] ] PUNCT bracis-28417 32 4 and and CCONJ bracis-28417 32 5 mgpt mgpt NOUN bracis-28417 32 6     SPACE bracis-28417 32 7 [ [ X bracis-28417 32 8 56 56 NUM bracis-28417 32 9 ] ] PUNCT bracis-28417 32 10 paving pave VERB bracis-28417 32 11 the the DET bracis-28417 32 12 way way NOUN bracis-28417 32 13 for for ADP bracis-28417 32 14 more more ADJ bracis-28417 32 15 inclusive inclusive ADJ bracis-28417 32 16 language language NOUN bracis-28417 32 17 understanding understanding NOUN bracis-28417 32 18 and and CCONJ bracis-28417 32 19 generation generation NOUN bracis-28417 32 20 by by ADP bracis-28417 32 21 leveraging leverage VERB bracis-28417 32 22 shared share VERB bracis-28417 32 23 knowledge knowledge NOUN bracis-28417 32 24 across across ADP bracis-28417 32 25 multiple multiple ADJ bracis-28417 32 26 languages language NOUN bracis-28417 32 27 . . PUNCT bracis-28417 33 1 however however ADV bracis-28417 33 2 , , PUNCT bracis-28417 33 3 there there PRON bracis-28417 33 4 are be VERB bracis-28417 33 5 limitations limitation NOUN bracis-28417 33 6 to to ADP bracis-28417 33 7 this this DET bracis-28417 33 8 approach approach NOUN bracis-28417 33 9 . . PUNCT bracis-28417 34 1 bloom bloom NOUN bracis-28417 34 2 , , PUNCT bracis-28417 34 3 a a DET bracis-28417 34 4 175b 175b NUM bracis-28417 34 5 - - PUNCT bracis-28417 34 6 parameter parameter NOUN bracis-28417 34 7 model model NOUN bracis-28417 34 8 pretrained pretraine VERB bracis-28417 34 9 on on ADP bracis-28417 34 10 46 46 NUM bracis-28417 34 11 languages language NOUN bracis-28417 34 12 , , PUNCT bracis-28417 34 13 performs perform VERB bracis-28417 34 14 worse bad ADJ bracis-28417 34 15 on on ADP bracis-28417 34 16 english english ADJ bracis-28417 34 17 tasks task NOUN bracis-28417 34 18 compared compare VERB bracis-28417 34 19 to to PART bracis-28417 34 20 opt opt VERB bracis-28417 34 21     SPACE bracis-28417 34 22 [ [ X bracis-28417 34 23 74 74 NUM bracis-28417 34 24 ] ] PUNCT bracis-28417 34 25 , , PUNCT bracis-28417 34 26 a a DET bracis-28417 34 27 similarly similarly ADV bracis-28417 34 28 sized sized ADJ bracis-28417 34 29 model model NOUN bracis-28417 34 30 pretrained pretraine VERB bracis-28417 34 31 on on ADP bracis-28417 34 32 english english ADJ bracis-28417 34 33 - - PUNCT bracis-28417 34 34 centric centric NOUN bracis-28417 34 35 corpora corpora NOUN bracis-28417 34 36 using use VERB bracis-28417 34 37 comparable comparable ADJ bracis-28417 34 38 computational computational ADJ bracis-28417 34 39 resources resource NOUN bracis-28417 34 40 and and CCONJ bracis-28417 34 41 data datum NOUN bracis-28417 34 42 size size NOUN bracis-28417 34 43 . . PUNCT bracis-28417 35 1 we we PRON bracis-28417 35 2 conjecture conjecture VERB bracis-28417 35 3 that that SCONJ bracis-28417 35 4 bloom bloom NOUN bracis-28417 35 5 ’s ’s PART bracis-28417 35 6 underperformance underperformance NOUN bracis-28417 35 7 may may AUX bracis-28417 35 8 be be AUX bracis-28417 35 9 attributed attribute VERB bracis-28417 35 10 to to ADP bracis-28417 35 11 its its PRON bracis-28417 35 12 relatively relatively ADV bracis-28417 35 13 limited limited ADJ bracis-28417 35 14 exposure exposure NOUN bracis-28417 35 15 to to ADP bracis-28417 35 16 english english ADJ bracis-28417 35 17 tokens token NOUN bracis-28417 35 18 during during ADP bracis-28417 35 19 the the DET bracis-28417 35 20 pretraining pretraine VERB bracis-28417 35 21 phase phase NOUN bracis-28417 35 22 . . PUNCT bracis-28417 36 1 consequently consequently ADV bracis-28417 36 2 , , PUNCT bracis-28417 36 3 this this DET bracis-28417 36 4 observation observation NOUN bracis-28417 36 5 suggests suggest VERB bracis-28417 36 6 that that SCONJ bracis-28417 36 7 monolingual monolingual ADJ bracis-28417 36 8 pretraining pretraine VERB bracis-28417 36 9 could could AUX bracis-28417 36 10 offer offer VERB bracis-28417 36 11 supplementary supplementary ADJ bracis-28417 36 12 advantages advantage NOUN bracis-28417 36 13 . . PUNCT bracis-28417 37 1 in in ADP bracis-28417 37 2 support support NOUN bracis-28417 37 3 of of ADP bracis-28417 37 4 this this DET bracis-28417 37 5 hypothesis hypothesis NOUN bracis-28417 37 6 , , PUNCT bracis-28417 37 7 models model NOUN bracis-28417 37 8 with with ADP bracis-28417 37 9 hundreds hundred NOUN bracis-28417 37 10 of of ADP bracis-28417 37 11 millions million NOUN bracis-28417 37 12 of of ADP bracis-28417 37 13 parameters parameter NOUN bracis-28417 37 14 pretrained pretraine VERB bracis-28417 37 15 on on ADP bracis-28417 37 16 monolingual monolingual ADJ bracis-28417 37 17 texts text NOUN bracis-28417 37 18 have have AUX bracis-28417 37 19 demonstrated demonstrate VERB bracis-28417 37 20 gains gain NOUN bracis-28417 37 21 over over ADP bracis-28417 37 22 multilingual multilingual ADJ bracis-28417 37 23 counterparts counterpart NOUN bracis-28417 37 24     SPACE bracis-28417 38 1 [ [ X bracis-28417 38 2 2 2 NUM bracis-28417 38 3 , , PUNCT bracis-28417 38 4 6,7,8 6,7,8 NUM bracis-28417 38 5 , , PUNCT bracis-28417 38 6 21 21 NUM bracis-28417 38 7 , , PUNCT bracis-28417 38 8 24 24 NUM bracis-28417 38 9 , , PUNCT bracis-28417 38 10 25 25 NUM bracis-28417 38 11 , , PUNCT bracis-28417 38 12 32 32 NUM bracis-28417 38 13 , , PUNCT bracis-28417 38 14 36 36 NUM bracis-28417 38 15 , , PUNCT bracis-28417 38 16 52 52 NUM bracis-28417 38 17 , , PUNCT bracis-28417 38 18 59 59 NUM bracis-28417 38 19 ] ] PUNCT bracis-28417 38 20 . . PUNCT bracis-28417 39 1 additionally additionally ADV bracis-28417 39 2 , , PUNCT bracis-28417 39 3 research research NOUN bracis-28417 39 4 has have AUX bracis-28417 39 5 indicated indicate VERB bracis-28417 39 6 that that SCONJ bracis-28417 39 7 language language NOUN bracis-28417 39 8 adaptation adaptation NOUN bracis-28417 39 9 is be AUX bracis-28417 39 10 beneficial beneficial ADJ bracis-28417 39 11 even even ADV bracis-28417 39 12 for for ADP bracis-28417 39 13 low low ADJ bracis-28417 39 14 - - PUNCT bracis-28417 39 15 resource resource NOUN bracis-28417 39 16 languages language NOUN bracis-28417 39 17     SPACE bracis-28417 40 1 [ [ X bracis-28417 40 2 4 4 NUM bracis-28417 40 3 , , PUNCT bracis-28417 40 4 13 13 NUM bracis-28417 40 5 , , PUNCT bracis-28417 40 6 38 38 NUM bracis-28417 40 7 , , PUNCT bracis-28417 40 8 72 72 NUM bracis-28417 40 9 ] ] PUNCT bracis-28417 40 10 . . PUNCT bracis-28417 41 1 however however ADV bracis-28417 41 2 , , PUNCT bracis-28417 41 3 there there PRON bracis-28417 41 4 is be VERB bracis-28417 41 5 a a DET bracis-28417 41 6 limited limited ADJ bracis-28417 41 7 number number NOUN bracis-28417 41 8 of of ADP bracis-28417 41 9 published publish VERB bracis-28417 41 10 research research NOUN bracis-28417 41 11 articles article NOUN bracis-28417 41 12 with with ADP bracis-28417 41 13 comprehensive comprehensive ADJ bracis-28417 41 14 evaluations evaluation NOUN bracis-28417 41 15 of of ADP bracis-28417 41 16 the the DET bracis-28417 41 17 benefits benefit NOUN bracis-28417 41 18 of of ADP bracis-28417 41 19 continued continue VERB bracis-28417 41 20 pretraining pretraine VERB bracis-28417 41 21 at at ADP bracis-28417 41 22 the the DET bracis-28417 41 23 multi multi ADJ bracis-28417 41 24 - - ADJ bracis-28417 41 25 billion billion NUM bracis-28417 41 26 - - PUNCT bracis-28417 41 27 parameter parameter NOUN bracis-28417 41 28 scale scale NOUN bracis-28417 41 29     SPACE bracis-28417 42 1 [ [ X bracis-28417 42 2 22 22 NUM bracis-28417 42 3 , , PUNCT bracis-28417 42 4 50 50 NUM bracis-28417 42 5 , , PUNCT bracis-28417 42 6 73 73 NUM bracis-28417 42 7 ] ] PUNCT bracis-28417 42 8 . . PUNCT bracis-28417 43 1 through through ADP bracis-28417 43 2 this this DET bracis-28417 43 3 study study NOUN bracis-28417 43 4 , , PUNCT bracis-28417 43 5 we we PRON bracis-28417 43 6 contribute contribute VERB bracis-28417 43 7 to to ADP bracis-28417 43 8 the the DET bracis-28417 43 9 literature literature NOUN bracis-28417 43 10 by by ADP bracis-28417 43 11 demonstrating demonstrate VERB bracis-28417 43 12 the the DET bracis-28417 43 13 effectiveness effectiveness NOUN bracis-28417 43 14 of of ADP bracis-28417 43 15 continued continue VERB bracis-28417 43 16 language language NOUN bracis-28417 43 17 - - PUNCT bracis-28417 43 18 specific specific ADJ bracis-28417 43 19 pretraining pretraining NOUN bracis-28417 43 20 for for ADP bracis-28417 43 21 portuguese portuguese ADJ bracis-28417 43 22 language language NOUN bracis-28417 43 23 models model NOUN bracis-28417 43 24 up up ADP bracis-28417 43 25 to to ADP bracis-28417 43 26 the the DET bracis-28417 43 27 65b 65b NUM bracis-28417 43 28 - - PUNCT bracis-28417 43 29 parameter parameter NOUN bracis-28417 43 30 scale scale NOUN bracis-28417 43 31 . . PUNCT bracis-28417 44 1 the the DET bracis-28417 44 2 question question NOUN bracis-28417 44 3 concerning concern VERB bracis-28417 44 4 whether whether SCONJ bracis-28417 44 5 it it PRON bracis-28417 44 6 is be AUX bracis-28417 44 7 advantageous advantageous ADJ bracis-28417 44 8 to to PART bracis-28417 44 9 train train VERB bracis-28417 44 10 models model NOUN bracis-28417 44 11 for for ADP bracis-28417 44 12 specific specific ADJ bracis-28417 44 13 languages language NOUN bracis-28417 44 14 is be AUX bracis-28417 44 15 closely closely ADV bracis-28417 44 16 associated associate VERB bracis-28417 44 17 with with ADP bracis-28417 44 18 the the DET bracis-28417 44 19 question question NOUN bracis-28417 44 20 of of ADP bracis-28417 44 21 whether whether SCONJ bracis-28417 44 22 it it PRON bracis-28417 44 23 is be AUX bracis-28417 44 24 beneficial beneficial ADJ bracis-28417 44 25 to to PART bracis-28417 44 26 train train NOUN bracis-28417 44 27 models model NOUN bracis-28417 44 28 for for ADP bracis-28417 44 29 particular particular ADJ bracis-28417 44 30 domains domain NOUN bracis-28417 44 31 of of ADP bracis-28417 44 32 knowledge knowledge NOUN bracis-28417 44 33 . . PUNCT bracis-28417 45 1 recent recent ADJ bracis-28417 45 2 studies study NOUN bracis-28417 45 3 , , PUNCT bracis-28417 45 4 such such ADJ bracis-28417 45 5 as as ADP bracis-28417 45 6 minerva minerva PROPN bracis-28417 45 7     SPACE bracis-28417 45 8 [ [ X bracis-28417 45 9 26 26 NUM bracis-28417 45 10 ] ] PUNCT bracis-28417 45 11 and and CCONJ bracis-28417 45 12 galactica galactica PROPN bracis-28417 45 13     SPACE bracis-28417 46 1 [ [ X bracis-28417 46 2 62 62 NUM bracis-28417 46 3 ] ] PUNCT bracis-28417 46 4 , , PUNCT bracis-28417 46 5 have have AUX bracis-28417 46 6 shown show VERB bracis-28417 46 7 that that SCONJ bracis-28417 46 8 domain domain NOUN bracis-28417 46 9 - - PUNCT bracis-28417 46 10 specific specific ADJ bracis-28417 46 11 pretraining pretraining NOUN bracis-28417 46 12 can can AUX bracis-28417 46 13 lead lead VERB bracis-28417 46 14 to to ADP bracis-28417 46 15 significant significant ADJ bracis-28417 46 16 improvements improvement NOUN bracis-28417 46 17 , , PUNCT bracis-28417 46 18 even even ADV bracis-28417 46 19 with with ADP bracis-28417 46 20 a a DET bracis-28417 46 21 smaller small ADJ bracis-28417 46 22 pretraining pretraine VERB bracis-28417 46 23 corpus corpus NOUN bracis-28417 46 24 compared compare VERB bracis-28417 46 25 to to ADP bracis-28417 46 26 large large ADJ bracis-28417 46 27 - - PUNCT bracis-28417 46 28 scale scale NOUN bracis-28417 46 29 , , PUNCT bracis-28417 46 30 general general ADJ bracis-28417 46 31 - - PUNCT bracis-28417 46 32 purpose purpose NOUN bracis-28417 46 33 pretraining pretraine VERB bracis-28417 46 34 corpora corpora PROPN bracis-28417 46 35 . . PUNCT bracis-28417 47 1 analogously analogously ADV bracis-28417 47 2 , , PUNCT bracis-28417 47 3 fu fu PROPN bracis-28417 47 4 et et PROPN bracis-28417 47 5 al al PROPN bracis-28417 47 6 . . PUNCT bracis-28417 47 7     SPACE bracis-28417 48 1 [ [ X bracis-28417 48 2 15 15 NUM bracis-28417 48 3 ] ] PUNCT bracis-28417 48 4 demonstrated demonstrate VERB bracis-28417 48 5 the the DET bracis-28417 48 6 feasibility feasibility NOUN bracis-28417 48 7 of of ADP bracis-28417 48 8 specializing specialize VERB bracis-28417 48 9 smaller small ADJ bracis-28417 48 10 models model NOUN bracis-28417 48 11 to to PART bracis-28417 48 12 perform perform VERB bracis-28417 48 13 multi multi ADJ bracis-28417 48 14 - - ADJ bracis-28417 48 15 step step ADJ bracis-28417 48 16 reasoning reasoning NOUN bracis-28417 48 17 , , PUNCT bracis-28417 48 18 a a DET bracis-28417 48 19 capability capability NOUN bracis-28417 48 20 typically typically ADV bracis-28417 48 21 exclusive exclusive ADJ bracis-28417 48 22 to to ADP bracis-28417 48 23 models model NOUN bracis-28417 48 24 with with ADP bracis-28417 48 25 at at ADP bracis-28417 48 26 least least ADJ bracis-28417 48 27 50b 50b NUM bracis-28417 48 28 parameters parameter NOUN bracis-28417 48 29 , , PUNCT bracis-28417 48 30 at at ADP bracis-28417 48 31 the the DET bracis-28417 48 32 expense expense NOUN bracis-28417 48 33 of of ADP bracis-28417 48 34 diminished diminished ADJ bracis-28417 48 35 performance performance NOUN bracis-28417 48 36 in in ADP bracis-28417 48 37 other other ADJ bracis-28417 48 38 , , PUNCT bracis-28417 48 39 more more ADV bracis-28417 48 40 general general ADJ bracis-28417 48 41 tasks task NOUN bracis-28417 48 42 . . PUNCT bracis-28417 49 1 pretraining pretraine VERB bracis-28417 49 2 with with ADP bracis-28417 49 3 a a DET bracis-28417 49 4 combination combination NOUN bracis-28417 49 5 of of ADP bracis-28417 49 6 general general ADJ bracis-28417 49 7 and and CCONJ bracis-28417 49 8 domain domain NOUN bracis-28417 49 9 - - PUNCT bracis-28417 49 10 specific specific ADJ bracis-28417 49 11 corpora corpora NOUN bracis-28417 49 12 can can AUX bracis-28417 49 13 potentially potentially ADV bracis-28417 49 14 enhance enhance VERB bracis-28417 49 15 performance performance NOUN bracis-28417 49 16 in in ADP bracis-28417 49 17 specialized specialized ADJ bracis-28417 49 18 tasks task NOUN bracis-28417 49 19 without without ADP bracis-28417 49 20 compromising compromise VERB bracis-28417 49 21 effectiveness effectiveness NOUN bracis-28417 49 22 in in ADP bracis-28417 49 23 general general ADJ bracis-28417 49 24 - - PUNCT bracis-28417 49 25 purpose purpose NOUN bracis-28417 49 26 tasks task NOUN bracis-28417 49 27 , , PUNCT bracis-28417 49 28 albeit albeit SCONJ bracis-28417 49 29 at at ADP bracis-28417 49 30 the the DET bracis-28417 49 31 cost cost NOUN bracis-28417 49 32 of of ADP bracis-28417 49 33 increased increase VERB bracis-28417 49 34 computational computational ADJ bracis-28417 49 35 demands demand NOUN bracis-28417 49 36 . . PUNCT bracis-28417 50 1 for for ADP bracis-28417 50 2 example example NOUN bracis-28417 50 3 , , PUNCT bracis-28417 50 4 bloomberggpt bloomberggpt NOUN bracis-28417 50 5     SPACE bracis-28417 51 1 [ [ X bracis-28417 51 2 68 68 NUM bracis-28417 51 3 ] ] X bracis-28417 51 4 , , PUNCT bracis-28417 51 5 a a DET bracis-28417 51 6 50b 50b NUM bracis-28417 51 7 - - PUNCT bracis-28417 51 8 parameter parameter NOUN bracis-28417 51 9 model model NOUN bracis-28417 51 10 pretrained pretraine VERB bracis-28417 51 11 on on ADP bracis-28417 51 12 heterogeneous heterogeneous ADJ bracis-28417 51 13 corpus corpus NOUN bracis-28417 51 14 in in ADP bracis-28417 51 15 which which PRON bracis-28417 51 16 more more ADJ bracis-28417 51 17 than than ADP bracis-28417 51 18 half half NOUN bracis-28417 51 19 of of ADP bracis-28417 51 20 texts text NOUN bracis-28417 51 21 are be AUX bracis-28417 51 22 from from ADP bracis-28417 51 23 the the DET bracis-28417 51 24 financial financial ADJ bracis-28417 51 25 domain domain NOUN bracis-28417 51 26 , , PUNCT bracis-28417 51 27 exhibits exhibit VERB bracis-28417 51 28 comparable comparable ADJ bracis-28417 51 29 performance performance NOUN bracis-28417 51 30 to to ADP bracis-28417 51 31 opt-66b opt-66b NOUN bracis-28417 51 32 in in ADP bracis-28417 51 33 general general ADJ bracis-28417 51 34 tasks task NOUN bracis-28417 51 35 . . PUNCT bracis-28417 52 1 however however ADV bracis-28417 52 2 , , PUNCT bracis-28417 52 3 bloomberggpt bloomberggpt PROPN bracis-28417 52 4 ’s ’s PART bracis-28417 52 5 pretraining pretraine VERB bracis-28417 52 6 dataset dataset NOUN bracis-28417 52 7 is be AUX bracis-28417 52 8 three three NUM bracis-28417 52 9 times time NOUN bracis-28417 52 10 larger large ADJ bracis-28417 52 11 , , PUNCT bracis-28417 52 12 and and CCONJ bracis-28417 52 13 consequently consequently ADV bracis-28417 52 14 used use VERB bracis-28417 52 15 more more ADJ bracis-28417 52 16 computational computational ADJ bracis-28417 52 17 resources resource NOUN bracis-28417 52 18 . . PUNCT bracis-28417 53 1 rather rather ADV bracis-28417 53 2 than than ADP bracis-28417 53 3 pursuing pursue VERB bracis-28417 53 4 a a DET bracis-28417 53 5 single single ADJ bracis-28417 53 6 model model NOUN bracis-28417 53 7 that that PRON bracis-28417 53 8 performs perform VERB bracis-28417 53 9 well well ADV bracis-28417 53 10 across across ADP bracis-28417 53 11 multiple multiple ADJ bracis-28417 53 12 domains domain NOUN bracis-28417 53 13 , , PUNCT bracis-28417 53 14 gururangan gururangan VERB bracis-28417 53 15 et et PROPN bracis-28417 53 16 al al PROPN bracis-28417 53 17 . . PUNCT bracis-28417 53 18     SPACE bracis-28417 54 1 [ [ X bracis-28417 54 2 17 17 NUM bracis-28417 54 3 ] ] X bracis-28417 54 4 propose propose VERB bracis-28417 54 5 an an DET bracis-28417 54 6 alternative alternative ADJ bracis-28417 54 7 approach approach NOUN bracis-28417 54 8 : : PUNCT bracis-28417 54 9 using use VERB bracis-28417 54 10 multiple multiple ADJ bracis-28417 54 11 expert expert NOUN bracis-28417 54 12 models model NOUN bracis-28417 54 13 , , PUNCT bracis-28417 54 14 each each PRON bracis-28417 54 15 trained train VERB bracis-28417 54 16 on on ADP bracis-28417 54 17 a a DET bracis-28417 54 18 domain domain NOUN bracis-28417 54 19 - - PUNCT bracis-28417 54 20 specific specific ADJ bracis-28417 54 21 subset subset NOUN bracis-28417 54 22 within within ADP bracis-28417 54 23 a a DET bracis-28417 54 24 broader broad ADJ bracis-28417 54 25 , , PUNCT bracis-28417 54 26 diverse diverse ADJ bracis-28417 54 27 dataset dataset NOUN bracis-28417 54 28 , , PUNCT bracis-28417 54 29 to to PART bracis-28417 54 30 function function VERB bracis-28417 54 31 as as ADP bracis-28417 54 32 a a DET bracis-28417 54 33 single single ADJ bracis-28417 54 34 general general ADJ bracis-28417 54 35 - - PUNCT bracis-28417 54 36 purpose purpose NOUN bracis-28417 54 37 model model NOUN bracis-28417 54 38 . . PUNCT bracis-28417 55 1 their their PRON bracis-28417 55 2 models model NOUN bracis-28417 55 3 outperform outperform VERB bracis-28417 55 4 dense dense ADJ bracis-28417 55 5 ones one NOUN bracis-28417 55 6 across across ADP bracis-28417 55 7 various various ADJ bracis-28417 55 8 domain domain NOUN bracis-28417 55 9 - - PUNCT bracis-28417 55 10 specific specific ADJ bracis-28417 55 11 tasks task NOUN bracis-28417 55 12 , , PUNCT bracis-28417 55 13 at at ADP bracis-28417 55 14 the the DET bracis-28417 55 15 expense expense NOUN bracis-28417 55 16 of of ADP bracis-28417 55 17 an an DET bracis-28417 55 18 increased increase VERB bracis-28417 55 19 parameter parameter NOUN bracis-28417 55 20 count count NOUN bracis-28417 55 21 , , PUNCT bracis-28417 55 22 consequently consequently ADV bracis-28417 55 23 leading lead VERB bracis-28417 55 24 to to ADP bracis-28417 55 25 larger large ADJ bracis-28417 55 26 memory memory NOUN bracis-28417 55 27 requirements requirement NOUN bracis-28417 55 28 for for ADP bracis-28417 55 29 efficient efficient ADJ bracis-28417 55 30 inference.footnote inference.footnote NUM bracis-28417 55 31 1 1 NUM bracis-28417 55 32 3 3 NUM bracis-28417 55 33 methodology methodology NOUN bracis-28417 55 34 in in ADP bracis-28417 55 35 this this DET bracis-28417 55 36 section section NOUN bracis-28417 55 37 , , PUNCT bracis-28417 55 38 we we PRON bracis-28417 55 39 outline outline VERB bracis-28417 55 40 the the DET bracis-28417 55 41 pretraining pretraine VERB bracis-28417 55 42 data datum NOUN bracis-28417 55 43 and and CCONJ bracis-28417 55 44 training training NOUN bracis-28417 55 45 details detail NOUN bracis-28417 55 46 used use VERB bracis-28417 55 47 to to PART bracis-28417 55 48 build build VERB bracis-28417 55 49 our our PRON bracis-28417 55 50 models model NOUN bracis-28417 55 51 , , PUNCT bracis-28417 55 52 including include VERB bracis-28417 55 53 data datum NOUN bracis-28417 55 54 sources source NOUN bracis-28417 55 55 , , PUNCT bracis-28417 55 56 preprocessing preprocesse VERB bracis-28417 55 57 techniques technique NOUN bracis-28417 55 58 , , PUNCT bracis-28417 55 59 architectures architecture NOUN bracis-28417 55 60 , , PUNCT bracis-28417 55 61 hyperparameters hyperparameter NOUN bracis-28417 55 62 , , PUNCT bracis-28417 55 63 and and CCONJ bracis-28417 55 64 optimization optimization NOUN bracis-28417 55 65 methods method NOUN bracis-28417 55 66 . . PUNCT bracis-28417 56 1 3.1 3.1 NUM bracis-28417 56 2 pretraining pretraine VERB bracis-28417 56 3 data datum NOUN bracis-28417 56 4 the the DET bracis-28417 56 5 pretraining pretraine VERB bracis-28417 56 6 data datum NOUN bracis-28417 56 7 is be AUX bracis-28417 56 8 derived derive VERB bracis-28417 56 9 from from ADP bracis-28417 56 10 the the DET bracis-28417 56 11 portuguese portuguese PROPN bracis-28417 56 12 subset subset NOUN bracis-28417 56 13 of of ADP bracis-28417 56 14 the the DET bracis-28417 56 15 clueweb clueweb NOUN bracis-28417 56 16 2022 2022 NUM bracis-28417 56 17 dataset dataset VERB bracis-28417 56 18     SPACE bracis-28417 56 19 [ [ X bracis-28417 56 20 40 40 NUM bracis-28417 56 21 , , PUNCT bracis-28417 56 22 41 41 NUM bracis-28417 56 23 ] ] PUNCT bracis-28417 56 24 . . PUNCT bracis-28417 57 1 to to PART bracis-28417 57 2 increase increase VERB bracis-28417 57 3 the the DET bracis-28417 57 4 datasets dataset NOUN bracis-28417 57 5 ’s ’s PART bracis-28417 57 6 quality quality NOUN bracis-28417 57 7 , , PUNCT bracis-28417 57 8 we we PRON bracis-28417 57 9 apply apply VERB bracis-28417 57 10 the the DET bracis-28417 57 11 quality quality NOUN bracis-28417 57 12 filters filter NOUN bracis-28417 57 13 from from ADP bracis-28417 57 14 massivetext massivetext ADJ bracis-28417 57 15     SPACE bracis-28417 58 1 [ [ X bracis-28417 58 2 45 45 NUM bracis-28417 58 3 ] ] PUNCT bracis-28417 58 4 , , PUNCT bracis-28417 58 5 modifying modify VERB bracis-28417 58 6 them they PRON bracis-28417 58 7 to to PART bracis-28417 58 8 accommodate accommodate VERB bracis-28417 58 9 the the DET bracis-28417 58 10 specific specific ADJ bracis-28417 58 11 requirements requirement NOUN bracis-28417 58 12 of of ADP bracis-28417 58 13 the the DET bracis-28417 58 14 portuguese portuguese ADJ bracis-28417 58 15 language language NOUN bracis-28417 58 16 . . PUNCT bracis-28417 59 1 we we PRON bracis-28417 59 2 normalize normalize VERB bracis-28417 59 3 the the DET bracis-28417 59 4 text text NOUN bracis-28417 59 5 with with ADP bracis-28417 59 6 ftfyfootnote ftfyfootnote NOUN bracis-28417 59 7 2 2 NUM bracis-28417 59 8 , , PUNCT bracis-28417 59 9 convert convert VERB bracis-28417 59 10 wikitexts wikitext NOUN bracis-28417 59 11 into into ADP bracis-28417 59 12 human human ADJ bracis-28417 59 13 - - PUNCT bracis-28417 59 14 readable readable ADJ bracis-28417 59 15 texts text NOUN bracis-28417 59 16 , , PUNCT bracis-28417 59 17 and and CCONJ bracis-28417 59 18 exclude exclude VERB bracis-28417 59 19 documents document NOUN bracis-28417 59 20 containing contain VERB bracis-28417 59 21 less less ADJ bracis-28417 59 22 than than ADP bracis-28417 59 23 200 200 NUM bracis-28417 59 24 unique unique ADJ bracis-28417 59 25 tokens token NOUN bracis-28417 59 26 . . PUNCT bracis-28417 60 1 these these DET bracis-28417 60 2 quality quality NOUN bracis-28417 60 3 filters filter NOUN bracis-28417 60 4 are be AUX bracis-28417 60 5 primarily primarily ADV bracis-28417 60 6 designed design VERB bracis-28417 60 7 for for ADP bracis-28417 60 8 web web NOUN bracis-28417 60 9 pages page NOUN bracis-28417 60 10 and and CCONJ bracis-28417 60 11 may may AUX bracis-28417 60 12 not not PART bracis-28417 60 13 seamlessly seamlessly ADV bracis-28417 60 14 transfer transfer VERB bracis-28417 60 15 to to ADP bracis-28417 60 16 other other ADJ bracis-28417 60 17 domains domain NOUN bracis-28417 60 18 . . PUNCT bracis-28417 61 1 there there PRON bracis-28417 61 2 is be VERB bracis-28417 61 3 potential potential ADJ bracis-28417 61 4 for for ADP bracis-28417 61 5 improvement improvement NOUN bracis-28417 61 6 by by ADP bracis-28417 61 7 employing employ VERB bracis-28417 61 8 more more ADV bracis-28417 61 9 automated automate VERB bracis-28417 61 10 methods method NOUN bracis-28417 61 11 ; ; PUNCT bracis-28417 61 12 however however ADV bracis-28417 61 13 , , PUNCT bracis-28417 61 14 this this DET bracis-28417 61 15 study study NOUN bracis-28417 61 16 did do AUX bracis-28417 61 17 not not PART bracis-28417 61 18 explore explore VERB bracis-28417 61 19 such such ADJ bracis-28417 61 20 approaches approach NOUN bracis-28417 61 21 due due ADP bracis-28417 61 22 to to ADP bracis-28417 61 23 the the DET bracis-28417 61 24 resource resource NOUN bracis-28417 61 25 - - PUNCT bracis-28417 61 26 intensive intensive ADJ bracis-28417 61 27 nature nature NOUN bracis-28417 61 28 of of ADP bracis-28417 61 29 pretraining pretraine VERB bracis-28417 61 30 experiments experiment NOUN bracis-28417 61 31 . . PUNCT bracis-28417 62 1 following follow VERB bracis-28417 62 2 the the DET bracis-28417 62 3 cleaning cleaning NOUN bracis-28417 62 4 process process NOUN bracis-28417 62 5 , , PUNCT bracis-28417 62 6 all all DET bracis-28417 62 7 documents document NOUN bracis-28417 62 8 are be AUX bracis-28417 62 9 concatenated concatenate VERB bracis-28417 62 10 using use VERB bracis-28417 62 11 an an DET bracis-28417 62 12 end end NOUN bracis-28417 62 13 - - PUNCT bracis-28417 62 14 of of ADP bracis-28417 62 15 - - PUNCT bracis-28417 62 16 sequence sequence NOUN bracis-28417 62 17 token token VERB bracis-28417 62 18 as as ADP bracis-28417 62 19 a a DET bracis-28417 62 20 separator separator NOUN bracis-28417 62 21 , , PUNCT bracis-28417 62 22 and and CCONJ bracis-28417 62 23 then then ADV bracis-28417 62 24 tokenized tokenize VERB bracis-28417 62 25 . . PUNCT bracis-28417 63 1 the the DET bracis-28417 63 2 gpt gpt PROPN bracis-28417 63 3 - - PUNCT bracis-28417 63 4 j j NOUN bracis-28417 63 5 tokenizer tokenizer NOUN bracis-28417 63 6 , , PUNCT bracis-28417 63 7 which which PRON bracis-28417 63 8 is be AUX bracis-28417 63 9 identical identical ADJ bracis-28417 63 10 to to ADP bracis-28417 63 11 the the DET bracis-28417 63 12 gpt-2 gpt-2 NOUN bracis-28417 63 13 tokenizer tokenizer NOUN bracis-28417 63 14     SPACE bracis-28417 64 1 [ [ X bracis-28417 64 2 44 44 NUM bracis-28417 64 3 ] ] PUNCT bracis-28417 64 4 , , PUNCT bracis-28417 64 5 produces produce VERB bracis-28417 64 6 7.8 7.8 NUM bracis-28417 64 7 billion billion NUM bracis-28417 64 8 tokens token NOUN bracis-28417 64 9 , , PUNCT bracis-28417 64 10 while while SCONJ bracis-28417 64 11 the the DET bracis-28417 64 12 llama llama NOUN bracis-28417 64 13 tokenizer tokenizer NOUN bracis-28417 64 14 produces produce VERB bracis-28417 64 15 7.3 7.3 NUM bracis-28417 64 16 billion billion NUM bracis-28417 64 17 tokens token NOUN bracis-28417 64 18 . . PUNCT bracis-28417 65 1 the the DET bracis-28417 65 2 discrepancy discrepancy NOUN bracis-28417 65 3 in in ADP bracis-28417 65 4 the the DET bracis-28417 65 5 total total ADJ bracis-28417 65 6 number number NOUN bracis-28417 65 7 of of ADP bracis-28417 65 8 tokens token NOUN bracis-28417 65 9 is be AUX bracis-28417 65 10 primarily primarily ADV bracis-28417 65 11 due due ADJ bracis-28417 65 12 to to ADP bracis-28417 65 13 the the DET bracis-28417 65 14 different different ADJ bracis-28417 65 15 tokenization tokenization NOUN bracis-28417 65 16 strategies strategy NOUN bracis-28417 65 17 each each DET bracis-28417 65 18 model model NOUN bracis-28417 65 19 employs employ VERB bracis-28417 65 20 , , PUNCT bracis-28417 65 21 byte byte NOUN bracis-28417 65 22 - - PUNCT bracis-28417 65 23 level level NOUN bracis-28417 65 24 bpe bpe PROPN bracis-28417 65 25 and and CCONJ bracis-28417 65 26 bpe bpe PROPN bracis-28417 65 27 based base VERB bracis-28417 65 28 on on ADP bracis-28417 65 29 sentencepiece sentencepiece PROPN bracis-28417 65 30     SPACE bracis-28417 66 1 [ [ X bracis-28417 66 2 23 23 NUM bracis-28417 66 3 ] ] PUNCT bracis-28417 66 4 , , PUNCT bracis-28417 66 5 respectively respectively ADV bracis-28417 66 6 along along ADP bracis-28417 66 7 with with ADP bracis-28417 66 8 the the DET bracis-28417 66 9 variation variation NOUN bracis-28417 66 10 of of ADP bracis-28417 66 11 the the DET bracis-28417 66 12 vocabularies vocabulary NOUN bracis-28417 66 13 used use VERB bracis-28417 66 14 by by ADP bracis-28417 66 15 each each DET bracis-28417 66 16 tokenizer tokenizer NOUN bracis-28417 66 17 . . PUNCT bracis-28417 67 1 we we PRON bracis-28417 67 2 extended extend VERB bracis-28417 67 3 the the DET bracis-28417 67 4 training training NOUN bracis-28417 67 5 of of ADP bracis-28417 67 6 three three NUM bracis-28417 67 7 models model NOUN bracis-28417 67 8 llama llama PROPN bracis-28417 67 9 7b 7b NOUN bracis-28417 67 10 and and CCONJ bracis-28417 67 11 65b 65b NUM bracis-28417 67 12     SPACE bracis-28417 67 13 [ [ PUNCT bracis-28417 67 14 63 63 NUM bracis-28417 67 15 ] ] PUNCT bracis-28417 67 16 as as ADV bracis-28417 67 17 well well ADV bracis-28417 67 18 as as ADP bracis-28417 67 19 gpt gpt NOUN bracis-28417 67 20 - - PUNCT bracis-28417 67 21 j j NOUN bracis-28417 67 22     SPACE bracis-28417 68 1 [ [ X bracis-28417 68 2 66 66 NUM bracis-28417 68 3 ] ] PUNCT bracis-28417 68 4 originally originally ADV bracis-28417 68 5 trained train VERB bracis-28417 68 6 on on ADP bracis-28417 68 7 english english ADJ bracis-28417 68 8 - - PUNCT bracis-28417 68 9 centric centric NOUN bracis-28417 68 10 corpora corpora NOUN bracis-28417 68 11 , , PUNCT bracis-28417 68 12 on on ADP bracis-28417 68 13 portuguese portuguese ADJ bracis-28417 68 14 texts text NOUN bracis-28417 68 15 ; ; PUNCT bracis-28417 68 16 these these DET bracis-28417 68 17 further further ADJ bracis-28417 68 18 pretrained pretraine VERB bracis-28417 68 19 models model NOUN bracis-28417 68 20 from from ADP bracis-28417 68 21 llama llama NOUN bracis-28417 68 22 are be AUX bracis-28417 68 23 denoted denote VERB bracis-28417 68 24 as as ADP bracis-28417 68 25 sabiá sabiá NOUN bracis-28417 68 26 , , PUNCT bracis-28417 68 27 while while SCONJ bracis-28417 68 28 the the DET bracis-28417 68 29 one one NOUN bracis-28417 68 30 derived derive VERB bracis-28417 68 31 from from ADP bracis-28417 68 32 gpt gpt NOUN bracis-28417 68 33 - - PUNCT bracis-28417 68 34 j j PROPN bracis-28417 68 35 is be AUX bracis-28417 68 36 referred refer VERB bracis-28417 68 37 to to ADP bracis-28417 68 38 as as SCONJ bracis-28417 68 39 sabiá-j.footnote sabiá-j.footnote PROPN bracis-28417 68 40 3 3 NUM bracis-28417 68 41 3.2 3.2 NUM bracis-28417 68 42 sabiá sabiá NOUN bracis-28417 68 43 models model NOUN bracis-28417 68 44 the the DET bracis-28417 68 45 llama llama PROPN bracis-28417 68 46 7b 7b NOUN bracis-28417 68 47 and and CCONJ bracis-28417 68 48 65b 65b NUM bracis-28417 68 49 models model NOUN bracis-28417 68 50 are be AUX bracis-28417 68 51 decoder decoder NOUN bracis-28417 68 52 - - PUNCT bracis-28417 68 53 only only ADV bracis-28417 68 54 transformer transformer NOUN bracis-28417 68 55 models model NOUN bracis-28417 68 56     SPACE bracis-28417 69 1 [ [ X bracis-28417 69 2 64 64 NUM bracis-28417 69 3 ] ] PUNCT bracis-28417 69 4 with with ADP bracis-28417 69 5 a a DET bracis-28417 69 6 similar similar ADJ bracis-28417 69 7 architecture architecture NOUN bracis-28417 69 8 to to ADP bracis-28417 69 9 palm palm NOUN bracis-28417 69 10 ’s ’s NOUN bracis-28417 69 11     SPACE bracis-28417 70 1 [ [ X bracis-28417 70 2 10 10 NUM bracis-28417 70 3 ] ] PUNCT bracis-28417 70 4 . . PUNCT bracis-28417 71 1 the the DET bracis-28417 71 2 models model NOUN bracis-28417 71 3 were be AUX bracis-28417 71 4 trained train VERB bracis-28417 71 5 using use VERB bracis-28417 71 6 a a DET bracis-28417 71 7 causal causal ADJ bracis-28417 71 8 language language NOUN bracis-28417 71 9 modeling model VERB bracis-28417 71 10 objective objective NOUN bracis-28417 71 11 on on ADP bracis-28417 71 12 a a DET bracis-28417 71 13 massive massive ADJ bracis-28417 71 14 dataset dataset NOUN bracis-28417 71 15 sourced source VERB bracis-28417 71 16 from from ADP bracis-28417 71 17 webpages webpage NOUN bracis-28417 71 18 , , PUNCT bracis-28417 71 19 code code NOUN bracis-28417 71 20 , , PUNCT bracis-28417 71 21 books book NOUN bracis-28417 71 22 , , PUNCT bracis-28417 71 23 and and CCONJ bracis-28417 71 24 scientific scientific ADJ bracis-28417 71 25 papers paper NOUN bracis-28417 71 26 . . PUNCT bracis-28417 72 1 the the DET bracis-28417 72 2 7b 7b PROPN bracis-28417 72 3 model model NOUN bracis-28417 72 4 was be AUX bracis-28417 72 5 trained train VERB bracis-28417 72 6 on on ADP bracis-28417 72 7 1 1 NUM bracis-28417 72 8 trillion trillion NUM bracis-28417 72 9 tokens token NOUN bracis-28417 72 10 and and CCONJ bracis-28417 72 11 the the DET bracis-28417 72 12 65b 65b ADJ bracis-28417 72 13 model model NOUN bracis-28417 72 14 was be AUX bracis-28417 72 15 trained train VERB bracis-28417 72 16 on on ADP bracis-28417 72 17 1.4 1.4 NUM bracis-28417 72 18 trillion trillion NUM bracis-28417 72 19 tokens token NOUN bracis-28417 72 20 . . PUNCT bracis-28417 73 1 while while SCONJ bracis-28417 73 2 the the DET bracis-28417 73 3 majority majority NOUN bracis-28417 73 4 of of ADP bracis-28417 73 5 the the DET bracis-28417 73 6 corpus corpus NOUN bracis-28417 73 7 is be AUX bracis-28417 73 8 in in ADP bracis-28417 73 9 english english PROPN bracis-28417 73 10 , , PUNCT bracis-28417 73 11 it it PRON bracis-28417 73 12 also also ADV bracis-28417 73 13 includes include VERB bracis-28417 73 14 an an DET bracis-28417 73 15 unspecified unspecified ADJ bracis-28417 73 16 amount amount NOUN bracis-28417 73 17 of of ADP bracis-28417 73 18 portuguese portuguese ADJ bracis-28417 73 19 text text NOUN bracis-28417 73 20 . . PUNCT bracis-28417 74 1 starting start VERB bracis-28417 74 2 from from ADP bracis-28417 74 3 the the DET bracis-28417 74 4 llama llama NOUN bracis-28417 74 5 weights weight NOUN bracis-28417 74 6 , , PUNCT bracis-28417 74 7 we we PRON bracis-28417 74 8 train train VERB bracis-28417 74 9 the the DET bracis-28417 74 10 sabiá sabiá NOUN bracis-28417 74 11 models model NOUN bracis-28417 74 12 on on ADP bracis-28417 74 13 our our PRON bracis-28417 74 14 portuguese portuguese ADJ bracis-28417 74 15 dataset dataset NOUN bracis-28417 74 16 ( ( PUNCT bracis-28417 74 17 see see VERB bracis-28417 74 18 sect sect NOUN bracis-28417 74 19 . . PUNCT bracis-28417 74 20     SPACE bracis-28417 74 21 3.1 3.1 NUM bracis-28417 74 22 ) ) PUNCT bracis-28417 74 23 using use VERB bracis-28417 74 24 the the DET bracis-28417 74 25 t5x t5x NOUN bracis-28417 74 26 and and CCONJ bracis-28417 74 27 seqio seqio NOUN bracis-28417 74 28 frameworks framework NOUN bracis-28417 74 29     SPACE bracis-28417 75 1 [ [ X bracis-28417 75 2 48 48 NUM bracis-28417 75 3 ] ] PUNCT bracis-28417 75 4 . . PUNCT bracis-28417 76 1 adhering adhere VERB bracis-28417 76 2 closely closely ADV bracis-28417 76 3 to to ADP bracis-28417 76 4 the the DET bracis-28417 76 5 hyperparameters hyperparameter NOUN bracis-28417 76 6 used use VERB bracis-28417 76 7 by by ADP bracis-28417 76 8 palm palm NOUN bracis-28417 76 9 , , PUNCT bracis-28417 76 10 we we PRON bracis-28417 76 11 use use VERB bracis-28417 76 12 the the DET bracis-28417 76 13 adafactor adafactor NOUN bracis-28417 76 14 optimizer optimizer NOUN bracis-28417 76 15     SPACE bracis-28417 77 1 [ [ X bracis-28417 77 2 55 55 NUM bracis-28417 77 3 ] ] PUNCT bracis-28417 77 4 without without ADP bracis-28417 77 5 factorization factorization NOUN bracis-28417 77 6 , , PUNCT bracis-28417 77 7 a a DET bracis-28417 77 8 first first ADJ bracis-28417 77 9 - - PUNCT bracis-28417 77 10 order order NOUN bracis-28417 77 11 momentum momentum NOUN bracis-28417 77 12 \(\beta \(\beta NOUN bracis-28417 77 13 _ _ NOUN bracis-28417 77 14 1 1 NUM bracis-28417 77 15 = = SYM bracis-28417 77 16 0.9\ 0.9\ NOUN bracis-28417 77 17 ) ) PUNCT bracis-28417 77 18 , , PUNCT bracis-28417 77 19 and and CCONJ bracis-28417 77 20 a a DET bracis-28417 77 21 second second ADJ bracis-28417 77 22 - - PUNCT bracis-28417 77 23 order order NOUN bracis-28417 77 24 momentum momentum NOUN bracis-28417 77 25 \(\beta \(\beta NOUN bracis-28417 77 26 _ _ NOUN bracis-28417 77 27 2 2 NUM bracis-28417 77 28 = = SYM bracis-28417 77 29 1 1 NUM bracis-28417 77 30 k^{-0.8}\ k^{-0.8}\ PROPN bracis-28417 77 31 ) ) PUNCT bracis-28417 77 32 , , PUNCT bracis-28417 77 33 where where SCONJ bracis-28417 77 34 k k PROPN bracis-28417 77 35 represents represent VERB bracis-28417 77 36 the the DET bracis-28417 77 37 step step NOUN bracis-28417 77 38 number number NOUN bracis-28417 77 39 . . PUNCT bracis-28417 78 1 we we PRON bracis-28417 78 2 apply apply VERB bracis-28417 78 3 global global ADJ bracis-28417 78 4 norm norm NOUN bracis-28417 78 5 clipping clip VERB bracis-28417 78 6 at at ADP bracis-28417 78 7 1.0 1.0 NUM bracis-28417 78 8 and and CCONJ bracis-28417 78 9 dynamic dynamic ADJ bracis-28417 78 10 weight weight NOUN bracis-28417 78 11 decay decay NOUN bracis-28417 78 12 of of ADP bracis-28417 78 13 \(lr^2\ \(lr^2\ NOUN bracis-28417 78 14 ) ) PUNCT bracis-28417 78 15 , , PUNCT bracis-28417 78 16 with with ADP bracis-28417 78 17 lr lr NOUN bracis-28417 78 18 denoting denote VERB bracis-28417 78 19 the the DET bracis-28417 78 20 current current ADJ bracis-28417 78 21 learning learning NOUN bracis-28417 78 22 rate rate NOUN bracis-28417 78 23 . . PUNCT bracis-28417 79 1 besides besides SCONJ bracis-28417 79 2 the the DET bracis-28417 79 3 standard standard ADJ bracis-28417 79 4 causal causal ADJ bracis-28417 79 5 language language NOUN bracis-28417 79 6 modeling modeling NOUN bracis-28417 79 7 loss loss NOUN bracis-28417 79 8 , , PUNCT bracis-28417 79 9 we we PRON bracis-28417 79 10 use use VERB bracis-28417 79 11 an an DET bracis-28417 79 12 auxiliary auxiliary ADJ bracis-28417 79 13 loss loss NOUN bracis-28417 79 14 of of ADP bracis-28417 79 15 \(10^{-4 \(10^{-4 NOUN bracis-28417 79 16 } } PUNCT bracis-28417 79 17 \log \log PROPN bracis-28417 79 18 ^2 ^2 PUNCT bracis-28417 79 19 ( ( PUNCT bracis-28417 79 20 \sum \sum NOUN bracis-28417 79 21 _ _ PUNCT bracis-28417 79 22 i i PRON bracis-28417 79 23 e^{z_i})\ e^{z_i})\ PROPN bracis-28417 79 24 ) ) PUNCT bracis-28417 79 25 , , PUNCT bracis-28417 79 26 where where SCONJ bracis-28417 79 27 z z NOUN bracis-28417 79 28 are be AUX bracis-28417 79 29 the the DET bracis-28417 79 30 logits logit NOUN bracis-28417 79 31 , , PUNCT bracis-28417 79 32 to to PART bracis-28417 79 33 decrease decrease VERB bracis-28417 79 34 the the DET bracis-28417 79 35 likelihood likelihood NOUN bracis-28417 79 36 of of ADP bracis-28417 79 37 loss loss NOUN bracis-28417 79 38 spikes spike NOUN bracis-28417 79 39 at at ADP bracis-28417 79 40 the the DET bracis-28417 79 41 65b 65b NUM bracis-28417 79 42 - - PUNCT bracis-28417 79 43 parameter parameter NOUN bracis-28417 79 44 scale scale NOUN bracis-28417 79 45 . . PUNCT bracis-28417 80 1 the the DET bracis-28417 80 2 learning learning NOUN bracis-28417 80 3 rate rate NOUN bracis-28417 80 4 is be AUX bracis-28417 80 5 linearly linearly ADV bracis-28417 80 6 increased increase VERB bracis-28417 80 7 from from ADP bracis-28417 80 8 0 0 NUM bracis-28417 80 9 to to ADP bracis-28417 80 10 1e-3 1e-3 PROPN bracis-28417 80 11 over over ADP bracis-28417 80 12 the the DET bracis-28417 80 13 initial initial ADJ bracis-28417 80 14 1,000 1,000 NUM bracis-28417 80 15 steps step NOUN bracis-28417 80 16 , , PUNCT bracis-28417 80 17 followed follow VERB bracis-28417 80 18 by by ADP bracis-28417 80 19 a a DET bracis-28417 80 20 constant constant ADJ bracis-28417 80 21 learning learning NOUN bracis-28417 80 22 rate rate NOUN bracis-28417 80 23 of of ADP bracis-28417 80 24 1e-3 1e-3 PROPN bracis-28417 80 25 for for ADP bracis-28417 80 26 an an DET bracis-28417 80 27 additional additional ADJ bracis-28417 80 28 9,000 9,000 NUM bracis-28417 80 29 steps step NOUN bracis-28417 80 30 . . PUNCT bracis-28417 81 1 the the DET bracis-28417 81 2 models model NOUN bracis-28417 81 3 were be AUX bracis-28417 81 4 trained train VERB bracis-28417 81 5 on on ADP bracis-28417 81 6 a a DET bracis-28417 81 7 tpu tpu NOUN bracis-28417 81 8 v2 v2 PROPN bracis-28417 81 9 - - PUNCT bracis-28417 81 10 512 512 NUM bracis-28417 81 11 , , PUNCT bracis-28417 81 12 using use VERB bracis-28417 81 13 batches batch NOUN bracis-28417 81 14 of of ADP bracis-28417 81 15 512 512 NUM bracis-28417 81 16 sequences sequence NOUN bracis-28417 81 17 , , PUNCT bracis-28417 81 18 each each PRON bracis-28417 81 19 containing contain VERB bracis-28417 81 20 2048 2048 NUM bracis-28417 81 21 tokens token NOUN bracis-28417 81 22 . . PUNCT bracis-28417 82 1 we we PRON bracis-28417 82 2 utilized utilize VERB bracis-28417 82 3 gradient gradient NOUN bracis-28417 82 4 checkpointing checkpointing NOUN bracis-28417 82 5 , , PUNCT bracis-28417 82 6 also also ADV bracis-28417 82 7 known know VERB bracis-28417 82 8 as as ADP bracis-28417 82 9 rematerialization rematerialization NOUN bracis-28417 82 10 , , PUNCT bracis-28417 82 11 to to PART bracis-28417 82 12 enable enable VERB bracis-28417 82 13 the the DET bracis-28417 82 14 use use NOUN bracis-28417 82 15 of of ADP bracis-28417 82 16 larger large ADJ bracis-28417 82 17 batches batch NOUN bracis-28417 82 18 , , PUNCT bracis-28417 82 19 thereby thereby ADV bracis-28417 82 20 increasing increase VERB bracis-28417 82 21 tpu tpu NOUN bracis-28417 82 22 utilization utilization NOUN bracis-28417 82 23 . . PUNCT bracis-28417 83 1 for for ADP bracis-28417 83 2 the the DET bracis-28417 83 3 7b 7b PROPN bracis-28417 83 4 model model NOUN bracis-28417 83 5 , , PUNCT bracis-28417 83 6 this this DET bracis-28417 83 7 configuration configuration NOUN bracis-28417 83 8 results result VERB bracis-28417 83 9 in in ADP bracis-28417 83 10 a a DET bracis-28417 83 11 throughput throughput NOUN bracis-28417 83 12 of of ADP bracis-28417 83 13 124,000 124,000 NUM bracis-28417 83 14 tokens token NOUN bracis-28417 83 15 / / SYM bracis-28417 83 16 sec sec PROPN bracis-28417 83 17 , , PUNCT bracis-28417 83 18 corresponding correspond VERB bracis-28417 83 19 to to ADP bracis-28417 83 20 a a DET bracis-28417 83 21 model model NOUN bracis-28417 83 22 flops flop NOUN bracis-28417 83 23 utilization utilization NOUN bracis-28417 83 24 ( ( PUNCT bracis-28417 83 25 mfu mfu NOUN bracis-28417 83 26 ) ) PUNCT bracis-28417 83 27     SPACE bracis-28417 84 1 [ [ X bracis-28417 84 2 10 10 NUM bracis-28417 84 3 ] ] PUNCT bracis-28417 84 4 of of ADP bracis-28417 84 5 45.2 45.2 NUM bracis-28417 84 6 % % NOUN bracis-28417 84 7 , , PUNCT bracis-28417 84 8 excluding exclude VERB bracis-28417 84 9 the the DET bracis-28417 84 10 self self NOUN bracis-28417 84 11 - - PUNCT bracis-28417 84 12 attention attention NOUN bracis-28417 84 13 operations operation NOUN bracis-28417 84 14 . . PUNCT bracis-28417 85 1 for for ADP bracis-28417 85 2 the the DET bracis-28417 85 3 65b 65b ADJ bracis-28417 85 4 model model NOUN bracis-28417 85 5 , , PUNCT bracis-28417 85 6 we we PRON bracis-28417 85 7 achieve achieve VERB bracis-28417 85 8 a a DET bracis-28417 85 9 throughput throughput NOUN bracis-28417 85 10 of of ADP bracis-28417 85 11 14,000 14,000 NUM bracis-28417 85 12 tokens token NOUN bracis-28417 85 13 / / SYM bracis-28417 85 14 sec sec PROPN bracis-28417 85 15 , , PUNCT bracis-28417 85 16 resulting result VERB bracis-28417 85 17 in in ADP bracis-28417 85 18 an an DET bracis-28417 85 19 mfu mfu NOUN bracis-28417 85 20 of of ADP bracis-28417 85 21 47.4 47.4 NUM bracis-28417 85 22 % % NOUN bracis-28417 85 23 . . PUNCT bracis-28417 86 1 the the DET bracis-28417 86 2 resulting result VERB bracis-28417 86 3 models model NOUN bracis-28417 86 4 were be AUX bracis-28417 86 5 trained train VERB bracis-28417 86 6 on on ADP bracis-28417 86 7 a a DET bracis-28417 86 8 total total NOUN bracis-28417 86 9 of of ADP bracis-28417 86 10 10.4 10.4 NUM bracis-28417 86 11 billion billion NUM bracis-28417 86 12 tokens token NOUN bracis-28417 86 13 , , PUNCT bracis-28417 86 14 or or CCONJ bracis-28417 86 15 1.52 1.52 NUM bracis-28417 86 16 epochs epoch NOUN bracis-28417 86 17 of of ADP bracis-28417 86 18 the the DET bracis-28417 86 19 portuguese portuguese ADJ bracis-28417 86 20 dataset dataset NOUN bracis-28417 86 21 . . PUNCT bracis-28417 87 1 this this PRON bracis-28417 87 2 equals equal VERB bracis-28417 87 3 to to ADP bracis-28417 87 4 10,000 10,000 NUM bracis-28417 87 5 training training NOUN bracis-28417 87 6 steps step NOUN bracis-28417 87 7 , , PUNCT bracis-28417 87 8 which which PRON bracis-28417 87 9 is be AUX bracis-28417 87 10 the the DET bracis-28417 87 11 same same ADJ bracis-28417 87 12 amount amount NOUN bracis-28417 87 13 used use VERB bracis-28417 87 14 to to PART bracis-28417 87 15 train train VERB bracis-28417 87 16 sabiá sabiá NOUN bracis-28417 87 17 . . PUNCT bracis-28417 88 1 we we PRON bracis-28417 88 2 noticed notice VERB bracis-28417 88 3 improvements improvement NOUN bracis-28417 88 4 in in ADP bracis-28417 88 5 few few ADJ bracis-28417 88 6 - - PUNCT bracis-28417 88 7 shot shot NOUN bracis-28417 88 8 tasks task NOUN bracis-28417 88 9 beyond beyond ADP bracis-28417 88 10 one one NUM bracis-28417 88 11 epoch epoch NOUN bracis-28417 88 12 , , PUNCT bracis-28417 88 13 which which PRON bracis-28417 88 14 corroborates corroborate VERB bracis-28417 88 15 results result NOUN bracis-28417 88 16 from from ADP bracis-28417 88 17 taylor taylor PROPN bracis-28417 88 18 et et PROPN bracis-28417 88 19 al al PROPN bracis-28417 88 20 . . PUNCT bracis-28417 88 21     SPACE bracis-28417 89 1 [ [ X bracis-28417 89 2 62 62 NUM bracis-28417 89 3 ] ] PUNCT bracis-28417 89 4 . . PUNCT bracis-28417 90 1 however however ADV bracis-28417 90 2 , , PUNCT bracis-28417 90 3 due due ADP bracis-28417 90 4 to to ADP bracis-28417 90 5 the the DET bracis-28417 90 6 high high ADJ bracis-28417 90 7 costs cost NOUN bracis-28417 90 8 of of ADP bracis-28417 90 9 pretraining pretraine VERB bracis-28417 90 10 , , PUNCT bracis-28417 90 11 we we PRON bracis-28417 90 12 did do AUX bracis-28417 90 13 not not PART bracis-28417 90 14 continue continue VERB bracis-28417 90 15 training.footnote training.footnote NUM bracis-28417 90 16 4 4 NUM bracis-28417 90 17 3.3 3.3 NUM bracis-28417 90 18 sabiá sabiá NOUN bracis-28417 90 19 - - PUNCT bracis-28417 90 20 j j PROPN bracis-28417 90 21 the the DET bracis-28417 90 22 gpt gpt PROPN bracis-28417 90 23 - - PUNCT bracis-28417 90 24 j j PROPN bracis-28417 90 25 model model NOUN bracis-28417 90 26 is be AUX bracis-28417 90 27 a a DET bracis-28417 90 28 6b 6b NUM bracis-28417 90 29 - - PUNCT bracis-28417 90 30 parameter parameter NOUN bracis-28417 90 31 decoder decoder NOUN bracis-28417 90 32 - - PUNCT bracis-28417 90 33 only only ADV bracis-28417 90 34 transformer transformer NOUN bracis-28417 90 35 model model NOUN bracis-28417 90 36 whose whose DET bracis-28417 90 37 architecture architecture NOUN bracis-28417 90 38 and and CCONJ bracis-28417 90 39 training training NOUN bracis-28417 90 40 hyperparameters hyperparameter NOUN bracis-28417 90 41 closely closely ADV bracis-28417 90 42 follow follow VERB bracis-28417 90 43 gpt-3 gpt-3 NOUN bracis-28417 90 44 6.7b 6.7b NOUN bracis-28417 90 45 . . PUNCT bracis-28417 91 1 the the DET bracis-28417 91 2 main main ADJ bracis-28417 91 3 differences difference NOUN bracis-28417 91 4 reside reside VERB bracis-28417 91 5 on on ADP bracis-28417 91 6 computing compute VERB bracis-28417 91 7 the the DET bracis-28417 91 8 mlp mlp NOUN bracis-28417 91 9 and and CCONJ bracis-28417 91 10 self self NOUN bracis-28417 91 11 - - PUNCT bracis-28417 91 12 attention attention NOUN bracis-28417 91 13 in in ADP bracis-28417 91 14 parallel parallel NOUN bracis-28417 91 15 , , PUNCT bracis-28417 91 16 applying apply VERB bracis-28417 91 17 attention attention NOUN bracis-28417 91 18 head head NOUN bracis-28417 91 19 with with ADP bracis-28417 91 20 dimension dimension NOUN bracis-28417 91 21 256 256 NUM bracis-28417 91 22 ( ( PUNCT bracis-28417 91 23 twice twice ADV bracis-28417 91 24 larger large ADJ bracis-28417 91 25 than than ADP bracis-28417 91 26 gpt-3 gpt-3 NOUN bracis-28417 91 27 6.7b 6.7b NOUN bracis-28417 91 28 ) ) PUNCT bracis-28417 91 29 , , PUNCT bracis-28417 91 30 and and CCONJ bracis-28417 91 31 using use VERB bracis-28417 91 32 rotary rotary ADJ bracis-28417 91 33 positional positional ADJ bracis-28417 91 34 embedding embed VERB bracis-28417 91 35 ( ( PUNCT bracis-28417 91 36 rope rope NOUN bracis-28417 91 37 ) ) PUNCT bracis-28417 91 38     SPACE bracis-28417 92 1 [ [ X bracis-28417 92 2 61 61 NUM bracis-28417 92 3 ] ] PUNCT bracis-28417 92 4 . . PUNCT bracis-28417 93 1 gpt gpt PROPN bracis-28417 93 2 - - PUNCT bracis-28417 93 3 j j PROPN bracis-28417 93 4 was be AUX bracis-28417 93 5 trained train VERB bracis-28417 93 6 on on ADP bracis-28417 93 7 400b 400b PROPN bracis-28417 93 8 tokens token NOUN bracis-28417 93 9 from from ADP bracis-28417 93 10 the the DET bracis-28417 93 11 pile pile ADJ bracis-28417 93 12 dataset dataset NOUN bracis-28417 93 13     SPACE bracis-28417 94 1 [ [ X bracis-28417 94 2 16 16 NUM bracis-28417 94 3 ] ] PUNCT bracis-28417 94 4 , , PUNCT bracis-28417 94 5 whose whose DET bracis-28417 94 6 97.4 97.4 NUM bracis-28417 94 7 % % NOUN bracis-28417 94 8 tokens token NOUN bracis-28417 94 9 are be AUX bracis-28417 94 10 in in ADP bracis-28417 94 11 english english PROPN bracis-28417 94 12 . . PUNCT bracis-28417 95 1 we we PRON bracis-28417 95 2 begin begin VERB bracis-28417 95 3 training train VERB bracis-28417 95 4 sabiá sabiá PROPN bracis-28417 95 5 - - PUNCT bracis-28417 95 6 j j PROPN bracis-28417 95 7 from from ADP bracis-28417 95 8 the the DET bracis-28417 95 9 released release VERB bracis-28417 95 10 gpt gpt NOUN bracis-28417 95 11 - - PUNCT bracis-28417 95 12 j j NOUN bracis-28417 95 13 checkpoint checkpoint NOUN bracis-28417 95 14 , , PUNCT bracis-28417 95 15 footnote footnote VERB bracis-28417 95 16 5 5 NUM bracis-28417 95 17 using use VERB bracis-28417 95 18 the the DET bracis-28417 95 19 mesh mesh NOUN bracis-28417 95 20 - - PUNCT bracis-28417 95 21 transformer transformer NOUN bracis-28417 95 22 - - PUNCT bracis-28417 95 23 jax jax NOUN bracis-28417 95 24 framework framework NOUN bracis-28417 95 25     SPACE bracis-28417 96 1 [ [ X bracis-28417 96 2 65 65 NUM bracis-28417 96 3 ] ] PUNCT bracis-28417 96 4 and and CCONJ bracis-28417 96 5 adamw adamw PROPN bracis-28417 96 6 optimizer optimizer NOUN bracis-28417 96 7     SPACE bracis-28417 97 1 [ [ X bracis-28417 97 2 30 30 NUM bracis-28417 97 3 ] ] PUNCT bracis-28417 97 4 with with ADP bracis-28417 97 5 a a DET bracis-28417 97 6 weight weight NOUN bracis-28417 97 7 decay decay NOUN bracis-28417 97 8 of of ADP bracis-28417 97 9 0.1 0.1 NUM bracis-28417 97 10 . . PUNCT bracis-28417 98 1 we we PRON bracis-28417 98 2 start start VERB bracis-28417 98 3 the the DET bracis-28417 98 4 pretraining pretraine VERB bracis-28417 98 5 by by ADP bracis-28417 98 6 warming warm VERB bracis-28417 98 7 up up ADP bracis-28417 98 8 the the DET bracis-28417 98 9 learning learning NOUN bracis-28417 98 10 rate rate NOUN bracis-28417 98 11 until until ADP bracis-28417 98 12 1.2e-5 1.2e-5 NUM bracis-28417 98 13 over over ADP bracis-28417 98 14 13,500 13,500 NUM bracis-28417 98 15 steps step NOUN bracis-28417 98 16 , , PUNCT bracis-28417 98 17 followed follow VERB bracis-28417 98 18 by by ADP bracis-28417 98 19 a a DET bracis-28417 98 20 cosine cosine NOUN bracis-28417 98 21 annealing anneal VERB bracis-28417 98 22 decay decay NOUN bracis-28417 98 23 during during ADP bracis-28417 98 24 135,518 135,518 NUM bracis-28417 98 25 steps step NOUN bracis-28417 98 26 until until ADP bracis-28417 98 27 the the DET bracis-28417 98 28 end end NOUN bracis-28417 98 29 learning learn VERB bracis-28417 98 30 rate rate NOUN bracis-28417 98 31 of of ADP bracis-28417 98 32 2.4e-6 2.4e-6 NUM bracis-28417 98 33 , , PUNCT bracis-28417 98 34 and and CCONJ bracis-28417 98 35 kept keep VERB bracis-28417 98 36 it it PRON bracis-28417 98 37 constant constant ADJ bracis-28417 98 38 from from ADP bracis-28417 98 39 there there ADV bracis-28417 98 40 on on ADP bracis-28417 98 41 . . PUNCT bracis-28417 99 1 we we PRON bracis-28417 99 2 train train VERB bracis-28417 99 3 on on ADP bracis-28417 99 4 a a DET bracis-28417 99 5 tpu tpu NOUN bracis-28417 99 6 v3 v3 PROPN bracis-28417 99 7 - - PUNCT bracis-28417 99 8 8 8 NUM bracis-28417 99 9 using use VERB bracis-28417 99 10 an an DET bracis-28417 99 11 effective effective ADJ bracis-28417 99 12 batch batch NOUN bracis-28417 99 13 size size NOUN bracis-28417 99 14 of of ADP bracis-28417 99 15 32 32 NUM bracis-28417 99 16 sequences sequence NOUN bracis-28417 99 17 of of ADP bracis-28417 99 18 2048 2048 NUM bracis-28417 99 19 tokens token NOUN bracis-28417 99 20 . . PUNCT bracis-28417 100 1 this this DET bracis-28417 100 2 results result VERB bracis-28417 100 3 in in ADP bracis-28417 100 4 a a DET bracis-28417 100 5 throughput throughput NOUN bracis-28417 100 6 of of ADP bracis-28417 100 7 5,200 5,200 NUM bracis-28417 100 8 tokens token NOUN bracis-28417 100 9 / / SYM bracis-28417 100 10 sec sec PROPN bracis-28417 100 11 , , PUNCT bracis-28417 100 12 corresponding correspond VERB bracis-28417 100 13 to to ADP bracis-28417 100 14 a a DET bracis-28417 100 15 mfu mfu NOUN bracis-28417 100 16 of of ADP bracis-28417 100 17 44.5 44.5 NUM bracis-28417 100 18 % % NOUN bracis-28417 100 19 without without ADP bracis-28417 100 20 self self NOUN bracis-28417 100 21 - - PUNCT bracis-28417 100 22 attention attention NOUN bracis-28417 100 23 . . PUNCT bracis-28417 101 1 the the DET bracis-28417 101 2 model model NOUN bracis-28417 101 3 was be AUX bracis-28417 101 4 trained train VERB bracis-28417 101 5 for for ADP bracis-28417 101 6 18 18 NUM bracis-28417 101 7     SPACE bracis-28417 101 8 d d NOUN bracis-28417 101 9 on on ADP bracis-28417 101 10 7.8b 7.8b PROPN bracis-28417 101 11 tokens token NOUN bracis-28417 101 12 , , PUNCT bracis-28417 101 13 or or CCONJ bracis-28417 101 14 one one NUM bracis-28417 101 15 epoch epoch NOUN bracis-28417 101 16 of of ADP bracis-28417 101 17 the the DET bracis-28417 101 18 portuguese portuguese NOUN bracis-28417 101 19 dataset.footnote dataset.footnote VERB bracis-28417 101 20 6 6 NUM bracis-28417 101 21 4 4 NUM bracis-28417 101 22 evaluation evaluation NOUN bracis-28417 101 23 on on ADP bracis-28417 101 24     SPACE bracis-28417 101 25 poeta poeta NOUN bracis-28417 101 26 we we PRON bracis-28417 101 27 evaluate evaluate VERB bracis-28417 101 28 the the DET bracis-28417 101 29 sabiá sabiá NOUN bracis-28417 101 30 models model NOUN bracis-28417 101 31 on on ADP bracis-28417 101 32 the the DET bracis-28417 101 33 portuguese portuguese ADJ bracis-28417 101 34 evaluation evaluation NOUN bracis-28417 101 35 tasks task NOUN bracis-28417 101 36 ( ( PUNCT bracis-28417 101 37 poeta poeta NOUN bracis-28417 101 38 ) ) PUNCT bracis-28417 101 39 benchmark benchmark NOUN bracis-28417 101 40 , , PUNCT bracis-28417 101 41 which which PRON bracis-28417 101 42 comprises comprise VERB bracis-28417 101 43 14 14 NUM bracis-28417 101 44 downstream downstream ADJ bracis-28417 101 45 nlp nlp NOUN bracis-28417 101 46 datasets dataset NOUN bracis-28417 101 47 in in ADP bracis-28417 101 48 portuguese portuguese PROPN bracis-28417 101 49 : : PUNCT bracis-28417 101 50 assin assin PROPN bracis-28417 101 51 2 2 NUM bracis-28417 101 52 rte rte NOUN bracis-28417 101 53 and and CCONJ bracis-28417 101 54 sts st VERB bracis-28417 101 55     SPACE bracis-28417 102 1 [ [ X bracis-28417 102 2 47 47 NUM bracis-28417 102 3 ] ] PUNCT bracis-28417 102 4 , , PUNCT bracis-28417 102 5 enem enem PROPN bracis-28417 102 6 challenge challenge NOUN bracis-28417 102 7     SPACE bracis-28417 103 1 [ [ X bracis-28417 103 2 57 57 NUM bracis-28417 103 3 ] ] PUNCT bracis-28417 103 4 , , PUNCT bracis-28417 103 5 enem enem PROPN bracis-28417 103 6 2022 2022 NUM bracis-28417 103 7     SPACE bracis-28417 104 1 [ [ X bracis-28417 104 2 37 37 NUM bracis-28417 104 3 ] ] PUNCT bracis-28417 104 4 , , PUNCT bracis-28417 104 5 faquad faquad ADV bracis-28417 104 6     SPACE bracis-28417 105 1 [ [ X bracis-28417 105 2 53 53 NUM bracis-28417 105 3 ] ] PUNCT bracis-28417 105 4 , , PUNCT bracis-28417 105 5 tweetsentbr tweetsentbr NOUN bracis-28417 105 6     SPACE bracis-28417 106 1 [ [ X bracis-28417 106 2 5 5 NUM bracis-28417 106 3 ] ] PUNCT bracis-28417 106 4 , , PUNCT bracis-28417 106 5 ag ag PROPN bracis-28417 106 6 news news PROPN bracis-28417 106 7     SPACE bracis-28417 107 1 [ [ X bracis-28417 107 2 75 75 NUM bracis-28417 107 3 ] ] PUNCT bracis-28417 107 4 , , PUNCT bracis-28417 107 5 imdb imdb PROPN bracis-28417 107 6     SPACE bracis-28417 108 1 [ [ X bracis-28417 108 2 31 31 NUM bracis-28417 108 3 ] ] PUNCT bracis-28417 108 4 , , PUNCT bracis-28417 109 1 massive massive ADJ bracis-28417 109 2     SPACE bracis-28417 109 3 [ [ X bracis-28417 109 4 14 14 NUM bracis-28417 109 5 ] ] PUNCT bracis-28417 109 6 , , PUNCT bracis-28417 109 7 mkqa mkqa PROPN bracis-28417 109 8     SPACE bracis-28417 110 1 [ [ X bracis-28417 110 2 29 29 NUM bracis-28417 110 3 ] ] PUNCT bracis-28417 110 4 , , PUNCT bracis-28417 110 5 boolq boolq ADV bracis-28417 110 6     SPACE bracis-28417 111 1 [ [ X bracis-28417 111 2 11 11 NUM bracis-28417 111 3 ] ] PUNCT bracis-28417 111 4 , , PUNCT bracis-28417 111 5 sst2 sst2 NOUN bracis-28417 111 6     SPACE bracis-28417 111 7 [ [ X bracis-28417 111 8 58 58 NUM bracis-28417 111 9 ] ] PUNCT bracis-28417 111 10 , , PUNCT bracis-28417 111 11 wsc wsc PROPN bracis-28417 111 12     SPACE bracis-28417 112 1 [ [ X bracis-28417 112 2 33 33 NUM bracis-28417 112 3 ] ] PUNCT bracis-28417 112 4 , , PUNCT bracis-28417 112 5 and and CCONJ bracis-28417 112 6 bluex bluex NOUN bracis-28417 112 7     SPACE bracis-28417 112 8 [ [ X bracis-28417 112 9 1 1 NUM bracis-28417 112 10 ] ] PUNCT bracis-28417 112 11 . . PUNCT bracis-28417 113 1 half half NOUN bracis-28417 113 2 of of ADP bracis-28417 113 3 them they PRON bracis-28417 113 4 ( ( PUNCT bracis-28417 113 5 assin assin PROPN bracis-28417 113 6 2 2 NUM bracis-28417 113 7 rte rte NOUN bracis-28417 113 8 and and CCONJ bracis-28417 113 9 sts st NOUN bracis-28417 113 10 , , PUNCT bracis-28417 113 11 bluex bluex PROPN bracis-28417 113 12 , , PUNCT bracis-28417 113 13 enem enem PROPN bracis-28417 113 14 challenge challenge NOUN bracis-28417 113 15 , , PUNCT bracis-28417 113 16 enem enem PROPN bracis-28417 113 17 2022 2022 NUM bracis-28417 113 18 , , PUNCT bracis-28417 113 19 faquad faquad ADJ bracis-28417 113 20 , , PUNCT bracis-28417 113 21 and and CCONJ bracis-28417 113 22 tweetsentbr tweetsentbr NOUN bracis-28417 113 23 ) ) PUNCT bracis-28417 113 24 were be AUX bracis-28417 113 25 originally originally ADV bracis-28417 113 26 written write VERB bracis-28417 113 27 in in ADP bracis-28417 113 28 portuguese portuguese PROPN bracis-28417 113 29 , , PUNCT bracis-28417 113 30 and and CCONJ bracis-28417 113 31 the the DET bracis-28417 113 32 remaining remain VERB bracis-28417 113 33 ones one NOUN bracis-28417 113 34 were be AUX bracis-28417 113 35 either either CCONJ bracis-28417 113 36 manually manually ADV bracis-28417 113 37 or or CCONJ bracis-28417 113 38 automatically automatically ADV bracis-28417 113 39 translated translate VERB bracis-28417 113 40 into into ADP bracis-28417 113 41 portuguese portuguese PROPN bracis-28417 113 42 from from ADP bracis-28417 113 43 their their PRON bracis-28417 113 44 originals original NOUN bracis-28417 113 45 in in ADP bracis-28417 113 46 english english PROPN bracis-28417 113 47 . . PUNCT bracis-28417 114 1 we we PRON bracis-28417 114 2 refer refer VERB bracis-28417 114 3 to to ADP bracis-28417 114 4 the the DET bracis-28417 114 5 first first ADJ bracis-28417 114 6 group group NOUN bracis-28417 114 7 as as ADP bracis-28417 114 8 “ " PUNCT bracis-28417 114 9 native native ADJ bracis-28417 114 10 ” " PUNCT bracis-28417 114 11 datasets dataset NOUN bracis-28417 114 12 and and CCONJ bracis-28417 114 13 the the DET bracis-28417 114 14 second second ADJ bracis-28417 114 15 group group NOUN bracis-28417 114 16 as as ADP bracis-28417 114 17 “ " PUNCT bracis-28417 114 18 translated translate VERB bracis-28417 114 19 ” " PUNCT bracis-28417 114 20 datasets.footnote datasets.footnote PROPN bracis-28417 114 21 7 7 NUM bracis-28417 114 22 the the DET bracis-28417 114 23 models model NOUN bracis-28417 114 24 were be AUX bracis-28417 114 25 evaluated evaluate VERB bracis-28417 114 26 in in ADP bracis-28417 114 27 a a DET bracis-28417 114 28 few few ADJ bracis-28417 114 29 - - PUNCT bracis-28417 114 30 shot shot NOUN bracis-28417 114 31 manner manner NOUN bracis-28417 114 32 using use VERB bracis-28417 114 33 the the DET bracis-28417 114 34 maximum maximum ADJ bracis-28417 114 35 number number NOUN bracis-28417 114 36 of of ADP bracis-28417 114 37 examples example NOUN bracis-28417 114 38 that that PRON bracis-28417 114 39 fits fit VERB bracis-28417 114 40 into into ADP bracis-28417 114 41 a a DET bracis-28417 114 42 2048 2048 NUM bracis-28417 114 43 - - PUNCT bracis-28417 114 44 token token VERB bracis-28417 114 45 context context NOUN bracis-28417 114 46 for for ADP bracis-28417 114 47 each each DET bracis-28417 114 48 task task NOUN bracis-28417 114 49 . . PUNCT bracis-28417 115 1 we we PRON bracis-28417 115 2 used use VERB bracis-28417 115 3 the the DET bracis-28417 115 4 gpt-2 gpt-2 NOUN bracis-28417 115 5 tokenizer tokenizer NOUN bracis-28417 115 6 as as ADP bracis-28417 115 7 a a DET bracis-28417 115 8 reference reference NOUN bracis-28417 115 9 because because SCONJ bracis-28417 115 10 it it PRON bracis-28417 115 11 results result VERB bracis-28417 115 12 in in ADP bracis-28417 115 13 more more ADJ bracis-28417 115 14 tokens token NOUN bracis-28417 115 15 . . PUNCT bracis-28417 116 1 this this PRON bracis-28417 116 2 allowed allow VERB bracis-28417 116 3 us we PRON bracis-28417 116 4 to to PART bracis-28417 116 5 comfortably comfortably ADV bracis-28417 116 6 fit fit VERB bracis-28417 116 7 prompts prompt NOUN bracis-28417 116 8 tokenized tokenize VERB bracis-28417 116 9 with with ADP bracis-28417 116 10 other other ADJ bracis-28417 116 11 tokenizers tokenizer NOUN bracis-28417 116 12 . . PUNCT bracis-28417 117 1 to to PART bracis-28417 117 2 evaluate evaluate VERB bracis-28417 117 3 the the DET bracis-28417 117 4 models model NOUN bracis-28417 117 5 , , PUNCT bracis-28417 117 6 we we PRON bracis-28417 117 7 manually manually ADV bracis-28417 117 8 select select VERB bracis-28417 117 9 a a DET bracis-28417 117 10 set set NOUN bracis-28417 117 11 of of ADP bracis-28417 117 12 few few ADJ bracis-28417 117 13 - - PUNCT bracis-28417 117 14 shot shot NOUN bracis-28417 117 15 examples example NOUN bracis-28417 117 16 for for ADP bracis-28417 117 17 each each DET bracis-28417 117 18 dataset dataset NOUN bracis-28417 117 19 on on ADP bracis-28417 117 20 poeta poeta PROPN bracis-28417 117 21 . . PUNCT bracis-28417 118 1 depending depend VERB bracis-28417 118 2 on on ADP bracis-28417 118 3 the the DET bracis-28417 118 4 dataset dataset NOUN bracis-28417 118 5 , , PUNCT bracis-28417 118 6 these these DET bracis-28417 118 7 examples example NOUN bracis-28417 118 8 are be AUX bracis-28417 118 9 balanced balance VERB bracis-28417 118 10 by by ADP bracis-28417 118 11 class class NOUN bracis-28417 118 12 ( ( PUNCT bracis-28417 118 13 except except SCONJ bracis-28417 118 14 for for ADP bracis-28417 118 15 faquad faquad ADJ bracis-28417 118 16 , , PUNCT bracis-28417 118 17 bluex bluex PROPN bracis-28417 118 18 , , PUNCT bracis-28417 118 19 enem enem PROPN bracis-28417 118 20 challenge challenge NOUN bracis-28417 118 21 , , PUNCT bracis-28417 118 22 enem enem PROPN bracis-28417 118 23 2022 2022 NUM bracis-28417 118 24 , , PUNCT bracis-28417 118 25 mkqa mkqa NOUN bracis-28417 118 26 , , PUNCT bracis-28417 118 27 and and CCONJ bracis-28417 118 28 wsc wsc PROPN bracis-28417 118 29 ) ) PUNCT bracis-28417 118 30 . . PUNCT bracis-28417 119 1 for for ADP bracis-28417 119 2 each each DET bracis-28417 119 3 test test NOUN bracis-28417 119 4 example example NOUN bracis-28417 119 5 , , PUNCT bracis-28417 119 6 the the DET bracis-28417 119 7 prompts prompt NOUN bracis-28417 119 8 are be AUX bracis-28417 119 9 built build VERB bracis-28417 119 10 with with ADP bracis-28417 119 11 the the DET bracis-28417 119 12 selected select VERB bracis-28417 119 13 few few ADJ bracis-28417 119 14 - - PUNCT bracis-28417 119 15 shot shot NOUN bracis-28417 119 16 examples example NOUN bracis-28417 119 17 in in ADP bracis-28417 119 18 alternating alternate VERB bracis-28417 119 19 order order NOUN bracis-28417 119 20 . . PUNCT bracis-28417 120 1 each each DET bracis-28417 120 2 task task NOUN bracis-28417 120 3 on on ADP bracis-28417 120 4 poeta poeta PROPN bracis-28417 120 5 has have VERB bracis-28417 120 6 a a DET bracis-28417 120 7 particular particular ADJ bracis-28417 120 8 instruction instruction NOUN bracis-28417 120 9 that that PRON bracis-28417 120 10 is be AUX bracis-28417 120 11 placed place VERB bracis-28417 120 12 at at ADP bracis-28417 120 13 the the DET bracis-28417 120 14 beginning beginning NOUN bracis-28417 120 15 of of ADP bracis-28417 120 16 the the DET bracis-28417 120 17 prompt prompt NOUN bracis-28417 120 18 . . PUNCT bracis-28417 121 1 following follow VERB bracis-28417 121 2 srivastava srivastava PROPN bracis-28417 121 3 et et PROPN bracis-28417 121 4 al al PROPN bracis-28417 121 5 . . PUNCT bracis-28417 121 6     SPACE bracis-28417 122 1 [ [ X bracis-28417 122 2 60 60 NUM bracis-28417 122 3 ] ] PUNCT bracis-28417 122 4 , , PUNCT bracis-28417 122 5 we we PRON bracis-28417 122 6 adopt adopt VERB bracis-28417 122 7 the the DET bracis-28417 122 8 normalized normalize VERB bracis-28417 122 9 preferred preferred ADJ bracis-28417 122 10 metric metric ADJ bracis-28417 122 11 ( ( PUNCT bracis-28417 122 12 npm npm PROPN bracis-28417 122 13 ) ) PUNCT bracis-28417 122 14 as as ADP bracis-28417 122 15 our our PRON bracis-28417 122 16 primary primary ADJ bracis-28417 122 17 evaluation evaluation NOUN bracis-28417 122 18 measure measure NOUN bracis-28417 122 19 : : PUNCT bracis-28417 122 20 $ $ SYM bracis-28417 122 21 $ $ SYM bracis-28417 122 22 \begin{aligned \begin{aligne VERB bracis-28417 122 23 } } PUNCT bracis-28417 122 24 \texttt \texttt PROPN bracis-28417 122 25 { { PUNCT bracis-28417 122 26 npm npm PROPN bracis-28417 122 27 } } PUNCT bracis-28417 122 28 = = SYM bracis-28417 122 29 \frac{1}{n \frac{1}{n NOUN bracis-28417 122 30 } } PUNCT bracis-28417 122 31 \sum \sum NOUN bracis-28417 122 32 _ _ PUNCT bracis-28417 122 33 { { PUNCT bracis-28417 122 34 i=1}^n i=1}^n NOUN bracis-28417 122 35 100 100 NUM bracis-28417 122 36 \times \time NOUN bracis-28417 122 37 \frac{\texttt \frac{\texttt NUM bracis-28417 122 38 { { PUNCT bracis-28417 122 39 [ [ PUNCT bracis-28417 122 40 raw raw ADJ bracis-28417 122 41 preferred preferred ADJ bracis-28417 122 42 metric]}_i metric]}_i NOUN bracis-28417 123 1 \texttt \texttt PROPN bracis-28417 123 2 { { PUNCT bracis-28417 123 3 [ [ X bracis-28417 123 4 random random ADJ bracis-28417 123 5 score]}_i}{\texttt score]}_i}{\texttt NOUN bracis-28417 123 6 { { PUNCT bracis-28417 123 7 [ [ X bracis-28417 123 8 high high ADJ bracis-28417 123 9 score]}_i score]}_i NOUN bracis-28417 124 1 \texttt \texttt ADP bracis-28417 124 2 { { PUNCT bracis-28417 124 3 [ [ X bracis-28417 124 4 random random ADJ bracis-28417 124 5 score]}_i score]}_i NOUN bracis-28417 124 6 } } PUNCT bracis-28417 124 7 \end{aligned}$$ \end{aligned}$$ X bracis-28417 124 8 ( ( PUNCT bracis-28417 124 9 1 1 NUM bracis-28417 124 10 ) ) PUNCT bracis-28417 124 11 where where SCONJ bracis-28417 124 12 n n PRON bracis-28417 124 13 is be AUX bracis-28417 124 14 the the DET bracis-28417 124 15 number number NOUN bracis-28417 124 16 of of ADP bracis-28417 124 17 evaluation evaluation NOUN bracis-28417 124 18 datasets dataset NOUN bracis-28417 124 19 , , PUNCT bracis-28417 124 20 \(\texttt \(\texttt X bracis-28417 124 21 { { PUNCT bracis-28417 124 22 [ [ PUNCT bracis-28417 124 23 raw raw ADJ bracis-28417 124 24 preferred preferred ADJ bracis-28417 124 25 metric]}_i\ metric]}_i\ NOUN bracis-28417 124 26 ) ) PUNCT bracis-28417 124 27 is be AUX bracis-28417 124 28 the the DET bracis-28417 124 29 score score NOUN bracis-28417 124 30 obtained obtain VERB bracis-28417 124 31 by by ADP bracis-28417 124 32 the the DET bracis-28417 124 33 model model NOUN bracis-28417 124 34 on on ADP bracis-28417 124 35 the the DET bracis-28417 124 36 i i PROPN bracis-28417 124 37 - - PUNCT bracis-28417 124 38 th th X bracis-28417 124 39 dataset dataset NOUN bracis-28417 124 40 , , PUNCT bracis-28417 124 41 \(\texttt \(\texttt NOUN bracis-28417 124 42 { { PUNCT bracis-28417 124 43 [ [ X bracis-28417 124 44 random random ADJ bracis-28417 124 45 score]}_i\ score]}_i\ NOUN bracis-28417 124 46 ) ) PUNCT bracis-28417 124 47 is be AUX bracis-28417 124 48 the the DET bracis-28417 124 49 score score NOUN bracis-28417 124 50 of of ADP bracis-28417 124 51 a a DET bracis-28417 124 52 random random ADJ bracis-28417 124 53 model model NOUN bracis-28417 124 54 ( ( PUNCT bracis-28417 124 55 e.g. e.g. ADV bracis-28417 124 56 , , PUNCT bracis-28417 124 57 50 50 NUM bracis-28417 124 58 % % NOUN bracis-28417 124 59 for for ADP bracis-28417 124 60 a a DET bracis-28417 124 61 binary binary ADJ bracis-28417 124 62 classification classification NOUN bracis-28417 124 63 task task NOUN bracis-28417 124 64 ) ) PUNCT bracis-28417 124 65 and and CCONJ bracis-28417 124 66 \(\texttt \(\texttt NOUN bracis-28417 124 67 { { PUNCT bracis-28417 124 68 [ [ PUNCT bracis-28417 124 69 high high ADJ bracis-28417 124 70 score]}_i\ score]}_i\ NOUN bracis-28417 124 71 ) ) PUNCT bracis-28417 124 72 is be AUX bracis-28417 124 73 the the DET bracis-28417 124 74 highest high ADJ bracis-28417 124 75 possible possible ADJ bracis-28417 124 76 score score NOUN bracis-28417 124 77 on on ADP bracis-28417 124 78 that that DET bracis-28417 124 79 dataset dataset NOUN bracis-28417 124 80 , , PUNCT bracis-28417 124 81 which which PRON bracis-28417 124 82 is be AUX bracis-28417 124 83 either either CCONJ bracis-28417 124 84 1 1 NUM bracis-28417 124 85 or or CCONJ bracis-28417 124 86 100 100 NUM bracis-28417 124 87 . . PUNCT bracis-28417 125 1 the the DET bracis-28417 125 2 preferred preferred ADJ bracis-28417 125 3 metric metric ADJ bracis-28417 125 4 and and CCONJ bracis-28417 125 5 random random ADJ bracis-28417 125 6 score score NOUN bracis-28417 125 7 for for ADP bracis-28417 125 8 each each DET bracis-28417 125 9 dataset dataset NOUN bracis-28417 125 10 are be AUX bracis-28417 125 11 presented present VERB bracis-28417 125 12 in in ADP bracis-28417 125 13 table table NOUN bracis-28417 125 14     SPACE bracis-28417 125 15 1 1 NUM bracis-28417 125 16 . . PUNCT bracis-28417 126 1 the the DET bracis-28417 126 2 rationale rationale NOUN bracis-28417 126 3 behind behind ADP bracis-28417 126 4 employing employ VERB bracis-28417 126 5 npm npm PROPN bracis-28417 126 6 rather rather ADV bracis-28417 126 7 than than ADP bracis-28417 126 8 a a DET bracis-28417 126 9 straightforward straightforward ADJ bracis-28417 126 10 average average NOUN bracis-28417 126 11 across across ADP bracis-28417 126 12 all all DET bracis-28417 126 13 datasets dataset NOUN bracis-28417 126 14 is be AUX bracis-28417 126 15 to to PART bracis-28417 126 16 mitigate mitigate VERB bracis-28417 126 17 the the DET bracis-28417 126 18 undue undue ADJ bracis-28417 126 19 influence influence NOUN bracis-28417 126 20 of of ADP bracis-28417 126 21 datasets dataset NOUN bracis-28417 126 22 with with ADP bracis-28417 126 23 inherently inherently ADV bracis-28417 126 24 high high ADJ bracis-28417 126 25 scores score NOUN bracis-28417 126 26 , , PUNCT bracis-28417 126 27 such such ADJ bracis-28417 126 28 as as ADP bracis-28417 126 29 binary binary ADJ bracis-28417 126 30 classification classification NOUN bracis-28417 126 31 datasets dataset NOUN bracis-28417 126 32 , , PUNCT bracis-28417 126 33 which which PRON bracis-28417 126 34 could could AUX bracis-28417 126 35 otherwise otherwise ADV bracis-28417 126 36 outweigh outweigh VERB bracis-28417 126 37 datasets dataset NOUN bracis-28417 126 38 characterized characterize VERB bracis-28417 126 39 by by ADP bracis-28417 126 40 lower low ADJ bracis-28417 126 41 scores score NOUN bracis-28417 126 42 . . PUNCT bracis-28417 127 1 table table NOUN bracis-28417 127 2 1 1 NUM bracis-28417 127 3 . . PUNCT bracis-28417 128 1 a a DET bracis-28417 128 2 summary summary NOUN bracis-28417 128 3 of of ADP bracis-28417 128 4 the the DET bracis-28417 128 5 datasets dataset NOUN bracis-28417 128 6 constituting constitute VERB bracis-28417 128 7 the the DET bracis-28417 128 8 poeta poeta NOUN bracis-28417 128 9 benchmark.full benchmark.full NOUN bracis-28417 128 10 size size NOUN bracis-28417 128 11 table table NOUN bracis-28417 128 12 5 5 NUM bracis-28417 128 13 results result VERB bracis-28417 128 14 the the DET bracis-28417 128 15 main main ADJ bracis-28417 128 16 results result NOUN bracis-28417 128 17 can can AUX bracis-28417 128 18 be be AUX bracis-28417 128 19 found find VERB bracis-28417 128 20 in in ADP bracis-28417 128 21 table table NOUN bracis-28417 128 22     SPACE bracis-28417 128 23 2 2 NUM bracis-28417 128 24 . . NOUN bracis-28417 128 25 models model NOUN bracis-28417 128 26 such such ADJ bracis-28417 128 27 as as ADP bracis-28417 128 28 bloomz bloomz ADJ bracis-28417 128 29 , , PUNCT bracis-28417 128 30 xglm xglm PROPN bracis-28417 128 31 and and CCONJ bracis-28417 128 32 bertin bertin PROPN bracis-28417 128 33 - - PUNCT bracis-28417 128 34 gpt gpt NOUN bracis-28417 128 35 struggled struggle VERB bracis-28417 128 36 to to PART bracis-28417 128 37 generate generate VERB bracis-28417 128 38 answers answer NOUN bracis-28417 128 39 in in ADP bracis-28417 128 40 portuguese portuguese PROPN bracis-28417 128 41 . . PUNCT bracis-28417 129 1 to to PART bracis-28417 129 2 address address VERB bracis-28417 129 3 this this DET bracis-28417 129 4 issue issue NOUN bracis-28417 129 5 , , PUNCT bracis-28417 129 6 we we PRON bracis-28417 129 7 adopted adopt VERB bracis-28417 129 8 an an DET bracis-28417 129 9 approach approach NOUN bracis-28417 129 10 akin akin ADJ bracis-28417 129 11 to to ADP bracis-28417 129 12 that that PRON bracis-28417 129 13 used use VERB bracis-28417 129 14 by by ADP bracis-28417 129 15 the the DET bracis-28417 129 16 xglm xglm PROPN bracis-28417 129 17 authors author NOUN bracis-28417 129 18 : : PUNCT bracis-28417 129 19 by by ADP bracis-28417 129 20 calculating calculate VERB bracis-28417 129 21 the the DET bracis-28417 129 22 likelihood likelihood NOUN bracis-28417 129 23 of of ADP bracis-28417 129 24 each each DET bracis-28417 129 25 candidate candidate NOUN bracis-28417 129 26 answer answer NOUN bracis-28417 129 27 string string NOUN bracis-28417 129 28 based base VERB bracis-28417 129 29 on on ADP bracis-28417 129 30 the the DET bracis-28417 129 31 input input NOUN bracis-28417 129 32 text text NOUN bracis-28417 129 33 and and CCONJ bracis-28417 129 34 subsequently subsequently ADV bracis-28417 129 35 selecting select VERB bracis-28417 129 36 the the DET bracis-28417 129 37 class class NOUN bracis-28417 129 38 with with ADP bracis-28417 129 39 the the DET bracis-28417 129 40 highest high ADJ bracis-28417 129 41 probability probability NOUN bracis-28417 129 42 . . PUNCT bracis-28417 130 1 for for ADP bracis-28417 130 2 faquad faquad ADJ bracis-28417 130 3 , , PUNCT bracis-28417 130 4 the the DET bracis-28417 130 5 only only ADJ bracis-28417 130 6 dataset dataset NOUN bracis-28417 130 7 in in ADP bracis-28417 130 8 the the DET bracis-28417 130 9 benchmark benchmark NOUN bracis-28417 130 10 without without ADP bracis-28417 130 11 predetermined predetermine VERB bracis-28417 130 12 candidate candidate NOUN bracis-28417 130 13 answers answer NOUN bracis-28417 130 14 , , PUNCT bracis-28417 130 15 we we PRON bracis-28417 130 16 allowed allow VERB bracis-28417 130 17 the the DET bracis-28417 130 18 models model NOUN bracis-28417 130 19 to to PART bracis-28417 130 20 generate generate VERB bracis-28417 130 21 answers answer NOUN bracis-28417 130 22 in in ADP bracis-28417 130 23 their their PRON bracis-28417 130 24 original original ADJ bracis-28417 130 25 format format NOUN bracis-28417 130 26 . . PUNCT bracis-28417 131 1 we we PRON bracis-28417 131 2 observe observe VERB bracis-28417 131 3 that that SCONJ bracis-28417 131 4 the the DET bracis-28417 131 5 llama llama PROPN bracis-28417 131 6 baselines baseline NOUN bracis-28417 131 7 significantly significantly ADV bracis-28417 131 8 outperform outperform VERB bracis-28417 131 9 models model NOUN bracis-28417 131 10 of of ADP bracis-28417 131 11 equivalent equivalent ADJ bracis-28417 131 12 size size NOUN bracis-28417 131 13 trained train VERB bracis-28417 131 14 with with ADP bracis-28417 131 15 fewer few ADJ bracis-28417 131 16 tokens token NOUN bracis-28417 131 17 , , PUNCT bracis-28417 131 18 such such ADJ bracis-28417 131 19 as as ADP bracis-28417 131 20 galactica galactica PROPN bracis-28417 131 21 and and CCONJ bracis-28417 131 22 opt opt VERB bracis-28417 131 23 . . PUNCT bracis-28417 132 1 furthermore furthermore ADV bracis-28417 132 2 , , PUNCT bracis-28417 132 3 despite despite SCONJ bracis-28417 132 4 being be AUX bracis-28417 132 5 trained train VERB bracis-28417 132 6 on on ADP bracis-28417 132 7 english english ADJ bracis-28417 132 8 - - PUNCT bracis-28417 132 9 centric centric NOUN bracis-28417 132 10 corpora corpora NOUN bracis-28417 132 11 , , PUNCT bracis-28417 132 12 llama-7b llama-7b X bracis-28417 132 13 surpasses surpass VERB bracis-28417 132 14 multilingual multilingual ADJ bracis-28417 132 15 bloom bloom NOUN bracis-28417 132 16 and and CCONJ bracis-28417 132 17 xglm xglm NOUN bracis-28417 132 18 of of ADP bracis-28417 132 19 similar similar ADJ bracis-28417 132 20 sizes size NOUN bracis-28417 132 21 . . PUNCT bracis-28417 133 1 the the DET bracis-28417 133 2 sabiá sabiá NOUN bracis-28417 133 3 models model NOUN bracis-28417 133 4 demonstrate demonstrate VERB bracis-28417 133 5 considerable considerable ADJ bracis-28417 133 6 improvement improvement NOUN bracis-28417 133 7 in in ADP bracis-28417 133 8 npm npm NOUN bracis-28417 133 9 compared compare VERB bracis-28417 133 10 to to ADP bracis-28417 133 11 their their PRON bracis-28417 133 12 respective respective ADJ bracis-28417 133 13 baseline baseline NOUN bracis-28417 133 14 models model NOUN bracis-28417 133 15 . . PUNCT bracis-28417 134 1 these these DET bracis-28417 134 2 npm npm PROPN bracis-28417 134 3 gains gain NOUN bracis-28417 134 4 are be AUX bracis-28417 134 5 more more ADV bracis-28417 134 6 substantial substantial ADJ bracis-28417 134 7 for for ADP bracis-28417 134 8 the the DET bracis-28417 134 9 smaller small ADJ bracis-28417 134 10 sabiá sabiá NOUN bracis-28417 134 11 - - PUNCT bracis-28417 134 12 j j PROPN bracis-28417 134 13 and and CCONJ bracis-28417 134 14 sabiá-7b sabiá-7b ADV bracis-28417 134 15 models model NOUN bracis-28417 134 16 . . PUNCT bracis-28417 135 1 notably notably ADV bracis-28417 135 2 , , PUNCT bracis-28417 135 3 sabiá-65b sabiá-65b PROPN bracis-28417 135 4 marginally marginally ADV bracis-28417 135 5 outperforms outperform VERB bracis-28417 135 6 openai openai PROPN bracis-28417 135 7 ’s ’s PART bracis-28417 135 8 gpt-3.5 gpt-3.5 NOUN bracis-28417 135 9 - - PUNCT bracis-28417 135 10 turbo turbo NOUN bracis-28417 135 11 , , PUNCT bracis-28417 135 12 which which PRON bracis-28417 135 13 serves serve VERB bracis-28417 135 14 as as ADP bracis-28417 135 15 the the DET bracis-28417 135 16 base base NOUN bracis-28417 135 17 model model NOUN bracis-28417 135 18 for for ADP bracis-28417 135 19 chatgpt chatgpt NOUN bracis-28417 135 20 . . PUNCT bracis-28417 136 1 table table NOUN bracis-28417 136 2 2 2 NUM bracis-28417 136 3 . . X bracis-28417 137 1 few few ADJ bracis-28417 137 2 - - PUNCT bracis-28417 137 3 shot shot NOUN bracis-28417 137 4 npm npm NOUN bracis-28417 137 5 results result NOUN bracis-28417 137 6 on on ADP bracis-28417 137 7 the the DET bracis-28417 137 8 poeta poeta NOUN bracis-28417 137 9 benchmark.full benchmark.full NOUN bracis-28417 137 10 size size NOUN bracis-28417 137 11 table table NOUN bracis-28417 137 12 through through ADP bracis-28417 137 13 our our PRON bracis-28417 137 14 portuguese portuguese ADJ bracis-28417 137 15 pretraining pretraine VERB bracis-28417 138 1 , , PUNCT bracis-28417 138 2 we we PRON bracis-28417 138 3 observed observe VERB bracis-28417 138 4 that that SCONJ bracis-28417 138 5 the the DET bracis-28417 138 6 improvement improvement NOUN bracis-28417 138 7 in in ADP bracis-28417 138 8 npm npm PROPN bracis-28417 138 9 was be AUX bracis-28417 138 10 higher high ADJ bracis-28417 138 11 in in ADP bracis-28417 138 12 native native ADJ bracis-28417 138 13 datasets dataset NOUN bracis-28417 138 14 than than ADP bracis-28417 138 15 that that PRON bracis-28417 138 16 in in ADP bracis-28417 138 17 translated translate VERB bracis-28417 138 18 datasets dataset NOUN bracis-28417 138 19 . . PUNCT bracis-28417 139 1 for for ADP bracis-28417 139 2 sabiá-65b sabiá-65b PROPN bracis-28417 139 3 , , PUNCT bracis-28417 139 4 improvements improvement NOUN bracis-28417 139 5 over over ADP bracis-28417 139 6 llama-65b llama-65b PROPN bracis-28417 139 7 were be AUX bracis-28417 139 8 mostly mostly ADV bracis-28417 139 9 from from ADP bracis-28417 139 10 the the DET bracis-28417 139 11 native native ADJ bracis-28417 139 12 subset subset NOUN bracis-28417 139 13 . . PUNCT bracis-28417 140 1 we we PRON bracis-28417 140 2 hypothesize hypothesize VERB bracis-28417 140 3 that that SCONJ bracis-28417 140 4 this this PRON bracis-28417 140 5 is be AUX bracis-28417 140 6 due due ADJ bracis-28417 140 7 to to ADP bracis-28417 140 8 the the DET bracis-28417 140 9 “ " PUNCT bracis-28417 140 10 mechanistic mechanistic ADJ bracis-28417 140 11 ” " PUNCT bracis-28417 140 12 nature nature NOUN bracis-28417 140 13 of of ADP bracis-28417 140 14 translated translate VERB bracis-28417 140 15 datasets dataset NOUN bracis-28417 140 16 : : PUNCT bracis-28417 140 17 since since SCONJ bracis-28417 140 18 they they PRON bracis-28417 140 19 were be AUX bracis-28417 140 20 translated translate VERB bracis-28417 140 21 from from ADP bracis-28417 140 22 english english PROPN bracis-28417 140 23 , , PUNCT bracis-28417 140 24 the the DET bracis-28417 140 25 baseline baseline PROPN bracis-28417 140 26 model model NOUN bracis-28417 140 27 already already ADV bracis-28417 140 28 possesses possess VERB bracis-28417 140 29 the the DET bracis-28417 140 30 knowledge knowledge NOUN bracis-28417 140 31 needed need VERB bracis-28417 140 32 to to PART bracis-28417 140 33 solve solve VERB bracis-28417 140 34 them they PRON bracis-28417 140 35 and and CCONJ bracis-28417 140 36 gains gain VERB bracis-28417 140 37 little little ADJ bracis-28417 140 38 from from ADP bracis-28417 140 39 learning learn VERB bracis-28417 140 40 the the DET bracis-28417 140 41 linguistic linguistic ADJ bracis-28417 140 42 , , PUNCT bracis-28417 140 43 syntactic syntactic ADJ bracis-28417 140 44 , , PUNCT bracis-28417 140 45 and and CCONJ bracis-28417 140 46 grammatical grammatical ADJ bracis-28417 140 47 knowledge knowledge NOUN bracis-28417 140 48 of of ADP bracis-28417 140 49 the the DET bracis-28417 140 50 target target NOUN bracis-28417 140 51 language language NOUN bracis-28417 140 52 . . PUNCT bracis-28417 141 1 for for ADP bracis-28417 141 2 instance instance NOUN bracis-28417 141 3 , , PUNCT bracis-28417 141 4 to to PART bracis-28417 141 5 answer answer VERB bracis-28417 141 6 the the DET bracis-28417 141 7 question question NOUN bracis-28417 141 8 “ " PUNCT bracis-28417 141 9 does do AUX bracis-28417 141 10 p p PROPN bracis-28417 141 11 o o PROPN bracis-28417 141 12 box box NOUN bracis-28417 141 13 come come VERB bracis-28417 141 14 before before ADP bracis-28417 141 15 street street NOUN bracis-28417 141 16 address address NOUN bracis-28417 141 17 ” " PUNCT bracis-28417 141 18 ( ( PUNCT bracis-28417 141 19 boolq boolq PROPN bracis-28417 141 20 dataset dataset PROPN bracis-28417 141 21 ) ) PUNCT bracis-28417 141 22 , , PUNCT bracis-28417 141 23 the the DET bracis-28417 141 24 model model NOUN bracis-28417 141 25 gains gain VERB bracis-28417 141 26 little little ADJ bracis-28417 141 27 from from ADP bracis-28417 141 28 additional additional ADJ bracis-28417 141 29 pretraining pretraining NOUN bracis-28417 141 30 on on ADP bracis-28417 141 31 a a DET bracis-28417 141 32 portuguese portuguese ADJ bracis-28417 141 33 corpus corpus NOUN bracis-28417 141 34 as as SCONJ bracis-28417 141 35 it it PRON bracis-28417 141 36 is be AUX bracis-28417 141 37 unlikely unlikely ADJ bracis-28417 141 38 that that SCONJ bracis-28417 141 39 the the DET bracis-28417 141 40 corpus corpus NOUN bracis-28417 141 41 would would AUX bracis-28417 141 42 provide provide VERB bracis-28417 141 43 new new ADJ bracis-28417 141 44 information information NOUN bracis-28417 141 45 regarding regard VERB bracis-28417 141 46 the the DET bracis-28417 141 47 formatting formatting NOUN bracis-28417 141 48 of of ADP bracis-28417 141 49 us we PRON bracis-28417 141 50 mailing mail VERB bracis-28417 141 51 addresses address NOUN bracis-28417 141 52 that that SCONJ bracis-28417 141 53 the the DET bracis-28417 141 54 model model NOUN bracis-28417 141 55 has have AUX bracis-28417 141 56 not not PART bracis-28417 141 57 already already ADV bracis-28417 141 58 encountered encounter VERB bracis-28417 141 59 during during ADP bracis-28417 141 60 its its PRON bracis-28417 141 61 initial initial ADJ bracis-28417 141 62 english english ADJ bracis-28417 141 63 - - PUNCT bracis-28417 141 64 centric centric ADJ bracis-28417 141 65 pretraining pretraining NOUN bracis-28417 141 66 . . PUNCT bracis-28417 142 1 conversely conversely ADV bracis-28417 142 2 , , PUNCT bracis-28417 142 3 language language NOUN bracis-28417 142 4 - - PUNCT bracis-28417 142 5 specific specific ADJ bracis-28417 142 6 pretraining pretraine VERB bracis-28417 142 7 introduces introduce NOUN bracis-28417 142 8 the the DET bracis-28417 142 9 specific specific ADJ bracis-28417 142 10 knowledge knowledge NOUN bracis-28417 142 11 required require VERB bracis-28417 142 12 to to PART bracis-28417 142 13 solve solve VERB bracis-28417 142 14 tasks task NOUN bracis-28417 142 15 in in ADP bracis-28417 142 16 the the DET bracis-28417 142 17 native native ADJ bracis-28417 142 18 subset subset NOUN bracis-28417 142 19 . . PUNCT bracis-28417 143 1 although although SCONJ bracis-28417 143 2 gpt gpt PROPN bracis-28417 143 3 - - PUNCT bracis-28417 143 4 j j NOUN bracis-28417 143 5 exhibited exhibit VERB bracis-28417 143 6 lower low ADJ bracis-28417 143 7 few few ADJ bracis-28417 143 8 - - PUNCT bracis-28417 143 9 shot shot NOUN bracis-28417 143 10 performance performance NOUN bracis-28417 143 11 in in ADP bracis-28417 143 12 english english ADJ bracis-28417 143 13 tasks task NOUN bracis-28417 143 14 relative relative ADJ bracis-28417 143 15 to to ADP bracis-28417 143 16 llama llama NOUN bracis-28417 143 17 , , PUNCT bracis-28417 143 18 we we PRON bracis-28417 143 19 use use VERB bracis-28417 143 20 it it PRON bracis-28417 143 21 in in ADP bracis-28417 143 22 this this DET bracis-28417 143 23 study study NOUN bracis-28417 143 24 to to PART bracis-28417 143 25 illustrate illustrate VERB bracis-28417 143 26 that that SCONJ bracis-28417 143 27 not not PART bracis-28417 143 28 only only ADV bracis-28417 143 29 highly highly ADV bracis-28417 143 30 optimized optimize VERB bracis-28417 143 31 models model NOUN bracis-28417 143 32 like like ADP bracis-28417 143 33 llama llama NOUN bracis-28417 143 34 can can AUX bracis-28417 143 35 benefit benefit VERB bracis-28417 143 36 from from ADP bracis-28417 143 37 extended extended ADJ bracis-28417 143 38 pretraining pretraining NOUN bracis-28417 143 39 . . PUNCT bracis-28417 144 1 we we PRON bracis-28417 144 2 chose choose VERB bracis-28417 144 3 not not PART bracis-28417 144 4 to to PART bracis-28417 144 5 use use VERB bracis-28417 144 6 bloom-7.1b bloom-7.1b NOUN bracis-28417 144 7 as as ADP bracis-28417 144 8 our our PRON bracis-28417 144 9 initial initial ADJ bracis-28417 144 10 checkpoint checkpoint NOUN bracis-28417 144 11 for for ADP bracis-28417 144 12 pretraining pretraine VERB bracis-28417 144 13 due due ADP bracis-28417 144 14 to to ADP bracis-28417 144 15 its its PRON bracis-28417 144 16 inferior inferior ADJ bracis-28417 144 17 performance performance NOUN bracis-28417 144 18 compared compare VERB bracis-28417 144 19 to to ADP bracis-28417 144 20 gpt gpt NOUN bracis-28417 144 21 - - PUNCT bracis-28417 144 22 j j NOUN bracis-28417 144 23 in in ADP bracis-28417 144 24 preliminary preliminary ADJ bracis-28417 144 25 few few ADJ bracis-28417 144 26 - - PUNCT bracis-28417 144 27 shot shot NOUN bracis-28417 144 28 experiments experiment NOUN bracis-28417 144 29 on on ADP bracis-28417 144 30 three three NUM bracis-28417 144 31 portuguese portuguese ADJ bracis-28417 144 32 datasets dataset NOUN bracis-28417 144 33 . . PUNCT bracis-28417 145 1 however however ADV bracis-28417 145 2 , , PUNCT bracis-28417 145 3 we we PRON bracis-28417 145 4 later later ADV bracis-28417 145 5 discovered discover VERB bracis-28417 145 6 that that SCONJ bracis-28417 145 7 its its PRON bracis-28417 145 8 performance performance NOUN bracis-28417 145 9 on on ADP bracis-28417 145 10 poeta poeta PROPN bracis-28417 145 11 surpassed surpass VERB bracis-28417 145 12 gpt gpt PROPN bracis-28417 145 13 - - PUNCT bracis-28417 145 14 j j NOUN bracis-28417 145 15 ’s ’s NOUN bracis-28417 145 16 . . PUNCT bracis-28417 146 1 nonetheless nonetheless ADV bracis-28417 146 2 , , PUNCT bracis-28417 146 3 bloom bloom NOUN bracis-28417 146 4 still still ADV bracis-28417 146 5 exhibits exhibit VERB bracis-28417 146 6 lower low ADJ bracis-28417 146 7 performance performance NOUN bracis-28417 146 8 compared compare VERB bracis-28417 146 9 to to ADP bracis-28417 146 10 llama llama PROPN bracis-28417 146 11 . . PUNCT bracis-28417 147 1 analogous analogous ADJ bracis-28417 147 2 to to ADP bracis-28417 147 3 sabiá sabiá NOUN bracis-28417 147 4 - - PUNCT bracis-28417 147 5 j j PROPN bracis-28417 147 6 , , PUNCT bracis-28417 147 7 bertin bertin PROPN bracis-28417 147 8 - - PUNCT bracis-28417 147 9 gpt gpt NOUN bracis-28417 147 10 is be AUX bracis-28417 147 11 a a DET bracis-28417 147 12 model model NOUN bracis-28417 147 13 pretrained pretraine VERB bracis-28417 147 14 on on ADP bracis-28417 147 15 spanish spanish ADJ bracis-28417 147 16 text text NOUN bracis-28417 147 17 starting start VERB bracis-28417 147 18 from from ADP bracis-28417 147 19 the the DET bracis-28417 147 20 gpt gpt PROPN bracis-28417 147 21 - - PUNCT bracis-28417 147 22 j j NOUN bracis-28417 147 23 weights weight NOUN bracis-28417 147 24 . . PUNCT bracis-28417 148 1 since since SCONJ bracis-28417 148 2 spanish spanish ADJ bracis-28417 148 3 and and CCONJ bracis-28417 148 4 portuguese portuguese ADJ bracis-28417 148 5 are be AUX bracis-28417 148 6 similar similar ADJ bracis-28417 148 7 languages language NOUN bracis-28417 148 8 , , PUNCT bracis-28417 148 9 it it PRON bracis-28417 148 10 is be AUX bracis-28417 148 11 reasonable reasonable ADJ bracis-28417 148 12 to to PART bracis-28417 148 13 expect expect VERB bracis-28417 148 14 that that SCONJ bracis-28417 148 15 bertin bertin NOUN bracis-28417 148 16 - - PUNCT bracis-28417 148 17 gpt gpt NOUN bracis-28417 148 18 would would AUX bracis-28417 148 19 perform perform VERB bracis-28417 148 20 better well ADJ bracis-28417 148 21 than than ADP bracis-28417 148 22 its its PRON bracis-28417 148 23 baseline baseline NOUN bracis-28417 148 24 model model NOUN bracis-28417 148 25 . . PUNCT bracis-28417 149 1 nevertheless nevertheless ADV bracis-28417 149 2 , , PUNCT bracis-28417 149 3 the the DET bracis-28417 149 4 observed observe VERB bracis-28417 149 5 npm npm NOUN bracis-28417 149 6 for for ADP bracis-28417 149 7 bertin bertin NOUN bracis-28417 149 8 - - PUNCT bracis-28417 149 9 gpt gpt NOUN bracis-28417 149 10 is be AUX bracis-28417 149 11 only only ADV bracis-28417 149 12 slightly slightly ADV bracis-28417 149 13 higher high ADJ bracis-28417 149 14 than than ADP bracis-28417 149 15 gpt gpt NOUN bracis-28417 149 16 - - PUNCT bracis-28417 149 17 j j NOUN bracis-28417 149 18 ’s ’s NOUN bracis-28417 149 19 . . PUNCT bracis-28417 150 1 a a DET bracis-28417 150 2 noteworthy noteworthy ADJ bracis-28417 150 3 comparison comparison NOUN bracis-28417 150 4 involves involve VERB bracis-28417 150 5 galactica galactica PROPN bracis-28417 150 6 , , PUNCT bracis-28417 150 7 a a DET bracis-28417 150 8 model model NOUN bracis-28417 150 9 pretrained pretraine VERB bracis-28417 150 10 on on ADP bracis-28417 150 11 scientific scientific ADJ bracis-28417 150 12 text text NOUN bracis-28417 150 13 , , PUNCT bracis-28417 150 14 predominantly predominantly ADV bracis-28417 150 15 in in ADP bracis-28417 150 16 english english PROPN bracis-28417 150 17 , , PUNCT bracis-28417 150 18 and and CCONJ bracis-28417 150 19 a a DET bracis-28417 150 20 similarly similarly ADV bracis-28417 150 21 - - PUNCT bracis-28417 150 22 sized sized ADJ bracis-28417 150 23 opt opt NOUN bracis-28417 150 24 model model NOUN bracis-28417 150 25 , , PUNCT bracis-28417 150 26 which which PRON bracis-28417 150 27 utilized utilize VERB bracis-28417 150 28 comparable comparable ADJ bracis-28417 150 29 pretraining pretraine VERB bracis-28417 150 30 compute compute NOUN bracis-28417 150 31 but but CCONJ bracis-28417 150 32 was be AUX bracis-28417 150 33 pretrained pretraine VERB bracis-28417 150 34 on on ADP bracis-28417 150 35 a a DET bracis-28417 150 36 larger large ADJ bracis-28417 150 37 and and CCONJ bracis-28417 150 38 more more ADV bracis-28417 150 39 diverse diverse ADJ bracis-28417 150 40 english english ADJ bracis-28417 150 41 - - PUNCT bracis-28417 150 42 centric centric NOUN bracis-28417 150 43 corpus corpus NOUN bracis-28417 150 44 . . PUNCT bracis-28417 151 1 in in ADP bracis-28417 151 2 their their PRON bracis-28417 151 3 study study NOUN bracis-28417 151 4 , , PUNCT bracis-28417 151 5 the the DET bracis-28417 151 6 authors author NOUN bracis-28417 151 7 demonstrate demonstrate VERB bracis-28417 151 8 that that SCONJ bracis-28417 151 9 galactica galactica PROPN bracis-28417 151 10 performs perform VERB bracis-28417 151 11 on on ADP bracis-28417 151 12 par par NOUN bracis-28417 151 13 with with ADP bracis-28417 151 14 opt opt NOUN bracis-28417 151 15 on on ADP bracis-28417 151 16 english english ADJ bracis-28417 151 17 tasks task NOUN bracis-28417 151 18 and and CCONJ bracis-28417 151 19 largely largely ADV bracis-28417 151 20 outperforms outperform VERB bracis-28417 151 21 opt opt NOUN bracis-28417 151 22 on on ADP bracis-28417 151 23 scientific scientific ADJ bracis-28417 151 24 - - PUNCT bracis-28417 151 25 related relate VERB bracis-28417 151 26 tasks task NOUN bracis-28417 151 27 . . PUNCT bracis-28417 152 1 conversely conversely ADV bracis-28417 152 2 , , PUNCT bracis-28417 152 3 opt opt VERB bracis-28417 152 4 significantly significantly ADV bracis-28417 152 5 outperforms outperform VERB bracis-28417 152 6 galactica galactica PROPN bracis-28417 152 7 in in ADP bracis-28417 152 8 portuguese portuguese ADJ bracis-28417 152 9 tasks task NOUN bracis-28417 152 10 . . PUNCT bracis-28417 153 1 this this DET bracis-28417 153 2 result result NOUN bracis-28417 153 3 underscores underscore VERB bracis-28417 153 4 the the DET bracis-28417 153 5 trade trade NOUN bracis-28417 153 6 - - PUNCT bracis-28417 153 7 offs off NOUN bracis-28417 153 8 associated associate VERB bracis-28417 153 9 with with ADP bracis-28417 153 10 domain domain NOUN bracis-28417 153 11 - - PUNCT bracis-28417 153 12 specific specific ADJ bracis-28417 153 13 specialization specialization NOUN bracis-28417 153 14 , , PUNCT bracis-28417 153 15 which which PRON bracis-28417 153 16 often often ADV bracis-28417 153 17 entails entail VERB bracis-28417 153 18 diminished diminished ADJ bracis-28417 153 19 performance performance NOUN bracis-28417 153 20 in in ADP bracis-28417 153 21 other other ADJ bracis-28417 153 22 tasks task NOUN bracis-28417 153 23 . . PUNCT bracis-28417 154 1 bloomz bloomz ADJ bracis-28417 154 2     SPACE bracis-28417 155 1 [ [ X bracis-28417 155 2 35 35 NUM bracis-28417 155 3 ] ] X bracis-28417 155 4 , , PUNCT bracis-28417 155 5 a a DET bracis-28417 155 6 multilingual multilingual ADJ bracis-28417 155 7 instruction instruction NOUN bracis-28417 155 8 - - PUNCT bracis-28417 155 9 tuned tune VERB bracis-28417 155 10 model model NOUN bracis-28417 155 11 , , PUNCT bracis-28417 155 12 demonstrated demonstrate VERB bracis-28417 155 13 superior superior ADJ bracis-28417 155 14 performance performance NOUN bracis-28417 155 15 compared compare VERB bracis-28417 155 16 to to ADP bracis-28417 155 17 its its PRON bracis-28417 155 18 baseline baseline ADJ bracis-28417 155 19 bloom bloom NOUN bracis-28417 155 20 model model NOUN bracis-28417 155 21 , , PUNCT bracis-28417 155 22 rivaling rival VERB bracis-28417 155 23 llama llama NOUN bracis-28417 155 24 of of ADP bracis-28417 155 25 equivalent equivalent ADJ bracis-28417 155 26 size.footnote size.footnote NUM bracis-28417 155 27 8 8 NUM bracis-28417 155 28 nevertheless nevertheless ADV bracis-28417 155 29 , , PUNCT bracis-28417 155 30 our our PRON bracis-28417 155 31 approach approach NOUN bracis-28417 155 32 of of ADP bracis-28417 155 33 pretraining pretraine VERB bracis-28417 155 34 in in ADP bracis-28417 155 35 portuguese portuguese PROPN bracis-28417 155 36 appears appear VERB bracis-28417 155 37 to to PART bracis-28417 155 38 yield yield VERB bracis-28417 155 39 superior superior ADJ bracis-28417 155 40 results result NOUN bracis-28417 155 41 , , PUNCT bracis-28417 155 42 as as SCONJ bracis-28417 155 43 sabiá sabiá NOUN bracis-28417 155 44 - - PUNCT bracis-28417 155 45 j j PROPN bracis-28417 155 46 surpasses surpass VERB bracis-28417 155 47 bloomz bloomz ADJ bracis-28417 155 48 despite despite SCONJ bracis-28417 155 49 originating originate VERB bracis-28417 155 50 from from ADP bracis-28417 155 51 a a DET bracis-28417 155 52 lower lower ADV bracis-28417 155 53 - - PUNCT bracis-28417 155 54 performing perform VERB bracis-28417 155 55 baseline baseline NOUN bracis-28417 155 56 model model NOUN bracis-28417 155 57 . . PUNCT bracis-28417 156 1 we we PRON bracis-28417 156 2 envision envision VERB bracis-28417 156 3 continued continued ADJ bracis-28417 156 4 pretraining pretraine VERB bracis-28417 156 5 and and CCONJ bracis-28417 156 6 instruction instruction NOUN bracis-28417 156 7 tuning tuning NOUN bracis-28417 156 8 as as ADP bracis-28417 156 9 complementary complementary ADJ bracis-28417 156 10 techniques technique NOUN bracis-28417 156 11 to to PART bracis-28417 156 12 be be AUX bracis-28417 156 13 combined combine VERB bracis-28417 156 14 in in ADP bracis-28417 156 15 future future ADJ bracis-28417 156 16 research research NOUN bracis-28417 156 17 . . PUNCT bracis-28417 157 1 5.1 5.1 NUM bracis-28417 157 2 results result NOUN bracis-28417 157 3 per per ADP bracis-28417 157 4     SPACE bracis-28417 157 5 dataset dataset NOUN bracis-28417 157 6 table table NOUN bracis-28417 157 7     SPACE bracis-28417 157 8 3 3 NUM bracis-28417 157 9 presents present VERB bracis-28417 157 10 the the DET bracis-28417 157 11 results result NOUN bracis-28417 157 12 per per ADP bracis-28417 157 13 poeta poeta PROPN bracis-28417 157 14 dataset dataset VERB bracis-28417 157 15 for for ADP bracis-28417 157 16 sabiá sabiá NOUN bracis-28417 157 17 models model NOUN bracis-28417 157 18 , , PUNCT bracis-28417 157 19 their their PRON bracis-28417 157 20 baselines baseline NOUN bracis-28417 157 21 , , PUNCT bracis-28417 157 22 and and CCONJ bracis-28417 157 23 for for ADP bracis-28417 157 24 the the DET bracis-28417 157 25 supervised supervised ADJ bracis-28417 157 26 state state NOUN bracis-28417 157 27 - - PUNCT bracis-28417 157 28 of of ADP bracis-28417 157 29 - - PUNCT bracis-28417 157 30 the the DET bracis-28417 157 31 - - PUNCT bracis-28417 157 32 art art NOUN bracis-28417 157 33 . . PUNCT bracis-28417 158 1 the the DET bracis-28417 158 2 sota sota PROPN bracis-28417 158 3 results result NOUN bracis-28417 158 4 reported report VERB bracis-28417 158 5 for for SCONJ bracis-28417 158 6 the the DET bracis-28417 158 7 translated translate VERB bracis-28417 158 8 datasets dataset NOUN bracis-28417 158 9 were be AUX bracis-28417 158 10 obtained obtain VERB bracis-28417 158 11 using use VERB bracis-28417 158 12 their their PRON bracis-28417 158 13 original original ADJ bracis-28417 158 14 english english ADJ bracis-28417 158 15 versions version NOUN bracis-28417 158 16     SPACE bracis-28417 159 1 [ [ X bracis-28417 159 2 46 46 NUM bracis-28417 159 3 , , PUNCT bracis-28417 159 4 51 51 NUM bracis-28417 159 5 , , PUNCT bracis-28417 159 6 71 71 NUM bracis-28417 159 7 , , PUNCT bracis-28417 159 8 76 76 NUM bracis-28417 159 9 ] ] PUNCT bracis-28417 159 10 . . PUNCT bracis-28417 160 1 since since SCONJ bracis-28417 160 2 the the DET bracis-28417 160 3 poeta poeta PROPN bracis-28417 160 4 benchmark benchmark NOUN bracis-28417 160 5 excludes exclude VERB bracis-28417 160 6 unanswerable unanswerable ADJ bracis-28417 160 7 examples example NOUN bracis-28417 160 8 of of ADP bracis-28417 160 9 the the DET bracis-28417 160 10 mkqa mkqa PROPN bracis-28417 160 11 dataset dataset PROPN bracis-28417 160 12 , , PUNCT bracis-28417 160 13 we we PRON bracis-28417 160 14 decided decide VERB bracis-28417 160 15 not not PART bracis-28417 160 16 to to PART bracis-28417 160 17 include include VERB bracis-28417 160 18 the the DET bracis-28417 160 19 sota sota ADJ bracis-28417 160 20 result result NOUN bracis-28417 160 21 for for ADP bracis-28417 160 22 this this DET bracis-28417 160 23 dataset dataset NOUN bracis-28417 160 24 . . PUNCT bracis-28417 161 1 in in ADP bracis-28417 161 2 more more ADV bracis-28417 161 3 challenging challenging ADJ bracis-28417 161 4 datasets dataset NOUN bracis-28417 161 5 , , PUNCT bracis-28417 161 6 such such ADJ bracis-28417 161 7 as as ADP bracis-28417 161 8 enem enem PROPN bracis-28417 161 9 challenge challenge NOUN bracis-28417 161 10 , , PUNCT bracis-28417 161 11 enem enem PROPN bracis-28417 161 12 2022 2022 NUM bracis-28417 161 13 , , PUNCT bracis-28417 161 14 and and CCONJ bracis-28417 161 15 bluex bluex PROPN bracis-28417 161 16 , , PUNCT bracis-28417 161 17 which which PRON bracis-28417 161 18 are be AUX bracis-28417 161 19 derived derive VERB bracis-28417 161 20 from from ADP bracis-28417 161 21 admission admission NOUN bracis-28417 161 22 exams exam NOUN bracis-28417 161 23 to to ADP bracis-28417 161 24 brazilian brazilian ADJ bracis-28417 161 25 universities university NOUN bracis-28417 161 26 , , PUNCT bracis-28417 161 27 we we PRON bracis-28417 161 28 see see VERB bracis-28417 161 29 the the DET bracis-28417 161 30 most most ADV bracis-28417 161 31 significant significant ADJ bracis-28417 161 32 gains gain NOUN bracis-28417 161 33 due due ADP bracis-28417 161 34 to to ADP bracis-28417 161 35 language language NOUN bracis-28417 161 36 - - PUNCT bracis-28417 161 37 specific specific ADJ bracis-28417 161 38 pretraining pretraining NOUN bracis-28417 161 39 . . PUNCT bracis-28417 162 1 substantial substantial ADJ bracis-28417 162 2 improvements improvement NOUN bracis-28417 162 3 are be AUX bracis-28417 162 4 also also ADV bracis-28417 162 5 observed observe VERB bracis-28417 162 6 in in ADP bracis-28417 162 7 tweetsentbr tweetsentbr NOUN bracis-28417 162 8 , , PUNCT bracis-28417 162 9 a a DET bracis-28417 162 10 dataset dataset NOUN bracis-28417 162 11 containing contain VERB bracis-28417 162 12 tweets tweet NOUN bracis-28417 162 13 with with ADP bracis-28417 162 14 an an DET bracis-28417 162 15 abundance abundance NOUN bracis-28417 162 16 of of ADP bracis-28417 162 17 slang slang PROPN bracis-28417 162 18 and and CCONJ bracis-28417 162 19 references reference NOUN bracis-28417 162 20 to to ADP bracis-28417 162 21 brazilian brazilian ADJ bracis-28417 162 22 popular popular ADJ bracis-28417 162 23 culture culture NOUN bracis-28417 162 24 . . PUNCT bracis-28417 163 1 we we PRON bracis-28417 163 2 hypothesize hypothesize VERB bracis-28417 163 3 that that SCONJ bracis-28417 163 4 this this DET bracis-28417 163 5 pretraining pretraine VERB bracis-28417 163 6 imparts impart VERB bracis-28417 163 7 specific specific ADJ bracis-28417 163 8 knowledge knowledge NOUN bracis-28417 163 9 about about ADP bracis-28417 163 10 the the DET bracis-28417 163 11 country country NOUN bracis-28417 163 12 ’s ’s PART bracis-28417 163 13 culture culture NOUN bracis-28417 163 14 , , PUNCT bracis-28417 163 15 literature literature NOUN bracis-28417 163 16 , , PUNCT bracis-28417 163 17 and and CCONJ bracis-28417 163 18 geography geography NOUN bracis-28417 163 19 that that PRON bracis-28417 163 20 is be AUX bracis-28417 163 21 less less ADV bracis-28417 163 22 frequently frequently ADV bracis-28417 163 23 encountered encounter VERB bracis-28417 163 24 and and CCONJ bracis-28417 163 25 learned learn VERB bracis-28417 163 26 during during ADP bracis-28417 163 27 the the DET bracis-28417 163 28 original original ADJ bracis-28417 163 29 pretraining pretraine VERB bracis-28417 163 30 with with ADP bracis-28417 163 31 more more ADV bracis-28417 163 32 diverse diverse ADJ bracis-28417 163 33 texts text NOUN bracis-28417 163 34 . . PUNCT bracis-28417 164 1 table table NOUN bracis-28417 164 2 3 3 NUM bracis-28417 164 3 . . PUNCT bracis-28417 164 4 results result NOUN bracis-28417 164 5 per per ADP bracis-28417 164 6 dataset dataset NOUN bracis-28417 164 7 . . PUNCT bracis-28417 165 1 \(^1\ \(^1\ PROPN bracis-28417 165 2 ) ) PUNCT bracis-28417 166 1 [ [ X bracis-28417 166 2 49 49 NUM bracis-28417 166 3 ] ] X bracis-28417 166 4 ; ; PUNCT bracis-28417 166 5 \(^2\ \(^2\ X bracis-28417 166 6 ) ) PUNCT bracis-28417 167 1 [ [ X bracis-28417 167 2 9 9 NUM bracis-28417 167 3 ] ] PUNCT bracis-28417 167 4 ; ; PUNCT bracis-28417 167 5 \(^3\ \(^3\ ADJ bracis-28417 167 6 ) ) PUNCT bracis-28417 168 1 [ [ X bracis-28417 168 2 34 34 NUM bracis-28417 168 3 ] ] X bracis-28417 168 4 ; ; PUNCT bracis-28417 168 5 \(^4\ \(^4\ NOUN bracis-28417 168 6 ) ) PUNCT bracis-28417 169 1 [ [ X bracis-28417 169 2 3 3 NUM bracis-28417 169 3 ] ] PUNCT bracis-28417 169 4 ; ; PUNCT bracis-28417 169 5 \(^5\ \(^5\ NOUN bracis-28417 169 6 ) ) PUNCT bracis-28417 170 1 [ [ X bracis-28417 170 2 71 71 NUM bracis-28417 170 3 ] ] X bracis-28417 170 4 ; ; PUNCT bracis-28417 170 5 \(^6\ \(^6\ NOUN bracis-28417 170 6 ) ) PUNCT bracis-28417 171 1 [ [ X bracis-28417 171 2 76 76 NUM bracis-28417 171 3 ] ] X bracis-28417 171 4 ; ; PUNCT bracis-28417 171 5 \(^7\ \(^7\ NOUN bracis-28417 171 6 ) ) PUNCT bracis-28417 172 1 [ [ X bracis-28417 172 2 46 46 NUM bracis-28417 172 3 ] ] X bracis-28417 172 4 ; ; PUNCT bracis-28417 172 5 \(^8\ \(^8\ NOUN bracis-28417 172 6 ) ) PUNCT bracis-28417 173 1 [ [ X bracis-28417 173 2 51].full 51].full NUM bracis-28417 173 3 size size NOUN bracis-28417 173 4 table table NOUN bracis-28417 173 5 certain certain ADJ bracis-28417 173 6 capabilities capability NOUN bracis-28417 173 7 only only ADV bracis-28417 173 8 emerge emerge VERB bracis-28417 173 9 at at ADP bracis-28417 173 10 scale scale NOUN bracis-28417 173 11 , , PUNCT bracis-28417 173 12 as as SCONJ bracis-28417 173 13 evidenced evidence VERB bracis-28417 173 14 by by ADP bracis-28417 173 15 [ [ X bracis-28417 173 16 67 67 NUM bracis-28417 173 17 ] ] PUNCT bracis-28417 173 18 . . PUNCT bracis-28417 174 1 for for ADP bracis-28417 174 2 example example NOUN bracis-28417 174 3 , , PUNCT bracis-28417 174 4 6 6 NUM bracis-28417 174 5 - - PUNCT bracis-28417 174 6 7b 7b NOUN bracis-28417 174 7 models model NOUN bracis-28417 174 8 perform perform VERB bracis-28417 174 9 close close ADV bracis-28417 174 10 to to ADP bracis-28417 174 11 the the DET bracis-28417 174 12 random random ADJ bracis-28417 174 13 baseline baseline NOUN bracis-28417 174 14 in in ADP bracis-28417 174 15 datasets dataset NOUN bracis-28417 174 16 such such ADJ bracis-28417 174 17 as as ADP bracis-28417 174 18 assin assin PROPN bracis-28417 174 19 2 2 NUM bracis-28417 174 20 rte rte NOUN bracis-28417 174 21 and and CCONJ bracis-28417 174 22 sts st NOUN bracis-28417 174 23 , , PUNCT bracis-28417 174 24 and and CCONJ bracis-28417 174 25 wsc wsc PROPN bracis-28417 174 26 . . PUNCT bracis-28417 175 1 however however ADV bracis-28417 175 2 , , PUNCT bracis-28417 175 3 at at ADP bracis-28417 175 4 the the DET bracis-28417 175 5 65b 65b ADJ bracis-28417 175 6 scale scale NOUN bracis-28417 175 7 , , PUNCT bracis-28417 175 8 we we PRON bracis-28417 175 9 observe observe VERB bracis-28417 175 10 substantial substantial ADJ bracis-28417 175 11 improvements improvement NOUN bracis-28417 175 12 , , PUNCT bracis-28417 175 13 approaching approach VERB bracis-28417 175 14 or or CCONJ bracis-28417 175 15 surpassing surpass VERB bracis-28417 175 16 state state NOUN bracis-28417 175 17 - - PUNCT bracis-28417 175 18 of of ADP bracis-28417 175 19 - - PUNCT bracis-28417 175 20 the the DET bracis-28417 175 21 - - PUNCT bracis-28417 175 22 art art NOUN bracis-28417 175 23 supervised supervise VERB bracis-28417 175 24 models model NOUN bracis-28417 175 25 on on ADP bracis-28417 175 26 the the DET bracis-28417 175 27 assin assin PROPN bracis-28417 175 28 2 2 NUM bracis-28417 175 29 rte rte NOUN bracis-28417 175 30 and and CCONJ bracis-28417 175 31 faquad faquad ADJ bracis-28417 175 32 datasets dataset NOUN bracis-28417 175 33 . . PUNCT bracis-28417 176 1 gpt-4 gpt-4 VERB bracis-28417 176 2     SPACE bracis-28417 176 3 [ [ X bracis-28417 176 4 39 39 NUM bracis-28417 176 5 ] ] PUNCT bracis-28417 176 6 results result NOUN bracis-28417 176 7 indicate indicate VERB bracis-28417 176 8 that that SCONJ bracis-28417 176 9 there there PRON bracis-28417 176 10 is be VERB bracis-28417 176 11 still still ADV bracis-28417 176 12 room room NOUN bracis-28417 176 13 for for ADP bracis-28417 176 14 improvement improvement NOUN bracis-28417 176 15 for for ADP bracis-28417 176 16 sabiá-65b sabiá-65b PROPN bracis-28417 176 17 in in ADP bracis-28417 176 18 the the DET bracis-28417 176 19 majority majority NOUN bracis-28417 176 20 of of ADP bracis-28417 176 21 the the DET bracis-28417 176 22 datasets dataset NOUN bracis-28417 176 23 evaluated evaluate VERB bracis-28417 176 24 in in ADP bracis-28417 176 25 this this DET bracis-28417 176 26 work work NOUN bracis-28417 176 27 . . PUNCT bracis-28417 177 1 nevertheless nevertheless ADV bracis-28417 177 2 , , PUNCT bracis-28417 177 3 sabiá-65b sabiá-65b PROPN bracis-28417 177 4 performs perform VERB bracis-28417 177 5 on on ADP bracis-28417 177 6 par par NOUN bracis-28417 177 7 with with ADP bracis-28417 177 8 gpt-4 gpt-4 NUM bracis-28417 177 9 in in ADP bracis-28417 177 10 datasets dataset NOUN bracis-28417 177 11 such such ADJ bracis-28417 177 12 as as ADP bracis-28417 177 13 assin assin PROPN bracis-28417 177 14 2 2 NUM bracis-28417 177 15 rte rte PROPN bracis-28417 177 16 , , PUNCT bracis-28417 177 17 enem enem NOUN bracis-28417 177 18 challenge challenge NOUN bracis-28417 177 19 , , PUNCT bracis-28417 177 20 and and CCONJ bracis-28417 177 21 faquad faquad ADJ bracis-28417 177 22 . . PUNCT bracis-28417 178 1 5.2 5.2 NUM bracis-28417 178 2 data datum NOUN bracis-28417 178 3 contamination contamination VERB bracis-28417 178 4 the the DET bracis-28417 178 5 pretraining pretraine VERB bracis-28417 178 6 data datum NOUN bracis-28417 178 7 for for ADP bracis-28417 178 8 sabiá sabiá NOUN bracis-28417 178 9 models model NOUN bracis-28417 178 10 were be AUX bracis-28417 178 11 collected collect VERB bracis-28417 178 12 up up ADP bracis-28417 178 13 until until ADP bracis-28417 178 14 february february PROPN bracis-28417 178 15 2022 2022 NUM bracis-28417 178 16 . . PUNCT bracis-28417 179 1 since since SCONJ bracis-28417 179 2 enem enem PROPN bracis-28417 179 3 2022 2022 NUM bracis-28417 179 4 was be AUX bracis-28417 179 5 publicly publicly ADV bracis-28417 179 6 released release VERB bracis-28417 179 7 in in ADP bracis-28417 179 8 november november PROPN bracis-28417 179 9 2022 2022 NUM bracis-28417 179 10 , , PUNCT bracis-28417 179 11 the the DET bracis-28417 179 12 model model NOUN bracis-28417 179 13 could could AUX bracis-28417 179 14 not not PART bracis-28417 179 15 have have VERB bracis-28417 179 16 access access NOUN bracis-28417 179 17 to to ADP bracis-28417 179 18 the the DET bracis-28417 179 19 answers answer NOUN bracis-28417 179 20 for for ADP bracis-28417 179 21 the the DET bracis-28417 179 22 questions question NOUN bracis-28417 179 23 present present ADJ bracis-28417 179 24 within within ADP bracis-28417 179 25 its its PRON bracis-28417 179 26 pretraining pretraine VERB bracis-28417 179 27 data datum NOUN bracis-28417 179 28 . . PUNCT bracis-28417 180 1 consequently consequently ADV bracis-28417 180 2 , , PUNCT bracis-28417 180 3 the the DET bracis-28417 180 4 improvements improvement NOUN bracis-28417 180 5 observed observe VERB bracis-28417 180 6 at at ADP bracis-28417 180 7 least least ADJ bracis-28417 180 8 for for ADP bracis-28417 180 9 enem enem PROPN bracis-28417 180 10 2022 2022 NUM bracis-28417 180 11 , , PUNCT bracis-28417 180 12 which which PRON bracis-28417 180 13 were be AUX bracis-28417 180 14 higher high ADJ bracis-28417 180 15 than than ADP bracis-28417 180 16 the the DET bracis-28417 180 17 average average NOUN bracis-28417 180 18 of of ADP bracis-28417 180 19 the the DET bracis-28417 180 20 datasets dataset NOUN bracis-28417 180 21 , , PUNCT bracis-28417 180 22 can can AUX bracis-28417 180 23 not not PART bracis-28417 180 24 be be AUX bracis-28417 180 25 attributed attribute VERB bracis-28417 180 26 to to ADP bracis-28417 180 27 data datum NOUN bracis-28417 180 28 contamination contamination NOUN bracis-28417 180 29 . . PUNCT bracis-28417 181 1 however however ADV bracis-28417 181 2 , , PUNCT bracis-28417 181 3 for for ADP bracis-28417 181 4 the the DET bracis-28417 181 5 other other ADJ bracis-28417 181 6 datasets dataset NOUN bracis-28417 181 7 , , PUNCT bracis-28417 181 8 the the DET bracis-28417 181 9 possibility possibility NOUN bracis-28417 181 10 of of ADP bracis-28417 181 11 data datum NOUN bracis-28417 181 12 contamination contamination NOUN bracis-28417 181 13 can can AUX bracis-28417 181 14 not not PART bracis-28417 181 15 be be AUX bracis-28417 181 16 ruled rule VERB bracis-28417 181 17 out out ADP bracis-28417 181 18 . . PUNCT bracis-28417 182 1 5.3 5.3 NUM bracis-28417 182 2 ablation ablation NOUN bracis-28417 182 3 : : PUNCT bracis-28417 182 4 english english ADJ bracis-28417 182 5 datasets dataset NOUN bracis-28417 182 6 in in ADP bracis-28417 182 7 this this DET bracis-28417 182 8 ablation ablation NOUN bracis-28417 182 9 study study NOUN bracis-28417 182 10 , , PUNCT bracis-28417 182 11 we we PRON bracis-28417 182 12 investigate investigate VERB bracis-28417 182 13 the the DET bracis-28417 182 14 potential potential ADJ bracis-28417 182 15 impact impact NOUN bracis-28417 182 16 of of ADP bracis-28417 182 17 portuguese portuguese NOUN bracis-28417 182 18 pretraining pretraine VERB bracis-28417 182 19 on on ADP bracis-28417 182 20 the the DET bracis-28417 182 21 performance performance NOUN bracis-28417 182 22 of of ADP bracis-28417 182 23 the the DET bracis-28417 182 24 model model NOUN bracis-28417 182 25 in in ADP bracis-28417 182 26 english english ADJ bracis-28417 182 27 datasets dataset NOUN bracis-28417 182 28 . . PUNCT bracis-28417 183 1 we we PRON bracis-28417 183 2 evaluated evaluate VERB bracis-28417 183 3 the the DET bracis-28417 183 4 llama-7b llama-7b PROPN bracis-28417 183 5 and and CCONJ bracis-28417 183 6 the the DET bracis-28417 183 7 sabiá-7b sabiá-7b ADJ bracis-28417 183 8 models model NOUN bracis-28417 183 9 in in ADP bracis-28417 183 10 english english ADJ bracis-28417 183 11 multiple multiple ADJ bracis-28417 183 12 - - PUNCT bracis-28417 183 13 choice choice NOUN bracis-28417 183 14 tasks task NOUN bracis-28417 183 15 . . PUNCT bracis-28417 184 1 for for ADP bracis-28417 184 2 simplicity simplicity NOUN bracis-28417 184 3 , , PUNCT bracis-28417 184 4 we we PRON bracis-28417 184 5 employed employ VERB bracis-28417 184 6 a a DET bracis-28417 184 7 few few ADJ bracis-28417 184 8 - - PUNCT bracis-28417 184 9 shot shot NOUN bracis-28417 184 10 evaluation evaluation NOUN bracis-28417 184 11 setup setup NOUN bracis-28417 184 12 with with ADP bracis-28417 184 13 10 10 NUM bracis-28417 184 14 randomly randomly ADV bracis-28417 184 15 selected select VERB bracis-28417 184 16 examples example NOUN bracis-28417 184 17 ( ( PUNCT bracis-28417 184 18 dynamic dynamic ADV bracis-28417 184 19 - - PUNCT bracis-28417 184 20 sampled sample VERB bracis-28417 184 21 prompt prompt NOUN bracis-28417 184 22 ) ) PUNCT bracis-28417 184 23 . . PUNCT bracis-28417 185 1 importantly importantly ADV bracis-28417 185 2 , , PUNCT bracis-28417 185 3 we we PRON bracis-28417 185 4 did do AUX bracis-28417 185 5 not not PART bracis-28417 185 6 incorporate incorporate VERB bracis-28417 185 7 any any DET bracis-28417 185 8 descriptions description NOUN bracis-28417 185 9 or or CCONJ bracis-28417 185 10 include include VERB bracis-28417 185 11 portuguese portuguese ADJ bracis-28417 185 12 keywords keyword NOUN bracis-28417 185 13 to to PART bracis-28417 185 14 delimit delimit VERB bracis-28417 185 15 the the DET bracis-28417 185 16 few few ADJ bracis-28417 185 17 - - PUNCT bracis-28417 185 18 shot shot NOUN bracis-28417 185 19 examples example NOUN bracis-28417 185 20 . . PUNCT bracis-28417 186 1 we we PRON bracis-28417 186 2 also also ADV bracis-28417 186 3 restricted restrict VERB bracis-28417 186 4 all all DET bracis-28417 186 5 the the DET bracis-28417 186 6 datasets dataset NOUN bracis-28417 186 7 to to ADP bracis-28417 186 8 350 350 NUM bracis-28417 186 9 test test NOUN bracis-28417 186 10 examples example NOUN bracis-28417 186 11 . . PUNCT bracis-28417 187 1 following follow VERB bracis-28417 187 2 llama llama PROPN bracis-28417 187 3 ’s ’s NOUN bracis-28417 187 4     SPACE bracis-28417 187 5 [ [ X bracis-28417 187 6 63 63 NUM bracis-28417 187 7 ] ] PUNCT bracis-28417 187 8 approach approach NOUN bracis-28417 187 9 , , PUNCT bracis-28417 187 10 given give VERB bracis-28417 187 11 the the DET bracis-28417 187 12 provided provide VERB bracis-28417 187 13 context context NOUN bracis-28417 187 14 , , PUNCT bracis-28417 187 15 we we PRON bracis-28417 187 16 select select VERB bracis-28417 187 17 the the DET bracis-28417 187 18 answer answer NOUN bracis-28417 187 19 with with ADP bracis-28417 187 20 the the DET bracis-28417 187 21 highest high ADJ bracis-28417 187 22 likelihood likelihood NOUN bracis-28417 187 23 normalized normalize VERB bracis-28417 187 24 by by ADP bracis-28417 187 25 the the DET bracis-28417 187 26 number number NOUN bracis-28417 187 27 of of ADP bracis-28417 187 28 characters character NOUN bracis-28417 187 29 . . PUNCT bracis-28417 188 1 the the DET bracis-28417 188 2 results result NOUN bracis-28417 188 3 in in ADP bracis-28417 188 4 table table NOUN bracis-28417 188 5     SPACE bracis-28417 188 6 4 4 NUM bracis-28417 188 7 indicate indicate VERB bracis-28417 188 8 that that SCONJ bracis-28417 188 9 the the DET bracis-28417 188 10 sabiá-7b sabiá-7b ADJ bracis-28417 188 11 model model NOUN bracis-28417 188 12 exhibits exhibit VERB bracis-28417 188 13 a a DET bracis-28417 188 14 slightly slightly ADV bracis-28417 188 15 reduced reduce VERB bracis-28417 188 16 performance performance NOUN bracis-28417 188 17 in in ADP bracis-28417 188 18 english english ADJ bracis-28417 188 19 tasks task NOUN bracis-28417 188 20 compared compare VERB bracis-28417 188 21 to to ADP bracis-28417 188 22 the the DET bracis-28417 188 23 baseline baseline NOUN bracis-28417 188 24 . . PUNCT bracis-28417 189 1 this this DET bracis-28417 189 2 result result NOUN bracis-28417 189 3 corroborates corroborate VERB bracis-28417 189 4 our our PRON bracis-28417 189 5 premise premise NOUN bracis-28417 189 6 that that SCONJ bracis-28417 189 7 model model NOUN bracis-28417 189 8 specialization specialization NOUN bracis-28417 189 9 invariably invariably ADV bracis-28417 189 10 entails entail VERB bracis-28417 189 11 a a DET bracis-28417 189 12 balancing balance VERB bracis-28417 189 13 act act NOUN bracis-28417 189 14 , , PUNCT bracis-28417 189 15 where where SCONJ bracis-28417 189 16 improvements improvement NOUN bracis-28417 189 17 in in ADP bracis-28417 189 18 one one NUM bracis-28417 189 19 domain domain NOUN bracis-28417 189 20 frequently frequently ADV bracis-28417 189 21 coincide coincide VERB bracis-28417 189 22 with with ADP bracis-28417 189 23 degradation degradation NOUN bracis-28417 189 24 in in ADP bracis-28417 189 25 another another PRON bracis-28417 189 26 . . PUNCT bracis-28417 190 1 table table NOUN bracis-28417 190 2 4 4 NUM bracis-28417 190 3 . . PUNCT bracis-28417 190 4 results result NOUN bracis-28417 190 5 in in ADP bracis-28417 190 6 english english ADJ bracis-28417 190 7 datasets.full datasets.full NUM bracis-28417 190 8 size size NOUN bracis-28417 190 9 table table NOUN bracis-28417 190 10 6 6 NUM bracis-28417 190 11 limitations limitation NOUN bracis-28417 190 12 owing owe VERB bracis-28417 190 13 to to ADP bracis-28417 190 14 the the DET bracis-28417 190 15 financial financial ADJ bracis-28417 190 16 constraints constraint NOUN bracis-28417 190 17 associated associate VERB bracis-28417 190 18 with with ADP bracis-28417 190 19 pretraining pretraine VERB bracis-28417 190 20 and and CCONJ bracis-28417 190 21 , , PUNCT bracis-28417 190 22 more more ADV bracis-28417 190 23 significantly significantly ADV bracis-28417 190 24 , , PUNCT bracis-28417 190 25 the the DET bracis-28417 190 26 manual manual ADJ bracis-28417 190 27 labor labor NOUN bracis-28417 190 28 involved involve VERB bracis-28417 190 29 in in ADP bracis-28417 190 30 collecting collect VERB bracis-28417 190 31 and and CCONJ bracis-28417 190 32 curating curate VERB bracis-28417 190 33 evaluation evaluation NOUN bracis-28417 190 34 datasets dataset NOUN bracis-28417 190 35 , , PUNCT bracis-28417 190 36 experiments experiment NOUN bracis-28417 190 37 were be AUX bracis-28417 190 38 conducted conduct VERB bracis-28417 190 39 exclusively exclusively ADV bracis-28417 190 40 in in ADP bracis-28417 190 41 portuguese portuguese NOUN bracis-28417 190 42 . . PUNCT bracis-28417 191 1 given give VERB bracis-28417 191 2 that that SCONJ bracis-28417 191 3 our our PRON bracis-28417 191 4 models model NOUN bracis-28417 191 5 started start VERB bracis-28417 191 6 pretraining pretraine VERB bracis-28417 191 7 from from ADP bracis-28417 191 8 english english PROPN bracis-28417 191 9 - - PUNCT bracis-28417 191 10 pretrained pretraine VERB bracis-28417 191 11 models model NOUN bracis-28417 191 12 and and CCONJ bracis-28417 191 13 that that SCONJ bracis-28417 191 14 portuguese portuguese ADJ bracis-28417 191 15 and and CCONJ bracis-28417 191 16 english english PROPN bracis-28417 191 17 exhibit exhibit VERB bracis-28417 191 18 relatively relatively ADV bracis-28417 191 19 close close ADJ bracis-28417 191 20 linguistic linguistic ADJ bracis-28417 191 21 proximity proximity NOUN bracis-28417 191 22 , , PUNCT bracis-28417 191 23 we we PRON bracis-28417 191 24 anticipate anticipate VERB bracis-28417 191 25 that that SCONJ bracis-28417 191 26 other other ADJ bracis-28417 191 27 researchers researcher NOUN bracis-28417 191 28 conducting conduct VERB bracis-28417 191 29 further far ADV bracis-28417 191 30 pretraining pretraine VERB bracis-28417 191 31 on on ADP bracis-28417 191 32 languages language NOUN bracis-28417 191 33 closely closely ADV bracis-28417 191 34 related relate VERB bracis-28417 191 35 to to ADP bracis-28417 191 36 english english PROPN bracis-28417 191 37 will will AUX bracis-28417 191 38 observe observe VERB bracis-28417 191 39 comparable comparable ADJ bracis-28417 191 40 improvements improvement NOUN bracis-28417 191 41 in in ADP bracis-28417 191 42 their their PRON bracis-28417 191 43 target target NOUN bracis-28417 191 44 tasks task NOUN bracis-28417 191 45 . . PUNCT bracis-28417 192 1 however however ADV bracis-28417 192 2 , , PUNCT bracis-28417 192 3 determining determine VERB bracis-28417 192 4 whether whether SCONJ bracis-28417 192 5 the the DET bracis-28417 192 6 benefits benefit NOUN bracis-28417 192 7 of of ADP bracis-28417 192 8 this this DET bracis-28417 192 9 method method NOUN bracis-28417 192 10 persist persist NOUN bracis-28417 192 11 for for ADP bracis-28417 192 12 languages language NOUN bracis-28417 192 13 more more ADV bracis-28417 192 14 distant distant ADJ bracis-28417 192 15 from from ADP bracis-28417 192 16 english english PROPN bracis-28417 192 17 remains remain VERB bracis-28417 192 18 an an DET bracis-28417 192 19 open open ADJ bracis-28417 192 20 research research NOUN bracis-28417 192 21 question question NOUN bracis-28417 192 22 . . PUNCT bracis-28417 193 1 portuguese portuguese PROPN bracis-28417 193 2 is be AUX bracis-28417 193 3 a a DET bracis-28417 193 4 language language NOUN bracis-28417 193 5 with with ADP bracis-28417 193 6 an an DET bracis-28417 193 7 abundance abundance NOUN bracis-28417 193 8 of of ADP bracis-28417 193 9 high high ADJ bracis-28417 193 10 - - PUNCT bracis-28417 193 11 quality quality NOUN bracis-28417 193 12 web web NOUN bracis-28417 193 13 - - PUNCT bracis-28417 193 14 based base VERB bracis-28417 193 15 texts text NOUN bracis-28417 193 16 . . PUNCT bracis-28417 194 1 thus thus ADV bracis-28417 194 2 , , PUNCT bracis-28417 194 3 the the DET bracis-28417 194 4 gains gain NOUN bracis-28417 194 5 observed observe VERB bracis-28417 194 6 with with ADP bracis-28417 194 7 the the DET bracis-28417 194 8 proposed propose VERB bracis-28417 194 9 method method NOUN bracis-28417 194 10 may may AUX bracis-28417 194 11 not not PART bracis-28417 194 12 necessarily necessarily ADV bracis-28417 194 13 extend extend VERB bracis-28417 194 14 to to ADP bracis-28417 194 15 low low ADJ bracis-28417 194 16 - - PUNCT bracis-28417 194 17 resource resource NOUN bracis-28417 194 18 languages language NOUN bracis-28417 194 19 with with ADP bracis-28417 194 20 limited limited ADJ bracis-28417 194 21 availability availability NOUN bracis-28417 194 22 of of ADP bracis-28417 194 23 quality quality NOUN bracis-28417 194 24 texts text NOUN bracis-28417 194 25 . . PUNCT bracis-28417 195 1 in in ADP bracis-28417 195 2 such such ADJ bracis-28417 195 3 cases case NOUN bracis-28417 195 4 , , PUNCT bracis-28417 195 5 parameter parameter NOUN bracis-28417 195 6 - - PUNCT bracis-28417 195 7 efficient efficient ADJ bracis-28417 195 8 methods method NOUN bracis-28417 195 9     SPACE bracis-28417 196 1 [ [ X bracis-28417 196 2 19 19 NUM bracis-28417 196 3 , , PUNCT bracis-28417 196 4 42 42 NUM bracis-28417 196 5 , , PUNCT bracis-28417 196 6 43 43 NUM bracis-28417 196 7 ] ] PUNCT bracis-28417 196 8 could could AUX bracis-28417 196 9 be be AUX bracis-28417 196 10 advantageous advantageous ADJ bracis-28417 196 11 , , PUNCT bracis-28417 196 12 as as SCONJ bracis-28417 196 13 evidenced evidence VERB bracis-28417 196 14 by by ADP bracis-28417 196 15 yong yong PROPN bracis-28417 196 16 et et PROPN bracis-28417 196 17 al al PROPN bracis-28417 196 18 . . PUNCT bracis-28417 196 19     SPACE bracis-28417 197 1 [ [ X bracis-28417 197 2 72 72 NUM bracis-28417 197 3 ] ] PUNCT bracis-28417 197 4 . . PUNCT bracis-28417 198 1 we we PRON bracis-28417 198 2 did do AUX bracis-28417 198 3 not not PART bracis-28417 198 4 use use VERB bracis-28417 198 5 these these DET bracis-28417 198 6 techniques technique NOUN bracis-28417 198 7 in in ADP bracis-28417 198 8 this this DET bracis-28417 198 9 study study NOUN bracis-28417 198 10 due due ADP bracis-28417 198 11 to to ADP bracis-28417 198 12 the the DET bracis-28417 198 13 training training NOUN bracis-28417 198 14 costs cost NOUN bracis-28417 198 15 , , PUNCT bracis-28417 198 16 which which PRON bracis-28417 198 17 are be AUX bracis-28417 198 18 approximately approximately ADV bracis-28417 198 19 equivalent equivalent ADJ bracis-28417 198 20 to to ADP bracis-28417 198 21 training train VERB bracis-28417 198 22 the the DET bracis-28417 198 23 entire entire ADJ bracis-28417 198 24 model.footnote model.footnote NOUN bracis-28417 198 25 9 9 NUM bracis-28417 198 26 7 7 NUM bracis-28417 198 27 conclusion conclusion NOUN bracis-28417 198 28 in in ADP bracis-28417 198 29 this this DET bracis-28417 198 30 study study NOUN bracis-28417 198 31 , , PUNCT bracis-28417 198 32 we we PRON bracis-28417 198 33 contributed contribute VERB bracis-28417 198 34 to to ADP bracis-28417 198 35 the the DET bracis-28417 198 36 expanding expand VERB bracis-28417 198 37 body body NOUN bracis-28417 198 38 of of ADP bracis-28417 198 39 scientific scientific ADJ bracis-28417 198 40 evidence evidence NOUN bracis-28417 198 41 that that SCONJ bracis-28417 198 42 specializing specialize VERB bracis-28417 198 43 models model NOUN bracis-28417 198 44 for for ADP bracis-28417 198 45 individual individual ADJ bracis-28417 198 46 languages language NOUN bracis-28417 198 47 leads lead VERB bracis-28417 198 48 to to ADP bracis-28417 198 49 improvements improvement NOUN bracis-28417 198 50 , , PUNCT bracis-28417 198 51 even even ADV bracis-28417 198 52 when when SCONJ bracis-28417 198 53 the the DET bracis-28417 198 54 baseline baseline NOUN bracis-28417 198 55 model model NOUN bracis-28417 198 56 is be AUX bracis-28417 198 57 large large ADJ bracis-28417 198 58 and and CCONJ bracis-28417 198 59 extensively extensively ADV bracis-28417 198 60 trained train VERB bracis-28417 198 61 . . PUNCT bracis-28417 199 1 we we PRON bracis-28417 199 2 achieved achieve VERB bracis-28417 199 3 this this PRON bracis-28417 199 4 for for ADP bracis-28417 199 5 the the DET bracis-28417 199 6 portuguese portuguese ADJ bracis-28417 199 7 language language NOUN bracis-28417 199 8 utilizing utilize VERB bracis-28417 199 9 a a DET bracis-28417 199 10 near near ADJ bracis-28417 199 11 state state NOUN bracis-28417 199 12 - - PUNCT bracis-28417 199 13 of of ADP bracis-28417 199 14 - - PUNCT bracis-28417 199 15 the the DET bracis-28417 199 16 - - PUNCT bracis-28417 199 17 art art NOUN bracis-28417 199 18 model model NOUN bracis-28417 199 19 with with ADP bracis-28417 199 20 65 65 NUM bracis-28417 199 21 billion billion NUM bracis-28417 199 22 parameters parameter NOUN bracis-28417 199 23 . . PUNCT bracis-28417 200 1 given give VERB bracis-28417 200 2 the the DET bracis-28417 200 3 relatively relatively ADV bracis-28417 200 4 low low ADJ bracis-28417 200 5 pretraining pretraine VERB bracis-28417 200 6 cost cost NOUN bracis-28417 200 7 and and CCONJ bracis-28417 200 8 significant significant ADJ bracis-28417 200 9 performance performance NOUN bracis-28417 200 10 gains gain NOUN bracis-28417 200 11 observed observe VERB bracis-28417 200 12 , , PUNCT bracis-28417 200 13 we we PRON bracis-28417 200 14 foresee foresee VERB bracis-28417 200 15 a a DET bracis-28417 200 16 future future ADJ bracis-28417 200 17 landscape landscape NOUN bracis-28417 200 18 consisting consist VERB bracis-28417 200 19 of of ADP bracis-28417 200 20 a a DET bracis-28417 200 21 diverse diverse ADJ bracis-28417 200 22 array array NOUN bracis-28417 200 23 of of ADP bracis-28417 200 24 models model NOUN bracis-28417 200 25 , , PUNCT bracis-28417 200 26 each each PRON bracis-28417 200 27 tailored tailor VERB bracis-28417 200 28 to to ADP bracis-28417 200 29 a a DET bracis-28417 200 30 specific specific ADJ bracis-28417 200 31 domain domain NOUN bracis-28417 200 32 , , PUNCT bracis-28417 200 33 rather rather ADV bracis-28417 200 34 than than ADP bracis-28417 200 35 a a DET bracis-28417 200 36 single single ADJ bracis-28417 200 37 , , PUNCT bracis-28417 200 38 all all PRON bracis-28417 200 39 - - PUNCT bracis-28417 200 40 encompassing encompass VERB bracis-28417 200 41 model model NOUN bracis-28417 200 42 . . PUNCT bracis-28417 201 1 notes note NOUN bracis-28417 201 2 1.to 1.to PRON bracis-28417 201 3 serve serve VERB bracis-28417 201 4 their their PRON bracis-28417 201 5 ensemble ensemble ADJ bracis-28417 201 6 with with ADP bracis-28417 201 7 a a DET bracis-28417 201 8 low low ADJ bracis-28417 201 9 latency latency NOUN bracis-28417 201 10 , , PUNCT bracis-28417 201 11 the the DET bracis-28417 201 12 weights weight NOUN bracis-28417 201 13 for for ADP bracis-28417 201 14 each each DET bracis-28417 201 15 expert expert NOUN bracis-28417 201 16 must must AUX bracis-28417 201 17 be be AUX bracis-28417 201 18 kept keep VERB bracis-28417 201 19 in in ADP bracis-28417 201 20 gpu gpu NOUN bracis-28417 201 21 memory memory NOUN bracis-28417 201 22 . . PUNCT bracis-28417 202 1 2.ftfy 2.ftfy NUM bracis-28417 202 2 normalization normalization NOUN bracis-28417 202 3 fixes fix VERB bracis-28417 202 4 mojibakes mojibake NOUN bracis-28417 202 5 and and CCONJ bracis-28417 202 6 remove remove VERB bracis-28417 202 7 remnant remnant ADJ bracis-28417 202 8 html html NOUN bracis-28417 202 9 tags tag NOUN bracis-28417 202 10 . . PUNCT bracis-28417 203 1 3.sabiá 3.sabiá NUM bracis-28417 203 2 is be AUX bracis-28417 203 3 a a DET bracis-28417 203 4 tribute tribute NOUN bracis-28417 203 5 to to ADP bracis-28417 203 6 the the DET bracis-28417 203 7 eponymous eponymous ADJ bracis-28417 203 8 bird bird NOUN bracis-28417 203 9 , , PUNCT bracis-28417 203 10 renowned renowne VERB bracis-28417 203 11 for for ADP bracis-28417 203 12 its its PRON bracis-28417 203 13 diverse diverse ADJ bracis-28417 203 14 and and CCONJ bracis-28417 203 15 intricate intricate ADJ bracis-28417 203 16 vocalizations vocalization NOUN bracis-28417 203 17 . . PUNCT bracis-28417 204 1 4.considering 4.considere VERB bracis-28417 204 2 the the DET bracis-28417 204 3 on on ADP bracis-28417 204 4 - - PUNCT bracis-28417 204 5 demand demand NOUN bracis-28417 204 6 pricing pricing NOUN bracis-28417 204 7 of of ADP bracis-28417 204 8 384 384 NUM bracis-28417 204 9 usd usd NOUN bracis-28417 204 10 per per ADP bracis-28417 204 11 hour hour NOUN bracis-28417 204 12 for for ADP bracis-28417 204 13 a a DET bracis-28417 204 14 tpu tpu NOUN bracis-28417 204 15 v2 v2 PROPN bracis-28417 204 16 - - PUNCT bracis-28417 204 17 512 512 NUM bracis-28417 204 18 , , PUNCT bracis-28417 204 19 pretraining pretraine VERB bracis-28417 204 20 sabiá-7b sabiá-7b ADV bracis-28417 204 21 and and CCONJ bracis-28417 204 22 sabiá-65b sabiá-65b PROPN bracis-28417 204 23 costs cost VERB bracis-28417 204 24 approximately approximately ADV bracis-28417 204 25 9,000 9,000 NUM bracis-28417 204 26 and and CCONJ bracis-28417 204 27 80,000 80,000 NUM bracis-28417 204 28 usd usd NOUN bracis-28417 204 29 , , PUNCT bracis-28417 204 30 respectively respectively ADV bracis-28417 204 31 . . PUNCT bracis-28417 205 1 5.https://huggingface.co/eleutherai/gpt-j-6b 5.https://huggingface.co/eleutherai/gpt-j-6b NUM bracis-28417 205 2 . . PUNCT bracis-28417 206 1 6.due 6.due NUM bracis-28417 206 2 to to ADP bracis-28417 206 3 constraints constraint NOUN bracis-28417 206 4 in in ADP bracis-28417 206 5 our our PRON bracis-28417 206 6 hardware hardware NOUN bracis-28417 206 7 budget budget NOUN bracis-28417 206 8 , , PUNCT bracis-28417 206 9 this this DET bracis-28417 206 10 model model NOUN bracis-28417 206 11 was be AUX bracis-28417 206 12 trained train VERB bracis-28417 206 13 with with ADP bracis-28417 206 14 fewer few ADJ bracis-28417 206 15 tokens token NOUN bracis-28417 206 16 compared compare VERB bracis-28417 206 17 to to AUX bracis-28417 206 18 sabiá sabiá VERB bracis-28417 206 19 . . PUNCT bracis-28417 207 1 7.the 7.the DET bracis-28417 207 2 massive massive ADJ bracis-28417 207 3 dataset dataset ADJ bracis-28417 207 4 underwent underwent NOUN bracis-28417 207 5 manual manual ADJ bracis-28417 207 6 translation translation NOUN bracis-28417 207 7 and and CCONJ bracis-28417 207 8 localization localization NOUN bracis-28417 207 9 ; ; PUNCT bracis-28417 207 10 however however ADV bracis-28417 207 11 , , PUNCT bracis-28417 207 12 given give VERB bracis-28417 207 13 that that SCONJ bracis-28417 207 14 the the DET bracis-28417 207 15 original original ADJ bracis-28417 207 16 text text NOUN bracis-28417 207 17 was be AUX bracis-28417 207 18 composed compose VERB bracis-28417 207 19 in in ADP bracis-28417 207 20 english english PROPN bracis-28417 207 21 , , PUNCT bracis-28417 207 22 it it PRON bracis-28417 207 23 has have AUX bracis-28417 207 24 been be AUX bracis-28417 207 25 categorized categorize VERB bracis-28417 207 26 as as ADP bracis-28417 207 27 a a DET bracis-28417 207 28 translated translate VERB bracis-28417 207 29 dataset dataset NOUN bracis-28417 207 30 . . PUNCT bracis-28417 208 1 8.this 8.this NUM bracis-28417 208 2 model model NOUN bracis-28417 208 3 was be AUX bracis-28417 208 4 used use VERB bracis-28417 208 5 in in ADP bracis-28417 208 6 the the DET bracis-28417 208 7 experiments experiment NOUN bracis-28417 208 8 : : PUNCT bracis-28417 208 9 https://huggingface.co/bigscience/bloomz-7b1-mt https://huggingface.co/bigscience/bloomz-7b1-mt X bracis-28417 208 10 . . PUNCT bracis-28417 209 1 9.although 9.although NUM bracis-28417 209 2 parameter parameter NOUN bracis-28417 209 3 - - PUNCT bracis-28417 209 4 efficient efficient ADJ bracis-28417 209 5 methods method NOUN bracis-28417 209 6 adjust adjust VERB bracis-28417 209 7 only only ADV bracis-28417 209 8 a a DET bracis-28417 209 9 fraction fraction NOUN bracis-28417 209 10 of of ADP bracis-28417 209 11 the the DET bracis-28417 209 12 weights weight NOUN bracis-28417 209 13 , , PUNCT bracis-28417 209 14 they they PRON bracis-28417 209 15 use use VERB bracis-28417 209 16 only only ADV bracis-28417 209 17 marginally marginally ADV bracis-28417 209 18 fewer few ADJ bracis-28417 209 19 training training NOUN bracis-28417 209 20 flops flop NOUN bracis-28417 209 21 , , PUNCT bracis-28417 209 22 as as SCONJ bracis-28417 209 23 activations activation NOUN bracis-28417 209 24 and and CCONJ bracis-28417 209 25 gradients gradient NOUN bracis-28417 209 26 are be AUX bracis-28417 209 27 computed compute VERB bracis-28417 209 28 for for ADP bracis-28417 209 29 the the DET bracis-28417 209 30 entire entire ADJ bracis-28417 209 31 model model NOUN bracis-28417 209 32 . . PUNCT bracis-28417 210 1 for for ADP bracis-28417 210 2 instance instance NOUN bracis-28417 210 3 , , PUNCT bracis-28417 210 4 lora lora PROPN bracis-28417 210 5     SPACE bracis-28417 211 1 [ [ X bracis-28417 211 2 20 20 NUM bracis-28417 211 3 ] ] PUNCT bracis-28417 211 4 , , PUNCT bracis-28417 211 5 a a DET bracis-28417 211 6 parameter parameter NOUN bracis-28417 211 7 - - PUNCT bracis-28417 211 8 efficient efficient ADJ bracis-28417 211 9 method method NOUN bracis-28417 211 10 , , PUNCT bracis-28417 211 11 improves improve VERB bracis-28417 211 12 training train VERB bracis-28417 211 13 throughput throughput NOUN bracis-28417 211 14 of of ADP bracis-28417 211 15 a a DET bracis-28417 211 16 gpt-3 gpt-3 NUM bracis-28417 211 17 175b 175b NUM bracis-28417 211 18 model model NOUN bracis-28417 211 19 by by ADP bracis-28417 211 20 only only ADV bracis-28417 211 21 nearly nearly ADV bracis-28417 211 22 32 32 NUM bracis-28417 211 23 % % NOUN bracis-28417 211 24 . . PUNCT bracis-28417 212 1 references reference NOUN bracis-28417 212 2 almeida almeida PROPN bracis-28417 212 3 , , PUNCT bracis-28417 212 4 t.s t.s PROPN bracis-28417 212 5 . . PROPN bracis-28417 212 6 , , PUNCT bracis-28417 212 7 laitz laitz PROPN bracis-28417 212 8 , , PUNCT bracis-28417 212 9 t. t. PROPN bracis-28417 212 10 , , PUNCT bracis-28417 212 11 bonás bonás PROPN bracis-28417 212 12 , , PUNCT bracis-28417 212 13 g.k g.k PROPN bracis-28417 212 14 . . PROPN bracis-28417 212 15 , , PUNCT bracis-28417 212 16 nogueira nogueira PROPN bracis-28417 212 17 , , PUNCT bracis-28417 212 18 r. r. PROPN bracis-28417 212 19 : : PUNCT bracis-28417 212 20 bluex bluex PROPN bracis-28417 212 21 : : PUNCT bracis-28417 212 22 a a DET bracis-28417 212 23 benchmark benchmark NOUN bracis-28417 212 24 based base VERB bracis-28417 212 25 on on ADP bracis-28417 212 26 brazilian brazilian ADJ bracis-28417 212 27 leading lead VERB bracis-28417 212 28 universities university NOUN bracis-28417 212 29 entrance entrance NOUN bracis-28417 212 30 exams exam NOUN bracis-28417 212 31 . . PUNCT bracis-28417 213 1 to to PART bracis-28417 213 2 appear appear VERB bracis-28417 213 3 ( ( PUNCT bracis-28417 213 4 2023 2023 NUM bracis-28417 213 5 ) ) PUNCT bracis-28417 213 6 google google NOUN bracis-28417 213 7 scholar scholar NOUN bracis-28417 213 8     SPACE bracis-28417 213 9 antoun antoun NOUN bracis-28417 213 10 , , PUNCT bracis-28417 213 11 w. w. PROPN bracis-28417 213 12 , , PUNCT bracis-28417 213 13 baly baly PROPN bracis-28417 213 14 , , PUNCT bracis-28417 213 15 f. f. PROPN bracis-28417 213 16 , , PUNCT bracis-28417 213 17 hajj hajj PROPN bracis-28417 213 18 , , PUNCT bracis-28417 213 19 h. h. PROPN bracis-28417 213 20 : : PUNCT bracis-28417 213 21 arabert arabert PROPN bracis-28417 213 22 : : PUNCT bracis-28417 213 23 transformer transformer NOUN bracis-28417 213 24 - - PUNCT bracis-28417 213 25 based base VERB bracis-28417 213 26 model model NOUN bracis-28417 213 27 for for ADP bracis-28417 213 28 arabic arabic ADJ bracis-28417 213 29 language language NOUN bracis-28417 213 30 understanding understanding NOUN bracis-28417 213 31 . . PUNCT bracis-28417 214 1 in in ADP bracis-28417 214 2 : : PUNCT bracis-28417 214 3 proceedings proceeding NOUN bracis-28417 214 4 of of ADP bracis-28417 214 5 the the DET bracis-28417 214 6 4th 4th ADJ bracis-28417 214 7 workshop workshop NOUN bracis-28417 214 8 on on ADP bracis-28417 214 9 open open ADJ bracis-28417 214 10 - - PUNCT bracis-28417 214 11 source source NOUN bracis-28417 214 12 arabic arabic ADJ bracis-28417 214 13 corpora corpora NOUN bracis-28417 214 14 and and CCONJ bracis-28417 214 15 processing processing NOUN bracis-28417 214 16 tools tool NOUN bracis-28417 214 17 , , PUNCT bracis-28417 214 18 with with ADP bracis-28417 214 19 a a DET bracis-28417 214 20 shared share VERB bracis-28417 214 21 task task NOUN bracis-28417 214 22 on on ADP bracis-28417 214 23 offensive offensive ADJ bracis-28417 214 24 language language NOUN bracis-28417 214 25 detection detection NOUN bracis-28417 214 26 . . PUNCT bracis-28417 215 1 pp pp ADJ bracis-28417 215 2 . . PUNCT bracis-28417 216 1 9–15 9–15 PROPN bracis-28417 216 2 . . PUNCT bracis-28417 217 1 european european ADJ bracis-28417 217 2 language language PROPN bracis-28417 217 3 resource resource NOUN bracis-28417 217 4 association association PROPN bracis-28417 217 5 , , PUNCT bracis-28417 217 6 marseille marseille PROPN bracis-28417 217 7 , , PUNCT bracis-28417 217 8 france france PROPN bracis-28417 217 9 ( ( PUNCT bracis-28417 217 10 2020 2020 NUM bracis-28417 217 11 ) ) PUNCT bracis-28417 217 12 google google PROPN bracis-28417 217 13 scholar scholar NOUN bracis-28417 217 14     SPACE bracis-28417 217 15 barros barros PROPN bracis-28417 217 16 , , PUNCT bracis-28417 217 17 t.m.d t.m.d NOUN bracis-28417 217 18 . . PROPN bracis-28417 217 19 , , PUNCT bracis-28417 217 20 et et PROPN bracis-28417 217 21 al al PROPN bracis-28417 217 22 . . PROPN bracis-28417 217 23 : : PUNCT bracis-28417 218 1 employing employ VERB bracis-28417 218 2 transformers transformer NOUN bracis-28417 218 3 and and CCONJ bracis-28417 218 4 emoji emoji NOUN bracis-28417 218 5 to to PART bracis-28417 218 6 perform perform VERB bracis-28417 218 7 sentiment sentiment NOUN bracis-28417 218 8 classification classification NOUN bracis-28417 218 9 of of ADP bracis-28417 218 10 social social ADJ bracis-28417 218 11 media medium NOUN bracis-28417 218 12 texts text NOUN bracis-28417 218 13 : : PUNCT bracis-28417 218 14 utilizando utilizando PROPN bracis-28417 218 15 transformers transformer NOUN bracis-28417 218 16 e e PROPN bracis-28417 218 17 emoji emoji PROPN bracis-28417 218 18 na na PROPN bracis-28417 218 19 classificação classificação PROPN bracis-28417 218 20 de de PROPN bracis-28417 218 21 sentimento sentimento PROPN bracis-28417 218 22 de de X bracis-28417 218 23 textos textos PROPN bracis-28417 218 24 oriundos oriundo NOUN bracis-28417 218 25 de de PROPN bracis-28417 218 26 redes redes PROPN bracis-28417 218 27 sociais sociais PROPN bracis-28417 218 28 ( ( PUNCT bracis-28417 218 29 2021 2021 NUM bracis-28417 218 30 ) ) PUNCT bracis-28417 218 31 google google PROPN bracis-28417 218 32 scholar scholar NOUN bracis-28417 218 33     SPACE bracis-28417 218 34 bhattacharjee bhattacharjee PROPN bracis-28417 218 35 , , PUNCT bracis-28417 218 36 a. a. PROPN bracis-28417 218 37 , , PUNCT bracis-28417 218 38 et et PROPN bracis-28417 218 39 al al PROPN bracis-28417 218 40 . . PROPN bracis-28417 218 41 : : PUNCT bracis-28417 219 1 banglabert banglabert NOUN bracis-28417 219 2 : : PUNCT bracis-28417 219 3 language language NOUN bracis-28417 219 4 model model NOUN bracis-28417 219 5 pretraining pretraine VERB bracis-28417 219 6 and and CCONJ bracis-28417 219 7 benchmarks benchmark NOUN bracis-28417 219 8 for for ADP bracis-28417 219 9 low low ADJ bracis-28417 219 10 - - PUNCT bracis-28417 219 11 resource resource NOUN bracis-28417 219 12 language language NOUN bracis-28417 219 13 understanding understanding NOUN bracis-28417 219 14 evaluation evaluation NOUN bracis-28417 219 15 in in ADP bracis-28417 219 16 bangla bangla PROPN bracis-28417 219 17 . . PUNCT bracis-28417 220 1 in in ADP bracis-28417 220 2 : : PUNCT bracis-28417 220 3 findings finding NOUN bracis-28417 220 4 of of ADP bracis-28417 220 5 the the DET bracis-28417 220 6 association association NOUN bracis-28417 220 7 for for ADP bracis-28417 220 8 computational computational ADJ bracis-28417 220 9 linguistics linguistic NOUN bracis-28417 220 10 : : PUNCT bracis-28417 220 11 naacl naacl PROPN bracis-28417 220 12 2022 2022 NUM bracis-28417 220 13 , , PUNCT bracis-28417 220 14 pp pp PROPN bracis-28417 220 15 . . PUNCT bracis-28417 220 16 1318–1327 1318–1327 NUM bracis-28417 220 17 . . PUNCT bracis-28417 221 1 association association PROPN bracis-28417 221 2 for for ADP bracis-28417 221 3 computational computational ADJ bracis-28417 221 4 linguistics linguistic NOUN bracis-28417 221 5 , , PUNCT bracis-28417 221 6 seattle seattle PROPN bracis-28417 221 7 , , PUNCT bracis-28417 221 8 united united PROPN bracis-28417 221 9 states states PROPN bracis-28417 221 10 ( ( PUNCT bracis-28417 221 11 2022 2022 NUM bracis-28417 221 12 ) ) PUNCT bracis-28417 221 13 . . PUNCT bracis-28417 222 1 https://doi.org/10.18653/v1/2022.findings-naacl.98 https://doi.org/10.18653/v1/2022.findings-naacl.98 PRON bracis-28417 222 2 brum brum NOUN bracis-28417 222 3 , , PUNCT bracis-28417 222 4 h. h. PROPN bracis-28417 222 5 , , PUNCT bracis-28417 222 6 volpe volpe PROPN bracis-28417 222 7 nunes nunes PROPN bracis-28417 222 8 , , PUNCT bracis-28417 222 9 m.d.g m.d.g NOUN bracis-28417 222 10 . . PUNCT bracis-28417 222 11 : : PUNCT bracis-28417 223 1 building build VERB bracis-28417 223 2 a a DET bracis-28417 223 3 sentiment sentiment NOUN bracis-28417 223 4 corpus corpus NOUN bracis-28417 223 5 of of ADP bracis-28417 223 6 tweets tweet NOUN bracis-28417 223 7 in in ADP bracis-28417 223 8 brazilian brazilian ADJ bracis-28417 223 9 portuguese portuguese PROPN bracis-28417 223 10 . . PUNCT bracis-28417 224 1 in in ADP bracis-28417 224 2 : : PUNCT bracis-28417 224 3 proceedings proceeding NOUN bracis-28417 224 4 of of ADP bracis-28417 224 5 the the DET bracis-28417 224 6 eleventh eleventh ADJ bracis-28417 224 7 international international ADJ bracis-28417 224 8 conference conference NOUN bracis-28417 224 9 on on ADP bracis-28417 224 10 language language NOUN bracis-28417 224 11 resources resource NOUN bracis-28417 224 12 and and CCONJ bracis-28417 224 13 evaluation evaluation NOUN bracis-28417 224 14 ( ( PUNCT bracis-28417 224 15 lrec lrec PROPN bracis-28417 224 16 2018 2018 NUM bracis-28417 224 17 ) ) PUNCT bracis-28417 224 18 . . PUNCT bracis-28417 225 1 european european ADJ bracis-28417 225 2 language language PROPN bracis-28417 225 3 resources resource NOUN bracis-28417 225 4 association association NOUN bracis-28417 225 5 ( ( PUNCT bracis-28417 225 6 elra elra ADV bracis-28417 225 7 ) ) PUNCT bracis-28417 225 8 , , PUNCT bracis-28417 225 9 miyazaki miyazaki PROPN bracis-28417 225 10 , , PUNCT bracis-28417 225 11 japan japan PROPN bracis-28417 225 12 ( ( PUNCT bracis-28417 225 13 may may PROPN bracis-28417 225 14 2018 2018 NUM bracis-28417 225 15 ) ) PUNCT bracis-28417 225 16 google google PROPN bracis-28417 225 17 scholar scholar NOUN bracis-28417 225 18     SPACE bracis-28417 225 19 cañete cañete NOUN bracis-28417 225 20 , , PUNCT bracis-28417 225 21 j. j. PROPN bracis-28417 225 22 , , PUNCT bracis-28417 225 23 chaperon chaperon PROPN bracis-28417 225 24 , , PUNCT bracis-28417 225 25 g. g. PROPN bracis-28417 225 26 , , PUNCT bracis-28417 225 27 fuentes fuentes PROPN bracis-28417 225 28 , , PUNCT bracis-28417 225 29 r. r. PROPN bracis-28417 225 30 , , PUNCT bracis-28417 225 31 ho ho PROPN bracis-28417 225 32 , , PUNCT bracis-28417 225 33 j.h j.h PROPN bracis-28417 225 34 . . PROPN bracis-28417 225 35 , , PUNCT bracis-28417 225 36 kang kang PROPN bracis-28417 225 37 , , PUNCT bracis-28417 225 38 h. h. PROPN bracis-28417 225 39 , , PUNCT bracis-28417 225 40 pérez pérez NOUN bracis-28417 225 41 , , PUNCT bracis-28417 225 42 j. j. PROPN bracis-28417 225 43 : : PUNCT bracis-28417 225 44 spanish spanish ADJ bracis-28417 225 45 pre pre ADJ bracis-28417 225 46 - - ADJ bracis-28417 225 47 trained train VERB bracis-28417 225 48 bert bert NOUN bracis-28417 225 49 model model NOUN bracis-28417 225 50 and and CCONJ bracis-28417 225 51 evaluation evaluation NOUN bracis-28417 225 52 data datum NOUN bracis-28417 225 53 . . PUNCT bracis-28417 226 1 in in ADP bracis-28417 226 2 : : PUNCT bracis-28417 226 3 pml4dc pml4dc VERB bracis-28417 226 4 at at ADP bracis-28417 226 5 iclr iclr NOUN bracis-28417 226 6 2020 2020 NUM bracis-28417 226 7 ( ( PUNCT bracis-28417 226 8 2020 2020 NUM bracis-28417 226 9 ) ) PUNCT bracis-28417 226 10 google google PROPN bracis-28417 226 11 scholar scholar NOUN bracis-28417 226 12     SPACE bracis-28417 226 13 carmo carmo PROPN bracis-28417 226 14 , , PUNCT bracis-28417 226 15 d. d. PROPN bracis-28417 226 16 , , PUNCT bracis-28417 226 17 piau piau NOUN bracis-28417 226 18 , , PUNCT bracis-28417 226 19 m. m. NOUN bracis-28417 226 20 , , PUNCT bracis-28417 226 21 campiotti campiotti PROPN bracis-28417 226 22 , , PUNCT bracis-28417 226 23 i. i. PROPN bracis-28417 226 24 , , PUNCT bracis-28417 226 25 nogueira nogueira PROPN bracis-28417 226 26 , , PUNCT bracis-28417 226 27 r. r. PROPN bracis-28417 226 28 , , PUNCT bracis-28417 226 29 lotufo lotufo PROPN bracis-28417 226 30 , , PUNCT bracis-28417 226 31 r. r. PROPN bracis-28417 226 32 : : PUNCT bracis-28417 226 33 ptt5 ptt5 PROPN bracis-28417 226 34 : : PUNCT bracis-28417 226 35 pretraining pretraine VERB bracis-28417 226 36 and and CCONJ bracis-28417 226 37 validating validate VERB bracis-28417 226 38 the the DET bracis-28417 226 39 t5 t5 PROPN bracis-28417 226 40 model model NOUN bracis-28417 226 41 on on ADP bracis-28417 226 42 brazilian brazilian ADJ bracis-28417 226 43 portuguese portuguese ADJ bracis-28417 226 44 data datum NOUN bracis-28417 226 45 . . PUNCT bracis-28417 227 1 arxiv arxiv PROPN bracis-28417 227 2 preprint preprint PROPN bracis-28417 227 3 arxiv:2008.09144 arxiv:2008.09144 ADP bracis-28417 227 4 ( ( PUNCT bracis-28417 227 5 2020 2020 NUM bracis-28417 227 6 ) ) PUNCT bracis-28417 227 7 chan chan PROPN bracis-28417 227 8 , , PUNCT bracis-28417 227 9 b. b. PROPN bracis-28417 227 10 , , PUNCT bracis-28417 227 11 schweter schweter ADV bracis-28417 227 12 , , PUNCT bracis-28417 227 13 s. s. PROPN bracis-28417 227 14 , , PUNCT bracis-28417 227 15 möller möller NOUN bracis-28417 227 16 , , PUNCT bracis-28417 227 17 t. t. PROPN bracis-28417 227 18 : : PUNCT bracis-28417 227 19 german german PROPN bracis-28417 227 20 ’s ’s PART bracis-28417 227 21 next next ADJ bracis-28417 227 22 language language NOUN bracis-28417 227 23 model model NOUN bracis-28417 227 24 . . PUNCT bracis-28417 228 1 in in ADP bracis-28417 228 2 : : PUNCT bracis-28417 228 3 proceedings proceeding NOUN bracis-28417 228 4 of of ADP bracis-28417 228 5 the the DET bracis-28417 228 6 28th 28th ADJ bracis-28417 228 7 international international ADJ bracis-28417 228 8 conference conference NOUN bracis-28417 228 9 on on ADP bracis-28417 228 10 computational computational ADJ bracis-28417 228 11 linguistics linguistic NOUN bracis-28417 228 12 , , PUNCT bracis-28417 228 13 pp pp ADV bracis-28417 228 14 . . PUNCT bracis-28417 229 1 6788–6796 6788–6796 NUM bracis-28417 229 2 . . PUNCT bracis-28417 230 1 international international ADJ bracis-28417 230 2 committee committee PROPN bracis-28417 230 3 on on ADP bracis-28417 230 4 computational computational ADJ bracis-28417 230 5 linguistics linguistic NOUN bracis-28417 230 6 , , PUNCT bracis-28417 230 7 barcelona barcelona PROPN bracis-28417 230 8 , , PUNCT bracis-28417 230 9 spain spain PROPN bracis-28417 230 10 ( ( PUNCT bracis-28417 230 11 online online PROPN bracis-28417 230 12 ) ) PUNCT bracis-28417 230 13 ( ( PUNCT bracis-28417 230 14 2020 2020 NUM bracis-28417 230 15 ) ) PUNCT bracis-28417 230 16 . . PUNCT bracis-28417 231 1 https://doi.org/10.18653/v1/2020.coling-main.598 https://doi.org/10.18653/v1/2020.coling-main.598 PROPN bracis-28417 231 2 chaves chaves PROPN bracis-28417 231 3 rodrigues rodrigues PROPN bracis-28417 231 4 , , PUNCT bracis-28417 231 5 r. r. PROPN bracis-28417 231 6 , , PUNCT bracis-28417 231 7 tanti tanti NOUN bracis-28417 231 8 , , PUNCT bracis-28417 231 9 m. m. NOUN bracis-28417 231 10 , , PUNCT bracis-28417 231 11 agerri agerri PROPN bracis-28417 231 12 , , PUNCT bracis-28417 231 13 r. r. PROPN bracis-28417 231 14 : : PUNCT bracis-28417 231 15 evaluation evaluation NOUN bracis-28417 231 16 of of ADP bracis-28417 231 17 portuguese portuguese ADJ bracis-28417 231 18 language language NOUN bracis-28417 231 19 models model NOUN bracis-28417 231 20 ( ( PUNCT bracis-28417 231 21 2023 2023 NUM bracis-28417 231 22 ) ) PUNCT bracis-28417 231 23 . . PUNCT bracis-28417 232 1 https://doi.org/10.5281/zenodo.7781848 https://doi.org/10.5281/zenodo.7781848 PROPN bracis-28417 232 2 , , PUNCT bracis-28417 232 3 https://github.com/ruanchaves/eplm https://github.com/ruanchaves/eplm NOUN bracis-28417 232 4 chowdhery chowdhery NOUN bracis-28417 232 5 , , PUNCT bracis-28417 232 6 a. a. NOUN bracis-28417 232 7 , , PUNCT bracis-28417 232 8 et et PROPN bracis-28417 232 9 al al PROPN bracis-28417 232 10 . . PUNCT bracis-28417 232 11 : : PUNCT bracis-28417 233 1 palm palm NOUN bracis-28417 233 2 : : PUNCT bracis-28417 233 3 scaling scale VERB bracis-28417 233 4 language language NOUN bracis-28417 233 5 modeling modeling NOUN bracis-28417 233 6 with with ADP bracis-28417 233 7 pathways pathway NOUN bracis-28417 233 8 . . PUNCT bracis-28417 234 1 arxiv arxiv PROPN bracis-28417 234 2 preprint preprint VERB bracis-28417 234 3 arxiv:2204.02311 arxiv:2204.02311 NOUN bracis-28417 234 4 ( ( PUNCT bracis-28417 234 5 2022 2022 NUM bracis-28417 234 6 ) ) PUNCT bracis-28417 234 7 clark clark PROPN bracis-28417 234 8 , , PUNCT bracis-28417 234 9 c. c. PROPN bracis-28417 234 10 , , PUNCT bracis-28417 234 11 lee lee PROPN bracis-28417 234 12 , , PUNCT bracis-28417 234 13 k. k. PROPN bracis-28417 234 14 , , PUNCT bracis-28417 234 15 chang chang PROPN bracis-28417 234 16 , , PUNCT bracis-28417 234 17 m.w m.w PROPN bracis-28417 234 18 . . PROPN bracis-28417 234 19 , , PUNCT bracis-28417 234 20 kwiatkowski kwiatkowski PROPN bracis-28417 234 21 , , PUNCT bracis-28417 234 22 t. t. PROPN bracis-28417 234 23 , , PUNCT bracis-28417 234 24 collins collins PROPN bracis-28417 234 25 , , PUNCT bracis-28417 234 26 m. m. NOUN bracis-28417 234 27 , , PUNCT bracis-28417 234 28 toutanova toutanova NOUN bracis-28417 234 29 , , PUNCT bracis-28417 234 30 k. k. PROPN bracis-28417 234 31 : : PUNCT bracis-28417 235 1 boolq boolq ADV bracis-28417 235 2 : : PUNCT bracis-28417 235 3 exploring explore VERB bracis-28417 235 4 the the DET bracis-28417 235 5 surprising surprising ADJ bracis-28417 235 6 difficulty difficulty NOUN bracis-28417 235 7 of of ADP bracis-28417 235 8 natural natural ADJ bracis-28417 235 9 yes yes NOUN bracis-28417 235 10 / / SYM bracis-28417 235 11 no no PRON bracis-28417 235 12 questions question NOUN bracis-28417 235 13 . . PUNCT bracis-28417 236 1 in in ADP bracis-28417 236 2 : : PUNCT bracis-28417 236 3 proceedings proceeding NOUN bracis-28417 236 4 of of ADP bracis-28417 236 5 the the DET bracis-28417 236 6 2019 2019 NUM bracis-28417 236 7 conference conference NOUN bracis-28417 236 8 of of ADP bracis-28417 236 9 the the DET bracis-28417 236 10 north north ADJ bracis-28417 236 11 american american ADJ bracis-28417 236 12 chapter chapter NOUN bracis-28417 236 13 of of ADP bracis-28417 236 14 the the DET bracis-28417 236 15 association association NOUN bracis-28417 236 16 for for ADP bracis-28417 236 17 computational computational ADJ bracis-28417 236 18 linguistics linguistic NOUN bracis-28417 236 19 : : PUNCT bracis-28417 236 20 human human ADJ bracis-28417 236 21 language language NOUN bracis-28417 236 22 technologies technology NOUN bracis-28417 236 23 , , PUNCT bracis-28417 236 24 volume volume NOUN bracis-28417 236 25 1 1 NUM bracis-28417 236 26 ( ( PUNCT bracis-28417 236 27 long long ADJ bracis-28417 236 28 and and CCONJ bracis-28417 236 29 short short ADJ bracis-28417 236 30 papers paper NOUN bracis-28417 236 31 ) ) PUNCT bracis-28417 236 32 , , PUNCT bracis-28417 236 33 pp pp PROPN bracis-28417 236 34 . . PUNCT bracis-28417 237 1 2924–2936 2924–2936 NUM bracis-28417 237 2 . . PUNCT bracis-28417 237 3 association association NOUN bracis-28417 237 4 for for ADP bracis-28417 237 5 computational computational ADJ bracis-28417 237 6 linguistics linguistic NOUN bracis-28417 237 7 , , PUNCT bracis-28417 237 8 minneapolis minneapolis PROPN bracis-28417 237 9 , , PUNCT bracis-28417 237 10 minnesota minnesota PROPN bracis-28417 237 11 ( ( PUNCT bracis-28417 237 12 2019 2019 NUM bracis-28417 237 13 ) ) PUNCT bracis-28417 237 14 . . PUNCT bracis-28417 238 1 https://doi.org/10.18653/v1/n19-1300 https://doi.org/10.18653/v1/n19-1300 NOUN bracis-28417 238 2 conneau conneau PROPN bracis-28417 238 3 , , PUNCT bracis-28417 238 4 a. a. PROPN bracis-28417 238 5 , , PUNCT bracis-28417 238 6 et et PROPN bracis-28417 238 7 al al PROPN bracis-28417 238 8 . . PROPN bracis-28417 238 9 : : PUNCT bracis-28417 239 1 unsupervised unsupervised ADJ bracis-28417 239 2 cross cross ADJ bracis-28417 239 3 - - ADJ bracis-28417 239 4 lingual lingual ADJ bracis-28417 239 5 representation representation NOUN bracis-28417 239 6 learning learn VERB bracis-28417 239 7 at at ADP bracis-28417 239 8 scale scale NOUN bracis-28417 239 9 . . PUNCT bracis-28417 240 1 in in ADP bracis-28417 240 2 : : PUNCT bracis-28417 240 3 proceedings proceeding NOUN bracis-28417 240 4 of of ADP bracis-28417 240 5 the the DET bracis-28417 240 6 58th 58th ADJ bracis-28417 240 7 annual annual ADJ bracis-28417 240 8 meeting meeting NOUN bracis-28417 240 9 of of ADP bracis-28417 240 10 the the DET bracis-28417 240 11 association association NOUN bracis-28417 240 12 for for ADP bracis-28417 240 13 computational computational ADJ bracis-28417 240 14 linguistics linguistic NOUN bracis-28417 240 15 , , PUNCT bracis-28417 240 16 pp pp X bracis-28417 240 17 . . PUNCT bracis-28417 240 18 8440–8451 8440–8451 NUM bracis-28417 240 19 ( ( PUNCT bracis-28417 240 20 2020 2020 NUM bracis-28417 240 21 ) ) PUNCT bracis-28417 240 22 google google PROPN bracis-28417 240 23 scholar scholar NOUN bracis-28417 240 24     SPACE bracis-28417 240 25 ebrahimi ebrahimi PROPN bracis-28417 240 26 , , PUNCT bracis-28417 240 27 a. a. PROPN bracis-28417 240 28 , , PUNCT bracis-28417 240 29 kann kann PROPN bracis-28417 240 30 , , PUNCT bracis-28417 240 31 k. k. PROPN bracis-28417 240 32 : : PUNCT bracis-28417 240 33 how how SCONJ bracis-28417 240 34 to to PART bracis-28417 240 35 adapt adapt VERB bracis-28417 240 36 your your PRON bracis-28417 240 37 pretrained pretraine VERB bracis-28417 240 38 multilingual multilingual ADJ bracis-28417 240 39 model model NOUN bracis-28417 240 40 to to ADP bracis-28417 240 41 1600 1600 NUM bracis-28417 240 42 languages language NOUN bracis-28417 240 43 . . PUNCT bracis-28417 241 1 in in ADP bracis-28417 241 2 : : PUNCT bracis-28417 241 3 proceedings proceeding NOUN bracis-28417 241 4 of of ADP bracis-28417 241 5 the the DET bracis-28417 241 6 59th 59th ADJ bracis-28417 241 7 annual annual ADJ bracis-28417 241 8 meeting meeting NOUN bracis-28417 241 9 of of ADP bracis-28417 241 10 the the DET bracis-28417 241 11 association association NOUN bracis-28417 241 12 for for ADP bracis-28417 241 13 computational computational ADJ bracis-28417 241 14 linguistics linguistic NOUN bracis-28417 241 15 and and CCONJ bracis-28417 241 16 the the DET bracis-28417 241 17 11th 11th ADJ bracis-28417 241 18 international international ADJ bracis-28417 241 19 joint joint ADJ bracis-28417 241 20 conference conference NOUN bracis-28417 241 21 on on ADP bracis-28417 241 22 natural natural ADJ bracis-28417 241 23 language language NOUN bracis-28417 241 24 processing processing NOUN bracis-28417 241 25 ( ( PUNCT bracis-28417 241 26 volume volume NOUN bracis-28417 241 27 1 1 NUM bracis-28417 241 28 : : PUNCT bracis-28417 241 29 long long ADJ bracis-28417 241 30 papers paper NOUN bracis-28417 241 31 ) ) PUNCT bracis-28417 241 32 , , PUNCT bracis-28417 241 33 pp pp ADP bracis-28417 241 34 . . PUNCT bracis-28417 242 1 4555–4567 4555–4567 NUM bracis-28417 242 2 . . PUNCT bracis-28417 243 1 association association NOUN bracis-28417 243 2 for for ADP bracis-28417 243 3 computational computational ADJ bracis-28417 243 4 linguistics linguistic NOUN bracis-28417 243 5 , , PUNCT bracis-28417 243 6 online online ADJ bracis-28417 243 7 ( ( PUNCT bracis-28417 243 8 aug aug PROPN bracis-28417 243 9 2021 2021 NUM bracis-28417 243 10 ) ) PUNCT bracis-28417 243 11 . . PUNCT bracis-28417 244 1 10.18653 10.18653 NUM bracis-28417 244 2 / / SYM bracis-28417 244 3 v1/2021.acl v1/2021.acl VERB bracis-28417 244 4 - - PUNCT bracis-28417 244 5 long.351 long.351 NOUN bracis-28417 244 6 google google PROPN bracis-28417 244 7 scholar scholar NOUN bracis-28417 244 8     SPACE bracis-28417 244 9 fitzgerald fitzgerald NOUN bracis-28417 244 10 , , PUNCT bracis-28417 244 11 j. j. PROPN bracis-28417 244 12 , , PUNCT bracis-28417 244 13 et et PROPN bracis-28417 244 14 al al PROPN bracis-28417 244 15 . . PUNCT bracis-28417 244 16 : : PUNCT bracis-28417 245 1 massive massive ADJ bracis-28417 245 2 : : PUNCT bracis-28417 245 3 a a DET bracis-28417 245 4 1m 1m NUM bracis-28417 245 5 - - PUNCT bracis-28417 245 6 example example NOUN bracis-28417 245 7 multilingual multilingual ADJ bracis-28417 245 8 natural natural ADJ bracis-28417 245 9 language language NOUN bracis-28417 245 10 understanding understanding NOUN bracis-28417 245 11 dataset dataset VERB bracis-28417 245 12 with with ADP bracis-28417 245 13 51 51 NUM bracis-28417 245 14 typologically typologically ADV bracis-28417 245 15 - - PUNCT bracis-28417 245 16 diverse diverse ADJ bracis-28417 245 17 languages language NOUN bracis-28417 245 18 ( ( PUNCT bracis-28417 245 19 2022 2022 NUM bracis-28417 245 20 ) ) PUNCT bracis-28417 245 21 google google PROPN bracis-28417 245 22 scholar scholar NOUN bracis-28417 245 23     SPACE bracis-28417 245 24 fu fu NOUN bracis-28417 245 25 , , PUNCT bracis-28417 245 26 y. y. PROPN bracis-28417 245 27 , , PUNCT bracis-28417 245 28 peng peng PROPN bracis-28417 245 29 , , PUNCT bracis-28417 245 30 h. h. PROPN bracis-28417 245 31 , , PUNCT bracis-28417 245 32 ou ou PROPN bracis-28417 245 33 , , PUNCT bracis-28417 245 34 l. l. PROPN bracis-28417 245 35 , , PUNCT bracis-28417 245 36 sabharwal sabharwal PROPN bracis-28417 245 37 , , PUNCT bracis-28417 245 38 a. a. NOUN bracis-28417 245 39 , , PUNCT bracis-28417 245 40 khot khot ADJ bracis-28417 245 41 , , PUNCT bracis-28417 245 42 t. t. PROPN bracis-28417 245 43 : : PUNCT bracis-28417 245 44 specializing specialize VERB bracis-28417 245 45 smaller small ADJ bracis-28417 245 46 language language NOUN bracis-28417 245 47 models model NOUN bracis-28417 245 48 towards towards ADP bracis-28417 245 49 multi multi ADJ bracis-28417 245 50 - - ADJ bracis-28417 245 51 step step ADJ bracis-28417 245 52 reasoning reasoning NOUN bracis-28417 245 53 . . PUNCT bracis-28417 246 1 arxiv arxiv PROPN bracis-28417 246 2 preprint preprint NOUN bracis-28417 246 3 arxiv:2301.12726 arxiv:2301.12726 NOUN bracis-28417 246 4 ( ( PUNCT bracis-28417 246 5 2023 2023 NUM bracis-28417 246 6 ) ) PUNCT bracis-28417 246 7 gao gao PROPN bracis-28417 246 8 , , PUNCT bracis-28417 246 9 l. l. PROPN bracis-28417 246 10 , , PUNCT bracis-28417 246 11 et et PROPN bracis-28417 246 12 al al PROPN bracis-28417 246 13 . . PUNCT bracis-28417 246 14 : : PUNCT bracis-28417 247 1 the the DET bracis-28417 247 2 pile pile NOUN bracis-28417 247 3 : : PUNCT bracis-28417 247 4 an an DET bracis-28417 247 5 800 800 NUM bracis-28417 247 6 gb gb NOUN bracis-28417 247 7 dataset dataset NOUN bracis-28417 247 8 of of ADP bracis-28417 247 9 diverse diverse ADJ bracis-28417 247 10 text text NOUN bracis-28417 247 11 for for ADP bracis-28417 247 12 language language NOUN bracis-28417 247 13 modeling modeling NOUN bracis-28417 247 14 . . PUNCT bracis-28417 248 1 arxiv arxiv PROPN bracis-28417 248 2 preprint preprint VERB bracis-28417 248 3 arxiv:2101.00027 arxiv:2101.00027 PROPN bracis-28417 248 4 ( ( PUNCT bracis-28417 248 5 2020 2020 NUM bracis-28417 248 6 ) ) PUNCT bracis-28417 248 7 gururangan gururangan NOUN bracis-28417 248 8 , , PUNCT bracis-28417 248 9 s. s. PROPN bracis-28417 248 10 , , PUNCT bracis-28417 248 11 et et PROPN bracis-28417 248 12 al al PROPN bracis-28417 248 13 . . PROPN bracis-28417 248 14 : : PUNCT bracis-28417 249 1 scaling scale VERB bracis-28417 249 2 expert expert ADJ bracis-28417 249 3 language language NOUN bracis-28417 249 4 models model NOUN bracis-28417 249 5 with with ADP bracis-28417 249 6 unsupervised unsupervised ADJ bracis-28417 249 7 domain domain NOUN bracis-28417 249 8 discovery discovery NOUN bracis-28417 249 9 . . PUNCT bracis-28417 250 1 arxiv arxiv PROPN bracis-28417 250 2 preprint preprint VERB bracis-28417 250 3 arxiv:2303.14177 arxiv:2303.14177 PRON bracis-28417 250 4 ( ( PUNCT bracis-28417 250 5 2023 2023 NUM bracis-28417 250 6 ) ) PUNCT bracis-28417 250 7 hoffmann hoffmann PROPN bracis-28417 250 8 , , PUNCT bracis-28417 250 9 j. j. PROPN bracis-28417 250 10 , , PUNCT bracis-28417 250 11 et et PROPN bracis-28417 250 12 al al PROPN bracis-28417 250 13 . . PROPN bracis-28417 250 14 : : PUNCT bracis-28417 251 1 training train VERB bracis-28417 251 2 compute compute NOUN bracis-28417 251 3 - - PUNCT bracis-28417 251 4 optimal optimal ADJ bracis-28417 251 5 large large ADJ bracis-28417 251 6 language language NOUN bracis-28417 251 7 models model NOUN bracis-28417 251 8 . . PUNCT bracis-28417 252 1 arxiv arxiv PROPN bracis-28417 252 2 preprint preprint NOUN bracis-28417 252 3 arxiv:2203.15556 arxiv:2203.15556 NOUN bracis-28417 252 4 ( ( PUNCT bracis-28417 252 5 2022 2022 NUM bracis-28417 252 6 ) ) PUNCT bracis-28417 252 7 houlsby houlsby ADJ bracis-28417 252 8 , , PUNCT bracis-28417 252 9 n. n. NOUN bracis-28417 252 10 , , PUNCT bracis-28417 252 11 et et PROPN bracis-28417 252 12 al al PROPN bracis-28417 252 13 . . PUNCT bracis-28417 252 14 : : PUNCT bracis-28417 253 1 parameter parameter NOUN bracis-28417 253 2 - - PUNCT bracis-28417 253 3 efficient efficient ADJ bracis-28417 253 4 transfer transfer NOUN bracis-28417 253 5 learning learning NOUN bracis-28417 253 6 for for ADP bracis-28417 253 7 nlp nlp NOUN bracis-28417 253 8 . . PUNCT bracis-28417 254 1 in in ADP bracis-28417 254 2 : : PUNCT bracis-28417 254 3 international international ADJ bracis-28417 254 4 conference conference NOUN bracis-28417 254 5 on on ADP bracis-28417 254 6 machine machine NOUN bracis-28417 254 7 learning learning NOUN bracis-28417 254 8 , , PUNCT bracis-28417 254 9 pp pp ADJ bracis-28417 254 10 . . PUNCT bracis-28417 255 1 2790–2799 2790–2799 NUM bracis-28417 255 2 . . PUNCT bracis-28417 256 1 pmlr pmlr NOUN bracis-28417 256 2 ( ( PUNCT bracis-28417 256 3 2019 2019 NUM bracis-28417 256 4 ) ) PUNCT bracis-28417 256 5 google google PROPN bracis-28417 256 6 scholar scholar NOUN bracis-28417 256 7     SPACE bracis-28417 256 8 hu hu PROPN bracis-28417 256 9 , , PUNCT bracis-28417 256 10 e.j e.j PROPN bracis-28417 256 11 . . PROPN bracis-28417 256 12 , , PUNCT bracis-28417 256 13 et et PROPN bracis-28417 256 14 al al PROPN bracis-28417 256 15 . . PUNCT bracis-28417 256 16 : : PUNCT bracis-28417 257 1 lora lora NOUN bracis-28417 257 2 : : PUNCT bracis-28417 257 3 low low ADJ bracis-28417 257 4 - - PUNCT bracis-28417 257 5 rank rank NOUN bracis-28417 257 6 adaptation adaptation NOUN bracis-28417 257 7 of of ADP bracis-28417 257 8 large large ADJ bracis-28417 257 9 language language NOUN bracis-28417 257 10 models model NOUN bracis-28417 257 11 . . PUNCT bracis-28417 258 1 in in ADP bracis-28417 258 2 : : PUNCT bracis-28417 258 3 international international ADJ bracis-28417 258 4 conference conference NOUN bracis-28417 258 5 on on ADP bracis-28417 258 6 learning learn VERB bracis-28417 258 7 representations representation NOUN bracis-28417 258 8 ( ( PUNCT bracis-28417 258 9 2022 2022 NUM bracis-28417 258 10 ) ) PUNCT bracis-28417 258 11 . . PUNCT bracis-28417 259 1 https://openreview.net/forum?id=nzevkeefyf9 https://openreview.net/forum?id=nzevkeefyf9 PROPN bracis-28417 259 2 kalyan kalyan PROPN bracis-28417 259 3 , , PUNCT bracis-28417 259 4 k.s k.s PROPN bracis-28417 259 5 . . PROPN bracis-28417 259 6 , , PUNCT bracis-28417 259 7 rajasekharan rajasekharan NOUN bracis-28417 259 8 , , PUNCT bracis-28417 259 9 a. a. NOUN bracis-28417 259 10 , , PUNCT bracis-28417 259 11 sangeetha sangeetha PROPN bracis-28417 259 12 , , PUNCT bracis-28417 259 13 s. s. PROPN bracis-28417 259 14 : : PUNCT bracis-28417 259 15 ammus ammus VERB bracis-28417 259 16 : : PUNCT bracis-28417 259 17 a a DET bracis-28417 259 18 survey survey NOUN bracis-28417 259 19 of of ADP bracis-28417 259 20 transformer transformer NOUN bracis-28417 259 21 - - PUNCT bracis-28417 259 22 based base VERB bracis-28417 259 23 pretrained pretraine VERB bracis-28417 259 24 models model NOUN bracis-28417 259 25 in in ADP bracis-28417 259 26 natural natural ADJ bracis-28417 259 27 language language NOUN bracis-28417 259 28 processing processing NOUN bracis-28417 259 29 . . PUNCT bracis-28417 260 1 arxiv arxiv PROPN bracis-28417 260 2 preprint preprint VERB bracis-28417 260 3 arxiv:2108.05542 arxiv:2108.05542 PROPN bracis-28417 260 4 ( ( PUNCT bracis-28417 260 5 2021 2021 NUM bracis-28417 260 6 ) ) PUNCT bracis-28417 260 7 kim kim PROPN bracis-28417 260 8 , , PUNCT bracis-28417 260 9 b. b. PROPN bracis-28417 260 10 , , PUNCT bracis-28417 260 11 et et PROPN bracis-28417 260 12 al al PROPN bracis-28417 260 13 . . PROPN bracis-28417 260 14 : : PUNCT bracis-28417 261 1 what what DET bracis-28417 261 2 changes change NOUN bracis-28417 261 3 can can AUX bracis-28417 261 4 large large ADJ bracis-28417 261 5 - - PUNCT bracis-28417 261 6 scale scale NOUN bracis-28417 261 7 language language NOUN bracis-28417 261 8 models model NOUN bracis-28417 261 9 bring bring VERB bracis-28417 261 10 ? ? PUNCT bracis-28417 262 1 intensive intensive ADJ bracis-28417 262 2 study study NOUN bracis-28417 262 3 on on ADP bracis-28417 262 4 hyperclova hyperclova NOUN bracis-28417 262 5 : : PUNCT bracis-28417 262 6 billions billion NOUN bracis-28417 262 7 - - PUNCT bracis-28417 262 8 scale scale NOUN bracis-28417 262 9 korean korean ADJ bracis-28417 262 10 generative generative NOUN bracis-28417 262 11 pretrained pretraine VERB bracis-28417 262 12 transformers transformer NOUN bracis-28417 262 13 . . PUNCT bracis-28417 263 1 in in ADP bracis-28417 263 2 : : PUNCT bracis-28417 263 3 proceedings proceeding NOUN bracis-28417 263 4 of of ADP bracis-28417 263 5 the the DET bracis-28417 263 6 2021 2021 NUM bracis-28417 263 7 conference conference NOUN bracis-28417 263 8 on on ADP bracis-28417 263 9 empirical empirical ADJ bracis-28417 263 10 methods method NOUN bracis-28417 263 11 in in ADP bracis-28417 263 12 natural natural ADJ bracis-28417 263 13 language language NOUN bracis-28417 263 14 processing processing NOUN bracis-28417 263 15 , , PUNCT bracis-28417 263 16 pp pp ADJ bracis-28417 263 17 . . PUNCT bracis-28417 264 1 3405–3424 3405–3424 NUM bracis-28417 264 2 ( ( PUNCT bracis-28417 264 3 2021k 2021k NUM bracis-28417 264 4 google google PROPN bracis-28417 264 5 scholar scholar NOUN bracis-28417 264 6     SPACE bracis-28417 264 7 kudo kudo PROPN bracis-28417 264 8 , , PUNCT bracis-28417 264 9 t. t. PROPN bracis-28417 264 10 , , PUNCT bracis-28417 264 11 richardson richardson PROPN bracis-28417 264 12 , , PUNCT bracis-28417 264 13 j. j. PROPN bracis-28417 264 14 : : PUNCT bracis-28417 264 15 sentencepiece sentencepiece PROPN bracis-28417 264 16 : : PUNCT bracis-28417 264 17 a a DET bracis-28417 264 18 simple simple ADJ bracis-28417 264 19 and and CCONJ bracis-28417 264 20 language language NOUN bracis-28417 264 21 independent independent PROPN bracis-28417 264 22 subword subword PROPN bracis-28417 264 23 tokenizer tokenizer NOUN bracis-28417 264 24 and and CCONJ bracis-28417 264 25 detokenizer detokenizer VERB bracis-28417 264 26 for for ADP bracis-28417 264 27 neural neural ADJ bracis-28417 264 28 text text NOUN bracis-28417 264 29 processing processing NOUN bracis-28417 264 30 . . PUNCT bracis-28417 265 1 in in ADP bracis-28417 265 2 : : PUNCT bracis-28417 265 3 proceedings proceeding NOUN bracis-28417 265 4 of of ADP bracis-28417 265 5 the the DET bracis-28417 265 6 2018 2018 NUM bracis-28417 265 7 conference conference NOUN bracis-28417 265 8 on on ADP bracis-28417 265 9 empirical empirical ADJ bracis-28417 265 10 methods method NOUN bracis-28417 265 11 in in ADP bracis-28417 265 12 natural natural ADJ bracis-28417 265 13 language language NOUN bracis-28417 265 14 processing processing NOUN bracis-28417 265 15 : : PUNCT bracis-28417 265 16 system system NOUN bracis-28417 265 17 demonstrations demonstration NOUN bracis-28417 265 18 , , PUNCT bracis-28417 265 19 pp pp ADV bracis-28417 265 20 . . PUNCT bracis-28417 265 21 66–71 66–71 NOUN bracis-28417 265 22 . . PUNCT bracis-28417 266 1 association association NOUN bracis-28417 266 2 for for ADP bracis-28417 266 3 computational computational ADJ bracis-28417 266 4 linguistics linguistic NOUN bracis-28417 266 5 , , PUNCT bracis-28417 266 6 brussels brussels PROPN bracis-28417 266 7 , , PUNCT bracis-28417 266 8 belgium belgium NOUN bracis-28417 266 9 ( ( PUNCT bracis-28417 266 10 2018 2018 NUM bracis-28417 266 11 ) ) PUNCT bracis-28417 266 12 . . PUNCT bracis-28417 267 1 https://doi.org/10.18653/v1/d18-2012 https://doi.org/10.18653/v1/d18-2012 PROPN bracis-28417 267 2 le le PROPN bracis-28417 267 3 , , PUNCT bracis-28417 267 4 h. h. PROPN bracis-28417 267 5 , , PUNCT bracis-28417 267 6 et et PROPN bracis-28417 267 7 al al PROPN bracis-28417 267 8 . . PROPN bracis-28417 267 9 : : PUNCT bracis-28417 268 1 flaubert flaubert PROPN bracis-28417 268 2 : : PUNCT bracis-28417 268 3 unsupervised unsupervised ADJ bracis-28417 268 4 language language NOUN bracis-28417 268 5 model model NOUN bracis-28417 268 6 pre pre NOUN bracis-28417 268 7 - - NOUN bracis-28417 268 8 training training NOUN bracis-28417 268 9 for for ADP bracis-28417 268 10 french french NOUN bracis-28417 268 11 . . PUNCT bracis-28417 269 1 in in ADP bracis-28417 269 2 : : PUNCT bracis-28417 269 3 proceedings proceeding NOUN bracis-28417 269 4 of of ADP bracis-28417 269 5 the the DET bracis-28417 269 6 twelfth twelfth ADJ bracis-28417 269 7 language language NOUN bracis-28417 269 8 resources resource NOUN bracis-28417 269 9 and and CCONJ bracis-28417 269 10 evaluation evaluation NOUN bracis-28417 269 11 conference conference NOUN bracis-28417 269 12 , , PUNCT bracis-28417 269 13 pp pp X bracis-28417 269 14 . . PUNCT bracis-28417 270 1 2479–2490 2479–2490 NUM bracis-28417 270 2 . . PUNCT bracis-28417 271 1 european european ADJ bracis-28417 271 2 language language PROPN bracis-28417 271 3 resources resource NOUN bracis-28417 271 4 association association PROPN bracis-28417 271 5 , , PUNCT bracis-28417 271 6 marseille marseille PROPN bracis-28417 271 7 , , PUNCT bracis-28417 271 8 france france PROPN bracis-28417 271 9 ( ( PUNCT bracis-28417 271 10 2020 2020 NUM bracis-28417 271 11 ) ) PUNCT bracis-28417 271 12 google google PROPN bracis-28417 271 13 scholar scholar NOUN bracis-28417 271 14     SPACE bracis-28417 271 15 lee lee PROPN bracis-28417 271 16 , , PUNCT bracis-28417 271 17 h. h. PROPN bracis-28417 271 18 , , PUNCT bracis-28417 271 19 yoon yoon PROPN bracis-28417 271 20 , , PUNCT bracis-28417 271 21 j. j. PROPN bracis-28417 271 22 , , PUNCT bracis-28417 271 23 hwang hwang PROPN bracis-28417 271 24 , , PUNCT bracis-28417 271 25 b. b. PROPN bracis-28417 271 26 , , PUNCT bracis-28417 271 27 joe joe PROPN bracis-28417 271 28 , , PUNCT bracis-28417 271 29 s. s. PROPN bracis-28417 271 30 , , PUNCT bracis-28417 271 31 min min PROPN bracis-28417 271 32 , , PUNCT bracis-28417 271 33 s. s. PROPN bracis-28417 271 34 , , PUNCT bracis-28417 271 35 gwon gwon VERB bracis-28417 271 36 , , PUNCT bracis-28417 271 37 y. y. NOUN bracis-28417 271 38 : : PUNCT bracis-28417 271 39 korealbert korealbert ADV bracis-28417 271 40 : : PUNCT bracis-28417 271 41 pretraining pretraine VERB bracis-28417 271 42 a a DET bracis-28417 271 43 lite lite ADJ bracis-28417 271 44 bert bert PROPN bracis-28417 271 45 model model NOUN bracis-28417 271 46 for for ADP bracis-28417 271 47 korean korean ADJ bracis-28417 271 48 language language NOUN bracis-28417 271 49 understanding understanding NOUN bracis-28417 271 50 . . PUNCT bracis-28417 272 1 in in ADP bracis-28417 272 2 : : PUNCT bracis-28417 272 3 2020 2020 NUM bracis-28417 272 4 25th 25th NOUN bracis-28417 272 5 international international ADJ bracis-28417 272 6 conference conference NOUN bracis-28417 272 7 on on ADP bracis-28417 272 8 pattern pattern NOUN bracis-28417 272 9 recognition recognition NOUN bracis-28417 272 10 ( ( PUNCT bracis-28417 272 11 icpr icpr PROPN bracis-28417 272 12 ) ) PUNCT bracis-28417 272 13 , , PUNCT bracis-28417 272 14 pp pp PROPN bracis-28417 272 15 . . PUNCT bracis-28417 272 16 5551–5557 5551–5557 X bracis-28417 272 17 . . PUNCT bracis-28417 273 1 ieee ieee NOUN bracis-28417 273 2 ( ( PUNCT bracis-28417 273 3 2021 2021 NUM bracis-28417 273 4 ) ) PUNCT bracis-28417 273 5 google google NOUN bracis-28417 273 6 scholar scholar NOUN bracis-28417 273 7     SPACE bracis-28417 273 8 lewkowycz lewkowycz NOUN bracis-28417 273 9 , , PUNCT bracis-28417 273 10 a. a. NOUN bracis-28417 273 11 , , PUNCT bracis-28417 273 12 et et PROPN bracis-28417 273 13 al al PROPN bracis-28417 273 14 . . PROPN bracis-28417 273 15 : : PUNCT bracis-28417 273 16 solving solve VERB bracis-28417 273 17 quantitative quantitative ADJ bracis-28417 273 18 reasoning reasoning NOUN bracis-28417 273 19 problems problem NOUN bracis-28417 273 20 with with ADP bracis-28417 273 21 language language NOUN bracis-28417 273 22 models model NOUN bracis-28417 273 23 . . PUNCT bracis-28417 274 1 arxiv arxiv PROPN bracis-28417 274 2 preprint preprint NOUN bracis-28417 274 3 arxiv:2206.14858 arxiv:2206.14858 NOUN bracis-28417 274 4 ( ( PUNCT bracis-28417 274 5 2022 2022 NUM bracis-28417 274 6 ) ) PUNCT bracis-28417 274 7 lin lin PROPN bracis-28417 274 8 , , PUNCT bracis-28417 274 9 x.v x.v PROPN bracis-28417 274 10 . . PROPN bracis-28417 274 11 , , PUNCT bracis-28417 274 12 et et PROPN bracis-28417 274 13 al al PROPN bracis-28417 274 14 . . PUNCT bracis-28417 274 15 : : PUNCT bracis-28417 275 1 few few ADJ bracis-28417 275 2 - - PUNCT bracis-28417 275 3 shot shot NOUN bracis-28417 275 4 learning learning NOUN bracis-28417 275 5 with with ADP bracis-28417 275 6 multilingual multilingual ADJ bracis-28417 275 7 generative generative ADJ bracis-28417 275 8 language language NOUN bracis-28417 275 9 models model NOUN bracis-28417 275 10 . . PUNCT bracis-28417 276 1 in in ADP bracis-28417 276 2 : : PUNCT bracis-28417 276 3 proceedings proceeding NOUN bracis-28417 276 4 of of ADP bracis-28417 276 5 the the DET bracis-28417 276 6 2022 2022 NUM bracis-28417 276 7 conference conference NOUN bracis-28417 276 8 on on ADP bracis-28417 276 9 empirical empirical ADJ bracis-28417 276 10 methods method NOUN bracis-28417 276 11 in in ADP bracis-28417 276 12 natural natural ADJ bracis-28417 276 13 language language NOUN bracis-28417 276 14 processing processing NOUN bracis-28417 276 15 , , PUNCT bracis-28417 276 16 pp pp ADP bracis-28417 276 17 . . PUNCT bracis-28417 277 1 9019–9052 9019–9052 NUM bracis-28417 277 2 ( ( PUNCT bracis-28417 277 3 2022 2022 NUM bracis-28417 277 4 ) ) PUNCT bracis-28417 277 5 google google PROPN bracis-28417 277 6 scholar scholar NOUN bracis-28417 277 7     SPACE bracis-28417 277 8 liu liu PROPN bracis-28417 277 9 , , PUNCT bracis-28417 277 10 y. y. PROPN bracis-28417 277 11 , , PUNCT bracis-28417 277 12 et et PROPN bracis-28417 277 13 al al PROPN bracis-28417 277 14 . . PUNCT bracis-28417 277 15 : : PUNCT bracis-28417 278 1 multilingual multilingual ADJ bracis-28417 278 2 denoising denoise VERB bracis-28417 278 3 pre pre NOUN bracis-28417 278 4 - - NOUN bracis-28417 278 5 training training NOUN bracis-28417 278 6 for for ADP bracis-28417 278 7 neural neural ADJ bracis-28417 278 8 machine machine NOUN bracis-28417 278 9 translation translation NOUN bracis-28417 278 10 . . PUNCT bracis-28417 279 1 trans trans PROPN bracis-28417 279 2 . . PUNCT bracis-28417 280 1 assoc assoc PROPN bracis-28417 280 2 . . PUNCT bracis-28417 281 1 comput comput PROPN bracis-28417 281 2 . . PUNCT bracis-28417 282 1 linguist linguist NOUN bracis-28417 282 2 . . PUNCT bracis-28417 283 1 8 8 NUM bracis-28417 283 2 , , PUNCT bracis-28417 283 3 726–742 726–742 NUM bracis-28417 283 4 ( ( PUNCT bracis-28417 283 5 2020 2020 NUM bracis-28417 283 6 ) ) PUNCT bracis-28417 283 7 article article NOUN bracis-28417 283 8     SPACE bracis-28417 283 9 google google PROPN bracis-28417 283 10 scholar scholar NOUN bracis-28417 283 11     SPACE bracis-28417 283 12 longpre longpre PROPN bracis-28417 283 13 , , PUNCT bracis-28417 283 14 s. s. PROPN bracis-28417 283 15 , , PUNCT bracis-28417 283 16 lu lu PROPN bracis-28417 283 17 , , PUNCT bracis-28417 283 18 y. y. PROPN bracis-28417 283 19 , , PUNCT bracis-28417 283 20 daiber daiber PROPN bracis-28417 283 21 , , PUNCT bracis-28417 283 22 j. j. PROPN bracis-28417 283 23 : : PUNCT bracis-28417 283 24 mkqa mkqa PROPN bracis-28417 283 25 : : PUNCT bracis-28417 283 26 a a DET bracis-28417 283 27 linguistically linguistically ADV bracis-28417 283 28 diverse diverse ADJ bracis-28417 283 29 benchmark benchmark NOUN bracis-28417 283 30 for for ADP bracis-28417 283 31 multilingual multilingual ADJ bracis-28417 283 32 open open ADJ bracis-28417 283 33 domain domain NOUN bracis-28417 283 34 question question NOUN bracis-28417 283 35 answering answering NOUN bracis-28417 283 36 . . PUNCT bracis-28417 284 1 trans trans PROPN bracis-28417 284 2 . . PUNCT bracis-28417 285 1 assoc assoc PROPN bracis-28417 285 2 . . PUNCT bracis-28417 286 1 comput comput PROPN bracis-28417 286 2 . . PUNCT bracis-28417 287 1 linguist linguist NOUN bracis-28417 287 2 . . PUNCT bracis-28417 288 1 9 9 NUM bracis-28417 288 2 , , PUNCT bracis-28417 288 3 1389–1406 1389–1406 NUM bracis-28417 288 4 ( ( PUNCT bracis-28417 288 5 2021 2021 NUM bracis-28417 288 6 ) ) PUNCT bracis-28417 288 7 article article NOUN bracis-28417 288 8     SPACE bracis-28417 288 9 google google PROPN bracis-28417 288 10 scholar scholar NOUN bracis-28417 288 11     SPACE bracis-28417 288 12 loshchilov loshchilov PROPN bracis-28417 288 13 , , PUNCT bracis-28417 288 14 i. i. NOUN bracis-28417 288 15 , , PUNCT bracis-28417 288 16 hutter hutter NOUN bracis-28417 288 17 , , PUNCT bracis-28417 288 18 f. f. PROPN bracis-28417 288 19 : : PUNCT bracis-28417 288 20 decoupled decouple VERB bracis-28417 288 21 weight weight NOUN bracis-28417 288 22 decay decay NOUN bracis-28417 288 23 regularization regularization NOUN bracis-28417 288 24 . . PUNCT bracis-28417 289 1 in in ADP bracis-28417 289 2 : : PUNCT bracis-28417 289 3 international international ADJ bracis-28417 289 4 conference conference NOUN bracis-28417 289 5 on on ADP bracis-28417 289 6 learning learn VERB bracis-28417 289 7 representations representation NOUN bracis-28417 289 8 ( ( PUNCT bracis-28417 289 9 2019 2019 NUM bracis-28417 289 10 ) ) PUNCT bracis-28417 289 11 google google PROPN bracis-28417 289 12 scholar scholar NOUN bracis-28417 289 13     SPACE bracis-28417 289 14 maas maas PROPN bracis-28417 289 15 , , PUNCT bracis-28417 289 16 a.l a.l PROPN bracis-28417 289 17 . . PROPN bracis-28417 289 18 , , PUNCT bracis-28417 289 19 daly daly PROPN bracis-28417 289 20 , , PUNCT bracis-28417 289 21 r.e r.e PROPN bracis-28417 289 22 . . PROPN bracis-28417 289 23 , , PUNCT bracis-28417 289 24 pham pham PROPN bracis-28417 289 25 , , PUNCT bracis-28417 289 26 p.t p.t PROPN bracis-28417 289 27 . . PROPN bracis-28417 289 28 , , PUNCT bracis-28417 289 29 huang huang PROPN bracis-28417 289 30 , , PUNCT bracis-28417 289 31 d. d. PROPN bracis-28417 289 32 , , PUNCT bracis-28417 289 33 ng ng PROPN bracis-28417 289 34 , , PUNCT bracis-28417 289 35 a.y a.y PROPN bracis-28417 289 36 . . PROPN bracis-28417 289 37 , , PUNCT bracis-28417 289 38 potts potts PROPN bracis-28417 289 39 , , PUNCT bracis-28417 289 40 c. c. PROPN bracis-28417 289 41 : : PUNCT bracis-28417 289 42 learning learn VERB bracis-28417 289 43 word word NOUN bracis-28417 289 44 vectors vector NOUN bracis-28417 289 45 for for ADP bracis-28417 289 46 sentiment sentiment NOUN bracis-28417 289 47 analysis analysis NOUN bracis-28417 289 48 . . PUNCT bracis-28417 290 1 in in ADP bracis-28417 290 2 : : PUNCT bracis-28417 290 3 proceedings proceeding NOUN bracis-28417 290 4 of of ADP bracis-28417 290 5 the the DET bracis-28417 290 6 49th 49th ADJ bracis-28417 290 7 annual annual ADJ bracis-28417 290 8 meeting meeting NOUN bracis-28417 290 9 of of ADP bracis-28417 290 10 the the DET bracis-28417 290 11 association association NOUN bracis-28417 290 12 for for ADP bracis-28417 290 13 computational computational ADJ bracis-28417 290 14 linguistics linguistic NOUN bracis-28417 290 15 : : PUNCT bracis-28417 290 16 human human ADJ bracis-28417 290 17 language language NOUN bracis-28417 290 18 technologies technology NOUN bracis-28417 290 19 , , PUNCT bracis-28417 290 20 pp pp ADP bracis-28417 290 21 . . PUNCT bracis-28417 291 1 142–150 142–150 NUM bracis-28417 291 2 . . PUNCT bracis-28417 292 1 association association NOUN bracis-28417 292 2 for for ADP bracis-28417 292 3 computational computational ADJ bracis-28417 292 4 linguistics linguistic NOUN bracis-28417 292 5 , , PUNCT bracis-28417 292 6 portland portland PROPN bracis-28417 292 7 , , PUNCT bracis-28417 292 8 oregon oregon PROPN bracis-28417 292 9 , , PUNCT bracis-28417 292 10 usa usa PROPN bracis-28417 292 11 ( ( PUNCT bracis-28417 292 12 2011 2011 NUM bracis-28417 292 13 ) ) PUNCT bracis-28417 292 14 google google PROPN bracis-28417 292 15 scholar scholar NOUN bracis-28417 292 16     SPACE bracis-28417 292 17 martin martin PROPN bracis-28417 292 18 , , PUNCT bracis-28417 292 19 l. l. PROPN bracis-28417 292 20 , , PUNCT bracis-28417 292 21 et et PROPN bracis-28417 292 22 al al PROPN bracis-28417 292 23 . . PROPN bracis-28417 292 24 : : PUNCT bracis-28417 293 1 camembert camembert NOUN bracis-28417 293 2 : : PUNCT bracis-28417 293 3 a a DET bracis-28417 293 4 tasty tasty ADJ bracis-28417 293 5 french french ADJ bracis-28417 293 6 language language NOUN bracis-28417 293 7 model model NOUN bracis-28417 293 8 . . PUNCT bracis-28417 294 1 in in ADP bracis-28417 294 2 : : PUNCT bracis-28417 294 3 proceedings proceeding NOUN bracis-28417 294 4 of of ADP bracis-28417 294 5 the the DET bracis-28417 294 6 58th 58th ADJ bracis-28417 294 7 annual annual ADJ bracis-28417 294 8 meeting meeting NOUN bracis-28417 294 9 of of ADP bracis-28417 294 10 the the DET bracis-28417 294 11 association association NOUN bracis-28417 294 12 for for ADP bracis-28417 294 13 computational computational ADJ bracis-28417 294 14 linguistics linguistic NOUN bracis-28417 294 15 . . PUNCT bracis-28417 295 1 pp pp ADJ bracis-28417 295 2 . . PUNCT bracis-28417 296 1 7203–7219 7203–7219 NUM bracis-28417 296 2 . . PUNCT bracis-28417 297 1 association association NOUN bracis-28417 297 2 for for ADP bracis-28417 297 3 computational computational ADJ bracis-28417 297 4 linguistics linguistic NOUN bracis-28417 297 5 , , PUNCT bracis-28417 297 6 online online ADV bracis-28417 297 7 ( ( PUNCT bracis-28417 297 8 2020 2020 NUM bracis-28417 297 9 ) ) PUNCT bracis-28417 297 10 . . PUNCT bracis-28417 298 1 https://doi.org/10.18653/v1/2020.acl-main.645 https://doi.org/10.18653/v1/2020.acl-main.645 PROPN bracis-28417 298 2 de de X bracis-28417 298 3 melo melo PROPN bracis-28417 298 4 , , PUNCT bracis-28417 298 5 g. g. PROPN bracis-28417 298 6 , , PUNCT bracis-28417 298 7 imaizumi imaizumi PROPN bracis-28417 298 8 , , PUNCT bracis-28417 298 9 v. v. PROPN bracis-28417 298 10 , , PUNCT bracis-28417 298 11 cozman cozman NOUN bracis-28417 298 12 , , PUNCT bracis-28417 298 13 f. f. PROPN bracis-28417 298 14 : : PROPN bracis-28417 298 15 winograd winograd PROPN bracis-28417 298 16 schemas schema NOUN bracis-28417 298 17 in in ADP bracis-28417 298 18 portuguese portuguese PROPN bracis-28417 298 19 . . PUNCT bracis-28417 299 1 in in ADP bracis-28417 299 2 : : PUNCT bracis-28417 299 3 anais anais PROPN bracis-28417 299 4 do do AUX bracis-28417 299 5 xvi xvi NOUN bracis-28417 299 6 encontro encontro VERB bracis-28417 299 7 nacional nacional ADJ bracis-28417 299 8 de de X bracis-28417 299 9 inteligência inteligência PROPN bracis-28417 299 10 artificial artificial PROPN bracis-28417 299 11 e e NOUN bracis-28417 299 12 computacional computacional NOUN bracis-28417 299 13 , , PUNCT bracis-28417 299 14 pp pp ADJ bracis-28417 299 15 . . PUNCT bracis-28417 300 1 787–798 787–798 NUM bracis-28417 300 2 . . PUNCT bracis-28417 301 1 sbc sbc NOUN bracis-28417 301 2 ( ( PUNCT bracis-28417 301 3 2019 2019 NUM bracis-28417 301 4 ) ) PUNCT bracis-28417 301 5 google google PROPN bracis-28417 301 6 scholar scholar NOUN bracis-28417 301 7     SPACE bracis-28417 301 8 moraes moraes PROPN bracis-28417 301 9 , , PUNCT bracis-28417 301 10 g. g. PROPN bracis-28417 301 11 , , PUNCT bracis-28417 301 12 bonifácio bonifácio PROPN bracis-28417 301 13 , , PUNCT bracis-28417 301 14 l.h l.h PROPN bracis-28417 301 15 . . PROPN bracis-28417 301 16 , , PUNCT bracis-28417 301 17 rodrigues rodrigues PROPN bracis-28417 301 18 de de PROPN bracis-28417 301 19 souza souza PROPN bracis-28417 301 20 , , PUNCT bracis-28417 301 21 l. l. PROPN bracis-28417 301 22 , , PUNCT bracis-28417 301 23 nogueira nogueira PROPN bracis-28417 301 24 , , PUNCT bracis-28417 301 25 r. r. PROPN bracis-28417 301 26 , , PUNCT bracis-28417 301 27 lotufo lotufo PROPN bracis-28417 301 28 , , PUNCT bracis-28417 301 29 r. r. PROPN bracis-28417 301 30 : : PUNCT bracis-28417 301 31 a a DET bracis-28417 301 32 cost cost NOUN bracis-28417 301 33 - - PUNCT bracis-28417 301 34 benefit benefit NOUN bracis-28417 301 35 analysis analysis NOUN bracis-28417 301 36 of of ADP bracis-28417 301 37 cross cross ADJ bracis-28417 301 38 - - ADJ bracis-28417 301 39 lingual lingual ADJ bracis-28417 301 40 transfer transfer NOUN bracis-28417 301 41 methods method NOUN bracis-28417 301 42 . . PUNCT bracis-28417 302 1 arxiv arxiv PROPN bracis-28417 302 2 preprint preprint VERB bracis-28417 302 3 arxiv:2105.06813 arxiv:2105.06813 PROPN bracis-28417 302 4 ( ( PUNCT bracis-28417 302 5 2021 2021 NUM bracis-28417 302 6 ) ) PUNCT bracis-28417 302 7 . . PUNCT bracis-28417 303 1 https://arxiv.org/abs/2105.06813 https://arxiv.org/abs/2105.06813 PROPN bracis-28417 303 2 muennighoff muennighoff PROPN bracis-28417 303 3 , , PUNCT bracis-28417 303 4 n. n. PROPN bracis-28417 303 5 , , PUNCT bracis-28417 303 6 et et PROPN bracis-28417 303 7 al al PROPN bracis-28417 303 8 . . PROPN bracis-28417 303 9 : : PUNCT bracis-28417 303 10 crosslingual crosslingual ADJ bracis-28417 303 11 generalization generalization NOUN bracis-28417 303 12 through through ADP bracis-28417 303 13 multitask multitask ADJ bracis-28417 303 14 finetuning finetuning NOUN bracis-28417 303 15 ( ( PUNCT bracis-28417 303 16 2022 2022 NUM bracis-28417 303 17 ) ) PUNCT bracis-28417 303 18 google google NOUN bracis-28417 303 19 scholar scholar NOUN bracis-28417 303 20     SPACE bracis-28417 303 21 nguyen nguyen NOUN bracis-28417 303 22 , , PUNCT bracis-28417 303 23 d.q d.q PROPN bracis-28417 303 24 . . PROPN bracis-28417 303 25 , , PUNCT bracis-28417 303 26 tuan tuan PROPN bracis-28417 303 27 nguyen nguyen PROPN bracis-28417 303 28 , , PUNCT bracis-28417 303 29 a. a. NOUN bracis-28417 303 30 : : PUNCT bracis-28417 303 31 phobert phobert NOUN bracis-28417 303 32 : : PUNCT bracis-28417 303 33 pre pre ADJ bracis-28417 303 34 - - ADJ bracis-28417 303 35 trained train VERB bracis-28417 303 36 language language NOUN bracis-28417 303 37 models model NOUN bracis-28417 303 38 for for ADP bracis-28417 303 39 vietnamese vietnamese NOUN bracis-28417 303 40 . . PUNCT bracis-28417 304 1 in in ADP bracis-28417 304 2 : : PUNCT bracis-28417 304 3 findings finding NOUN bracis-28417 304 4 of of ADP bracis-28417 304 5 the the DET bracis-28417 304 6 association association NOUN bracis-28417 304 7 for for ADP bracis-28417 304 8 computational computational ADJ bracis-28417 304 9 linguistics linguistic NOUN bracis-28417 304 10 : : PUNCT bracis-28417 304 11 emnlp emnlp PROPN bracis-28417 304 12 2020 2020 NUM bracis-28417 304 13 , , PUNCT bracis-28417 304 14 pp pp ADV bracis-28417 304 15 . . PUNCT bracis-28417 305 1 1037–1042 1037–1042 NUM bracis-28417 305 2 . . PUNCT bracis-28417 305 3 association association NOUN bracis-28417 305 4 for for ADP bracis-28417 305 5 computational computational ADJ bracis-28417 305 6 linguistics linguistic NOUN bracis-28417 305 7 , , PUNCT bracis-28417 305 8 online online ADV bracis-28417 305 9 ( ( PUNCT bracis-28417 305 10 2020 2020 NUM bracis-28417 305 11 ) ) PUNCT bracis-28417 305 12 . . PUNCT bracis-28417 306 1 https://doi.org/10.18653/v1/2020.findings-emnlp.92 https://doi.org/10.18653/v1/2020.findings-emnlp.92 PROPN bracis-28417 306 2 nunes nunes PROPN bracis-28417 306 3 , , PUNCT bracis-28417 306 4 d. d. PROPN bracis-28417 306 5 , , PUNCT bracis-28417 306 6 primi primi PROPN bracis-28417 306 7 , , PUNCT bracis-28417 306 8 r. r. PROPN bracis-28417 306 9 , , PUNCT bracis-28417 306 10 pires pire NOUN bracis-28417 306 11 , , PUNCT bracis-28417 306 12 r. r. PROPN bracis-28417 306 13 , , PUNCT bracis-28417 306 14 lotufo lotufo PROPN bracis-28417 306 15 , , PUNCT bracis-28417 306 16 r. r. PROPN bracis-28417 306 17 , , PUNCT bracis-28417 306 18 nogueira nogueira PROPN bracis-28417 306 19 , , PUNCT bracis-28417 306 20 r. r. PROPN bracis-28417 306 21 : : PUNCT bracis-28417 306 22 evaluating evaluate VERB bracis-28417 306 23 gpt-3.5 gpt-3.5 PUNCT bracis-28417 306 24 and and CCONJ bracis-28417 306 25 gpt-4 gpt-4 NOUN bracis-28417 306 26 models model NOUN bracis-28417 306 27 on on ADP bracis-28417 306 28 brazilian brazilian ADJ bracis-28417 306 29 university university NOUN bracis-28417 306 30 admission admission NOUN bracis-28417 306 31 exams exam NOUN bracis-28417 306 32 ( ( PUNCT bracis-28417 306 33 2023 2023 NUM bracis-28417 306 34 ) ) PUNCT bracis-28417 306 35 google google NOUN bracis-28417 306 36 scholar scholar NOUN bracis-28417 306 37     SPACE bracis-28417 306 38 ogueji ogueji NOUN bracis-28417 306 39 , , PUNCT bracis-28417 306 40 k. k. PROPN bracis-28417 306 41 , , PUNCT bracis-28417 306 42 zhu zhu PROPN bracis-28417 306 43 , , PUNCT bracis-28417 306 44 y. y. PROPN bracis-28417 306 45 , , PUNCT bracis-28417 306 46 lin lin PROPN bracis-28417 306 47 , , PUNCT bracis-28417 306 48 j. j. PROPN bracis-28417 306 49 : : PUNCT bracis-28417 306 50 small small ADJ bracis-28417 306 51 data datum NOUN bracis-28417 306 52 ? ? PUNCT bracis-28417 307 1 no no DET bracis-28417 307 2 problem problem NOUN bracis-28417 307 3 ! ! PUNCT bracis-28417 308 1 exploring explore VERB bracis-28417 308 2 the the DET bracis-28417 308 3 viability viability NOUN bracis-28417 308 4 of of ADP bracis-28417 308 5 pretrained pretraine VERB bracis-28417 308 6 multilingual multilingual ADJ bracis-28417 308 7 language language NOUN bracis-28417 308 8 models model NOUN bracis-28417 308 9 for for ADP bracis-28417 308 10 low low ADV bracis-28417 308 11 - - PUNCT bracis-28417 308 12 resourced resource VERB bracis-28417 308 13 languages language NOUN bracis-28417 308 14 . . PUNCT bracis-28417 309 1 in in ADP bracis-28417 309 2 : : PUNCT bracis-28417 309 3 proceedings proceeding NOUN bracis-28417 309 4 of of ADP bracis-28417 309 5 the the DET bracis-28417 309 6 1st 1st ADJ bracis-28417 309 7 workshop workshop NOUN bracis-28417 309 8 on on ADP bracis-28417 309 9 multilingual multilingual ADJ bracis-28417 309 10 representation representation NOUN bracis-28417 309 11 learning learning NOUN bracis-28417 309 12 , , PUNCT bracis-28417 309 13 pp pp ADJ bracis-28417 309 14 . . PUNCT bracis-28417 310 1 116–126 116–126 NUM bracis-28417 310 2 . . PUNCT bracis-28417 310 3 association association NOUN bracis-28417 310 4 for for ADP bracis-28417 310 5 computational computational ADJ bracis-28417 310 6 linguistics linguistic NOUN bracis-28417 310 7 , , PUNCT bracis-28417 310 8 punta punta PROPN bracis-28417 310 9 cana cana PROPN bracis-28417 310 10 , , PUNCT bracis-28417 310 11 dominican dominican PROPN bracis-28417 310 12 republic republic NOUN bracis-28417 310 13 ( ( PUNCT bracis-28417 310 14 2021 2021 NUM bracis-28417 310 15 ) ) PUNCT bracis-28417 310 16 google google NOUN bracis-28417 310 17 scholar scholar NOUN bracis-28417 310 18     SPACE bracis-28417 310 19 openai openai PROPN bracis-28417 310 20 : : PUNCT bracis-28417 310 21 gpt-4 gpt-4 NUM bracis-28417 310 22 technical technical ADJ bracis-28417 310 23 report report NOUN bracis-28417 310 24 ( ( PUNCT bracis-28417 310 25 2023 2023 NUM bracis-28417 310 26 ) ) PUNCT bracis-28417 310 27 google google NOUN bracis-28417 310 28 scholar scholar NOUN bracis-28417 310 29     SPACE bracis-28417 310 30 overwijk overwijk NOUN bracis-28417 310 31 , , PUNCT bracis-28417 310 32 a. a. NOUN bracis-28417 310 33 , , PUNCT bracis-28417 310 34 xiong xiong PROPN bracis-28417 310 35 , , PUNCT bracis-28417 310 36 c. c. PROPN bracis-28417 310 37 , , PUNCT bracis-28417 310 38 callan callan PROPN bracis-28417 310 39 , , PUNCT bracis-28417 310 40 j. j. PROPN bracis-28417 310 41 : : PUNCT bracis-28417 310 42 clueweb clueweb PROPN bracis-28417 310 43 22 22 NUM bracis-28417 310 44 : : SYM bracis-28417 310 45 10 10 NUM bracis-28417 310 46 billion billion NUM bracis-28417 310 47 web web NOUN bracis-28417 310 48 documents document NOUN bracis-28417 310 49 with with ADP bracis-28417 310 50 rich rich ADJ bracis-28417 310 51 information information NOUN bracis-28417 310 52 . . PUNCT bracis-28417 311 1 in in ADP bracis-28417 311 2 : : PUNCT bracis-28417 311 3 proceedings proceeding NOUN bracis-28417 311 4 of of ADP bracis-28417 311 5 the the DET bracis-28417 311 6 45th 45th ADJ bracis-28417 311 7 international international PROPN bracis-28417 311 8 acm acm PROPN bracis-28417 311 9 sigir sigir PROPN bracis-28417 311 10 conference conference PROPN bracis-28417 311 11 on on ADP bracis-28417 311 12 research research NOUN bracis-28417 311 13 and and CCONJ bracis-28417 311 14 development development NOUN bracis-28417 311 15 in in ADP bracis-28417 311 16 information information NOUN bracis-28417 311 17 retrieval retrieval NOUN bracis-28417 311 18 , , PUNCT bracis-28417 311 19 pp pp ADP bracis-28417 311 20 . . PUNCT bracis-28417 312 1 3360–3362 3360–3362 NUM bracis-28417 312 2 ( ( PUNCT bracis-28417 312 3 2022 2022 NUM bracis-28417 312 4 ) ) PUNCT bracis-28417 312 5 google google NOUN bracis-28417 312 6 scholar scholar NOUN bracis-28417 312 7     SPACE bracis-28417 312 8 overwijk overwijk NOUN bracis-28417 312 9 , , PUNCT bracis-28417 312 10 a. a. NOUN bracis-28417 312 11 , , PUNCT bracis-28417 312 12 xiong xiong PROPN bracis-28417 312 13 , , PUNCT bracis-28417 312 14 c. c. PROPN bracis-28417 312 15 , , PUNCT bracis-28417 312 16 liu liu PROPN bracis-28417 312 17 , , PUNCT bracis-28417 312 18 x. x. PROPN bracis-28417 312 19 , , PUNCT bracis-28417 312 20 vandenberg vandenberg PROPN bracis-28417 312 21 , , PUNCT bracis-28417 312 22 c. c. PROPN bracis-28417 312 23 , , PUNCT bracis-28417 312 24 callan callan PROPN bracis-28417 312 25 , , PUNCT bracis-28417 312 26 j. j. PROPN bracis-28417 312 27 : : PUNCT bracis-28417 312 28 clueweb clueweb PROPN bracis-28417 312 29 22 22 NUM bracis-28417 312 30 : : SYM bracis-28417 312 31 10 10 NUM bracis-28417 312 32 billion billion NUM bracis-28417 312 33 web web NOUN bracis-28417 312 34 documents document NOUN bracis-28417 312 35 with with ADP bracis-28417 312 36 visual visual ADJ bracis-28417 312 37 and and CCONJ bracis-28417 312 38 semantic semantic ADJ bracis-28417 312 39 information information NOUN bracis-28417 312 40 ( ( PUNCT bracis-28417 312 41 2022 2022 NUM bracis-28417 312 42 ) ) PUNCT bracis-28417 312 43 google google PROPN bracis-28417 312 44 scholar scholar NOUN bracis-28417 312 45     SPACE bracis-28417 312 46 pfeiffer pfeiffer PROPN bracis-28417 312 47 , , PUNCT bracis-28417 312 48 j. j. PROPN bracis-28417 312 49 , , PUNCT bracis-28417 312 50 kamath kamath PROPN bracis-28417 312 51 , , PUNCT bracis-28417 312 52 a. a. PROPN bracis-28417 312 53 , , PUNCT bracis-28417 312 54 rücklé rücklé PROPN bracis-28417 312 55 , , PUNCT bracis-28417 312 56 a. a. PROPN bracis-28417 312 57 , , PUNCT bracis-28417 312 58 cho cho PROPN bracis-28417 312 59 , , PUNCT bracis-28417 312 60 k. k. PROPN bracis-28417 312 61 , , PUNCT bracis-28417 312 62 gurevych gurevych PROPN bracis-28417 312 63 , , PUNCT bracis-28417 312 64 i. i. NOUN bracis-28417 312 65 : : PUNCT bracis-28417 312 66 adapterfusion adapterfusion NOUN bracis-28417 312 67 : : PUNCT bracis-28417 312 68 non non ADJ bracis-28417 312 69 - - ADJ bracis-28417 312 70 destructive destructive ADJ bracis-28417 312 71 task task NOUN bracis-28417 312 72 composition composition NOUN bracis-28417 312 73 for for ADP bracis-28417 312 74 transfer transfer NOUN bracis-28417 312 75 learning learning NOUN bracis-28417 312 76 . . PUNCT bracis-28417 313 1 in in ADP bracis-28417 313 2 : : PUNCT bracis-28417 313 3 proceedings proceeding NOUN bracis-28417 313 4 of of ADP bracis-28417 313 5 the the DET bracis-28417 313 6 16th 16th ADJ bracis-28417 313 7 conference conference NOUN bracis-28417 313 8 of of ADP bracis-28417 313 9 the the DET bracis-28417 313 10 european european ADJ bracis-28417 313 11 chapter chapter NOUN bracis-28417 313 12 of of ADP bracis-28417 313 13 the the DET bracis-28417 313 14 association association NOUN bracis-28417 313 15 for for ADP bracis-28417 313 16 computational computational ADJ bracis-28417 313 17 linguistics linguistic NOUN bracis-28417 313 18 : : PUNCT bracis-28417 313 19 main main ADJ bracis-28417 313 20 volume volume NOUN bracis-28417 313 21 , , PUNCT bracis-28417 313 22 pp pp X bracis-28417 313 23 . . PUNCT bracis-28417 314 1 487–503 487–503 NUM bracis-28417 314 2 . . PUNCT bracis-28417 315 1 association association NOUN bracis-28417 315 2 for for ADP bracis-28417 315 3 computational computational ADJ bracis-28417 315 4 linguistics linguistic NOUN bracis-28417 315 5 , , PUNCT bracis-28417 315 6 online online ADV bracis-28417 315 7 ( ( PUNCT bracis-28417 315 8 2021 2021 NUM bracis-28417 315 9 ) ) PUNCT bracis-28417 315 10 . . PUNCT bracis-28417 316 1 https://doi.org/10.18653/v1/2021.eacl-main.39 https://doi.org/10.18653/v1/2021.eacl-main.39 PROPN bracis-28417 316 2 pfeiffer pfeiffer PROPN bracis-28417 316 3 , , PUNCT bracis-28417 316 4 j. j. PROPN bracis-28417 316 5 , , PUNCT bracis-28417 316 6 vulić vulić PROPN bracis-28417 316 7 , , PUNCT bracis-28417 316 8 i. i. NOUN bracis-28417 316 9 , , PUNCT bracis-28417 316 10 gurevych gurevych PROPN bracis-28417 316 11 , , PUNCT bracis-28417 316 12 i. i. NOUN bracis-28417 316 13 , , PUNCT bracis-28417 316 14 ruder ruder PROPN bracis-28417 316 15 , , PUNCT bracis-28417 316 16 s. s. PROPN bracis-28417 316 17 : : PUNCT bracis-28417 316 18 mad mad ADJ bracis-28417 316 19 - - PUNCT bracis-28417 316 20 x x NOUN bracis-28417 316 21 : : PUNCT bracis-28417 316 22 an an DET bracis-28417 316 23 adapter adapter NOUN bracis-28417 316 24 - - PUNCT bracis-28417 316 25 based base VERB bracis-28417 316 26 framework framework NOUN bracis-28417 316 27 for for ADP bracis-28417 316 28 multi multi ADJ bracis-28417 316 29 - - ADJ bracis-28417 316 30 task task ADJ bracis-28417 316 31 cross cross ADJ bracis-28417 316 32 - - ADJ bracis-28417 316 33 lingual lingual ADJ bracis-28417 316 34 transfer transfer NOUN bracis-28417 316 35 . . PUNCT bracis-28417 317 1 arxiv arxiv PROPN bracis-28417 317 2 preprint preprint VERB bracis-28417 317 3 arxiv:2005.00052 arxiv:2005.00052 PROPN bracis-28417 317 4 ( ( PUNCT bracis-28417 317 5 2020 2020 NUM bracis-28417 317 6 ) ) PUNCT bracis-28417 318 1 radford radford PROPN bracis-28417 318 2 , , PUNCT bracis-28417 318 3 a. a. PROPN bracis-28417 318 4 , , PUNCT bracis-28417 318 5 wu wu PROPN bracis-28417 318 6 , , PUNCT bracis-28417 318 7 j. j. PROPN bracis-28417 318 8 , , PUNCT bracis-28417 318 9 child child NOUN bracis-28417 318 10 , , PUNCT bracis-28417 318 11 r. r. PROPN bracis-28417 318 12 , , PUNCT bracis-28417 318 13 luan luan PROPN bracis-28417 318 14 , , PUNCT bracis-28417 318 15 d. d. PROPN bracis-28417 318 16 , , PUNCT bracis-28417 318 17 amodei amodei PROPN bracis-28417 318 18 , , PUNCT bracis-28417 318 19 d. d. PROPN bracis-28417 318 20 , , PUNCT bracis-28417 318 21 sutskever sutskever PROPN bracis-28417 318 22 , , PUNCT bracis-28417 318 23 i. i. PROPN bracis-28417 318 24 , , PUNCT bracis-28417 318 25 et et PROPN bracis-28417 318 26 al al PROPN bracis-28417 318 27 . . PUNCT bracis-28417 318 28 : : PUNCT bracis-28417 319 1 language language NOUN bracis-28417 319 2 models model NOUN bracis-28417 319 3 are be AUX bracis-28417 319 4 unsupervised unsupervised ADJ bracis-28417 319 5 multitask multitask ADJ bracis-28417 319 6 learners learner NOUN bracis-28417 319 7 . . PUNCT bracis-28417 320 1 openai openai PROPN bracis-28417 320 2 blog blog NOUN bracis-28417 320 3 1(8 1(8 NUM bracis-28417 320 4 ) ) PUNCT bracis-28417 320 5 , , PUNCT bracis-28417 320 6 9 9 NUM bracis-28417 320 7 ( ( PUNCT bracis-28417 320 8 2019 2019 NUM bracis-28417 320 9 ) ) PUNCT bracis-28417 320 10 google google PROPN bracis-28417 320 11 scholar scholar NOUN bracis-28417 320 12     SPACE bracis-28417 320 13 rae rae PROPN bracis-28417 320 14 , , PUNCT bracis-28417 320 15 j.w j.w PROPN bracis-28417 320 16 . . PROPN bracis-28417 320 17 , , PUNCT bracis-28417 320 18 et et PROPN bracis-28417 320 19 al al PROPN bracis-28417 320 20 . . PUNCT bracis-28417 320 21 : : PUNCT bracis-28417 321 1 scaling scale VERB bracis-28417 321 2 language language NOUN bracis-28417 321 3 models model NOUN bracis-28417 321 4 : : PUNCT bracis-28417 321 5 methods method NOUN bracis-28417 321 6 , , PUNCT bracis-28417 321 7 analysis analysis NOUN bracis-28417 321 8 & & CCONJ bracis-28417 321 9 insights insight NOUN bracis-28417 321 10 from from ADP bracis-28417 321 11 training train VERB bracis-28417 321 12 gopher gopher NOUN bracis-28417 321 13 . . PUNCT bracis-28417 322 1 arxiv arxiv PROPN bracis-28417 322 2 preprint preprint NOUN bracis-28417 322 3 arxiv:2112.11446 arxiv:2112.11446 NOUN bracis-28417 322 4 ( ( PUNCT bracis-28417 322 5 2021 2021 NUM bracis-28417 322 6 ) ) PUNCT bracis-28417 322 7 raffel raffel NOUN bracis-28417 322 8 , , PUNCT bracis-28417 322 9 c. c. PROPN bracis-28417 322 10 , , PUNCT bracis-28417 322 11 et et PROPN bracis-28417 322 12 al al PROPN bracis-28417 322 13 . . PROPN bracis-28417 322 14 : : PUNCT bracis-28417 322 15 exploring explore VERB bracis-28417 322 16 the the DET bracis-28417 322 17 limits limit NOUN bracis-28417 322 18 of of ADP bracis-28417 322 19 transfer transfer NOUN bracis-28417 322 20 learning learn VERB bracis-28417 322 21 with with ADP bracis-28417 322 22 a a DET bracis-28417 322 23 unified unified ADJ bracis-28417 322 24 text text NOUN bracis-28417 322 25 - - PUNCT bracis-28417 322 26 to to ADP bracis-28417 322 27 - - PUNCT bracis-28417 322 28 text text NOUN bracis-28417 322 29 transformer transformer NOUN bracis-28417 322 30 . . PUNCT bracis-28417 323 1 j. j. PROPN bracis-28417 323 2 mach mach PROPN bracis-28417 323 3 . . PUNCT bracis-28417 324 1 learn learn VERB bracis-28417 324 2 . . PUNCT bracis-28417 325 1 res re NOUN bracis-28417 325 2 . . PUNCT bracis-28417 326 1 21(1 21(1 NUM bracis-28417 326 2 ) ) PUNCT bracis-28417 327 1 , , PUNCT bracis-28417 327 2 5485–5551 5485–5551 NUM bracis-28417 327 3 ( ( PUNCT bracis-28417 327 4 2020 2020 NUM bracis-28417 327 5 ) ) PUNCT bracis-28417 327 6 mathscinet mathscinet NOUN bracis-28417 327 7     SPACE bracis-28417 327 8 google google PROPN bracis-28417 327 9 scholar scholar NOUN bracis-28417 327 10     SPACE bracis-28417 327 11 real real ADJ bracis-28417 327 12 , , PUNCT bracis-28417 327 13 l. l. PROPN bracis-28417 327 14 , , PUNCT bracis-28417 327 15 fonseca fonseca PROPN bracis-28417 327 16 , , PUNCT bracis-28417 327 17 e. e. PROPN bracis-28417 327 18 , , PUNCT bracis-28417 327 19 gonçalo gonçalo PROPN bracis-28417 327 20 oliveira oliveira PROPN bracis-28417 327 21 , , PUNCT bracis-28417 327 22 h. h. PROPN bracis-28417 327 23 : : PUNCT bracis-28417 327 24 the the DET bracis-28417 327 25 assin assin PROPN bracis-28417 327 26 2 2 NUM bracis-28417 327 27 shared share VERB bracis-28417 327 28 task task NOUN bracis-28417 327 29 : : PUNCT bracis-28417 327 30 a a DET bracis-28417 327 31 quick quick ADJ bracis-28417 327 32 overview overview NOUN bracis-28417 327 33 . . PUNCT bracis-28417 328 1 in in ADP bracis-28417 328 2 : : PUNCT bracis-28417 328 3 quaresma quaresma PROPN bracis-28417 328 4 , , PUNCT bracis-28417 328 5 p. p. PROPN bracis-28417 328 6 , , PUNCT bracis-28417 328 7 vieira vieira PROPN bracis-28417 328 8 , , PUNCT bracis-28417 328 9 r. r. PROPN bracis-28417 328 10 , , PUNCT bracis-28417 328 11 aluísio aluísio NOUN bracis-28417 328 12 , , PUNCT bracis-28417 328 13 s. s. PROPN bracis-28417 328 14 , , PUNCT bracis-28417 328 15 moniz moniz PROPN bracis-28417 328 16 , , PUNCT bracis-28417 328 17 h. h. PROPN bracis-28417 328 18 , , PUNCT bracis-28417 328 19 batista batista PROPN bracis-28417 328 20 , , PUNCT bracis-28417 328 21 f. f. PROPN bracis-28417 328 22 , , PUNCT bracis-28417 328 23 gonçalves gonçalves PROPN bracis-28417 328 24 , , PUNCT bracis-28417 328 25 t. t. PROPN bracis-28417 328 26 ( ( PUNCT bracis-28417 328 27 eds eds PROPN bracis-28417 328 28 . . PUNCT bracis-28417 328 29 ) ) PUNCT bracis-28417 328 30 propor propor ADJ bracis-28417 328 31 2020 2020 NUM bracis-28417 328 32 . . PUNCT bracis-28417 329 1 lncs lncs PROPN bracis-28417 329 2 ( ( PUNCT bracis-28417 329 3 lnai lnai ADJ bracis-28417 329 4 ) ) PUNCT bracis-28417 329 5 , , PUNCT bracis-28417 329 6 vol vol NOUN bracis-28417 329 7 . . PROPN bracis-28417 329 8 12037 12037 NUM bracis-28417 329 9 , , PUNCT bracis-28417 329 10 pp pp ADP bracis-28417 329 11 . . PUNCT bracis-28417 330 1 406–412 406–412 NUM bracis-28417 330 2 . . PUNCT bracis-28417 330 3 springer springer NOUN bracis-28417 330 4 , , PUNCT bracis-28417 330 5 cham cham PROPN bracis-28417 330 6 ( ( PUNCT bracis-28417 330 7 2020 2020 NUM bracis-28417 330 8 ) ) PUNCT bracis-28417 330 9 . . PUNCT bracis-28417 331 1 https://doi.org/10.1007/978-3-030-41505-1_39 https://doi.org/10.1007/978-3-030-41505-1_39 PROPN bracis-28417 331 2 chapter chapter NOUN bracis-28417 331 3     SPACE bracis-28417 331 4 google google PROPN bracis-28417 331 5 scholar scholar NOUN bracis-28417 331 6     SPACE bracis-28417 331 7 roberts roberts PROPN bracis-28417 331 8 , , PUNCT bracis-28417 331 9 a. a. PROPN bracis-28417 331 10 , , PUNCT bracis-28417 331 11 et et PROPN bracis-28417 331 12 al al PROPN bracis-28417 331 13 . . PUNCT bracis-28417 331 14 : : PUNCT bracis-28417 332 1 scaling scale VERB bracis-28417 332 2 up up ADP bracis-28417 332 3 models model NOUN bracis-28417 332 4 and and CCONJ bracis-28417 332 5 data datum NOUN bracis-28417 332 6 with with ADP bracis-28417 332 7 t5x t5x NOUN bracis-28417 332 8 and and CCONJ bracis-28417 332 9 seqio seqio NOUN bracis-28417 332 10 . . PUNCT bracis-28417 333 1 arxiv arxiv PROPN bracis-28417 333 2 preprint preprint NOUN bracis-28417 333 3 arxiv:2203.17189 arxiv:2203.17189 PROPN bracis-28417 333 4 13 13 NUM bracis-28417 333 5 ( ( PUNCT bracis-28417 333 6 2022 2022 NUM bracis-28417 333 7 ) ) PUNCT bracis-28417 333 8 rosa rosa PROPN bracis-28417 333 9 , , PUNCT bracis-28417 333 10 g.m g.m PROPN bracis-28417 333 11 . . PROPN bracis-28417 333 12 , , PUNCT bracis-28417 333 13 bonifacio bonifacio PROPN bracis-28417 333 14 , , PUNCT bracis-28417 333 15 l.h l.h PROPN bracis-28417 333 16 . . PROPN bracis-28417 333 17 , , PUNCT bracis-28417 333 18 de de PROPN bracis-28417 333 19 souza souza PROPN bracis-28417 333 20 , , PUNCT bracis-28417 333 21 l.r l.r PROPN bracis-28417 333 22 . . PROPN bracis-28417 333 23 , , PUNCT bracis-28417 333 24 lotufo lotufo PROPN bracis-28417 333 25 , , PUNCT bracis-28417 333 26 r. r. PROPN bracis-28417 333 27 , , PUNCT bracis-28417 333 28 nogueira nogueira PROPN bracis-28417 333 29 , , PUNCT bracis-28417 333 30 r. r. PROPN bracis-28417 333 31 : : PUNCT bracis-28417 333 32 a a DET bracis-28417 333 33 cost cost NOUN bracis-28417 333 34 - - PUNCT bracis-28417 333 35 benefit benefit NOUN bracis-28417 333 36 analysis analysis NOUN bracis-28417 333 37 of of ADP bracis-28417 333 38 cross cross ADJ bracis-28417 333 39 - - ADJ bracis-28417 333 40 lingual lingual ADJ bracis-28417 333 41 transfer transfer NOUN bracis-28417 333 42 methods method NOUN bracis-28417 333 43 . . PUNCT bracis-28417 334 1 arxiv arxiv PROPN bracis-28417 334 2 preprint preprint VERB bracis-28417 334 3 arxiv:2105.06813 arxiv:2105.06813 PROPN bracis-28417 334 4 ( ( PUNCT bracis-28417 334 5 2021 2021 NUM bracis-28417 334 6 ) ) PUNCT bracis-28417 334 7 la la PROPN bracis-28417 334 8 rosa rosa PROPN bracis-28417 334 9 , , PUNCT bracis-28417 334 10 j.d j.d PROPN bracis-28417 334 11 . . PROPN bracis-28417 334 12 , , PUNCT bracis-28417 334 13 fernández fernández PROPN bracis-28417 334 14 , , PUNCT bracis-28417 334 15 a. a. NOUN bracis-28417 334 16 : : PUNCT bracis-28417 334 17 zero zero NUM bracis-28417 334 18 - - PUNCT bracis-28417 334 19 shot shot NOUN bracis-28417 334 20 reading reading NOUN bracis-28417 334 21 comprehension comprehension NOUN bracis-28417 334 22 and and CCONJ bracis-28417 334 23 reasoning reasoning NOUN bracis-28417 334 24 for for ADP bracis-28417 334 25 spanish spanish ADJ bracis-28417 334 26 with with ADP bracis-28417 334 27 bertin bertin PROPN bracis-28417 334 28 gpt gpt PROPN bracis-28417 334 29 - - PUNCT bracis-28417 334 30 j-6b j-6b PROPN bracis-28417 334 31 . . PUNCT bracis-28417 335 1 in in ADP bracis-28417 335 2 : : PUNCT bracis-28417 335 3 y y PROPN bracis-28417 335 4 gómez gómez NOUN bracis-28417 335 5 , , PUNCT bracis-28417 335 6 m.m m.m PROPN bracis-28417 335 7 . . PROPN bracis-28417 335 8 , , PUNCT bracis-28417 335 9 ( ( PUNCT bracis-28417 335 10 eds ed NOUN bracis-28417 335 11 . . PUNCT bracis-28417 335 12 ) ) PUNCT bracis-28417 335 13 proceedings proceeding NOUN bracis-28417 335 14 of of ADP bracis-28417 335 15 the the DET bracis-28417 335 16 iberian iberian PROPN bracis-28417 335 17 languages languages PROPN bracis-28417 335 18 evaluation evaluation NOUN bracis-28417 335 19 forum forum NOUN bracis-28417 335 20 ( ( PUNCT bracis-28417 335 21 iberlef iberlef PROPN bracis-28417 335 22 2022 2022 NUM bracis-28417 335 23 ) ) PUNCT bracis-28417 335 24 . . PUNCT bracis-28417 336 1 ceur ceur PROPN bracis-28417 336 2 workshop workshop NOUN bracis-28417 336 3 proceedings proceeding NOUN bracis-28417 336 4 ( ( PUNCT bracis-28417 336 5 2022 2022 NUM bracis-28417 336 6 ) ) PUNCT bracis-28417 336 7 google google NOUN bracis-28417 336 8 scholar scholar NOUN bracis-28417 336 9     SPACE bracis-28417 336 10 sakaguchi sakaguchi ADJ bracis-28417 336 11 , , PUNCT bracis-28417 336 12 k. k. PROPN bracis-28417 336 13 , , PUNCT bracis-28417 336 14 bras bras PROPN bracis-28417 336 15 , , PUNCT bracis-28417 336 16 r.l r.l PROPN bracis-28417 336 17 . . PROPN bracis-28417 336 18 , , PUNCT bracis-28417 336 19 bhagavatula bhagavatula PROPN bracis-28417 336 20 , , PUNCT bracis-28417 336 21 c. c. NOUN bracis-28417 336 22 , , PUNCT bracis-28417 336 23 choi choi NOUN bracis-28417 336 24 , , PUNCT bracis-28417 336 25 y. y. NOUN bracis-28417 336 26 : : PUNCT bracis-28417 336 27 winogrande winogrande PROPN bracis-28417 336 28 : : PUNCT bracis-28417 336 29 an an DET bracis-28417 336 30 adversarial adversarial ADJ bracis-28417 336 31 winograd winograd ADJ bracis-28417 336 32 schema schema NOUN bracis-28417 336 33 challenge challenge NOUN bracis-28417 336 34 at at ADP bracis-28417 336 35 scale scale NOUN bracis-28417 336 36 . . PUNCT bracis-28417 337 1 commun commun PROPN bracis-28417 337 2 . . PUNCT bracis-28417 338 1 acm acm PROPN bracis-28417 338 2 64(9 64(9 NUM bracis-28417 338 3 ) ) PUNCT bracis-28417 338 4 , , PUNCT bracis-28417 338 5 99–106 99–106 NUM bracis-28417 338 6 ( ( PUNCT bracis-28417 338 7 2021 2021 NUM bracis-28417 338 8 ) ) PUNCT bracis-28417 338 9 article article NOUN bracis-28417 338 10     SPACE bracis-28417 338 11 google google PROPN bracis-28417 338 12 scholar scholar NOUN bracis-28417 338 13     SPACE bracis-28417 338 14 sarti sarti NOUN bracis-28417 338 15 , , PUNCT bracis-28417 338 16 g. g. PROPN bracis-28417 338 17 , , PUNCT bracis-28417 338 18 nissim nissim PROPN bracis-28417 338 19 , , PUNCT bracis-28417 338 20 m. m. NOUN bracis-28417 338 21 : : PUNCT bracis-28417 338 22 it5 it5 NOUN bracis-28417 338 23 : : PUNCT bracis-28417 338 24 large large ADJ bracis-28417 338 25 - - PUNCT bracis-28417 338 26 scale scale NOUN bracis-28417 338 27 text text NOUN bracis-28417 338 28 - - PUNCT bracis-28417 338 29 to to ADP bracis-28417 338 30 - - PUNCT bracis-28417 338 31 text text NOUN bracis-28417 338 32 pretraining pretraine VERB bracis-28417 338 33 for for ADP bracis-28417 338 34 italian italian ADJ bracis-28417 338 35 language language NOUN bracis-28417 338 36 understanding understanding NOUN bracis-28417 338 37 and and CCONJ bracis-28417 338 38 generation generation NOUN bracis-28417 338 39 . . PUNCT bracis-28417 339 1 arxiv arxiv PROPN bracis-28417 339 2 preprint preprint NOUN bracis-28417 339 3 arxiv:2203.03759 arxiv:2203.03759 PROPN bracis-28417 339 4 ( ( PUNCT bracis-28417 339 5 2022 2022 NUM bracis-28417 339 6 ) ) PUNCT bracis-28417 339 7 sayama sayama NOUN bracis-28417 339 8 , , PUNCT bracis-28417 339 9 h.f h.f PROPN bracis-28417 339 10 . . PROPN bracis-28417 339 11 , , PUNCT bracis-28417 339 12 araujo araujo PROPN bracis-28417 339 13 , , PUNCT bracis-28417 339 14 a.v a.v PROPN bracis-28417 339 15 . . PROPN bracis-28417 339 16 , , PUNCT bracis-28417 339 17 fernandes fernandes PROPN bracis-28417 339 18 , , PUNCT bracis-28417 339 19 e.r e.r PROPN bracis-28417 339 20 . . PROPN bracis-28417 339 21 : : PUNCT bracis-28417 340 1 faquad faquad ADJ bracis-28417 340 2 : : PUNCT bracis-28417 340 3 reading read VERB bracis-28417 340 4 comprehension comprehension NOUN bracis-28417 340 5 dataset dataset VERB bracis-28417 340 6 in in ADP bracis-28417 340 7 the the DET bracis-28417 340 8 domain domain NOUN bracis-28417 340 9 of of ADP bracis-28417 340 10 brazilian brazilian ADJ bracis-28417 340 11 higher high ADJ bracis-28417 340 12 education education NOUN bracis-28417 340 13 . . PUNCT bracis-28417 341 1 in in ADP bracis-28417 341 2 : : PUNCT bracis-28417 341 3 2019 2019 NUM bracis-28417 341 4 8th 8th ADJ bracis-28417 341 5 brazilian brazilian ADJ bracis-28417 341 6 conference conference NOUN bracis-28417 341 7 on on ADP bracis-28417 341 8 intelligent intelligent ADJ bracis-28417 341 9 systems system NOUN bracis-28417 341 10 ( ( PUNCT bracis-28417 341 11 bracis bracis NOUN bracis-28417 341 12 ) ) PUNCT bracis-28417 341 13 , , PUNCT bracis-28417 341 14 pp pp ADP bracis-28417 341 15 . . PUNCT bracis-28417 342 1 443–448 443–448 NUM bracis-28417 342 2 ( ( PUNCT bracis-28417 342 3 2019 2019 NUM bracis-28417 342 4 ) ) PUNCT bracis-28417 342 5 . . PUNCT bracis-28417 343 1 https://doi.org/10.1109/bracis.2019.00084 https://doi.org/10.1109/bracis.2019.00084 PROPN bracis-28417 343 2 scao scao PROPN bracis-28417 343 3 , , PUNCT bracis-28417 343 4 t.l t.l PROPN bracis-28417 343 5 . . PROPN bracis-28417 343 6 , , PUNCT bracis-28417 343 7 et et PROPN bracis-28417 343 8 al al PROPN bracis-28417 343 9 . . PROPN bracis-28417 343 10 : : PUNCT bracis-28417 344 1 bloom bloom NOUN bracis-28417 344 2 : : PUNCT bracis-28417 344 3 a a DET bracis-28417 344 4 176b 176b ADJ bracis-28417 344 5 - - PUNCT bracis-28417 344 6 parameter parameter NOUN bracis-28417 344 7 open open ADJ bracis-28417 344 8 - - PUNCT bracis-28417 344 9 access access NOUN bracis-28417 344 10 multilingual multilingual ADJ bracis-28417 344 11 language language NOUN bracis-28417 344 12 model model NOUN bracis-28417 344 13 . . PUNCT bracis-28417 345 1 arxiv arxiv PROPN bracis-28417 345 2 preprint preprint VERB bracis-28417 345 3 arxiv:2211.05100 arxiv:2211.05100 NOUN bracis-28417 345 4 ( ( PUNCT bracis-28417 345 5 2022 2022 NUM bracis-28417 345 6 ) ) PUNCT bracis-28417 345 7 shazeer shazeer NOUN bracis-28417 345 8 , , PUNCT bracis-28417 345 9 n. n. NOUN bracis-28417 345 10 , , PUNCT bracis-28417 345 11 stern stern ADJ bracis-28417 345 12 , , PUNCT bracis-28417 345 13 m. m. NOUN bracis-28417 345 14 : : PUNCT bracis-28417 345 15 adafactor adafactor NOUN bracis-28417 345 16 : : PUNCT bracis-28417 345 17 adaptive adaptive ADJ bracis-28417 345 18 learning learning NOUN bracis-28417 345 19 rates rate NOUN bracis-28417 345 20 with with ADP bracis-28417 345 21 sublinear sublinear NOUN bracis-28417 345 22 memory memory NOUN bracis-28417 345 23 cost cost NOUN bracis-28417 345 24 . . PUNCT bracis-28417 346 1 in in ADP bracis-28417 346 2 : : PUNCT bracis-28417 346 3 international international ADJ bracis-28417 346 4 conference conference NOUN bracis-28417 346 5 on on ADP bracis-28417 346 6 machine machine NOUN bracis-28417 346 7 learning learning NOUN bracis-28417 346 8 , , PUNCT bracis-28417 346 9 pp pp ADP bracis-28417 346 10 . . PUNCT bracis-28417 347 1 4596–4604 4596–4604 NUM bracis-28417 347 2 . . PUNCT bracis-28417 348 1 pmlr pmlr NOUN bracis-28417 348 2 ( ( PUNCT bracis-28417 348 3 2018 2018 NUM bracis-28417 348 4 ) ) PUNCT bracis-28417 348 5 google google PROPN bracis-28417 348 6 scholar scholar NOUN bracis-28417 348 7     SPACE bracis-28417 348 8 shliazhko shliazhko NOUN bracis-28417 348 9 , , PUNCT bracis-28417 348 10 o. o. PROPN bracis-28417 348 11 , , PUNCT bracis-28417 348 12 fenogenova fenogenova PROPN bracis-28417 348 13 , , PUNCT bracis-28417 348 14 a. a. PROPN bracis-28417 348 15 , , PUNCT bracis-28417 348 16 tikhonova tikhonova PROPN bracis-28417 348 17 , , PUNCT bracis-28417 348 18 m. m. NOUN bracis-28417 348 19 , , PUNCT bracis-28417 348 20 mikhailov mikhailov PROPN bracis-28417 348 21 , , PUNCT bracis-28417 348 22 v. v. PROPN bracis-28417 348 23 , , PUNCT bracis-28417 348 24 kozlova kozlova PROPN bracis-28417 348 25 , , PUNCT bracis-28417 348 26 a. a. PROPN bracis-28417 348 27 , , PUNCT bracis-28417 348 28 shavrina shavrina NOUN bracis-28417 348 29 , , PUNCT bracis-28417 348 30 t. t. NOUN bracis-28417 348 31 : : PUNCT bracis-28417 348 32 mgpt mgpt NOUN bracis-28417 348 33 : : PUNCT bracis-28417 348 34 few few ADJ bracis-28417 348 35 - - PUNCT bracis-28417 348 36 shot shot NOUN bracis-28417 348 37 learners learner NOUN bracis-28417 348 38 go go VERB bracis-28417 348 39 multilingual multilingual ADJ bracis-28417 348 40 . . PUNCT bracis-28417 349 1 arxiv arxiv PROPN bracis-28417 349 2 preprint preprint VERB bracis-28417 349 3 arxiv:2204.07580 arxiv:2204.07580 NOUN bracis-28417 349 4 ( ( PUNCT bracis-28417 349 5 2022 2022 NUM bracis-28417 349 6 ) ) PUNCT bracis-28417 349 7 silveira silveira PROPN bracis-28417 349 8 , , PUNCT bracis-28417 349 9 i.c i.c PROPN bracis-28417 349 10 . . PROPN bracis-28417 349 11 , , PUNCT bracis-28417 349 12 maua maua PROPN bracis-28417 349 13 , , PUNCT bracis-28417 349 14 d.d d.d PROPN bracis-28417 349 15 . . PROPN bracis-28417 349 16 : : PUNCT bracis-28417 349 17 advances advance VERB bracis-28417 349 18 in in ADP bracis-28417 349 19 automatically automatically ADV bracis-28417 349 20 solving solve VERB bracis-28417 349 21 the the DET bracis-28417 349 22 enem enem NOUN bracis-28417 349 23 . . PUNCT bracis-28417 350 1 in in ADP bracis-28417 350 2 : : PUNCT bracis-28417 350 3 2018 2018 NUM bracis-28417 350 4 7th 7th ADJ bracis-28417 350 5 brazilian brazilian ADJ bracis-28417 350 6 conference conference NOUN bracis-28417 350 7 on on ADP bracis-28417 350 8 intelligent intelligent ADJ bracis-28417 350 9 systems system NOUN bracis-28417 350 10 ( ( PUNCT bracis-28417 350 11 bracis bracis NOUN bracis-28417 350 12 ) ) PUNCT bracis-28417 350 13 , , PUNCT bracis-28417 350 14 pp pp PROPN bracis-28417 350 15 . . PUNCT bracis-28417 350 16 43–48 43–48 NUM bracis-28417 350 17 . . PUNCT bracis-28417 350 18 ieee ieee NOUN bracis-28417 350 19 computer computer NOUN bracis-28417 350 20 society society NOUN bracis-28417 350 21 , , PUNCT bracis-28417 350 22 los los PROPN bracis-28417 350 23 alamitos alamitos PROPN bracis-28417 350 24 , , PUNCT bracis-28417 350 25 ca ca PROPN bracis-28417 350 26 , , PUNCT bracis-28417 350 27 usa usa PROPN bracis-28417 350 28 ( ( PUNCT bracis-28417 350 29 oct oct PROPN bracis-28417 350 30 2018 2018 NUM bracis-28417 350 31 ) ) PUNCT bracis-28417 350 32 . . PUNCT bracis-28417 351 1 https://doi.org/10.1109/bracis.2018.00016 https://doi.org/10.1109/bracis.2018.00016 PROPN bracis-28417 351 2 socher socher NOUN bracis-28417 351 3 , , PUNCT bracis-28417 351 4 r. r. PROPN bracis-28417 351 5 , , PUNCT bracis-28417 351 6 et et PROPN bracis-28417 351 7 al al PROPN bracis-28417 351 8 . . PUNCT bracis-28417 351 9 : : PUNCT bracis-28417 352 1 recursive recursive ADJ bracis-28417 352 2 deep deep ADJ bracis-28417 352 3 models model NOUN bracis-28417 352 4 for for ADP bracis-28417 352 5 semantic semantic ADJ bracis-28417 352 6 compositionality compositionality NOUN bracis-28417 352 7 over over ADP bracis-28417 352 8 a a DET bracis-28417 352 9 sentiment sentiment NOUN bracis-28417 352 10 treebank treebank VERB bracis-28417 352 11 . . PUNCT bracis-28417 353 1 in in ADP bracis-28417 353 2 : : PUNCT bracis-28417 353 3 proceedings proceeding NOUN bracis-28417 353 4 of of ADP bracis-28417 353 5 the the DET bracis-28417 353 6 2013 2013 NUM bracis-28417 353 7 conference conference NOUN bracis-28417 353 8 on on ADP bracis-28417 353 9 empirical empirical ADJ bracis-28417 353 10 methods method NOUN bracis-28417 353 11 in in ADP bracis-28417 353 12 natural natural ADJ bracis-28417 353 13 language language NOUN bracis-28417 353 14 processing processing NOUN bracis-28417 353 15 , , PUNCT bracis-28417 353 16 pp pp ADJ bracis-28417 353 17 . . PUNCT bracis-28417 354 1 1631–1642 1631–1642 NUM bracis-28417 354 2 . . PUNCT bracis-28417 355 1 association association NOUN bracis-28417 355 2 for for ADP bracis-28417 355 3 computational computational ADJ bracis-28417 355 4 linguistics linguistic NOUN bracis-28417 355 5 , , PUNCT bracis-28417 355 6 seattle seattle PROPN bracis-28417 355 7 , , PUNCT bracis-28417 355 8 washington washington PROPN bracis-28417 355 9 , , PUNCT bracis-28417 355 10 usa usa PROPN bracis-28417 355 11 ( ( PUNCT bracis-28417 355 12 2013 2013 NUM bracis-28417 355 13 ) ) PUNCT bracis-28417 355 14 google google PROPN bracis-28417 355 15 scholar scholar NOUN bracis-28417 355 16     SPACE bracis-28417 355 17 souza souza PROPN bracis-28417 355 18 , , PUNCT bracis-28417 355 19 f. f. PROPN bracis-28417 355 20 , , PUNCT bracis-28417 355 21 nogueira nogueira PROPN bracis-28417 355 22 , , PUNCT bracis-28417 355 23 r. r. PROPN bracis-28417 355 24 , , PUNCT bracis-28417 355 25 lotufo lotufo PROPN bracis-28417 355 26 , , PUNCT bracis-28417 355 27 r. r. PROPN bracis-28417 355 28 : : PUNCT bracis-28417 355 29 bertimbau bertimbau PROPN bracis-28417 355 30 : : PUNCT bracis-28417 355 31 pretrained pretraine VERB bracis-28417 355 32 bert bert NOUN bracis-28417 355 33 models model NOUN bracis-28417 355 34 for for ADP bracis-28417 355 35 brazilian brazilian ADJ bracis-28417 355 36 portuguese portuguese NOUN bracis-28417 355 37 . . PUNCT bracis-28417 356 1 in in ADP bracis-28417 356 2 : : PUNCT bracis-28417 356 3 cerri cerri PROPN bracis-28417 356 4 , , PUNCT bracis-28417 356 5 r. r. PROPN bracis-28417 356 6 , , PUNCT bracis-28417 356 7 prati prati PROPN bracis-28417 356 8 , , PUNCT bracis-28417 356 9 r.c r.c PROPN bracis-28417 356 10 . . PROPN bracis-28417 356 11 ( ( PUNCT bracis-28417 356 12 eds ed NOUN bracis-28417 356 13 . . PUNCT bracis-28417 356 14 ) ) PUNCT bracis-28417 356 15 bracis bracis PROPN bracis-28417 356 16 2020 2020 NUM bracis-28417 356 17 . . PUNCT bracis-28417 357 1 lncs lncs PROPN bracis-28417 357 2 ( ( PUNCT bracis-28417 357 3 lnai lnai ADJ bracis-28417 357 4 ) ) PUNCT bracis-28417 357 5 , , PUNCT bracis-28417 357 6 vol vol NOUN bracis-28417 357 7 . . PUNCT bracis-28417 357 8 12319 12319 NUM bracis-28417 357 9 , , PUNCT bracis-28417 357 10 pp pp ADP bracis-28417 357 11 . . PUNCT bracis-28417 358 1 403–417 403–417 NUM bracis-28417 358 2 . . PUNCT bracis-28417 358 3 springer springer NOUN bracis-28417 358 4 , , PUNCT bracis-28417 358 5 cham cham PROPN bracis-28417 358 6 ( ( PUNCT bracis-28417 358 7 2020 2020 NUM bracis-28417 358 8 ) ) PUNCT bracis-28417 358 9 . . PUNCT bracis-28417 359 1 https://doi.org/10.1007/978-3-030-61377-8_28 https://doi.org/10.1007/978-3-030-61377-8_28 PROPN bracis-28417 359 2 chapter chapter NOUN bracis-28417 359 3     SPACE bracis-28417 359 4 google google PROPN bracis-28417 359 5 scholar scholar NOUN bracis-28417 359 6     SPACE bracis-28417 359 7 srivastava srivastava PROPN bracis-28417 359 8 , , PUNCT bracis-28417 359 9 a. a. PROPN bracis-28417 359 10 , , PUNCT bracis-28417 359 11 et et PROPN bracis-28417 359 12 al al PROPN bracis-28417 359 13 . . PUNCT bracis-28417 359 14 : : PUNCT bracis-28417 360 1 beyond beyond ADP bracis-28417 360 2 the the DET bracis-28417 360 3 imitation imitation NOUN bracis-28417 360 4 game game NOUN bracis-28417 360 5 : : PUNCT bracis-28417 360 6 quantifying quantify VERB bracis-28417 360 7 and and CCONJ bracis-28417 360 8 extrapolating extrapolate VERB bracis-28417 360 9 the the DET bracis-28417 360 10 capabilities capability NOUN bracis-28417 360 11 of of ADP bracis-28417 360 12 language language NOUN bracis-28417 360 13 models model NOUN bracis-28417 360 14 . . PUNCT bracis-28417 361 1 arxiv arxiv PROPN bracis-28417 361 2 preprint preprint NOUN bracis-28417 361 3 arxiv:2206.04615 arxiv:2206.04615 NOUN bracis-28417 361 4 ( ( PUNCT bracis-28417 361 5 2022 2022 NUM bracis-28417 361 6 ) ) PUNCT bracis-28417 361 7 su su PROPN bracis-28417 361 8 , , PUNCT bracis-28417 361 9 j. j. PROPN bracis-28417 361 10 , , PUNCT bracis-28417 361 11 lu lu PROPN bracis-28417 361 12 , , PUNCT bracis-28417 361 13 y. y. PROPN bracis-28417 361 14 , , PUNCT bracis-28417 361 15 pan pan PROPN bracis-28417 361 16 , , PUNCT bracis-28417 361 17 s. s. PROPN bracis-28417 361 18 , , PUNCT bracis-28417 361 19 wen wen PROPN bracis-28417 361 20 , , PUNCT bracis-28417 361 21 b. b. PROPN bracis-28417 361 22 , , PUNCT bracis-28417 361 23 liu liu PROPN bracis-28417 361 24 , , PUNCT bracis-28417 361 25 y. y. NOUN bracis-28417 361 26 : : PUNCT bracis-28417 362 1 roformer roformer NOUN bracis-28417 362 2 : : PUNCT bracis-28417 362 3 enhanced enhanced ADJ bracis-28417 362 4 transformer transformer NOUN bracis-28417 362 5 with with ADP bracis-28417 362 6 rotary rotary ADJ bracis-28417 362 7 position position NOUN bracis-28417 362 8 embedding embed VERB bracis-28417 362 9 . . PUNCT bracis-28417 363 1 arxiv arxiv PROPN bracis-28417 363 2 preprint preprint VERB bracis-28417 363 3 arxiv:2104.09864 arxiv:2104.09864 NOUN bracis-28417 363 4 ( ( PUNCT bracis-28417 363 5 2021 2021 NUM bracis-28417 363 6 ) ) PUNCT bracis-28417 363 7 taylor taylor PROPN bracis-28417 363 8 , , PUNCT bracis-28417 363 9 r. r. PROPN bracis-28417 363 10 et et PROPN bracis-28417 363 11 al al PROPN bracis-28417 363 12 . . PROPN bracis-28417 363 13 : : PUNCT bracis-28417 364 1 galactica galactica NOUN bracis-28417 364 2 : : PUNCT bracis-28417 364 3 a a DET bracis-28417 364 4 large large ADJ bracis-28417 364 5 language language NOUN bracis-28417 364 6 model model NOUN bracis-28417 364 7 for for ADP bracis-28417 364 8 science science NOUN bracis-28417 364 9 . . PUNCT bracis-28417 365 1 arxiv arxiv PROPN bracis-28417 365 2 preprint preprint VERB bracis-28417 365 3 arxiv:2211.09085 arxiv:2211.09085 NUM bracis-28417 365 4 ( ( PUNCT bracis-28417 365 5 2022 2022 NUM bracis-28417 365 6 ) ) PUNCT bracis-28417 365 7 touvron touvron PROPN bracis-28417 365 8 , , PUNCT bracis-28417 365 9 h. h. PROPN bracis-28417 365 10 , , PUNCT bracis-28417 365 11 et et PROPN bracis-28417 365 12 al al PROPN bracis-28417 365 13 . . PUNCT bracis-28417 365 14 : : PUNCT bracis-28417 366 1 llama llama NOUN bracis-28417 366 2 : : PUNCT bracis-28417 366 3 open open ADJ bracis-28417 366 4 and and CCONJ bracis-28417 366 5 efficient efficient ADJ bracis-28417 366 6 foundation foundation NOUN bracis-28417 366 7 language language NOUN bracis-28417 366 8 models model NOUN bracis-28417 366 9 . . PUNCT bracis-28417 367 1 arxiv arxiv PROPN bracis-28417 367 2 preprint preprint PROPN bracis-28417 367 3 arxiv:2302.13971 arxiv:2302.13971 PROPN bracis-28417 367 4 ( ( PUNCT bracis-28417 367 5 2023 2023 NUM bracis-28417 367 6 ) ) PUNCT bracis-28417 367 7 vaswani vaswani NOUN bracis-28417 367 8 , , PUNCT bracis-28417 367 9 a. a. NOUN bracis-28417 367 10 , , PUNCT bracis-28417 367 11 et et PROPN bracis-28417 367 12 al al PROPN bracis-28417 367 13 . . PUNCT bracis-28417 367 14 : : PUNCT bracis-28417 368 1 attention attention NOUN bracis-28417 368 2 is be AUX bracis-28417 368 3 all all PRON bracis-28417 368 4 you you PRON bracis-28417 368 5 need need VERB bracis-28417 368 6 . . PUNCT bracis-28417 369 1 in in ADP bracis-28417 369 2 : : PUNCT bracis-28417 369 3 advances advance NOUN bracis-28417 369 4 in in ADP bracis-28417 369 5 neural neural ADJ bracis-28417 369 6 information information NOUN bracis-28417 369 7 processing processing NOUN bracis-28417 369 8 systems system NOUN bracis-28417 369 9 30 30 NUM bracis-28417 369 10 ( ( PUNCT bracis-28417 369 11 2017 2017 NUM bracis-28417 369 12 ) ) PUNCT bracis-28417 369 13 google google PROPN bracis-28417 369 14 scholar scholar NOUN bracis-28417 369 15     SPACE bracis-28417 369 16 wang wang PROPN bracis-28417 369 17 , , PUNCT bracis-28417 369 18 b. b. PROPN bracis-28417 369 19 : : PUNCT bracis-28417 369 20 mesh mesh NOUN bracis-28417 369 21 - - PUNCT bracis-28417 369 22 transformer transformer NOUN bracis-28417 369 23 - - PUNCT bracis-28417 369 24 jax jax NOUN bracis-28417 369 25 : : PUNCT bracis-28417 369 26 model model ADJ bracis-28417 369 27 - - PUNCT bracis-28417 369 28 parallel parallel ADJ bracis-28417 369 29 implementation implementation NOUN bracis-28417 369 30 of of ADP bracis-28417 369 31 transformer transformer ADJ bracis-28417 369 32 language language NOUN bracis-28417 369 33 model model NOUN bracis-28417 369 34 with with ADP bracis-28417 369 35 jax jax PROPN bracis-28417 369 36 . . PUNCT bracis-28417 370 1 https://github.com/kingoflolz/mesh-transformer-jax https://github.com/kingoflolz/mesh-transformer-jax PROPN bracis-28417 370 2 ( ( PUNCT bracis-28417 370 3 2021 2021 NUM bracis-28417 370 4 ) ) PUNCT bracis-28417 370 5 wang wang PROPN bracis-28417 370 6 , , PUNCT bracis-28417 370 7 b. b. PROPN bracis-28417 370 8 , , PUNCT bracis-28417 370 9 komatsuzaki komatsuzaki NOUN bracis-28417 370 10 , , PUNCT bracis-28417 370 11 a. a. NOUN bracis-28417 370 12 : : PUNCT bracis-28417 370 13 gpt gpt PROPN bracis-28417 370 14 - - PUNCT bracis-28417 370 15 j-6b j-6b PROPN bracis-28417 370 16 : : PUNCT bracis-28417 370 17 a a DET bracis-28417 370 18 6 6 NUM bracis-28417 370 19 billion billion NUM bracis-28417 370 20 parameter parameter NOUN bracis-28417 370 21 autoregressive autoregressive ADJ bracis-28417 370 22 language language NOUN bracis-28417 370 23 model model NOUN bracis-28417 370 24 ( ( PUNCT bracis-28417 370 25 2021 2021 NUM bracis-28417 370 26 ) ) PUNCT bracis-28417 370 27 google google PROPN bracis-28417 370 28 scholar scholar NOUN bracis-28417 370 29     SPACE bracis-28417 370 30 wei wei PROPN bracis-28417 370 31 , , PUNCT bracis-28417 370 32 j. j. PROPN bracis-28417 370 33 , , PUNCT bracis-28417 370 34 et et PROPN bracis-28417 370 35 al al PROPN bracis-28417 370 36 . . PUNCT bracis-28417 370 37 : : PUNCT bracis-28417 370 38 emergent emergent ADJ bracis-28417 370 39 abilities ability NOUN bracis-28417 370 40 of of ADP bracis-28417 370 41 large large ADJ bracis-28417 370 42 language language NOUN bracis-28417 370 43 models model NOUN bracis-28417 370 44 . . PUNCT bracis-28417 371 1 transactions transaction NOUN bracis-28417 371 2 on on ADP bracis-28417 371 3 machine machine NOUN bracis-28417 371 4 learning learn VERB bracis-28417 371 5 research research NOUN bracis-28417 371 6 ( ( PUNCT bracis-28417 371 7 2022 2022 NUM bracis-28417 371 8 ) ) PUNCT bracis-28417 371 9 , , PUNCT bracis-28417 371 10 survey survey NOUN bracis-28417 371 11 certification certification NOUN bracis-28417 371 12 google google PROPN bracis-28417 371 13 scholar scholar PROPN bracis-28417 371 14     SPACE bracis-28417 371 15 wu wu PROPN bracis-28417 371 16 , , PUNCT bracis-28417 371 17 s. s. PROPN bracis-28417 371 18 , , PUNCT bracis-28417 371 19 et et PROPN bracis-28417 371 20 al al PROPN bracis-28417 371 21 . . PUNCT bracis-28417 371 22 : : PUNCT bracis-28417 372 1 bloomberggpt bloomberggpt NOUN bracis-28417 372 2 : : PUNCT bracis-28417 372 3 a a DET bracis-28417 372 4 large large ADJ bracis-28417 372 5 language language NOUN bracis-28417 372 6 model model NOUN bracis-28417 372 7 for for ADP bracis-28417 372 8 finance finance NOUN bracis-28417 372 9 ( ( PUNCT bracis-28417 372 10 2023 2023 NUM bracis-28417 372 11 ) ) PUNCT bracis-28417 372 12 google google NOUN bracis-28417 372 13 scholar scholar NOUN bracis-28417 372 14     SPACE bracis-28417 372 15 xue xue PROPN bracis-28417 372 16 , , PUNCT bracis-28417 372 17 l. l. PROPN bracis-28417 372 18 , , PUNCT bracis-28417 372 19 et et PROPN bracis-28417 372 20 al al PROPN bracis-28417 372 21 . . PUNCT bracis-28417 372 22 : : PUNCT bracis-28417 373 1 byt5 byt5 PROPN bracis-28417 373 2 : : PUNCT bracis-28417 373 3 towards towards ADP bracis-28417 373 4 a a DET bracis-28417 373 5 token token ADJ bracis-28417 373 6 - - PUNCT bracis-28417 373 7 free free ADJ bracis-28417 373 8 future future NOUN bracis-28417 373 9 with with ADP bracis-28417 373 10 pre pre ADJ bracis-28417 373 11 - - ADJ bracis-28417 373 12 trained train VERB bracis-28417 373 13 byte byte NOUN bracis-28417 373 14 - - PUNCT bracis-28417 373 15 to to ADP bracis-28417 373 16 - - PUNCT bracis-28417 373 17 byte byte NOUN bracis-28417 373 18 models model NOUN bracis-28417 373 19 . . PUNCT bracis-28417 374 1 trans trans PROPN bracis-28417 374 2 . . PUNCT bracis-28417 375 1 assoc assoc PROPN bracis-28417 375 2 . . PUNCT bracis-28417 376 1 comput comput PROPN bracis-28417 376 2 . . PUNCT bracis-28417 377 1 linguist linguist NOUN bracis-28417 377 2 . . PUNCT bracis-28417 378 1 10 10 NUM bracis-28417 378 2 , , PUNCT bracis-28417 378 3 291–306 291–306 NUM bracis-28417 378 4 ( ( PUNCT bracis-28417 378 5 2022 2022 NUM bracis-28417 378 6 ) ) PUNCT bracis-28417 378 7 article article NOUN bracis-28417 378 8     SPACE bracis-28417 378 9 google google PROPN bracis-28417 378 10 scholar scholar NOUN bracis-28417 378 11     SPACE bracis-28417 378 12 xue xue PROPN bracis-28417 378 13 , , PUNCT bracis-28417 378 14 l. l. PROPN bracis-28417 378 15 , , PUNCT bracis-28417 378 16 et et PROPN bracis-28417 378 17 al al PROPN bracis-28417 378 18 . . PUNCT bracis-28417 378 19 : : PUNCT bracis-28417 379 1 mt5 mt5 ADJ bracis-28417 379 2 : : PUNCT bracis-28417 379 3 a a DET bracis-28417 379 4 massively massively ADV bracis-28417 379 5 multilingual multilingual ADJ bracis-28417 379 6 pre pre ADJ bracis-28417 379 7 - - ADJ bracis-28417 379 8 trained train VERB bracis-28417 379 9 text text NOUN bracis-28417 379 10 - - PUNCT bracis-28417 379 11 to to ADP bracis-28417 379 12 - - PUNCT bracis-28417 379 13 text text NOUN bracis-28417 379 14 transformer transformer NOUN bracis-28417 379 15 . . PUNCT bracis-28417 380 1 arxiv arxiv PROPN bracis-28417 380 2 preprint preprint VERB bracis-28417 380 3 arxiv:2010.11934 arxiv:2010.11934 NOUN bracis-28417 380 4 ( ( PUNCT bracis-28417 380 5 2020 2020 NUM bracis-28417 380 6 ) ) PUNCT bracis-28417 380 7 yang yang PROPN bracis-28417 380 8 , , PUNCT bracis-28417 380 9 z. z. PROPN bracis-28417 380 10 , , PUNCT bracis-28417 380 11 dai dai PROPN bracis-28417 380 12 , , PUNCT bracis-28417 380 13 z. z. PROPN bracis-28417 380 14 , , PUNCT bracis-28417 380 15 yang yang PROPN bracis-28417 380 16 , , PUNCT bracis-28417 380 17 y. y. PROPN bracis-28417 380 18 , , PUNCT bracis-28417 380 19 carbonell carbonell PROPN bracis-28417 380 20 , , PUNCT bracis-28417 380 21 j. j. PROPN bracis-28417 380 22 , , PUNCT bracis-28417 380 23 salakhutdinov salakhutdinov PROPN bracis-28417 380 24 , , PUNCT bracis-28417 380 25 r. r. PROPN bracis-28417 380 26 , , PUNCT bracis-28417 380 27 le le X bracis-28417 380 28 , , PUNCT bracis-28417 380 29 q.v q.v PROPN bracis-28417 380 30 . . PROPN bracis-28417 380 31 : : PUNCT bracis-28417 381 1 xlnet xlnet PROPN bracis-28417 381 2 : : PUNCT bracis-28417 381 3 generalized generalize VERB bracis-28417 381 4 autoregressive autoregressive ADJ bracis-28417 381 5 pretraining pretraine VERB bracis-28417 381 6 for for ADP bracis-28417 381 7 language language NOUN bracis-28417 381 8 understanding understanding NOUN bracis-28417 381 9 . . PUNCT bracis-28417 382 1 curran curran PROPN bracis-28417 382 2 associates associates PROPN bracis-28417 382 3 inc inc PROPN bracis-28417 382 4 . . PROPN bracis-28417 382 5 , , PUNCT bracis-28417 382 6 red red ADJ bracis-28417 382 7 hook hook NOUN bracis-28417 382 8 , , PUNCT bracis-28417 382 9 ny ny PROPN bracis-28417 382 10 , , PUNCT bracis-28417 382 11 usa usa PROPN bracis-28417 382 12 ( ( PUNCT bracis-28417 382 13 2019 2019 NUM bracis-28417 382 14 ) ) PUNCT bracis-28417 382 15 google google PROPN bracis-28417 382 16 scholar scholar NOUN bracis-28417 382 17     SPACE bracis-28417 382 18 yong yong PROPN bracis-28417 382 19 , , PUNCT bracis-28417 382 20 z.x z.x PROPN bracis-28417 382 21 . . PROPN bracis-28417 382 22 , , PUNCT bracis-28417 382 23 et et PROPN bracis-28417 382 24 al al PROPN bracis-28417 382 25 . . PUNCT bracis-28417 382 26 : : PUNCT bracis-28417 383 1 bloom+ bloom+ X bracis-28417 383 2 1 1 NUM bracis-28417 383 3 : : PUNCT bracis-28417 383 4 adding add VERB bracis-28417 383 5 language language NOUN bracis-28417 383 6 support support NOUN bracis-28417 383 7 to to PART bracis-28417 383 8 bloom bloom VERB bracis-28417 383 9 for for ADP bracis-28417 383 10 zero zero NUM bracis-28417 383 11 - - PUNCT bracis-28417 383 12 shot shot NOUN bracis-28417 383 13 prompting prompting NOUN bracis-28417 383 14 . . PUNCT bracis-28417 384 1 arxiv arxiv PROPN bracis-28417 384 2 preprint preprint VERB bracis-28417 384 3 arxiv:2212.09535 arxiv:2212.09535 PROPN bracis-28417 384 4 ( ( PUNCT bracis-28417 384 5 2022 2022 NUM bracis-28417 384 6 ) ) PUNCT bracis-28417 384 7 zeng zeng PROPN bracis-28417 384 8 , , PUNCT bracis-28417 384 9 a. a. PROPN bracis-28417 384 10 , , PUNCT bracis-28417 384 11 et et PROPN bracis-28417 384 12 al al PROPN bracis-28417 384 13 . . PROPN bracis-28417 384 14 : : PUNCT bracis-28417 385 1 glm-130b glm-130b ADV bracis-28417 385 2 : : PUNCT bracis-28417 385 3 an an DET bracis-28417 385 4 open open ADJ bracis-28417 385 5 bilingual bilingual ADJ bracis-28417 385 6 pre pre ADJ bracis-28417 385 7 - - ADJ bracis-28417 385 8 trained train VERB bracis-28417 385 9 model model NOUN bracis-28417 385 10 . . PUNCT bracis-28417 386 1 arxiv arxiv PROPN bracis-28417 386 2 preprint preprint NOUN bracis-28417 386 3 arxiv:2210.02414 arxiv:2210.02414 NUM bracis-28417 386 4 ( ( PUNCT bracis-28417 386 5 2022 2022 NUM bracis-28417 386 6 ) ) PUNCT bracis-28417 386 7 zhang zhang PROPN bracis-28417 386 8 , , PUNCT bracis-28417 386 9 s. s. PROPN bracis-28417 386 10 , , PUNCT bracis-28417 386 11 et et PROPN bracis-28417 386 12 al al PROPN bracis-28417 386 13 . . PROPN bracis-28417 386 14 : : PUNCT bracis-28417 387 1 opt opt PROPN bracis-28417 387 2 : : PUNCT bracis-28417 387 3 open open ADJ bracis-28417 387 4 pre pre ADJ bracis-28417 387 5 - - ADJ bracis-28417 387 6 trained train VERB bracis-28417 387 7 transformer transformer NOUN bracis-28417 387 8 language language NOUN bracis-28417 387 9 models model NOUN bracis-28417 387 10 . . PUNCT bracis-28417 388 1 arxiv arxiv PROPN bracis-28417 388 2 preprint preprint NOUN bracis-28417 388 3 arxiv:2205.01068 arxiv:2205.01068 PROPN bracis-28417 388 4 ( ( PUNCT bracis-28417 388 5 2022 2022 NUM bracis-28417 388 6 ) ) PUNCT bracis-28417 388 7 zhang zhang PROPN bracis-28417 388 8 , , PUNCT bracis-28417 388 9 x. x. PROPN bracis-28417 388 10 , , PUNCT bracis-28417 388 11 zhao zhao PROPN bracis-28417 388 12 , , PUNCT bracis-28417 388 13 j.j j.j PROPN bracis-28417 388 14 . . PROPN bracis-28417 388 15 , , PUNCT bracis-28417 388 16 lecun lecun ADJ bracis-28417 388 17 , , PUNCT bracis-28417 388 18 y. y. NOUN bracis-28417 388 19 : : PUNCT bracis-28417 388 20 character character NOUN bracis-28417 388 21 - - PUNCT bracis-28417 388 22 level level NOUN bracis-28417 388 23 convolutional convolutional ADJ bracis-28417 388 24 networks network NOUN bracis-28417 388 25 for for ADP bracis-28417 388 26 text text NOUN bracis-28417 388 27 classification classification NOUN bracis-28417 388 28 . . PUNCT bracis-28417 389 1 in in ADP bracis-28417 389 2 : : PUNCT bracis-28417 389 3 nips nip NOUN bracis-28417 389 4 ( ( PUNCT bracis-28417 389 5 2015 2015 NUM bracis-28417 389 6 ) ) PUNCT bracis-28417 389 7 google google PROPN bracis-28417 389 8 scholar scholar NOUN bracis-28417 389 9     SPACE bracis-28417 389 10 zoph zoph NOUN bracis-28417 389 11 , , PUNCT bracis-28417 389 12 b. b. PROPN bracis-28417 389 13 : : PUNCT bracis-28417 389 14 designing design VERB bracis-28417 389 15 effective effective ADJ bracis-28417 389 16 sparse sparse ADJ bracis-28417 389 17 expert expert NOUN bracis-28417 389 18 models model NOUN bracis-28417 389 19 . . PUNCT bracis-28417 390 1 in in ADP bracis-28417 390 2 : : PUNCT bracis-28417 390 3 2022 2022 NUM bracis-28417 390 4 ieee ieee NOUN bracis-28417 390 5 international international PROPN bracis-28417 390 6 parallel parallel ADJ bracis-28417 390 7 and and CCONJ bracis-28417 390 8 distributed distributed ADJ bracis-28417 390 9 processing processing NOUN bracis-28417 390 10 symposium symposium NOUN bracis-28417 390 11 workshops workshop NOUN bracis-28417 390 12 ( ( PUNCT bracis-28417 390 13 ipdpsw ipdpsw NOUN bracis-28417 390 14 ) ) PUNCT bracis-28417 390 15 , , PUNCT bracis-28417 390 16 p. p. NOUN bracis-28417 390 17 1044 1044 NUM bracis-28417 390 18 . . PUNCT bracis-28417 391 1 ieee ieee NOUN bracis-28417 391 2 ( ( PUNCT bracis-28417 391 3 2022 2022 NUM bracis-28417 391 4 ) ) PUNCT bracis-28417 391 5 google google PROPN bracis-28417 391 6 scholar scholar NOUN bracis-28417 391 7     SPACE bracis-28417 391 8 download download NOUN bracis-28417 391 9 references reference NOUN bracis-28417 391 10 acknowledgments acknowledgment NOUN bracis-28417 391 11 we we PRON bracis-28417 391 12 thank thank VERB bracis-28417 391 13 google google PROPN bracis-28417 391 14 cloud cloud PROPN bracis-28417 391 15 for for ADP bracis-28417 391 16 the the DET bracis-28417 391 17 generous generous ADJ bracis-28417 391 18 tpu tpu PROPN bracis-28417 391 19 grant grant PROPN bracis-28417 391 20 . . PUNCT bracis-28417 392 1 author author NOUN bracis-28417 392 2 information information NOUN bracis-28417 392 3 authors author NOUN bracis-28417 392 4 and and CCONJ bracis-28417 392 5 affiliations affiliation NOUN bracis-28417 392 6 maritaca maritaca PROPN bracis-28417 392 7 ai ai VERB bracis-28417 392 8 , , PUNCT bracis-28417 392 9 campinas campinas PROPN bracis-28417 392 10 , , PUNCT bracis-28417 392 11 brazil brazil PROPN bracis-28417 392 12 ramon ramon NOUN bracis-28417 392 13 pires pire NOUN bracis-28417 392 14 , , PUNCT bracis-28417 392 15     SPACE bracis-28417 392 16 hugo hugo PROPN bracis-28417 392 17 abonizio abonizio PROPN bracis-28417 392 18 , , PUNCT bracis-28417 392 19     SPACE bracis-28417 392 20 thales thales PROPN bracis-28417 392 21 sales sales PROPN bracis-28417 392 22 almeida almeida PROPN bracis-28417 392 23     SPACE bracis-28417 392 24 & & CCONJ bracis-28417 392 25     SPACE bracis-28417 392 26 rodrigo rodrigo PROPN bracis-28417 392 27 nogueira nogueira PROPN bracis-28417 392 28 authors author NOUN bracis-28417 392 29 ramon ramon NOUN bracis-28417 392 30 piresview piresview NOUN bracis-28417 392 31 author author NOUN bracis-28417 392 32 publications publication NOUN bracis-28417 392 33 search search NOUN bracis-28417 392 34 author author NOUN bracis-28417 392 35 on on ADP bracis-28417 392 36 : : PUNCT bracis-28417 392 37 pubmed pubmed PROPN bracis-28417 392 38     SPACE bracis-28417 392 39 google google PROPN bracis-28417 392 40 scholar scholar PROPN bracis-28417 392 41 hugo hugo PROPN bracis-28417 392 42 abonizioview abonizioview PROPN bracis-28417 392 43 author author NOUN bracis-28417 392 44 publications publication NOUN bracis-28417 392 45 search search NOUN bracis-28417 392 46 author author NOUN bracis-28417 392 47 on on ADP bracis-28417 392 48 : : PUNCT bracis-28417 392 49 pubmed pubmed PROPN bracis-28417 392 50     SPACE bracis-28417 392 51 google google PROPN bracis-28417 392 52 scholar scholar NOUN bracis-28417 392 53 thales thales PROPN bracis-28417 392 54 sales sale NOUN bracis-28417 392 55 almeidaview almeidaview NOUN bracis-28417 392 56 author author NOUN bracis-28417 392 57 publications publication VERB bracis-28417 392 58 search search NOUN bracis-28417 392 59 author author NOUN bracis-28417 392 60 on on ADP bracis-28417 392 61 : : PUNCT bracis-28417 392 62 pubmed pubmed PROPN bracis-28417 392 63     SPACE bracis-28417 392 64 google google PROPN bracis-28417 392 65 scholar scholar PROPN bracis-28417 392 66 rodrigo rodrigo PROPN bracis-28417 392 67 nogueiraview nogueiraview PROPN bracis-28417 392 68 author author NOUN bracis-28417 392 69 publications publication VERB bracis-28417 392 70 search search NOUN bracis-28417 392 71 author author NOUN bracis-28417 392 72 on on ADP bracis-28417 392 73 : : PUNCT bracis-28417 392 74 pubmed pubmed PROPN bracis-28417 392 75     SPACE bracis-28417 392 76 google google PROPN bracis-28417 392 77 scholar scholar NOUN bracis-28417 392 78 corresponding correspond VERB bracis-28417 392 79 author author NOUN bracis-28417 392 80 correspondence correspondence NOUN bracis-28417 392 81 to to ADP bracis-28417 392 82 ramon ramon NOUN bracis-28417 392 83 pires pire NOUN bracis-28417 392 84 . . PUNCT bracis-28417 393 1 editor editor NOUN bracis-28417 393 2 information information NOUN bracis-28417 393 3 editors editor NOUN bracis-28417 393 4 and and CCONJ bracis-28417 393 5 affiliations affiliation NOUN bracis-28417 393 6 federal federal PROPN bracis-28417 393 7 university university PROPN bracis-28417 393 8 of of ADP bracis-28417 393 9 são são PROPN bracis-28417 393 10 carlos carlos PROPN bracis-28417 393 11 , , PUNCT bracis-28417 393 12 são são PROPN bracis-28417 393 13 carlos carlos PROPN bracis-28417 393 14 , , PUNCT bracis-28417 393 15 brazil brazil PROPN bracis-28417 393 16 murilo murilo PROPN bracis-28417 393 17 c. c. PROPN bracis-28417 393 18 naldi naldi PROPN bracis-28417 393 19 centro centro PROPN bracis-28417 393 20 universitario universitario PROPN bracis-28417 393 21 da da PROPN bracis-28417 393 22 fei fei PROPN bracis-28417 393 23 , , PUNCT bracis-28417 393 24 são são PROPN bracis-28417 393 25 bernardo bernardo PROPN bracis-28417 393 26 do do AUX bracis-28417 393 27 campo campo PROPN bracis-28417 393 28 , , PUNCT bracis-28417 393 29 brazil brazil PROPN bracis-28417 393 30 reinaldo reinaldo PROPN bracis-28417 393 31 a. a. PROPN bracis-28417 393 32 c. c. PROPN bracis-28417 393 33 bianchi bianchi PROPN bracis-28417 393 34 rights right NOUN bracis-28417 393 35 and and CCONJ bracis-28417 393 36 permissions permission NOUN bracis-28417 393 37 reprints reprint NOUN bracis-28417 393 38 and and CCONJ bracis-28417 393 39 permissions permission VERB bracis-28417 393 40 copyright copyright NOUN bracis-28417 393 41 information information NOUN bracis-28417 393 42 © © ADP bracis-28417 393 43 2023 2023 NUM bracis-28417 393 44 the the DET bracis-28417 393 45 author(s author(s NOUN bracis-28417 393 46 ) ) PUNCT bracis-28417 393 47 , , PUNCT bracis-28417 393 48 under under ADP bracis-28417 393 49 exclusive exclusive ADJ bracis-28417 393 50 license license NOUN bracis-28417 393 51 to to ADP bracis-28417 393 52 springer springer NOUN bracis-28417 393 53 nature nature PROPN bracis-28417 393 54 switzerland switzerland PROPN bracis-28417 393 55 ag ag PROPN bracis-28417 393 56 about about ADP bracis-28417 393 57 this this DET bracis-28417 393 58 paper paper NOUN bracis-28417 393 59 cite cite VERB bracis-28417 393 60 this this DET bracis-28417 393 61 paper paper NOUN bracis-28417 393 62 pires pire NOUN bracis-28417 393 63 , , PUNCT bracis-28417 393 64 r. r. PROPN bracis-28417 393 65 , , PUNCT bracis-28417 393 66 abonizio abonizio PROPN bracis-28417 393 67 , , PUNCT bracis-28417 393 68 h. h. PROPN bracis-28417 393 69 , , PUNCT bracis-28417 393 70 almeida almeida PROPN bracis-28417 393 71 , , PUNCT bracis-28417 393 72 t.s t.s PROPN bracis-28417 393 73 . . PROPN bracis-28417 393 74 , , PUNCT bracis-28417 393 75 nogueira nogueira PROPN bracis-28417 393 76 , , PUNCT bracis-28417 393 77 r. r. PROPN bracis-28417 393 78 ( ( PUNCT bracis-28417 393 79 2023 2023 NUM bracis-28417 393 80 ) ) PUNCT bracis-28417 393 81 . . PUNCT bracis-28417 394 1 sabiá sabiá PROPN bracis-28417 394 2 : : PUNCT bracis-28417 395 1 portuguese portuguese VERB bracis-28417 395 2 large large ADJ bracis-28417 395 3 language language NOUN bracis-28417 395 4 models model NOUN bracis-28417 395 5 . . PUNCT bracis-28417 396 1 in in ADP bracis-28417 396 2 : : PUNCT bracis-28417 396 3 naldi naldi PROPN bracis-28417 396 4 , , PUNCT bracis-28417 396 5 m.c m.c PROPN bracis-28417 396 6 . . PROPN bracis-28417 396 7 , , PUNCT bracis-28417 396 8 bianchi bianchi PROPN bracis-28417 396 9 , , PUNCT bracis-28417 396 10 r.a.c r.a.c ADP bracis-28417 396 11 . . PUNCT bracis-28417 396 12 ( ( PUNCT bracis-28417 396 13 eds ed NOUN bracis-28417 396 14 ) ) PUNCT bracis-28417 396 15 intelligent intelligent ADJ bracis-28417 396 16 systems system NOUN bracis-28417 396 17 . . PUNCT bracis-28417 397 1 bracis bracis PROPN bracis-28417 397 2 2023 2023 NUM bracis-28417 397 3 . . PUNCT bracis-28417 398 1 lecture lecture NOUN bracis-28417 398 2 notes note NOUN bracis-28417 398 3 in in ADP bracis-28417 398 4 computer computer NOUN bracis-28417 398 5 science science NOUN bracis-28417 398 6 ( ( PUNCT bracis-28417 398 7 ) ) PUNCT bracis-28417 398 8 , , PUNCT bracis-28417 398 9 vol vol NOUN bracis-28417 398 10 14197 14197 NUM bracis-28417 398 11 . . PUNCT bracis-28417 399 1 springer springer NOUN bracis-28417 399 2 , , PUNCT bracis-28417 399 3 cham cham PROPN bracis-28417 399 4 . . PUNCT bracis-28417 400 1 https://doi.org/10.1007/978-3-031-45392-2_15 https://doi.org/10.1007/978-3-031-45392-2_15 PROPN bracis-28417 400 2 download download NOUN bracis-28417 400 3 citation citation NOUN bracis-28417 400 4 .ris .ris PUNCT bracis-28417 401 1 .enw .enw PROPN bracis-28417 401 2 .bib .bib PUNCT bracis-28417 402 1 doi doi PROPN bracis-28417 402 2 : : PUNCT bracis-28417 402 3 https://doi.org/10.1007/978-3-031-45392-2_15 https://doi.org/10.1007/978-3-031-45392-2_15 VERB bracis-28417 402 4 published publish VERB bracis-28417 402 5 : : PUNCT bracis-28417 402 6 12 12 NUM bracis-28417 402 7 october october PROPN bracis-28417 402 8 2023 2023 NUM bracis-28417 402 9 publisher publisher NOUN bracis-28417 402 10 name name NOUN bracis-28417 402 11 : : PUNCT bracis-28417 402 12 springer springer NOUN bracis-28417 402 13 , , PUNCT bracis-28417 402 14 cham cham PROPN bracis-28417 402 15 print print PROPN bracis-28417 402 16 isbn isbn PROPN bracis-28417 402 17 : : PUNCT bracis-28417 402 18 978 978 NUM bracis-28417 402 19 - - SYM bracis-28417 402 20 3 3 NUM bracis-28417 402 21 - - PUNCT bracis-28417 402 22 031 031 NUM bracis-28417 402 23 - - PUNCT bracis-28417 402 24 45391 45391 NUM bracis-28417 402 25 - - SYM bracis-28417 402 26 5 5 NUM bracis-28417 402 27 online online ADJ bracis-28417 402 28 isbn isbn NOUN bracis-28417 402 29 : : PUNCT bracis-28417 402 30 978 978 NUM bracis-28417 402 31 - - SYM bracis-28417 402 32 3 3 NUM bracis-28417 402 33 - - PUNCT bracis-28417 402 34 031 031 NUM bracis-28417 402 35 - - PUNCT bracis-28417 402 36 45392 45392 NUM bracis-28417 402 37 - - SYM bracis-28417 402 38 2 2 NUM bracis-28417 402 39 ebook ebook NOUN bracis-28417 402 40 packages package NOUN bracis-28417 402 41 : : PUNCT bracis-28417 402 42 computer computer NOUN bracis-28417 402 43 sciencecomputer sciencecomputer NOUN bracis-28417 402 44 science science NOUN bracis-28417 402 45 ( ( PUNCT bracis-28417 402 46 r0 r0 NOUN bracis-28417 402 47 ) ) PUNCT bracis-28417 402 48 share share VERB bracis-28417 402 49 this this DET bracis-28417 402 50 paper paper NOUN bracis-28417 402 51 anyone anyone PRON bracis-28417 402 52 you you PRON bracis-28417 402 53 share share VERB bracis-28417 402 54 the the DET bracis-28417 402 55 following follow VERB bracis-28417 402 56 link link NOUN bracis-28417 402 57 with with ADP bracis-28417 402 58 will will AUX bracis-28417 402 59 be be AUX bracis-28417 402 60 able able ADJ bracis-28417 402 61 to to PART bracis-28417 402 62 read read VERB bracis-28417 402 63 this this DET bracis-28417 402 64 content content NOUN bracis-28417 402 65 : : PUNCT bracis-28417 402 66 get get VERB bracis-28417 402 67 shareable shareable ADJ bracis-28417 402 68 linksorry linksorry NOUN bracis-28417 402 69 , , PUNCT bracis-28417 402 70 a a DET bracis-28417 402 71 shareable shareable ADJ bracis-28417 402 72 link link NOUN bracis-28417 402 73 is be AUX bracis-28417 402 74 not not PART bracis-28417 402 75 currently currently ADV bracis-28417 402 76 available available ADJ bracis-28417 402 77 for for ADP bracis-28417 402 78 this this DET bracis-28417 402 79 article article NOUN bracis-28417 402 80 . . PUNCT bracis-28417 403 1 copy copy VERB bracis-28417 403 2 shareable shareable ADJ bracis-28417 403 3 link link NOUN bracis-28417 403 4 to to PART bracis-28417 403 5 clipboard clipboard NOUN bracis-28417 403 6 provided provide VERB bracis-28417 403 7 by by ADP bracis-28417 403 8 the the DET bracis-28417 403 9 springer springer NOUN bracis-28417 403 10 nature nature PROPN bracis-28417 403 11 sharedit sharedit PROPN bracis-28417 403 12 content content NOUN bracis-28417 403 13 - - PUNCT bracis-28417 403 14 sharing share VERB bracis-28417 403 15 initiative initiative NOUN bracis-28417 403 16 publish publish VERB bracis-28417 403 17 with with ADP bracis-28417 403 18 us us PROPN bracis-28417 403 19 policies policy NOUN bracis-28417 403 20 and and CCONJ bracis-28417 403 21 ethics ethic NOUN bracis-28417 403 22 search search NOUN bracis-28417 403 23 search search NOUN bracis-28417 403 24 by by ADP bracis-28417 403 25 keyword keyword NOUN bracis-28417 403 26 or or CCONJ bracis-28417 403 27 author author NOUN bracis-28417 403 28 search search NOUN bracis-28417 403 29 navigation navigation NOUN bracis-28417 403 30 find find VERB bracis-28417 403 31 a a DET bracis-28417 403 32 journal journal NOUN bracis-28417 403 33 publish publish VERB bracis-28417 403 34 with with ADP bracis-28417 403 35 us we PRON bracis-28417 403 36 track track VERB bracis-28417 403 37 your your PRON bracis-28417 403 38 research research NOUN bracis-28417 403 39 discover discover VERB bracis-28417 403 40 content content NOUN bracis-28417 403 41 journals journal NOUN bracis-28417 403 42 a a DET bracis-28417 403 43 - - PUNCT bracis-28417 403 44 z z NOUN bracis-28417 403 45 books book NOUN bracis-28417 403 46 a a DET bracis-28417 403 47 - - PUNCT bracis-28417 403 48 z z NOUN bracis-28417 403 49 publish publish NOUN bracis-28417 403 50 with with ADP bracis-28417 403 51 us us PROPN bracis-28417 403 52 journal journal PROPN bracis-28417 403 53 finder finder PROPN bracis-28417 403 54 publish publish VERB bracis-28417 403 55 your your PRON bracis-28417 403 56 research research NOUN bracis-28417 403 57 language language NOUN bracis-28417 403 58 editing edit VERB bracis-28417 403 59 open open ADJ bracis-28417 403 60 access access NOUN bracis-28417 403 61 publishing publishing NOUN bracis-28417 403 62 products product NOUN bracis-28417 403 63 and and CCONJ bracis-28417 403 64 services service NOUN bracis-28417 403 65 our our PRON bracis-28417 403 66 products product NOUN bracis-28417 403 67 librarians librarian VERB bracis-28417 403 68 societies society NOUN bracis-28417 403 69 partners partner NOUN bracis-28417 403 70 and and CCONJ bracis-28417 403 71 advertisers advertiser NOUN bracis-28417 403 72 our our PRON bracis-28417 403 73 brands brand NOUN bracis-28417 403 74 springer springer NOUN bracis-28417 403 75 nature nature PROPN bracis-28417 403 76 portfolio portfolio PROPN bracis-28417 403 77 bmc bmc PROPN bracis-28417 403 78 palgrave palgrave PROPN bracis-28417 403 79 macmillan macmillan PROPN bracis-28417 403 80 apress apress PROPN bracis-28417 403 81 discover discover VERB bracis-28417 403 82 your your PRON bracis-28417 403 83 privacy privacy NOUN bracis-28417 403 84 choices choice NOUN bracis-28417 403 85 / / SYM bracis-28417 403 86 manage manage NOUN bracis-28417 403 87 cookies cookie NOUN bracis-28417 403 88 your your PRON bracis-28417 403 89 us us PROPN bracis-28417 404 1 state state NOUN bracis-28417 404 2 privacy privacy NOUN bracis-28417 404 3 rights right NOUN bracis-28417 404 4 accessibility accessibility NOUN bracis-28417 404 5 statement statement NOUN bracis-28417 404 6 terms term NOUN bracis-28417 404 7 and and CCONJ bracis-28417 404 8 conditions condition NOUN bracis-28417 404 9 privacy privacy NOUN bracis-28417 404 10 policy policy NOUN bracis-28417 404 11 help help NOUN bracis-28417 404 12 and and CCONJ bracis-28417 404 13 support support VERB bracis-28417 404 14 legal legal ADJ bracis-28417 404 15 notice notice NOUN bracis-28417 404 16 cancel cancel VERB bracis-28417 404 17 contracts contract NOUN bracis-28417 404 18 here here ADV bracis-28417 404 19 129.74.145.123 129.74.145.123 NUM bracis-28417 404 20 hesburgh hesburgh PROPN bracis-28417 404 21 library library PROPN bracis-28417 404 22 er er INTJ bracis-28417 404 23 unit unit NOUN bracis-28417 404 24 ( ( PUNCT bracis-28417 404 25 3005732405 3005732405 NUM bracis-28417 404 26 ) ) PUNCT bracis-28417 404 27 northeast northeast ADJ bracis-28417 404 28 research research NOUN bracis-28417 404 29 libraries library NOUN bracis-28417 404 30 ( ( PUNCT bracis-28417 404 31 nerl nerl PROPN bracis-28417 404 32 ) ) PUNCT bracis-28417 404 33 ( ( PUNCT bracis-28417 404 34 8200828607 8200828607 NUM bracis-28417 404 35 ) ) PUNCT bracis-28417 404 36 nerl nerl VERB bracis-28417 404 37 ta ta X bracis-28417 404 38 account account NOUN bracis-28417 404 39 ( ( PUNCT bracis-28417 404 40 3006206169 3006206169 NUM bracis-28417 404 41 ) ) PUNCT bracis-28417 404 42 university university NOUN bracis-28417 404 43 of of ADP bracis-28417 404 44 notre notre PROPN bracis-28417 404 45 dame dame PROPN bracis-28417 404 46 hesburgh hesburgh PROPN bracis-28417 404 47 library library NOUN bracis-28417 404 48 ( ( PUNCT bracis-28417 404 49 3000184373 3000184373 NUM bracis-28417 404 50 ) ) PUNCT bracis-28417 405 1 © © ADP bracis-28417 405 2 2025 2025 NUM bracis-28417 405 3 springer springer NOUN bracis-28417 405 4 nature nature NOUN